CODA: Depth-Aligned Scene Completion and Object Decomposition from a Single RGB-D Image

Dongwon Son¹Junhyek Han¹Yoontae Cho¹Minseok Lee¹Hong-seok Choi²Jiwook Choi²Hyungjin Kim²Beomjoon Kim¹

¹ KAIST² Samsung Heavy Industries Co., Ltd.

Complete Once, Decompose Afterward

Complete the scene. Then separate its objects.

CODA overview: observed partial point cloud, depth-grounded complete object meshes, and the illustrated shelf after two seconds of simulated gravity. — From one unsegmented RGB-D observation to complete environment and movable-object meshes.

TL;DR

CODA completes an entire scene from one RGB-D view, then separates it into individual objects. Explicit depth grounding keeps the reconstructed geometry aligned with the observation—so we can test whether the objects stay in place under gravity.

METHOD OVERVIEW

The complete CODA pipeline

CODA pipeline: a single RGB-D view conditions coarse occupancy completion, fine scene-mesh reconstruction, and object decomposition. Observed-surface initialization and depth-guided feature matching ground the geometry. A texture flow assigns object-identity codes, and nearest-code assignment separates environment and object meshes. — **Figure 2.** CODA first completes coarse occupancy, then reconstructs the scene surface, and finally separates it into environment and object meshes. RGB-D conditioning grounds all three stages; the texture flow produces object-identity codes for decomposition.

RECONSTRUCTION SPEED → ACCURACY → RETENTION

Five methods. Speed, accuracy, and retention.

Download video ↗

Reconstruction time and results
Method	Reconstruction time ↓	HomebrewedDB (HB)			Custom
Method	Reconstruction time ↓	Retained ↑	F@5 complete ↑	F@5 input view ↑	Retained ↑	F@5 complete ↑	F@5 input view ↑
CODA	5.59 s	78.52%	0.6789	0.9214	76.61%	0.6879	0.9331
ZeroGrasp	1.54 s	17.70%	0.5674	0.9184	7.13%	0.5746	0.9188
SAM3D	85.08 s	35.71%	0.5223	0.7240	24.94%	0.5685	0.7899
RecGen	100.15 s	47.14%	0.5929	0.8369	25.84%	0.6007	0.8336
ASTRA	28.1 min	76.30%	0.5501	0.6877	—	—	—

What do F@5 complete and F@5 input view measure?

F@5 is a surface F1 score with a 5 mm tolerance. It balances how much of the reconstructed surface is close to the reference surface and how much of the reference surface is covered by the reconstruction. Scores range from 0 to 1; higher is better.

F@5 complete compares the reconstructed object with its full reference shape, including parts hidden from the camera. It measures how accurately the method completes the whole object.

F@5 input view measures agreement with the surface observed in the input RGB-D image. It measures how well the reconstruction aligns with the observed depth.

FROM OBSERVATION TO SIMULATION

One view. A complete, separated scene.

15 recorded examples

01 RGB-D observation
02 Partial point cloud
03 Reconstruction
04 Retention test

CUSTOM

Loading example…

Compare CODA withLinked camera · Linked replay time

Loading scene…

Native reconstruction meshes

CODA

Z ↑

Observed points

Colors distinguish predicted objects.Drag to orbit · Scroll to zoom

HomebrewedDB HB

Six tabletop observations

Custom scenes

Nine observations across shelves and tables

FIGURE 7 · DIVERSITY AND ALIGNMENT

One observation. Eight possible completions.

Explore supplementary examples from the paper’s diversity evaluation in 3D. Compare how the saved completions vary behind the observed surface, while checking their alignment with the same RGB-D points.

Mustard bottle

Eight fixed seeds per method, from the same input.

White box: target object, marked for display.

The 3D view uses saved surface samples from the paper’s reconstructions.

Completion Region Observed RGB points Ground truth

CODARecGen

Loading saved completions…

Linked camera · Drag to orbit · Scroll to zoom

How to read this comparison

RGB points are the observed target surface; the optional gray reference is ground truth. Colors distinguish the eight fixed seeds. GT-best shows each method’s highest-scoring sample using ground truth—an oracle choice, not an input-only selection.

F_v is mean visible F1 and F* is best complete F1, both at 10 mm. D_vis and D_hid measure pairwise voxel disagreement across all eight samples: lower visible disagreement means greater consistency; hidden diversity describes variation, not correctness. Values are the saved evaluation metrics, independent of the display controls.