Fitting benchmark
Question: given a manufacturer CAD whose upholstery is not shaped like the real product and four photos of the real product, does the fit recover the true shape?
Method (worker/showroom/bench/synthetic.py)
For each CC0 model (Poly Haven sofa_02, ArmChair_01, Sofa_01):
- Ground truth: the original model, normalised (metres, Y up, front +Z).
- Photos: Cycles renders of the ground truth with its own textures, front / back / left / right, 1024 x 768, studio backdrop with floor shadows. Cameras are jittered like a real shoot: one tripod height (pitch 5-12 deg, +-1.5 deg per photo), yaw +-3 deg, distance +-4 %, target +-2 cm, roll +-1 deg. The pipeline is told the lens (50 mm equivalent, as EXIF would) but not the camera poses.
- CAD given to the pipeline: the ground truth with every upholstered part flattened (crown height -15 %, surface pulled 6 mm inwards with a smooth falloff) and decimated by 50 %.
- The production code path (
process_geometry: masks with SAM 2, registration, spline fit) runs on CAD + photos.
Metrics
- Surface error of the upholstered parts against the ground truth: symmetric point-to-surface distance (mm), mean and 95th percentile, for the CAD as given and after fitting.
- Silhouette IoU per photo for the registered CAD and for the fitted model.
- Camera error of the registration (yaw, pitch, distance) against the true jittered cameras.
What silhouettes can and cannot recover: four outlines constrain the outer shape (crowns, arm and back profiles, overall proportions). The 6 mm inset in the middle of panels that face the camera is mostly invisible in silhouettes, so it is expected to stay; the mean error therefore has a floor, while the large errors (p95) are what the fit should remove.
Results
Run of 2026-11-17 on the demo server (CPU, 3 threads; coarse camera search, then 120 registration + 250 fit iterations at 256 px; data/bench/final4).
Upholstery surface error vs ground truth (symmetric point-to-surface, mm)
| model | soft parts | CAD as given: mean / p95 / max | after fitting: mean / p95 / max |
|---|---|---|---|
| sofa_02 (tufted sofa) | 2 | 17.1 / 69.3 / 109.0 | 16.5 / 44.3 / 78.1 |
| ArmChair_01 | 1 | 15.0 / 53.9 / 155.4 | 6.7 / 16.5 / 73.9 |
| Sofa_01 | 3 | 9.8 / 32.8 / 96.8 | 15.4 / 37.5 / 58.6 (worse, see below) |
Silhouette IoU per photo (front / back / left / right) and camera recovery (mean over photos)
| model | IoU, registered CAD | IoU, fitted | yaw err (deg) | pitch err (deg) | distance err (%) | pipeline s |
|---|---|---|---|---|---|---|
| sofa_02 | 0.930 / 0.828 / 0.910 / 0.930 | 0.955 / 0.937 / 0.927 / 0.931 | 0.9 | 3.2 | 2.8 | 101 |
| ArmChair_01 | 0.857 / 0.799 / 0.860 / 0.868 | 0.941 / 0.921 / 0.930 / 0.937 | 0.8 | 1.6 | 2.0 | 145 |
| Sofa_01 | 0.945 / 0.838 / 0.763 / 0.886 | 0.946 / 0.908 / 0.830 / 0.894 | 0.7 | 2.6 | 2.5 | 138 |
Real photos (Amazon Berkeley Objects armchair, 4 frames of a real 360 catalogue spin, no ground truth): after the coarse camera search (found a consistent -30 deg turntable offset and a ~140 mm-equivalent lens) the silhouette IoU is 0.979 / 0.973 / 0.976 / 0.977.
Reading the numbers
- The fit removes the large shape errors that silhouettes can see: on the armchair the mean error more than halves (15.0 -> 6.7 mm) and p95 drops 54 -> 17 mm; on the tufted sofa p95 drops 69 -> 44 mm and the worst error 109 -> 78 mm.
- The 6 mm inset of panels that face a camera is not visible in silhouettes, so the mean has a floor; on Sofa_01 the mean gets worse: in its left photo the light seat cushion merges with the light backdrop, the mask misses it (left IoU 0.83) and the fit pulls the cushion back. Mask quality is the current limit; a photometric term or manual mask touch-up would address it.
- Camera registration recovers yaw within ~1 deg, pitch within ~3 deg and distance within ~3 %.
- Bugs found by earlier benchmark runs and fixed: focal not rescaled after padding tight photo crops (cameras ~12 % too far); gradient-only registration stuck in local optima on real photos.
- Pipeline time per product on 3-4 CPU threads: ~2 minutes (SAM 2 masks ~10 s, camera search ~10 s, fit ~100 s).