A deflated box-derived evaluation yardstick, and a (later-overturned) negative for frozen-backbone slot-binding

⚠ SUPERSEDED. The negative conclusion below was later traced to an implementation bug (a per-dimension target-whitening step). With it corrected, the milestone passes — see the resolved companion report. This version is retained for the scientific trail.

Summary

This line of work asks whether an object/entity ontology emerges from unlabeled real observation, as part of a state-over-token learning thesis. This note reports a full diagnostic investigation on COCO with a frozen DINO ViT-B/16 backbone, an iterative slot-attention binding core, and spatial-broadcast reconstruction (DINOSAUR-style). It contains two results that point in opposite directions and must be kept apart:

1. A measurement-artifact correction (positive). The convenient box-derived evaluation gold (bounding boxes painted as masks) caps even an oracle clustering of the features at FG-ARI of roughly 0.13–0.15 — below the 0.20 pass bar. The bar was therefore structurally unreachable under that yardstick regardless of emergence. With real instance-segmentation gold (COCO panoptic, things-only, per-image paired), the oracle ceiling rises to about 0.24 — above the bar.

2. A clean negative (as understood at the time). Re-probing trained checkpoints against the fixed yardstick: zero-training k-means on the same frozen features already scores about 0.21 (clears the bar), while the trained slot-binding encoder masks score only about 0.09–0.13 — below the no-training baseline. The decoder masks (about 0.22–0.23) only tie k-means, and via a readout shown unreliable on synthetic data. So the trained binding appeared not to extract object structure beyond naive clustering of the frozen features.

Method (the diagnostic ladder)

Each apparent failure was first interrogated as plumbing, not conclusion, and falsified before the next step:

• Seven confounds eliminated (each adversarially checked): training objective (contrastive to reconstruction), optimizer schedule (warmup / grad-clip / cosine), backbone (supervised to DINO self-supervised), per-dimension target normalization, decoder capacity, slot-vector collapse (a diversity penalty), and a second, distinct collapse mode — decoder single-slot domination — addressed by a mask-entropy balance gate. Two collapse modes at two layers implies anti-collapse is required at every layer, not just one.

• Scale tested: roughly 8× images (to 40k COCO) × 5× steps (to 150k). The slot-to-object alignment trend was effectively flat across ten checkpoints — scale within reach moved nothing.

• Positive control: the same code on synthetic feature grids where objects are cleanly separable. The encoder binds them (FG-ARI 0.54, well above a shuffle baseline of about 0.07) — so the binding core is sound; the decoder readout is the broken part (near chance there).

• Feature-separability audit: object structure is present but weak in DINO-COCO features (k-NN object purity of roughly 0.70–0.80, well above chance), and higher resolution / better features (DINO-B/8, DINOv2 at 448) did not lift the oracle ceiling — which localized the cap to the gold, not the features.

• Yardstick fix and adversarial validation: real segmentation gold, things-only, per-image paired (boxes derived from the same segments); the box-calibration arm reproduced the depressed score (pipeline validated) while the segmentation gold reached about 0.24.

Why it matters

The honest conclusion is sharper than the pre-correction read ("flat / scale ceiling," which was itself a box-gold artifact). At the time, the bottleneck appeared to be neither the metric nor scale, but that frozen-backbone slot-reconstruction did not beat k-means on this data. The evidence-based next lever was a reconstruction path that actually trains the encoder toward object grouping (a stronger decoder giving real grouping gradient) and/or unfreezing features — not more scale, and not a different gold.

This was published as a negative result because the measurement-artifact half is a reusable caution: box-derived gold deflates object-discovery metrics below common pass bars, and an oracle-clustering calibration exposes it cheaply. (Note: the slot-binding negative was subsequently overturned; the measurement caution stands.)