Handoff — seg3d: tree precedence over hard classes, and the mega-instance problem

Goal

Miro's framing: "Almost everything other than, perhaps, ground should have precedence over greenery."

How it works now (verified)

Stage order

fuse.fuse_segment — src/iolabs_point_cloud_segmentation_3d/fuse.py:1884-1892:

_load_state → _paint_ground → _resolve_surface → _paint_asphalt → _paint_lines_stage
→ _paint_masks   (detector sidecars: guardrails, then verticalsigns — trees land here)
→ _paint_guardrail_json → _paint_signs_json   (both: UNCLASSIFIED rows only)
→ _paint_vegetation → _paint_vehicles         (vehicles: UNCLASSIFIED rows only)
→ build_instances → decimate_with_map

Paint ranks

So today tree is a plain rank-0 detector class: it is painted after ground, asphalt and the lane lines with no gate at all, and it therefore overwrites asphalt (11), solid_line (64) and dashed_line (65). It also beats a plain guardrail (66) and wall (67) row, because the guardrails sidecar is loaded before the verticalsigns one and rank-0 rows are last-write-wins (fuse.py:316-320). Only guardrail_support (72) and guardrail_top_rail (74) already beat it, via PAINT_LAST_CLASSES.

Vehicles are a separate mechanism and this is the part that matters most: a point hanging over the carriageway is UNCLASSIFIED at _paint_masks time (asphalt only paints the road surface). _paint_vehicles (fuse.py:1647) then sweeps leftover = state.classification == UNCLASSIFIED_CODE (fuse.py:1682) inside the corridor polygon into vehicle (73). A mega-tree that already claimed those rows blocks that sweep — and a soft gate on tree will not unblock it, because UNCLASSIFIED is itself soft. This is the 066 vehicle-streak failure.

The changes

Proposed precedence table

Change 1 — the one-liner do this first

src/iolabs_point_cloud_segmentation_3d/masks.py:110:

PAINT_FIRST_CLASSES: frozenset[str] = frozenset({
-    "low_vegetation", "medium_vegetation",
+    "low_vegetation", "medium_vegetation", "tree",
})

No other code change is needed for it: _PAINT_RANK (masks.py:116) is derived from the frozenset, and _paint_mask_rows gates every rank < 0 row on soft_codes. PAINT_FIRST gives exactly the semantics Miro asked for — "beats ground and unclassified, loses to everything else" — because the gate is literally (UNCLASSIFIED_CODE, ground). Verified, not assumed.

Two second-order effects to know about, both wanted:

• Inside the PAINT_FIRST group the first claim per row wins (fuse.py:311-315). Trees and low/medium vegetation cannot contest a row (both come from the verticalsigns sidecar, where one row carries one instance), so adding tree to an unordered frozenset is safe. Do not turn it into a tuple.
• build_instances (fuse.py:92) collects an instance's rows by matching the final classification against its seg_class. Tree instances will therefore shrink, and a tree instance that lost every row disappears from the instance table. That is correct behaviour; just expect the instance counts in stats to move.

Change 2 — the follow-on, required the one-liner alone does not fix 066

The vehicle sweep only considers UNCLASSIFIED rows, so tree paint over the carriageway still wins by squatting. Let the sweep reclaim tree rows — src/iolabs_point_cloud_segmentation_3d/fuse.py:1682:

-    leftover = state.classification == UNCLASSIFIED_CODE
+    leftover = np.isin(
+        state.classification,
+        (UNCLASSIFIED_CODE, BY_NAME["tree"].las_code),
+    )

Rationale to put in the docstring: the sweep is already the pipeline's "what is hanging over the road" rule, and a detector tree whose crown allegedly covers the carriageway between the outer asphalt edges is a vehicle streak far more often than a canopy. Gate it behind a config key so it can be turned off per dataset: add vehicle_reclaim_tree: bool = True next to vehicle_enabled (_config_model.py:118, docstring at _config_model.py:67, default in seg3d.default.json:17) and follow that field's existing pattern exactly.

Alternatives considered and rejected: moving the tree paint after _paint_vehicles (invasive, breaks the "masks are one pass" invariant and the instance bookkeeping), and dropping tree from the mask sidecar in seg3d (loses real trees).

Change 3 — docs + tests that will fail

• docs/fusion_spec.md:158-167 — the paragraph naming PAINT_FIRST_CLASSES as "low_vegetation and medium_vegetation". Also the step-6 sentence at docs/fusion_spec.md:142 ("Detector classes overwrite anything") and step 6f (vehicles).
• masks.py:103-112 — the PAINT_FIRST_CLASSES comment argues from hedges only; extend it with the mega-instance argument (a 78 m DBSCAN tree instance swallowing vehicle streaks and road slivers on 066), so the next reader knows why tree is in there.
• fuse.py:1334-1344 (_paint_masks docstring) and fuse.py:277-281 (_paint_mask_rows docstring) both enumerate the vegetation classes.
• Tests: tests/test_fusion.py:3949 test_paint_order_puts_vegetation_first (expects [0,3,1,4,2] for a pool that does not contain tree — still passes, but add a tree case), tests/test_fusion.py:4488 test_paint_order_realises_the_documented_group_ordering and tests/test_fusion.py:4405 test_paint_mask_rows_matches_the_per_row_reference (both drive off _MASK_CLASS_POOL at tests/test_fusion.py:4373, which already includes "tree" — these will start exercising the new gate for free and should be checked, not just re-run). Add: a test that a tree mask row over asphalt/solid_line leaves the hard class alone, and a fuse_segment-level test that a tree row over the carriageway ends as vehicle (73).

Not to be changed: paint_signs_from_json (fuse.py:687) and _sign_footprint_rows (fuse.py:645) already paint UNCLASSIFIED rows only (fuse.py:683), so the JSON-only tree cylinders already obey the desired precedence. Change 2's reclaim covers them too.

Mega-instance splitting — optional second step

The idea: break a tree instance where its footprint is cut by a non-vegetation class, then drop fragments below a min point count / min extent.

Recommendation: do not implement it now. The evidence says precedence removes the visible damage. The gallery's "Hard structure removed" section is exactly the class of error changes 1+2 handle (road panel under the gantry on 093, road slivers, lane markings, barriers). What splitting would add on top is only instance-level hygiene — the 78 m instance stays one instance id even after its non-tree rows are stripped, so the instance table and any per-instance geometry (bbox, footprint, height) stay wrong even though the per-point classification is right.

Decide it on the number, not the picture: after changes 1+2, re-run and look at the tree instance table for 066 (stats / instances.csv). If long instances are still there and anyone consumes tree instance geometry downstream, then split; the natural place is a post-pass over MaskInstance.global_rows before build_instances (fuse.py:1885-1898), splitting on connected components of the surviving rows in XY and dropping fragments under a configurable min size. Note that the real fix for instance identity is upstream in the detector's DBSCAN, not here — flag it back rather than growing seg3d a clustering stage.

Verification

Inputs

• seg3d stages (edges / ground / mesh / guardrails), A1 branch_000 no-angle-filter: /home/ai/seg3d_out/260905/ — read its HANDOFF.md first; 060/066/085 are symlinks into /home/ai/seg3d_out/260904/ (which has the full stack incl. LAS + orbit videos and the recipe).
• verticalsigns masks with the current tree paint: /home/ai/veg373_work/returns/out/r9/ (tuning segments 018 060 066 085) and /home/ai/veg373_work/returns/out/val_r9/ (003 022 023 087 093 094). Pass one of these roots as the signs masks dir; do not regenerate them.
• Detector side, read-only context: /home/ai/dev/wt-vs-382, branch feat/AI3D-382-returns-recall, PR #10; mask writer is src/iolabs_point_cloud_detection_verticalsigns/detect.py:956 _dump_point_masks.

Segments and what to look at

1. 066 — the mega-instance segment. Moving-vehicle streaks currently painted tree must become vehicle (73). This is the headline check for change 2.
2. 093 — road panel under the gantry: must become asphalt (11), change 1.
3. 003 — lamp mast / gantry column swallowed by a tree instance. expect no fix Neither change helps here: seg3d has no independent claim on those rows, so they stay tree unless the detector emits them as sign/gate. Report it back to the verticalsigns side rather than inventing a seg3d rule.
4. 085 — the acceptance segment, because it has human ground truth.
5. 018 / 060 — regression only: tree-heavy verge, must not lose tree points to the vehicle sweep.

Checks

• Before/after point counts by LAS class, per segment, full resolution. Expected direction: tree (5) down; asphalt (11), lines (64/65), guardrail (66), wall (67) and vehicle (73) up. Any class other than tree going down is a bug.
• Also count on the decimated output, not only full-res. tree has priority tier 4 and vehicle tier 0 (classes.py:84, classes.py:85, classes.py:159 priority_lut), so in a voxel holding both, the tree representative still wins the tie and the QC LAS can look unchanged even when the full-res fix worked.
• 085 ground-truth match: /home/ai/veg373_work/recall/gt/match085.py against segment_085_recap_classified*.laz in the same directory. Acceptance: tree precision/recall must not drop; points that GT calls vehicle / road / barrier and the old fuse called tree must move to their own class. Record the confusion delta, not just the headline numbers.
• Renders of 066, 093, 085 before/after, same camera. Prior evidence to compare against: before/after gallery (sections "Hard structure removed" and "Still wrong") and the AI3D-382 summary.
• uv run pytest tests/test_fusion.py green, plus the new tests above.
• Publish a short HTML image report of the pass (changed / improved / not improved / results / next) — Miro's standing rule for every pass.

Risks

• watch Guardrail shoulder over-paint is a known, live seg3d issue and must not get worse. Change 1 makes guardrail (66) and wall (67) beat tree on contested rows, which by construction grows guardrail coverage. Diff guardrail point counts per segment before/after and eyeball the shoulder on 060 and 085. If guardrail grows visibly, say so explicitly in the report — do not absorb it silently. The detector-side guardrail_skip_green_points colour gate (default off, see _paint_masks) is the existing lever if it needs damping.
• watch Change 2 can eat real canopy. pavement.classify_above_corridor (pavement.py:138) is a pure XY-in-corridor test with no height gate, so a genuine tree crown overhanging the carriageway between the outer asphalt edges becomes vehicle. This is the reason for the config flag. Quantify on a tree-lined segment (018, 060) and on 085 against GT before defaulting it to true.
• low/medium vegetation vs tree ordering. They now share rank −1 and the first-claim-wins rule. Safe today because both come from the one verticalsigns sidecar where a row carries exactly one instance — but if a future detector ever emits vegetation into a second sidecar, this becomes order-dependent. Leave a comment saying so.
• Instance table churn. Tree instances shrink and some vanish (see change 1). Anything downstream reading tree instance counts or per-instance geometry will move; check instances.csv consumers.
• Do not touch PAINT_LAST_CLASSES. The support/top-rail ordering encodes the detector's own claim precedence and is unrelated to this ticket.

House rules

• Work in a git worktree of /home/ai/dev/3dai.iolabs.pointcloud.3dsegmentation (HEAD a584a8a, branch main). Do not commit on main.
• Commit often, one-line messages prefixed with the seg3d ticket — placeholder AI3D-XXX; ask Miro for the real number before the first commit.
• No co-author trailers. No push. No PR unless asked.
• Do not edit or run anything in /home/ai/dev/wt-vs-382 — that is the verticalsigns side and it belongs to another agent.
• Treat /home/ai/seg3d_out/ and /home/ai/veg373_work/ inputs as read-only; write new fuse output to a fresh run root.
• Runtime budget is under 20 min per segment; stream per file and keep memory under the 32 GB box.
• Order of work: change 1 → tests → measure → change 2 behind its config flag → measure → 085 GT → report page. Do not do both changes and measure once; their effects are separable and Miro will want them separated.