AI3D-382 · using `number_of_returns` to fix tree recall and foliage-vs-structure — final summary

7 Sep 2026 · branch feat/AI3D-382-returns-recall (worktree /home/ai/dev/wt-vs-382, not pushed) · 9 detection passes, 146 commits, 1 811 tests · pass pages: 2–3 · 4 · 5 · 6 · 7 · 8 · 9 · background: DEM (Digital Elevation Model) bias explainer

The task. The new no-scan-angle-filter dataset carries a per-point number_of_returns (multi-return = foliage). Use it to (1) improve tree recall — the v6 hedge gate and the 10 m candidate slab were tuned on scan-angle-filtered data that cut tree tops and trunks off — and (2) separate foliage from signs, guardrails and walls.
The loop rule (Miro). After each pass: run detection on ≤ 3 segments, look at the renders, measure whether the last change helped, and throw the code away ruthlessly if it did not. Stop only when detection is perfect, needs a human decision or annotation, or needs better data. Nine passes ran; seven ideas were kept and eight were thrown away. Tuning happened on 018/060/066/085/086; 003/022/023/087/093/094 were never touched until the final gate.
Result on the six untouched validation segments: canopy recall above 10 m 0.018 → 0.769, 4–10 m 0.750 → 0.903, foliage-cell recall 0.494 → 0.860 — positive on 6/6 segments — while hard-structure contamination inside the tree paint fell from 111 cells to 26 (points −75 %). +1 real sign, 0 signs and 0 delineators lost, 0 trees lost.
Stopping condition reached: needs a human decision. What remains is product definition (do attached shrubs and grass under a tree count as “tree”? — 67 % of the residual false positives), mega-instance splitting (seg3d’s domain), and a pole-vs-trunk discriminator (its own ticket). See section 6.

1. Final gate — six validation segments that were never used for tuning

Segments 003 / 022 / 023 / 087 / 093 / 094. base = cfg/corpus.json (every new knob OFF) vs r9 = cfg/r9.json (the shipped stack). 12 detector runs, all rc = 0, zero errors, mask_file_missing = 0 everywhere. Source: eval/val_judgement.md.

metric	base	r9	delta	read
M1 canopy recall > 10 m (2.37 M pts)	0.0175	0.7693	+0.752	the upper canopy was simply not painted before
M1b canopy recall 4–10 m (7.84 M pts)	0.7503	0.9031	+0.153	positive on 6/6 segments
M2 foliage-cell recall, segment-wide (14 449 cells)	0.4942	0.8601	+0.366	unmasked remainder 16.1 M → 4.5 M pts
M3 hard-contamination cells	111	26	−85	down on 6/6
M3 hard points inside tree paint	137 009	34 057	−75 %	hard fraction 0.00363 → 0.00068
M6 truth-spot recall (48 tree spots)	1.000	1.000	0	both `not_tree` spots bit-identical
M4 signs	4	5	+1 real	0 removed; rendered and confirmed
M4 delineators	88	88	0	rail half-post CSVs md5-identical on 6/6
M4 tree detections	331	322	−9	all 9 rendered: road / mast / barrier slivers, 0 vegetation
M5 runtime, worst segment	308 s	447 s = 7.4 min	+45 %	37 % of the 20 min/segment budget
old-data parity, packaged defaults	identical		—	`clusters.csv`, `tree_clusters.csv`, `point_masks.npz` row-for-row and bit-for-bit; one crown-circle float inside the P2 overlay tolerance
cluster-verdict churn (sign path)	22 / 3 964 = 0.55 %		—	the pass-2 blocking instability was 7.6 %

085 human ground truth — progression across the whole ticket

1 872 942 annotator-labelled points, 100.000 % matched within 5 cm. “Veg-tolerant precision” counts annotator classes 74 (low veg) and 75 (medium veg) as acceptable, because whether attached shrubs count as tree is exactly open item 1. Source: eval/r9_judgement.md.

metric	base	r6pr (pass 7)	r8 (pass 8)	r9
precision	0.489	0.4884	0.5028	0.5431
veg-tolerant precision	0.741	0.7490	0.7705	0.8528
recall (GT tree, class 5)	0.755	0.7547	0.7547	0.7547
hard FP (guardrail + road + barrier)	—	52 300	46 519	26 965
GT guardrail points painted tree	—	—	25 772	6 276

Recall did not move by a single point across nine passes — every gain in precision was paid for with zero true-tree loss. The remaining 085 error is 19 892 road + 6 276 rail that the seg3d guardrail model itself misses, both inside the two mega-instances that also hold every true positive.

2. What was kept

knob	what it does	evidence
canopy extension `canopy_extension_enabled`, `min_multi_return_fraction 0.20`, `min_triple_return_fraction 0.02`	Above the 10 m candidate slab, paint each 1 m column cell whose points are multi-return foliage and continue upward. This is the whole recall win: the old slab simply stopped at 10 m.	pass 2–3 — truth-spot recall > 10 m 0.019 → 0.879 on the training segments; +0.752 on val; +1.7 % runtime
returns stripper on the sign path `returns_stripper.enabled`, `tau 0.30`, `radius 0.75`, `reject_promoted_poles` (+ `min_fraction 0.03`)	Drop multi-return (foliage) points from sign candidates before the shape features, so a plate buried in a crown is still recognisable. Guards: min cluster height 1.5 m, min 20 dropped, keep-only sparse fallback, min remaining 0.5, max surviving mrf 0.25, neighbour rules computed on unstripped geometry, and a guard that rejects poles promoted purely by stripping.	pass 2–3 +2 real signs on 018; pass 5 pole guard; val: +1 real sign, 0 lost, churn 0.55 %
sparse-sign guard `sign_post.min_points_per_m = 100`	Reject airborne “ghost” signs with no post: reason `sparse_sign`.	pass 2–3 — both 018 ghosts gone; real signs run 1 014–2 951 pts/m, ghosts 45–51 → 20× margin
hard-blob unpaint `hard_blob_unpaint_enabled`	After painting, find 1 m cells inside a tree instance with ≥ 300 pts and multi-return fraction < 0.05 — hard flat structure — and remove the paint there.	pass 2–3 — M3 hard cells 44 → 23, points −79 %; removed a billboard, lane markings, sign boards, road streaks
… + instance reject `hard_blob_reject_instance_enabled`	If a tree instance is mostly hard blob, drop the whole detection (`hard_structure_returns`).	pass 4 — 4 rejected, all structure; val: 9 rejected, 9/9 road / mast / barrier
… + survivor test `reject_max_survivor_mrf 0.1`, `reject_min_survivor_points 2000`	Only reject when what survives the unpaint is small and structure-like, so a real tree with a hard blob in it keeps its detection.	pass 6 — byte-neutral on the corpus (a safety rail, not a win); val: all 9 rejects tripped it correctly
… + cell hysteresis `hard_blob_cell_hysteresis = 0.5`	8-connected relaxation: a cell at 0.5× the point floor joins an existing hard blob, so a blob is not fragmented by its own sparse edge.	pass 7 — monotone, +5.4 k structure points unpainted, no crown touched
… + low band 0.3 m with canopy-column guard `low_band_min_elev_m 0.3`, `canopy_column_min_points 50`, `canopy_column_min_mrf 0.20`	Extend the hard-blob test down into the 0.3–1 m skirt, except under a column that has real canopy above it (so trunk bases and understorey survive).	pass 8 / pass 9 — 066 moving-vehicle streaks below 1 m −81 %, crowns bit-identical, 085 hard FP 52.3 k → 46.5 k, 0 GT-tree lost
guardrail-mask unpaint `guardrail_mask_unpaint_enabled`, `classes = []` (all), masks from `rail_halfpost.models_dir`	Remove tree paint from points that the seg3d guardrail model already claims. Point-level, so it reaches rails buried inside a mega-instance that no per-cluster rule can touch. Defaults OFF; a run without masks is unaffected.	pass 9 — 085 GT guardrail paint −75.7 %, hard FP −42 %, veg-tolerant precision 0.771 → 0.853, 0 GT-tree lost, 0 detections changed
performance lazy overlay fits, voxel-weighted lobe DBSCAN, convex-hull MEC prefilter	`crown_circles()` was 86 % of runtime and was computed for every cluster though only emitted detections consume it. Now fitted lazily at the two emission sites.	segment 002 1 355 s → 198 s (6.8×); decisions and masks bit-identical, only the crown overlay circles move
per-record TCS ground `tcs_ground.per_segment = false` (code path kept, off)	The per-segment ground union is implemented, reviewed and tested — supported seeding, record clamp, ELM rejection — but it never reached per-record parity, so the shipped default stays per-record.	pass 4 – pass 7; handed to tablecloth: segment-ground handoff

3. What was thrown away

Every one of these was implemented, run on ≤ 3 segments, measured, looked at, and deleted or defaulted off. That is the loop working.

idea	verdict	why
hedge-gate returns veto (`max_multi/triple_return_fraction`)	code removed	Fired on nothing: 0 gate-evaluated parents vetoed corpus-wide, 0 hedge verdicts overruled. The tree/hedge separation it needs does not exist in the returns signal (AUC 0.70).
returns stripper on the rail half-post path	code removed	Inert: the emitted half-post set was identical with and without it on all four segments. At tau 0.30 the emitted crest band loses ~100 % of its points and `min_remaining_fraction` restores it whole — a KD-tree per band cluster bought zero output change.
hard-blob mrf hysteresis 0.02	default 0.0, code kept	All cost: it punched two axis-aligned 5-cell rectangles out of a continuous green shrub bank on 018 (26 127 points, greener than the paint retained a metre away) and never once fired on the 060 billboard it was written for.
per-segment TCS, naive min-Z seeding	reverted	Below-ground noise density scales with the number of drive passes; the union DEM sank to −8 m, road entered the candidate slab, and a real 018 sign was lost on a 27 mm shift.
per-segment TCS, supported seeding (lowest 5 cm bin with ≥ 3 pts)	reverted	Fixes every catastrophe but buys ~nothing (GT recall +0.001 — the earlier “win” was ground leakage) for +9–22 % runtime, and still misses 2 of 7 canaries. Per-cell support cannot separate isolated noise from sparse ground under a canopy.
per-segment TCS, ELM seed rejection	reverted	Made clamp-off worse: it empties 15 % of populated cells on the two worst segments and hands them to interpolation, because an 11×11 median on a road corridor sits on the verge or the barrier.
DEM percentile / cell size / noise trim changes	do not touch	Every alternative loses more true tree than hard FP. Least-bad (`cell_m 1.5`) removes 3.0 k hard FP for 4.4 k GT-tree points (ratio 0.68); a noise trim loses 135 k of 156 k GT-tree points. Keep `percentile p8` / 0.75 m.
lower hard-cell point floor 300 → 100	do not lower	The residual rails live in dense mixed cells (n p50 = 206, mrf p50 = 0.73) — they fail the mrf test, not the point test. Would capture 2 630 points = 5.8 % of the residual.
elongation gate on joined hard cells (L ≥ 5 m & W ≤ 2 m)	do not add	New cells 8-join onto the wide mega-instance blobs, so the joined component is never elongated: 10 points captured on 085, 0 on 066.

4. Before / after

Panels are, left to right, true RGB | number_of_returns (grey = 1, orange = 2, red = ≥ 3) | BEFORE | AFTER; top row oblique, bottom row the same points top-down. The BEFORE column always comes first. Click any image for full resolution; ← / → (or swipe) step through the images of that card only; Esc closes.

unmaskedtreemedium_vegetationsign-ish (sign / pole / delineator)guardrailother

A. The recall win: paint the canopy above the 10 m slab >10 m recall 0.018 → 0.769

ce_085_big — BEFORE first · 085 — **canopy extension** (pass 2). Base paint stops at the 10 m candidate slab and leaves the whole upper canopy grey; with per-cell `mrf ≥ 0.20` + `mr3 ≥ 0.02` the crowns are painted to their tops.

B. Signs found under foliage, and nothing lost +1 real sign on val, 0 lost

zoom_sign_018_ADDED_1 — BEFORE first · 018 — **returns stripper on the sign path** (pass 2). Dropping the multi-return foliage that swallows a plate reveals a real sign the baseline missed completely. The r1 tuning also cost a delineator, which is why the pass shipped only after the guards (min height 1.5 m, unstripped-geometry neighbour rules, promoted-pole guard).

sign_ADDED_023 — BEFORE first · 023 (−2906.34, −1913.61), h 2.52 m, 5 206 pts, ml 0.469 — the **+1 sign on the untouched validation split**. A genuine ~2.5 m board/cabinet on the verge in front of the tree wall; base paints nothing there. Magenta covers exactly panel + post, no foliage tail. **0 signs and 0 delineators lost on all six val segments.**

C. Hard structure taken back out of the tree paint 111 → 26 hard cells

b_066_streaks — BEFORE first · 066 carriageway — **hard-blob low band 0.3–1 m** (pass 8/9). The “streaks” turned out to be **moving vehicles** smeared along the lane and admitted by the 0.3–10 m slab. Sub-1 m paint in the window drops **27 885 → 17 258 (−38 %)**, every band ≥ 2 m bit-identical, crowns untouched (canopy-column guard).

treerej_022_-2907_-1925 — BEFORE first · 022 (−2906.96, −1925.15) — one of the **9 tree detections removed on the validation split** by `hard_blob_reject_instance`: a thin all-single-return sliver on the median barrier in the middle of the roadway. All nine were rendered and judged: **nine road-surface / mast / barrier artefacts, zero vegetation.**

hardcell_003_-3329_-3671 — 003 (−3329, −3671), 20 082 pts — **the residual we do NOT fix** (identical in base and r9): a lamp/camera mast painted tree full height. Vertical, single-return, but a genuine column — it needs a **pole-vs-trunk discriminator**, not another unpaint knob. Open item 3.

D. The guardrails finally go — and no tree moves 085 rail paint −75.7 %, 0 GT-tree lost

e03_085_2290_1230 — BEFORE first · 085 (2289.72, 1229.92) — **guardrail-mask unpaint** (pass 9). r8 draws a solid bright-green line down the near w-beam; r9 strips it to a broken residual (the residual is where the *seg3d model’s own mask* stops). Conifer top-left and shrub bank bottom-right identical in both. 085 GT guardrail paint **25 772 → 6 276 (−75.7 %)**.

blob1_087_2475_1352 — BEFORE first · 087 (2475.00, 1352.05), **295 539 pts, 93 × 35 m** — the largest block of tree paint removed anywhere on the validation split. Base draws a continuous green line down both road edges; r9 deletes exactly that line and nothing else. Crowns above are identical (denser under r9). **7 of 7 largest unpainted blobs were guardrail beam or single-return artefact — none was vegetation.**

t1_018_crown_lowrej_c0_s0 — BEFORE first · 018 truth crown `lowrej_018_c0_s0`, 13.2 m — **the cost check**. r8 and r9 are indistinguishable: full crown, full understorey, only the guardrail line at the bottom edge lightly thinned. **0 GT-tree points lost** on 085 (tp 156 209 in both roots).

h1_085_2282_1219 — BEFORE first · 085 (2282.00, 1219.00) — **the honest cost**. A real hedge grows *through* the rail, so its rail-embedded foliage sits inside the seg3d guardrail mask and goes with the beam: **3 500 GT low/medium_veg points, 0.6 %** of the segment’s veg paint. The bush is still detected and still painted. Escape hatch if it ever bites: `guardrail_mask_unpaint_classes = ["w_beam"]` + mask erosion.

E. The one dead end worth documenting: the ground DEM 4 passes, shipped nothing

dem_bias — **The four-pass detour that produced nothing shippable.** Merging all drive passes into one segment-wide ground union multiplies rare below-ground noise until most 0.2 m cells contain one, min-Z seeding takes it, and the DEM sinks (z_ground p5 −3.0 m, worst −8.1 m on 066). Road then reads as 0.3 m “above ground” and enters the tree slab. Supported seeding, a record clamp and ELM rejection each fixed part of it and none reached per-record parity → shipped `per_segment = false`, handed the problem to tablecloth. Full explainer: DEM bias.

5. Open items — this is where the loop stops

Stopping condition 2 was reached: what remains needs a human decision, annotation, or another team’s code.

#	item	why it stops here
1	Do attached shrubs and grass under a tree count as “tree”? product definition	67 % of the remaining false positives against the 085 human GT are annotator classes 74 (low veg) and 75 (medium veg) inside our tree paint. If they count, precision is 0.853, not 0.543. This is a definition, not a bug — nothing should be tuned until it is answered.
2	Mega-instance splitting seg3d	Single verge instances 78–100 m long swallow rails, road and vehicles along with every true positive (085 instance 3 = 6.9 M points and zero GT trees). No per-cluster rule can reach inside them; only a point-level mask or an instance split can.
3	Lamp / camera masts and gantry columns painted tree separate ticket	Vertical, single-return, tall — and a genuine column, so no unpaint knob distinguishes them from a trunk. Needs a pole-vs-trunk discriminator. Visible on 003 at (−3329, −3671), 20 082 points, identical in base and r9.
4	Tree vs hedge cannot be separated by returns needs labels	Measured AUC 0.70 — not a discriminator. Miro’s hedge definition is a size/band property, not a foliage property. Any progress needs labelled hedges, not more features.
5	`return_number` export available, not needed	Only `number_of_returns` is exported today; `return_number` exists in the LAS on N: and would allow first/last-echo logic. Nothing in this ticket needed it — do not spend the export effort speculatively.
6	Per-segment TCS ground handed over	Four passes of work, reviewed and tested, shipped OFF. Handed to the tablecloth repo with the seeding, clamp and noise-rejection findings: handoff-tablecloth-segment-ground-default-20260906-1420.
7	seg3d fuse: add `tree` to `PAINT_FIRST_CLASSES` one-line change	In `3dsegmentation/masks.py`, guardrail classes are PAINT_LAST (they already override tree), but `tree` (LAS 5) has rank 0, so road and vehicle rows we paint tree still leak through the fuse. Low/medium vegetation are already PAINT_FIRST.

6. Where everything is

what	where
worktree / branch	`/home/ai/dev/wt-vs-382`, branch `feat/AI3D-382-returns-recall` (146 commits, not pushed — house rule)
recommended config	`configs/a1nr_returns_recommended.json` in the repo (documented in `README.md`); the run-shaped copy is `cfg/r9.json` in the scratch tree
tests	1 811 — `uv run pytest -q`; lint `uv tool run ruff check src tests scripts`
scratch root	`/home/ai/veg373_work/returns/`
configs	`cfg/` — `corpus.json` (base) … `r9.json` (shipped), `v6mv_nr.json` (baseline)
detector outputs	`out/` — `r1`…`r9`, `val_base`, `val_r9`, `parity_final_old_002`
judgements	`eval/r1_judgement.md` … `eval/r9_judgement.md`, `eval/val_judgement.md` (the final gate)
renders	`analysis/r1_renders/` … `analysis/r9_renders/`, `analysis/val_renders/`, `analysis/dem_bias/`
diagnoses	`diag/DIAG_treepaint.md` (066 vehicles, 085 GT breakdown), `diag2/DIAG_dem_hardcells.md` (DEM alternatives, hard cells)
data	`/home/ai/veg373_work/returns/data/A1NR/lane_points` (14 truth segments, no scan-angle filter, with `number_of_returns`)
seg3d guardrail masks	`/home/ai/seg3d_out/260905/guardrails_A1B0` → `rail_halfpost.models_dir`
run command	`uv run verticalsigns-detect --data-dir <data> --segments 085 --out <out> --config cfg/r9.json --dump-point-masks --log-level INFO`

AI3D-382 · using number_of_returns to fix tree recall and foliage-vs-structure — final summary