3D Segmentation Fusion — Plan

2026-07-16 · Miro · repo 3dai.iolabs.pointcloud.3dsegmentation

Summary

Build iolabs-point-cloud-segmentation-3d: a fusion pipeline that takes the per-segment run3_points.npz clouds of 260416_Abschnitt_4_5 and the outputs of asphaltedge (pavement corridor), modelling-lines XML (solid/dashed markings), guardrails (guardrails/walls) and verticalsigns (signs, gates, delineators, trees), and emits one voxel-decimated (5 cm ≤ the 7 cm cap), classified point cloud per segment (LAS 1.4 with classification codes + a class-colored PLY for instant viewing). Guardrails and verticalsigns get small “dump point mask” hooks (authorized) so per-point memberships come from the detectors themselves, not geometric re-derivation. Demos on segments 066–074 (the only ones with asphalt edges; 066 has a gantry + poles, 071 a delineator, all have guardrails) plus asphaltedge re-runs on 2–3 extra segments to get delineator-rich (131/132) and tree-rich (076/105) demos. Visual QC: own top-down class renders + CloudCompare on Windows via Codex computer-use, with screen video.

All four sources verified to share the same world frame as the input points (checked seg 066: XML lane vertices land inside the segment bbox at matching Z; guardrail polylines and verticalsigns positions likewise match the cloud frame).

Key decisions

Topic	Decision	Why
Input	`lane_points/segment_NNN/*_run3_points.npz` (XYZ f64 + intensity + RGB + scan_angle), ~1.8 M pts/segment	Common input of all three detectors; same world frame as every output
Pavement	Polygon from asphaltedge run4 `left/right_polyline_points` (left + reversed right), point-in-polygon on XY ∧ membership in `run4_road_surface.npz` (1 mm-rounded XYZ hash join, same trick asphaltedge io.py uses)	“Between the edges” exactly as detected; surface test keeps cars/noise out of the asphalt class
Lines	Newest `run7_lanes_.xml`, features `Type ∈ {Axis of the Edge, Center Lines}`. Element-level rule (per `lane.py:333-380`): `<Spline>` control-point polylines → solid, `<Line>` start/end segments → dashed* (a Center Lines feature may contain both). Paint pavement points within 0.20 m XY of the line geometry (\|Δz\| < 0.5 m), via KD-tree on 5 cm-resampled line vertices. Cross-check against the pipeline's own `lane_points_{solid,dashed}.npz` per-point exports in QC	XML is the contract the user named; solid/dashed maps to spline vs dash-segment representation, verified in modellinglines source
Guardrails/walls	Hook in guardrails `detect.py`: keep the per-record candidate mask (detect.py:212) + candidate→cell ids, join to per-instance cell sets (geometry.py:533), dump sidecar NPZ (record file, row indices, instance id/type). Walls = instances with `type="concrete"` (same stream)	Detector-faithful per-point mask; re-run is ~18 s/segment
Signs/gates/delineators/trees	Hook in verticalsigns: thread original (file, row) indices through `grid.accumulate_candidates`, dump indices per accepted detection (detect.py:536-550) + tree path (trees.py). Run with `tree_run.config.json` (out_ml-style combined pass: QC-verified classical+ML verifier, vegetation-RF trees)	Same fidelity argument; single re-run yields signs+gates+delineators+trees together
Priority on overlap	sign/gate/delineator/tree/guardrail > solid/dashed line > asphalt > unclassified	Thin/rare features must survive decimation and overlap
Decimation	5 cm voxel grid; per voxel keep one representative point of the highest-priority class present (closest to voxel centroid), carrying original RGB + intensity + class	Under the 7 cm cap; priority pick preserves thin features (lines, delineators)
Output	`segment_NNN_segmented.laz` (LAS 1.4, classification + RGB + intensity) + `…_classcolor.ply` (RGB = class palette)	LAS = the deliverable with the mask; PLY opens in CloudCompare pre-colored for eyeballing
Orchestration	Workflow tool fan-out: Sonnet/Opus agents for the parallel build stages, Codex (GPT 5.6 Sol) + Cursor (Grok 4.5) for adversarial review passes, Fable orchestrates & integrates	Per instruction; Grok/Sonnet/Opus effectively free right now

Class codes & palette

Class	LAS code	Color	Source
Unclassified	1	dark grey	—
Asphalt / pavement	2 (ground)	grey	asphaltedge corridor ∧ road surface
Solid line	64	white	XML splines
Dashed line	65	yellow	XML line segments
Guardrail	66	orange	guardrails hook (type w_beam/cable)
Wall	67	brick	guardrails hook (type concrete)
Sign	68	blue	verticalsigns (sign, pole_other)
Gate / gantry	69	purple	verticalsigns (gantry_or_gate)
Delineator	70	cyan	verticalsigns
Tree	5 (high veg)	green	verticalsigns vegetation-RF

Phases

1 · Mask hooks in guardrails + verticalsigns ~2 h wall

guardrails: opt-in --dump-point-masks; capture candidate row indices per record + flat cell ids, join with instance cluster_flat sets after detect_instances; write segment_NNN/point_masks.npz (per record: indices, instance_id; plus instance type table). Guard: zero behavior change without the flag; existing tests stay green.
verticalsigns: retain (file_id, row_idx) through accumulate_candidates concat; at accepted detections dump the cluster's original indices keyed by detection index; same for the tree path. Also behind a flag.
Re-run both detectors on the demo segments (066–074 + extras) with hooks on.
Built as parallel Workflow stages (Opus/Sonnet agents, one per repo), each verified by running the detector on one segment and asserting mask sanity (indices in range, XY of masked points near reported polylines/positions). Review pass: Codex Sol.

2 · Fusion package in this repo ~2–3 h wall

Modules: io_npz (points loader, stable record order), pavement (edge polygon + surface hash join), lines_xml (parser + painter), masks (hook sidecar ingestion), fuse (priority merge), voxel (decimation), writer (laspy LAS 1.4 + PLY), CLI seg3d-fuse --dataset-root … --segments 066-074 --voxel 0.05.
Deps: numpy, scipy (cKDTree), shapely or matplotlib.path, laspy, lxml/ElementTree.
Unit tests on synthetic micro-clouds (polygon test, painter radius, priority, voxel pick) + an integration test on segment 071 (smallest with all sources).
Alignment guards baked in: assert ≥99% of each source's geometry falls in the segment bbox, else abort loudly.
Workflow: parallel module builders (Sonnet/Opus), Fable integrates; adversarial review = Codex Sol + Cursor Grok 4.5 before the demo runs.

3 · Extend asphaltedge coverage for richer demos ~15–25 min compute

Run asphaltedge run_segments.py on 131, 132 (25 delineators + sign) and 076 or 105 (trees), ~4–5 min each (NPZ-load dominated), to output/seg3d.
If a run fails on those segments (detector was only ever validated on 66–74), demos fall back to 066–074 — not blocking.

4 · Produce demo clouds + visual inspection ~1 h wall

Fuse segments 066–074 (+ extras) → LAZ + class-colored PLY, plus own top-down per-class PNG renders as fast QC.
CloudCompare on Windows via Codex computer-use: open PLYs, orbit each demo cloud, record screen video of the inspection; save video + screenshots back to the repo.
Acceptance: lines lie on pavement, guardrails hug the barrier geometry, gantry/delineators/trees masked, pavement fills exactly the between-edges corridor.

5 · Wrap-up ~15 min

README with usage + class table; commit/push this repo; commit the hook changes in guardrails + verticalsigns (separate branches, 1-line messages).
Deliver: file list of produced clouds, renders, video, and a short findings note.

Risks

coverage Asphalt edges exist only for 066–074; extra segments need fresh asphaltedge runs that may misbehave outside the validated range. Fallback: demo on 066–074 only.
granularity Guardrail membership is cell-based (0.10 m occupancy cells) → a few cm of spillover around rails is expected; acceptable at 5 cm voxels.
trees No tree detections in 066–074; tree class only shows if phase 3 succeeds on a tree-rich segment.
windows CloudCompare-via-computer-use (and its screen recording) can be flaky; fallback = own renders + CloudCompare launched manually later. The clouds themselves don't depend on it.
xml-lines Some “Center Lines” features mix splines and dashes (a feature is typed “Center Lines” if any of its segments is dashed); painter classifies per geometry element (Spline=solid, Line=dashed), so mixed features are handled correctly. QC cross-check against lane_points_{solid,dashed}.npz catches systematic mislabels.

Open questions

Voxel size: I picked 5 cm (cap is 7). OK?
Output: LAZ (classification+RGB) plus class-colored PLY per segment. OK, or single format?
Should the demo set include the extra asphaltedge runs (131/132 + one tree segment), or strictly 066–074?

Approve (with any tweaks) and I start with phases 1+2 in parallel.