Tablecloth: one ground surface for each segment

Summary

0 · Words used in this report

record

One drive pass along the segment. One npz file contains one record. A segment has 6 to 8 records.

segment

A part of the road, approximately 100 m long.

cell

A square of 0.2 m × 0.2 m in the horizontal plane.

seed

The first estimate of the ground height in one cell.

surface

The model of the ground. The SMRF filter calculates it from the seeds.

gate

The test that keeps a point as ground: z − surface ≤ 0.05 + 1.25 × gradient. The test has no lower limit. Thus the program keeps all points below the surface as ground.

ghost return

A false point below the true ground. A reflection from a wet or smooth surface causes it. The false point is a mirror image of an object above the road.

record scope

The old method. The program makes one surface for each record.

segment scope

The new method. The program makes one surface for each segment.

combiner

The rule that selects one seed from the lowest points of all records in one cell.

GT

Ground truth. A person gave a class to 1,872,942 points in segment 085.

DEM bias

The height of the surface minus the height of the true ground. A negative value shows that the surface is too low.

recall

The fraction of the true ground points that the program keeps as ground. A high value is good.

leak

The fraction of a class (for example trees) that the program keeps as ground. A low value is good.

1 · The problem: the minimum of K records is biased low

The correct test uses the human ground truth of segment 085. Each point gets the label of the ground-truth point within 5 cm. This table shows what the person labelled the points that create the union minimum:

Source: render/gt_test_lowpoints.py. 70,618 points in the area that the human mapped. More than two thirds are road. They are a lower carriageway, and ground below trees and below an embankment. They are not artefacts.

The real cause is the order statistic. This table shows each order statistic of the record minima minus the height of the human road, on the 101,783 cells of segment 085 that have 3 or more ground-truth ground points:

Source: render/gt_road_bias.py. Each step up the order removes approximately one half of the low tail. This is the behaviour of the minimum of K values that contain noise. There is no separate population of noise. The remaining error is a few centimetres for each point. It is not visible in a rendering.

Result. In a cell, the records can see more than one real surface: the carriageway, a lower carriageway, and the ground below vegetation. The min rule always takes the lowest surface. The SMRF filter then puts the ground model at that lowest surface. The true carriageway is then more than 5 cm above the model, and the gate removes it. This is why the road recall decreases from 0.915 to 0.794.

2 · What ships, and what was rejected

What ships. Segment scope pools every record, makes one seed raster with the per-cell minimum, then runs the unchanged SMRF and the unchanged gate. The minimum over the pooled points equals the minimum of the per-record minima, so this is exactly the old min option without any machinery. Segment 018 under the new default gives 7 mask files that are byte-identical to the old min run.

What was rejected. The mode combiner put the record minima into groups and used the largest group. Its purpose was to reject "ghost returns". Section 1 shows that those points are real surfaces, so the purpose was invalid. The code is preserved on the branch archive/segment-seed-combiners-do-not-merge. Do not merge that branch.

The one seed knob. ground_percentile replaces the strict minimum with a percentile of the pooled points. On the open road a higher percentile is better. Below vegetation it is much worse, because there the ground is the lowest few per cent of the returns. Section 3 gives the numbers. Keep the value at 0.0.

The measurements below were made before the deletion. They stay in this report because they are the evidence for the decision.

DEM bias against the human ground truth of segment 085

How to read these plots

• The four maps: the axes are the coordinates x and y in metres. Each pixel is one cell that has 3 or more true ground points. The colour is the surface height minus the true ground height. Blue = the surface is too low. Red = the surface is too high. White = correct. All four maps use the same colour scale.
• What to compare: look at the second map (simple minimum). Large blue areas cover the carriageway. In the third map (mode) these areas are white.
• Histogram: the horizontal axis is the DEM bias in metres. The vertical axis is the number of cells, on a logarithmic scale. The dashed line is at −0.05 m, which is the limit of the gate. Cells to the left of the dashed line cause the loss of true ground points. A good curve is narrow and near 0.
• Bar chart: each group of bars is one class of the ground truth. The height is the fraction of that class that the program keeps as ground. The bars for road and ground must be high. The bars for vegetation and objects must be low.
• The two outcome maps: each point is a ground-truth point. Red = a ground point that the program did not keep (an error). Green = an object point that the program kept (also an error). Grey = correct. Fewer coloured points is better.

combiner	bias, 5th percentile (m)	fraction more than 0.05 m too low	road recall	ground recall	low vegetation leak	medium vegetation leak	tree leak	guardrail leak
one surface for each record	−0.042	0.043	0.915	0.998	0.119	0.580	0.837	0.010
simple minimum	−0.115	0.152	0.794	0.999	0.082	0.260	0.655	0.005
mode (new default)	−0.024	0.031	0.941	0.999	0.305	0.927	0.991	0.016
mode_ref (optional)	−0.036	0.038	0.926	0.999	0.238	0.893	0.988	0.006

How to read the table. The first four columns show the quality of the ground surface. Lower values in columns 2 and 3 are better. Higher values in columns 4 and 5 are better. The last four columns show the errors. Lower values are better. Source: diag/combiner_harness_v5_085.md. In that file, mode has the name mode_med, and mode_ref has the name mode_desc_0.50_s2.

The tree leak is high for all methods. The gate has no lower limit, thus the program keeps all points below the surface. In a forest the surface is at the top of the trees, because no record sees the ground. The old method already had a tree leak of 0.837.

3 · The real problem: the surface below vegetation

The gate has no lower limit. Every point below the ground model becomes ground. A model that is too high therefore makes vegetation into ground. Until 2026-09-08 no measurement looked at the height of the model below vegetation. This is that measurement. It uses the 6,436 cells of segment 085 where the human labelled ground points and also vegetation more than 0.5 m above them.

Source: render/gt_veg_surface_error.py. Every percentile above 0 lifts the model into the canopy. Keep ground_percentile at 0.0.

The gate decides, not the seed. The pooled minimum is below the road more often than a per-record surface is, and a gate of 0.05 m is too narrow to absorb that. This table uses the pooled-minimum surface on segment 085 and changes only max_elev_diff_m:

Source: render/gt_gate_sweep.py. Compare the line for 0.15 with the production values today (one surface for each record, gate 0.05): road 0.915, medium vegetation 0.580, trees 0.837, low vegetation 0.119, guardrail 0.010. A gate of 0.15 m is better on the road, on medium vegetation and on trees. It is worse on low vegetation and on guardrails. A wider gate keeps more object points as ground. Do the verticalsigns sign test before you change it. The gate is unchanged in the code.

Why the model is too high below vegetation

Most drive passes see only the top of a bush. Usually only one pass has the correct angle to see the ground below the bush. The mode combiner uses the majority, thus the surface stays at the top of the bush. The program then keeps the full bush as ground.

The old method gave the correct result for the one pass that saw the ground, and the incorrect result for the other passes. For this reason its medium-vegetation leak was already 0.580.

Source: render/gt_lowroad_recall.py, segment 085, 1,164,894 ground-truth ground points, of which 154,789 are more than 1 m below the median of the record minima. The mode combiner keeps almost all of them because its model is above them. The same effect keeps the vegetation.

The mode_ref variant can move the seed down to a lower group, if 2 or more records agree. This decreases the leak, but it also decreases the accuracy of the ground surface. A larger permitted movement admits the ghost returns again. Thus mode_ref is optional and is not the default.

Vertical sections where the surface follows a hedge

How to read these plots

• All four plots: the horizontal axis is the distance along the section in metres. The vertical axis is the height in metres.
• Plots 1 and 2 (segment 085): the colour of each point is its ground-truth class. The lines are the surfaces, as in section 1: red = simple minimum, green = mode, grey = one record.
• Plots 3 and 4 (segment 018): the upper panel shows the old method, and the lower panel shows the mode combiner. The colour shows if the program keeps the point as ground.
• Result to see: the green line is above the true ground and inside the vegetation. All points below the green line become ground points. This is the cost of the correction.

In the forest beside the road all methods put the surface at the top of the trees. The grey lines of the single records are also 3 m to 7 m above the true ground. No record sees the ground, thus the filter fills these cells from the road. The mode combiner uses the majority, thus it does this more frequently. In the hedge of segment 018 the program keeps the vegetation up to 3 m above the ground. The kept fraction in that section increases from 0.27 to 0.73. This is the meaning of the increase of the tree leak from 0.837 to 0.991.

Where the classification changes

How to read these plots

• Both maps: the axes are the coordinates x and y in metres. Each pixel is a cell of 0.5 m.
• Green = the new method keeps more points as ground. Purple = the new method keeps fewer points as ground. Black = no change.
• Result to see: in the first map the green areas are along the vegetation at the side of the road. The carriageway does not change. In the second map (simple minimum) large purple areas cover the carriageway. This is the error that section 1 shows.

For segment 018, the mode combiner adds 3.70 million points and removes 0.57 million points. The kept fraction increases by 0.085. The added points are in a small number of cells that contain dense vegetation. The largest cell, at x = −3273 and y = −2187, gains 38,700 points. For segment 086 the kept fraction increases from 0.625 to 0.668.

4 · Full views of segment 018

Segment 018: 7 records, 37.0 million points

How to read these plots

• The four views in the first row: the axes are the coordinates x and y in metres. The grey image is a shaded relief of the points that the program keeps as ground. The cell size is 0.1 m. Red areas are the points that the program removes.
• What to compare: in the view of the simple minimum the carriageway has more red. This shows the loss of true road points. In the view of the mode combiner the carriageway is grey again.
• The three maps in the second row: the colour is the fraction of points that the program keeps as ground in each cell of 0.5 m. A bright colour means a high fraction.
• The bar chart: one bar for each record. The height is the kept fraction of that record. Compare the bars of the three methods. No bar of the mode combiner is lower than the bar of the old method.

5 · Numbers for each segment

How to read the table. "Kept" is the fraction of all points that the program classifies as ground. "Worst record" is the largest decrease of the kept fraction of one record, against the old method. Only records with 100,000 points or more are included. A negative value shows a loss. "Surface difference" is the 5th percentile of the segment surface minus the record surface. A negative value shows that the shared surface is below the surface of one record. This occurs where a record cannot see the true ground, or where the majority of the records contain ghost returns.

Segments 065 and 066 of branch 0 of A1 are not used in this report. Their data is corrupt. Source: diag/combiner_harness_v5.md. Full runs of the software: segment 018 gives 0.754 → 0.839, segment 086 gives 0.625 → 0.668 (diag/ab_default_vs_record.md).

6 · Changes to the software

• Removed on 2026-09-08: segment_seed_combiner, segment_seed_link_m, segment_seed_ref_drop_m, segment_seed_min_support, ClusterStats, cluster_record_minima, combine_record_seeds, build_segment_seed_surface, the stack of per-record grids with its memory limit, and tests/test_seed_combiners.py. 1,089 lines fewer.
• Added: accumulate_segment_seed_grid. It streams the records into one pooled seed grid. With ground_percentile > 0 the program uses the existing in-memory path.
• Configuration: ground_scope is segment (default) or record. ground_percentile (default 0.0) is the only seed knob.
• Module segment_cloud.py: SegmentCloud, build_segment_cloud, build_segment_surface, classify_segment, classify_segment_with_surface.
• Not changed: the SMRF finalisation, the gate, the record path, and the CSF path.
• Tests: 181 pass and 6 are skipped. Two tests prove that the pooled grid equals the minimum over a stack of per-record minima, and that the percentile path matches the record path.

# the new default (one pooled surface for each segment)
uv run python -m iolabs_point_cloud_tablecloth.runner --data-dir <lane_points> --segments 018,085 --out <out>
# the old behaviour, identical to the files of 2026-09-05
uv run python -m iolabs_point_cloud_tablecloth.runner ... --set ground_scope=record

7 · Your decisions

1. Set max_elev_diff_m. This is now the decision that matters. 0.05 keeps the vegetation low but loses road points. 0.15 is better than production on the road, on medium vegetation and on trees. Section 3 has the table.
2. Do the three acceptance tests of verticalsigns before any change of the gate. (a) The kept fraction of each segment must stay within ±0.05 of the old value. (b) The road sign in segment 018 at x = −3212.87 and y = −2195.74 must stay in the point cloud. (c) The ground height must not decrease more than 0.5 m at the reference positions, and the tree crowns must not get holes. A wider gate makes test (b) more difficult.
3. Keep ground_percentile at 0.0 until a new measurement says something else.
4. Jira tag, push of the working branch, release to Nexus. None of these is done. The archive branch is pushed.
5. Make the directory ground_A1B0 again only after the acceptance tests.

More data