comma.ai’s public challenge asks entrants to ship an archive.zip that inflates to video. The published score combines archive bytes, SegNet distortion, and PoseNet distortion on the public test clip.
comma-lab is a public experiment log and submission repo maintained by Alejandro Pena. It publishes measured runs, rejected branches, and the current promoted operating point in one place.
Apr 6, 2026, 1:57 PM CDT
Track A remains separate: `current_workflow` 0.00 at 167 bytes. The robust run ledger currently contains 35 measured runs, 10 promotions, and 15 explicit rejections.
robust_current · libsvtav1 · 524x394 · film-grain=22 · lanczos/lanczos · rgb24(pc)
Score vs bytes
Track B runs only. Better runs move toward the lower-left. The x-axis uses log scaling. The y-axis is linear. The severe AV1 bug run at 97.45 is omitted here so the operating range stays legible; it remains documented in the search-path section.
Lower is better. The plot is generated directly from scorer-backed artifacts in the repository.
Search path
A branch view of the measured search path: x265 reductions, ROI failures, the AV1 byte-layout failure, and the later hardening step that moved the floor to 2.12.
This graph is selective by design. It shows the runs that changed the operating point or changed the lab’s understanding of the evaluator.
Original vs inflated output
Browser preview assets for inspection only. Official scoring still comes from the scorer-backed reports.
Original
Source contest video excerpt.
Inflated output
Current promoted inflate path rendered to a browser preview.
Original zoom
Crop focused on task-relevant structure.
Inflated zoom
Same crop after compression and inflate.
Compression: 864,486 bytes. Distortion: PoseNet 0.09384175, SegNet 0.00575313. Total score: 2.12.
Walkthrough
-
Evaluator path
The scorer resizes frames and measures task distortion. It is sensitive to pipeline details, not just visual appearance.
PoseNet sees both frames. SegNet sees only the last frame in each pair. That asymmetry is why small encoding or decode-path changes can move the score.
-
Critical bug
The main AV1 failure was a byte-layout error, not a codec-limit problem.
The failed path emitted rawvideo as yuv444p bytes. The corrected path forces rgb24, which matches the evaluator’s raw-frame expectation.
-
Current operating point
The current honest floor is 2.12 at 864,486 bytes.
Against the first honest baseline (4.06 at 3,735,828 bytes), the lab reduced bytes while preserving task signal well enough to keep the score moving down.
Why 2.12 beat 2.18
| Metric | Prior | Current | Delta |
|---|---|---|---|
| current_workflow score | 2.18 | 2.12 | -0.06 |
| archive bytes | 864,455 | 864,486 | +31 |
| PoseNet distortion | 0.10656800 | 0.09384175 | -0.01272625 |
| SegNet distortion | 0.00569617 | 0.00575313 | +0.00005696 |
The bytes barely moved. The score change came primarily from lower PoseNet distortion.
Local neighborhood
This table isolates the local AV1 neighborhood around the promoted floor. It shows which nearby changes improved the result and which did not.
| Variant | Changed axis | Score | Bytes | Δ score | Δ bytes | Verdict |
|---|---|---|---|---|---|---|
| crf 33 | crf 33 | 2.20 | 920,457 | promoted | ||
| crf 34 | crf 34 | 2.19 | 864,455 | -0.01 | -56,002 | promoted |
| crf 35 | crf 35 | 2.21 | 808,036 | +0.02 | -56,419 | rejected |
| unsharp 0.30 | postfilter unsharp=9:9:0.30:9:9:0.0 | 2.20 | 864,455 | -0.01 | +56,419 | rejected |
| film-grain 0 | film-grain 0 | 3.33 | 719,096 | +1.13 | -145,359 | rejected |
| 522x392 | geometry 522x392 | 2.23 | 862,238 | +0.04 | -2,217 | rejected |
| lanczos upscale | upscale lanczos | 2.18 | 864,455 | -0.01 | +0 | promoted |
| color hardening | explicit bt709/tv -> rgb24(pc) | 2.12 | 864,486 | -0.06 | +31 | promoted |
On narrow screens, swipe horizontally to inspect the full table.
References
Primary artifacts
Turning points
- Initial honest floor — robust_current-baseline-cpu-2026-04-03 (4.06)
- 512x384 / crf23 — robust_current-medium23-cpu-2026-04-03 (3.62)
- ROI branch rejection — robust_current-dynamic-main-roi-cpu-2026-04-05 (4.47)
- Current floor — robust_current-av1-524x394-colorspace-hardening-promoted-cpu-2026-04-06 (2.12)
The landing page stays as the brief. Full detail lives in the linked artifacts.