/* #hutter_publication_handoff @hutter_metrics @um_optimizations @hutter_run_costs @variant_protocol @pprog_p8v2_gen1_report_20260804 Handoff to ../hutter: publication owns the rendering, cmpr-src owns the data The division, so neither side has to guess: cmpr-src GENERATES facts about runs. The measured byte counts, the per-position dumps, the diagnostics, the costs. No rate, no extrapolation, no ranking. cmpr-src DERIVES rates, extrapolations and rankings from those facts, in one place (tests/pprog/gen1-report), so a change of reporting policy costs a re-render and not a re-run. ../hutter PUBLISHES everything a human looks at: the renderings, the chart, whatever goes out. So what crosses the boundary is data plus a spec plus the rules any published number has to obey. Five things to read, in this order. ## 1. The data, and how to make more of it tests/pprog/gen1-pos/*.pos one file per (variant, sample), named ..pos. THIS IS WHAT THE RENDERING READS. Two header lines then one tab-separated row per position: # axes M dumped # pos byte rec causal tok_fired tok_a tok_b tok_c settled s1 w1 s2 w2 conv Field meanings are in section 15.1 of p8v2-viz-spec.md. Note `dumped`: the diagnostic settling is capped, so a dump can cover a prefix of the sample and says which. tests/pprog/gen1.tsv one row per (variant, sample), MEASURED FACTS ONLY, plus a p7 tests/pprog/gen2.tsv baseline row per sample so the comparison is in the file rather than in someone's head. Byte counts are split by the three size classes of #hutter_metrics: p_header/p_k1/p_backward are class (a), p_token is class (b), p_gap + p_trace is class (c). (There was a p_k0 column until 2026-08-06. The k=0 background is gone; see the k=0 note below.) tests/pprog/gen2-pos/*.pos generation 2's dumps, same format. Generation 2 is #pprog_p8v2_gen2_goal_20260806: the baseline advances to v002 and every surviving alternative is re-run off it -- v002, v003 and v013..v020. tests/pprog/axes.json THE AXES AS DATA: per axis its letter, name, the question it decides, which alternative the baseline uses and what to watch; per alternative its id, digit, label, the prose of its OFRA block verbatim, and its worked examples; per worked example the numbers and the provenance (variant, sample, position, source file) that makes it checkable. Emitted by tests/pprog/p8v2-axes.py, which runs no compression. THIS REPLACES ANY HAND-COPIED LIST OF WHAT THE VARIANTS VARY -- a copy goes stale the moment an axis moves. tests/pprog/p8v2-replay.py the parent-side Python replay of LSA, learning and settling, used to compute the sweep-level worked examples and CHECKED against the retained model and the .pos dump before any example is emitted. Also the reference a JS viewer can be conformance-checked against. Regenerating, from cmpr-src: P8V2_LADDER="e64 e1k e10k" P8V2_SLOW_MAX=1000 tests/pprog/acceptance-p8v2 P8V2_GEN=2 P8V2_LADDER="e64 e1k e10k" P8V2_SLOW_MAX=1000 tests/pprog/acceptance-p8v2 python3 tests/pprog/p8v2-axes.py about 30 seconds for all 11 variants. Without SLOW_MAX the two settling variants (v006, v007) run at 10^4 and take 80-90 s each -- they settle one window per candidate byte, which is quadratic -- and the whole thing is about 4 minutes. Rungs above 10^4 add TSV rows but no dumps, on purpose: the panel reads 128 positions and a 10^6 dump is 50 MB. Re-rendering from existing dumps costs nothing and needs no run at all. Iterating on the visualization should never require re-running a compression. ## 2. The spec tests/pprog/p8v2-viz-spec.md section 15 -- THE GENERATION PANEL: the .pos format, what the panel shows, the cell encoding (hue = which level supplied the prediction, lightness = the settled weight, ring = trace membership, strike = settled argmax disagrees), the divergence strip, and six acceptance criteria. Sections 1-14 are the older standalone rendering of a single model; section 0's self-containment contract covers both. ## 3. The rules any published number has to obey #hutter_metrics the three size classes and what may be done with each. The short version: only class (c), the memory trace, is a per-byte rate; class (b) is bytes with a cap and extrapolates in log DSS; class (a) is bytes. NEVER publish a single per-byte rate over the whole model file -- at 1 KB that ranked the keep rules exactly backwards, which is why the block exists. Every number derived for a DSS other than the sample's own says EXTRAPOLATED and carries its basis. #um_optimizations an optimization changes how the machine computes and never what is reported; reporting stays in UM terms. If a published figure disagrees with the structural line or the query layer, the figure is wrong. #hutter_run_costs anything expensive is published with the shape that makes it expensive, not just a wall clock. A wall clock alone does not say whether the next size up is twice the cost or a hundred times. ## 4. The renderer that already exists tests/pprog/gen1-viz.py implements section 15. Standard library only -- no numpy, no matplotlib on this machine -- emitting one self-contained HTML file of inline SVG, both surfaces, no external request of any kind. It is publication tooling, so it is ../hutter's to take, move and rename. tests/pprog/gen1.html STALE. Built from the pre-enwik9 dumps, before the samples were real prefixes. Do not publish it; regenerate from the current dumps. ## 5. What is NOT a result, and must not be published as one - THE PICKS. Generation 1 is eleven variants over six axes with nothing pruned and no axis decided; #variant_protocol says the programmer picks and agents do not. Publishing a winner would be inventing one. - ANY CHART POINT. No end-to-end enwik9-scale measurement exists. The memchain interpreter reached 10^6 only after the append-only collapse, and the compression ladder's published points came from p8-opt.c, a direct-optimization program, not from this interpreter. - A SMALL-PREFIX RATE AS AN ESTIMATE. The model is learned from the same bytes it is measured on, so a small prefix is memorised and its rate is a LOWER BOUND on the asymptote, not an estimate of it. Measured on the baseline: 0.219, 0.317, 0.584, 0.730, 0.770 at 64, 10^3, 10^4, 10^5, 10^6, still rising at 10^6. - ROUND-TRIP FAILURE AS A BUG. Step 1 does not require it; v006 and v007 fail it by construction and the TSV carries a column rather than a gate. ## Provenance of the data as it stands Samples are real prefixes of ../hutter/enwik9. The build is the one with the append-only chain and the capped diagnostic dump (#um_optimizations), which produces byte-identical models to the version before it -- verified at 10^4 and 10^6 -- so dumps and TSV rows from either are comparable. Generation 1 is #wordsv2-v001 .. v011 of tests/pprog/p8v2-words.md; what each varies is in that file, and the axis vector is in every dump's header and every TSV row. ## THE k=0 BACKGROUND IS GONE, 2026-08-06 The model file lost its first 256-byte section. It used to sit at offset 48, between the header and the k=1 argmax table; the layout is now 48 + 256 (k=1 table) + BWD + TB + G + SC and EVERY OFFSET PAST THE HEADER MOVED DOWN 256. The magic is still P8V2, because the program version did not change; a model built before 2026-08-06 is caught by the size identity, not by the magic, so anything reading these files should check the size and fail loudly rather than misread the argmax table as a background. Every model under tests/pprog/models/p8v2/ has been regenerated. The standing instruction in #hutter_e64_viewer_prereq_20260805 -- "read the 256 bytes at offset 48 and treat them as provisional" -- IS WITHDRAWN. There are no such bytes. An unclamped position starts settling at 256 zeros; #f-p8-assign assigns 256 zeros when nothing fires; and #mem_cell_markov_1-norm sends no backward message at all when the byte has no predecessor, where it used to substitute the background. ../hutter's docs/pprog/build-p8v2-e64 and p8v2-e64.tpl.html have been updated for all four of these. ALSO CHANGED IN THE SAME PUSH, and it moves numbers: the accumulating settling updates (A2, A3) now pin the nesting of their entrywise lsa_add as MESSAGE-OUTER, ENTRY-INNER. lsa_add draws from the shared entropy stream, so the two nestings consume it in different orders; generation 1's v002 and v003 had been generated with different nestings, which made axis A not a like-for-like comparison. Both TSVs and all dumps and models were regenerated afterwards. NUMBERS FROM BEFORE 2026-08-06 ARE NOT COMPARABLE WITH NUMBERS FROM AFTER IT, in either direction; the pre-removal gen1.tsv is not kept. Written 2026-08-05, k=0 and generation-2 sections added 2026-08-06. */