/* #pprog_p8v2_gen1_report_20260804 @pprog_p8v2_impl_goal_20260804 @variant_protocol @pp_wordsv2 @hutter_metrics @hutter_run_costs @um_optimizations @hutter_publication_handoff @pprog_p8v2_questions_20260727 P8v2 generation 1: what was built, what it did, and what needs a ruling (2026-08-04; remeasured on real enwik9 prefixes and reported by class 2026-08-05) The first version of this block reported the wrong thing twice over, and both are fixed rather than patched: 1. THE SAMPLES WERE NOT ENWIK9. 2. THE NUMBERS MIXED THE SIZE CLASSES. #hutter_metrics is the answer to that and governs from now on: (a) fixed, (b) sub-linear and capped by the architecture, (c) linear in the DSS. The split that follows from it is now real: GENERATING (#pp_wordsv2's report lines and tests/pprog/acceptance-p8v2) emits measured facts and refuses to print a derived quantity; DISPLAYING (tests/pprog/gen1-report, tests/pprog/p8v2-diag.py) derives every rate, extrapolation, ranking and diagnostic from them and runs no compressions; PUBLISHING is ../hutter's. A reporting change now costs a re-render, not a re-run. ## What to look at tests/pprog/gen1.tsv one row per (variant, sample), MEASURED FACTS ONLY: the three classes as byte counts, the binary size, round-trip, the k=2 diagnostics, the settling cost and its shape. tests/pprog/gen1-report the derived view. Applies #hutter_metrics: a, model_b(DSS), S/U, k, the ranking by a, the a-against-M trend, and the k=2 diagnostics. Every extrapolated column is labelled EXTRAPOLATED with its basis. tests/pprog/p8v2-diag.py the k=2 diagnostics, derived from (sample, model file). tests/pprog/gen1-pos/ per-position dumps, kept for samples up to 10^4. tests/pprog/acceptance-p8v2 runs the generation; the ladder above 10^4 is opt-in via P8V2_LADDER because #hutter_run_costs says what it costs. #hutter_publication_handoff what ../hutter reads to publish any of this, and the list of things that are not results. ## The numbers All measured on real prefixes of ../hutter/enwik9, all in one file (tests/pprog/gen1.tsv) with a p7 baseline row per rung, all on the same build. a is the class (c) rate -- trace bytes over M -- and it is the only class whose differences survive to 10^9. The derived view is tests/pprog/gen1-report. The baseline against p7, on the same bytes: M p8v2 a p7 a p8v2/p7 model_B k=2 hits net entries removed / model_B 64 0.2188 0.4844 0.45 188 57/62 0.11 10^3 0.3170 0.6110 0.52 908 808/998 0.35 10^4 0.5844 0.8354 0.70 4296 5406/9998 0.63 10^5 0.7295 0.8800 0.83 9008 39549/99998 2.07 Net trace entries removed per byte of token model goes 0.11, 0.35, 0.63, 2.07 across the same four decades, because the model grows sub-linearly while what it explains grows linearly. ## The keep rules, ranked the way #hutter_metrics requires At 10^5, by a, with p7 for scale: v008 E2, two replay rounds 0.7287 the best in the generation, by 0.0008 v001 the baseline, keep all 0.7295 v010 F2, 256 by support 0.8382 v011 F3, 256 by sharpness 0.8731 p7 -- 0.8800 ## Points where generation 1 had to read the goal rather than quote it These are the ones that change what gets built next. Q1-Q4 and Q8 stand as written on 2026-08-04; Q5-Q7 are decisions I made on your behalf and are the ones most worth overruling. Q1. D1. The goal says "surprising under the k=1 argmax". Taken literally the baseline's trace is p7's exactly, so the k=2 layer contributes nothing to class (c) and the model is pure added cost. Generation 1 read D1 as p7's RULE SHAPE (one causal pass, no settling, no removal) with the COMPOSITE argmax as the predictor, which is what #omega-p8-frozen now says. If the literal reading was meant, D1 is a control that measures the token section's cost and nothing else, and the baseline has to move to D2. Q2. Eleven runs, not eight. The protocol's own formula over the six axes as the goal lists them is 1 + (2+1+1+2+2+2) = 11; the goal says 8 in four places (and says the cross product is 72, where it is 324). Generation 1 ran 11. If 8 was the intent, three alternatives were meant to be dropped and the text does not say which three. Q3. E2's reading. "Relearn k=1 after the k=2 patterns exist" is implemented as relearning k=1 over the RESIDUAL -- the positions no kept k=2 rule covers -- because the other reading (relearn over the sequence the round-1 model reconstructs) is a no-op whenever the model round-trips, and it does. Is that the axis you wanted? Q4. Where axis D goes now. Under A1 the token message ASSIGNS, so wherever a k=2 rule fires it overwrites the backward message entirely; the settled argmax at a candidate position is then just the causal composite prediction, which is wrong there by construction, so no candidate is ever removable. Axis D cannot do anything while axis A is A1. The options are to pair D with A2/A3 in generation 2, to change the D2/D3 predicate so it does not consult the level that already got it wrong, or to close the axis until f is decided. Q5. The message rule, which I changed. The frozen rule was going to be w_p - (255 - w_s), read as LSA division by the source's doubt. It annihilates every non-absolute message: at these sample sizes a learned weight is 1 or 2 LSA units, so one hop off a clamped position takes every message to 0 and the whole window settles to byte 0 (measured: 10 of 64 positions correct). It is now min(w_p, w_s) -- a message is as strong as its weakest link, the same arithmetic the AND gate already uses -- and the same measurement is 64 of 64. Both forms are recorded in #omega-p8-hyper. Q6. Size accounting for the token section, now settled by #hutter_metrics rather than by me: it is class (b), sub-linear and capped at 65536 * 4 = 262144 bytes at k=2, reported as bytes and extrapolated in log DSS. #pprog_p8v2_questions_20260727 (2) had it in the fixed cost, which is right for F2/F3 (a budget of 256) and wrong for F1, which keeps every distinct context. Measured aside, offered because it bears on the log-DSS rule: on real enwik9 the distinct 2-gram count grows like M^0.31 over the last decade measured (227 at 10^3, 1074 at 10^4, 2252 at 10^5, 5800 at 10^6, 11845 at 10^7), which is Heaps-like and faster than log. Extrapolating the power law from 10^7 gives ~49000 contexts at 10^9 against the log rule's ~15000. At k=2 it does not matter -- both are under the cap and both are below 2e-4 of S/U -- but at k=3 the cap stops binding and the two rules diverge by the same factor against a much larger base. Q7. Settling sweeps. The goal froze W = 128 but not the sweep count. A message travels one position per sweep, so anything less than W leaves the far end of the window unsettled (measured: at 24 sweeps over 64 positions, 10 of 64 correct). It is now W sweeps, and it is why settling a window is O(W^2) and therefore why D2/D3 are quadratic in M (#hutter_run_costs). Q8. SN of a token, minor. The event form is settled ("The 3-token at position 3 is "abc"."); the RULE form was not, and --patterns prints rules, so it drops the position clause: "The 2-token is "ab"." -> "The input byte is c." 37. Also, concatenating two byte renderings is ambiguous to a reader when a byte is not printable ("0x20x" is 0x20 then x, but only because the hex form is fixed-width). Neither blocks anything. ## Q9 -- turned up by generation 1, and fixed during it The interpreter could not reach the DSS, and it was not p8v2's fault. #pp_pattern implemented the length-N copy chain literally: chain_shift moved every one of the N cells one position toward the tail on every single input byte, so a run was O(M*N), and every run uses N = M. Measured on the bare p3 memchain, net of pacing: 0.17 s, 0.51 s, 1.63 s at N = 2*10^4, 4*10^4, 8*10^4 -- ratios of 3.0 and 3.2 per doubling against the 4 that quadratic predicts. That was ~2 minutes at 10^6, ~3 hours at 10^7, and 10^18 byte moves at 10^9. It is the same bottleneck p8-opt.c was written to get around, and it is why no point on the ladder had ever been measured end-to-end at enwik9 scale by this interpreter. Fixed. Settling happens inside the uncertainty window; everything to the left of it has settled and will not change again, so it has no reason to move leftward byte by byte. The chain is now stored append-only -- the byte is written at a cursor, the cursor advances, and mem_cell_i is the VIEW buf[cursor - 1 - i] -- so a step is O(1) and a run is linear. With the diagnostic dump also capped, a 10^6 run went from 213.3 s to 24.4 s, 8.7x, and the 10^7 rung became affordable. It is an OPTIMIZATION and #um_optimizations is the block that says what that means: it changes how the machine computes and nothing that is reported, reporting stays in UM terms, and every collapse admitted carries the rule that unfolds it. The unfolding here is that the UM still has N cells and 256*N atomic identity patterns; the structural line still counts 2560000 of them at N = 10^4, and --patterns still enumerates all 256 atoms of input -> mem_cell_1 even though not one copy is ever performed. The acceptance test holds both. The model file, the round-trip and every reported number are byte-identical across the change, which was the condition for admitting it at all. What remains open is only the scope: p1, p3, p4, p7 and p8 still shift literally, because their pp_main blocks index mem[] under the shifting convention. The p7 baseline in the generation TSV therefore stops at 10^5. Whether to carry the collapse across to them -- and if so whether that belongs here, to #run_ofra_strip_goal_20260729 / #run_ofra_split_goal_20260729, or to its own push -- is the question. ## Not done, and why - p1, p3, p4, p7 and p8 STILL SHIFT THE CHAIN LITERALLY. The append-only collapse (#um_optimizations) is applied to the wordsv2 kind only, because their pp_main blocks index mem[] under the shifting convention -- the tail is mem[N-1], the j-th input byte is mem[M-1-j] -- so it is not a one-block change for them and re-rolling five shipped fixtures is not this push's business. The visible consequence is that the p7 baseline rows in the TSV stop at 10^5 and say why; a p7 run at 10^6 takes upwards of ten minutes. - ONE CMPR WART, left alone deliberately: if a --rewritepl call returns nothing, the fill loop in #handle_run_ofra caches the empty block under its staleness key, and every later run of that variant then splices the emptiness in without another call, so a transient failure looks permanent and structural. It bit once here (#pp_settle for v003). The workaround is `rm .cmpr/pprog/blocks/-` and rerun. Recorded rather than fixed because it is a cmpr change and this push is Hutter. - tests/pprog/gen1.html and tests/pprog/gen1-viz.py are left on disk and are no longer produced or referenced by anything here: rendering and publication are ../hutter's. The .html is stale (it was built from the pre-enwik9 dumps) and nothing in cmpr-src needs it. Section 15 of tests/pprog/p8v2-viz-spec.md still specifies the generation panel, so the renderer can be moved across intact. ## Green p1, p3, p4, p7, p7-query, explain and p8 acceptance all pass; tests/run-all is 64/64; the build is warning-clean and so is every generated child program. No fixture other than p8v2's re-rolled a block or recompiled a binary. The two line changes outside p8v2 are in acceptance-p7-query and acceptance-explain, which grepped for "requires the markov kind" in an error message that now names both kinds. */