p8v2 generation 1

Eleven variants over six axes, measured on real prefixes of enwik9 at four sizes. Built 2026-08-04, remeasured and reported by size class 2026-08-05. Rendered from #pprog_p8v2_gen1_report_20260804, #hutter_metrics and #hutter_publication_handoff.

WE NEED FEEDBACK FROM THE PROGRAMMER HERE. Nothing has been pruned and no axis has been decided. Per #variant_protocol the picks are the programmer's, by taste and by where the work is going; measurement is evidence put in front of that judgement and never the decision rule. There is no winner on this page, and no chart point.

Step f on e64 → The 64-byte prefix settled one time step at a time, with the settling rules as live controls. Where this page reports what each variant DID, that one lets you drive f and watch it happen — and it is where axes A, B and C are visible at all, since under D1 the trace never consults the settled state. Its engine reproduces 9 of the 11 recorded runs exactly. Open the generation 1 panel → Every variant over the same input positions, with a divergence strip marking the columns where they disagree — decomposed into recorded, level and settled, because the combined strip saturates and a saturated strip is not a list of positions worth looking at. Self-contained inline SVG, both surfaces, no external request. Spec: section 15 of p8v2-viz-spec.md; renderer gen1-viz.py, standard library only. Open the generation 2 panel → The same panel over generation 2's ten variants, post-k=0 removal (#pprog_p8v2_gen2_report_20260806). Generation 1's runs were re-run after the same removal, so the two panels are comparable with each other and with nothing published before 2026-08-06. Same renderer: it takes the generation off the posdir name. Open the generation 3 panel → The baseline against generation 3's three axis-A alternatives, the first that set a pattern's fall-off from its own support rather than from a constant (#pprog_p8v2_gen3_report_20260810). Read the conv gutter carefully here: under a rule that fires a pattern only on some time steps, a window can look settled because nothing has happened yet, and one alternative reports a mean of 1.00 time steps for exactly that reason. The column measures the wrong thing for a scheduled rule, and generation 3 says so rather than correcting it silently. Open the generation 4 panel → The A7 pair — the k=1 argmax delivered at min(8, w_s) with fall-off 1, one alternative because the two constants only make sense together — against the baseline and against B2, where the stored learned weights are the delivery and every B1 strategy has an exact fact to be compared with (#pprog_p8v2_gen4_report_20260815). The ground truth cuts both ways: at e64 the stated 8 beats the stored truth, by e10k the stored matrix wins. The instrument for this generation is p8v2-gen4-e64.

What changed on 2026-08-05

An earlier version of this page reported the wrong thing twice over. Both are fixed rather than patched, and both are worth stating because the corrected numbers reverse the earlier ranking.

The samples were not enwik9. Two of three were repo-derived, and the one carrying every headline was yes 'the quick brown fox…' — 66 distinct 2-grams in 10 000 bytes where a real enwik9 prefix of that size has 1074. Everything below is measured on real prefixes of enwik9, at four sizes, so the rate can be read as a trend rather than a point.
The numbers mixed the size classes. A single per-byte rate over the whole model file is an artifact of the sample size. At 1 KB it ranked the keep rules exactly backwards. #hutter_metrics is the answer to that and governs from here on.

What the terms mean

CMP, UM and cmpr terms are assumed and only cited. Everything below was invented by this experiment or its chain, so it is defined here.

The three size classes

From #hutter_metrics. Every byte of the model file S belongs to exactly one class, and they are never summed into a single per-byte rate.

(a) FIXED — size does not depend on the corpus size: the self-extracting binary, the model-file header, the k=0 background table, the k=1 byte model, and B2's stored backward LPP where a variant uses it. No extrapolation; contributes fixed/DSS, which at 10⁹ is around 2.6×10⁻⁵. Reported as bytes.

(b) SUB-LINEAR — the token section: kept k-rules times entry size, capped by the architecture at 65536 × 4 = 262 144 bytes at k=2. Extrapolated as model_b(M) × log(DSS)/log(M), capped. Reported as bytes at the sample and bytes extrapolated, never as a per-byte rate — dividing a capped quantity by a tiny M and multiplying by 10⁹ charges it thousands of times over. That is the error above.

(c) LINEAR — the memory trace: the gap section plus the recorded bytes. Reported as the per-byte rate a = trace(M)/M. At 10⁹ this is more than 99.99% of S, so a is the only quantity whose differences survive to the chart, and it is the headline everywhere below.

Composite, and the only route to k:
S(DSS) = fixed + model_b(DSS) + a·DSS; S/U = fixed/DSS + model_b(DSS)/DSS + a; k = log(S/U)/log(0.99).

The samples

Real prefixes of enwik9, named by size: e64 (64 bytes, for eyeballing — the prefix the fixture works its examples on), e1k, e10k, e100k. The ladder above 10⁴ is opt-in via P8V2_LADDER because of what it costs; rungs above 10⁴ add table rows but no position dumps, since the panel reads 128 positions and a 10⁶ dump is 50 MB.

The axes

Defined in #pprog_p8v2_impl_goal_20260804, § “The axes”; the alternatives are OFRA blocks in p8v2-words.md. The axes vector reads A B C D E F, one digit each, so 111111 is the baseline and 111113 is the baseline with F3 swapped in.

axis	the open point, and its alternatives
A	Settling update. A1 assignment, the longest k that fires assigns (baseline). A2 LSA addition with decay at 2f. A3 addition, no decay, renormalised per position to constant sum.
B	Backward direction of the k=1 pattern, which settling needs. B1 reconstruct the LPP by normalising the p7 forward table (baseline). B2 store the full LPP both directions — 65536 LSA bytes, the control, and visible in class (a) below.
C	AND gate arithmetic at partial activation. C1 min(w_a, w_b) (baseline). C2 sum with threshold, the traditional neuron shape.
D	Sparsification — which bytes the trace records. D1 p7's rule frozen, under the composite argmax (baseline). D2 surprising under the settled argmax over W, one pass. D3 greedy: drop a byte if removing it does not change the most probable sequence within W, re-settling after each removal.
E	Replay. E1 single pass, k=1 frozen from p7 (baseline). E2 two rounds, relearning k=1 over the residual. E3 replay until the trace stops shrinking, cap 4.
F	Token creation and pruning in omega. F1 collect everything, never prune (baseline). F2 fixed budget per k by total LSA support. F3 fixed budget per k by sharpness of the output distribution.

The baseline against p7

The k=2 layer against the order-1 Markov model it is built on, on the same bytes, at every rung. All columns MEASURED.

M	p8v2 a	p7 a	ratio	model B	k=2 hits	net removed / model B
64	0.1875	0.4844	0.39	188	58/62	0.11
1000	0.2920	0.6110	0.48	908	833/998	0.39
10000	0.5901	0.8354	0.71	4296	5349/9998	0.61

The k=2 layer is a real win at every size measured, and it round-trips byte-identically. On real text it removes 17% to 55% of the class (c) bytes.
a rises with M. The model is learned from the same bytes it is measured on, so at small M it has simply memorised the sample: most contexts were seen once, and the argmax is whatever followed that once. Every small-prefix number in this project's history, p7's included, is a lower bound on the asymptote and not an estimate of it. The trend was measured rather than assumed — #hutter_metrics asserted the opposite before it was.
The advantage over p7 shrinks monotonically across four decades, and the k=2 hit rate falls the same way (92%, 81%, 54%, 40%) while coverage stays at 100% because F1 keeps every context it sees. Both models converge on their true conditionals as M grows, and a k=2 argmax stops being much better than a k=1 argmax once neither is memorising. Extrapolated naively, the advantage would be small at 10⁹. This is the finding of generation 1 and it is invisible at any single sample size. It does not say the design is wrong — one argmax successor per context is the weakest possible use of a token model, and the point of the ladder to k=64 is that longer contexts stay sharp where short ones wash out — but it does say that measuring at 10³ and extrapolating would have misled in the optimistic direction.
The counter-fact, and the one thing that improves with scale: net trace entries removed per byte of token model, the last column above, because the model grows sub-linearly while what it explains grows linearly. That is the argument for the token model, and no rate on a small prefix could have made it.

Every run, by size class

Ranked by a, the class (c) rate, exactly as gen1-report ranks them. The first six columns are MEASURED. model B @1e9, S/U and k are EXTRAPOLATED to a corpus of 10⁹ from the sample size in the heading, by the composite above, and are not a measurement of anything at that size. Byte counts are exact; a and S/U to four decimals.

M = 64 bytes (e64)

variant	axes	fixed B	model B	trace B	a	model B @1e9	S/U	k	round-trip
v001	111111	25832	188	12	0.1875	936	0.187527	166.5	yes
v002	211111	25832	188	12	0.1875	936	0.187527	166.5	yes
v003	311111	25864	188	12	0.1875	936	0.187527	166.5	yes
v004	121111	91368	188	12	0.1875	936	0.187592	166.5	yes
v005	112111	25832	188	12	0.1875	936	0.187527	166.5	yes
v006	111211	25832	188	12	0.1875	936	0.187527	166.5	yes
v007	111311	25832	188	12	0.1875	936	0.187527	166.5	yes
v008	111121	25944	188	12	0.1875	936	0.187527	166.5	yes
v009	111131	25944	188	12	0.1875	936	0.187527	166.5	yes
v010	111112	25832	188	12	0.1875	936	0.187527	166.5	yes
v011	111113	25832	188	12	0.1875	936	0.187527	166.5	yes
p7	------	17288	0	31	0.4844	0	0.484392	72.1	n/a

M = 1000 bytes (e1k)

variant	axes	fixed B	model B	trace B	a	model B @1e9	S/U	k	round-trip
v001	111111	25832	908	292	0.2920	2724	0.292029	122.5	yes
v002	211111	25832	908	292	0.2920	2724	0.292029	122.5	yes
v003	311111	25864	908	292	0.2920	2724	0.292029	122.5	yes
v004	121111	91368	908	292	0.2920	2724	0.292094	122.5	yes
v005	112111	25832	908	292	0.2920	2724	0.292029	122.5	yes
v006	111211	25832	908	292	0.2920	2724	0.292029	122.5	no
v007	111311	25832	908	292	0.2920	2724	0.292029	122.5	no
v010	111112	25832	908	292	0.2920	2724	0.292029	122.5	yes
v011	111113	25832	908	292	0.2920	2724	0.292029	122.5	yes
v008	111121	25944	908	318	0.3180	2724	0.318029	114.0	yes
v009	111131	25944	908	318	0.3180	2724	0.318029	114.0	yes
p7	------	17288	0	611	0.6110	0	0.611017	49.0	n/a

M = 10000 bytes (e10k)

variant	axes	fixed B	model B	trace B	a	model B @1e9	S/U	k	round-trip
v008	111121	25944	4296	5882	0.5882	9665	0.588236	52.8	yes
v001	111111	25832	4296	5901	0.5901	9665	0.590135	52.5	yes
v002	211111	25832	4296	5901	0.5901	9665	0.590135	52.5	yes
v003	311111	25864	4296	5901	0.5901	9665	0.590136	52.5	yes
v004	121111	91368	4296	5901	0.5901	9665	0.590201	52.5	yes
v005	112111	25832	4296	5901	0.5901	9665	0.590135	52.5	yes
v009	111131	25944	4296	5928	0.5928	9665	0.592836	52.0	yes
v010	111112	25832	1024	7834	0.7834	2303	0.783428	24.3	yes
v011	111113	25832	380	7878	0.7878	854	0.787827	23.7	yes
p7	------	17288	0	8354	0.8354	0	0.835417	17.9	n/a

v006, v007 skipped at this size: D2/D3 settle one full window per candidate recorded byte, O(SC*W^2*256), quadratic in M; 10000 bytes is over the 1000 limit

The class (c) rate against sample size

MEASURED, all of it. Read the trend, not any one point — a single a is a point on a curve that is still rising at 10⁶.

variant	e64	e1k	e10k	e100k
v001	0.1875	0.2920	0.5901	the baseline — every axis at its baseline alternative
v002	0.1875	0.2920	0.5901	A2 — LSA ADDITION against DECAY: the position's state accumulates every pattern application that reaches it (LSA addition, per entry), and then decays.
v003	0.1875	0.2920	0.5901	A3 — LSA addition with NO decay, renormalized per position to constant sum.
v004	0.1875	0.2920	0.5901	B2 — the control: store the full LPP in both directions -- the whole 65536-byte LSA matrix over ordered byte pairs, which is what was learned anyway.
v005	0.1875	0.2920	0.5901	C2 — input summation and thresholding, the traditional neuron shape.
v006	0.1875	0.2920	—	D2 — start from the set #omega-p8-frozen records, then drop what settling can recover.
v007	0.1875	0.2920	—	D3 — the placeholder #omega-p8 gives, made precise: greedy removal with re-settling.
v008	0.1875	0.3180	0.5882	E2 — two rounds. Round 1 is #omega-p8-replay-1.
v009	0.1875	0.3180	0.5928	E3 — replay until the trace stops shrinking, capped at 4 rounds (#omega-p8-hyper).
v010	0.1875	0.2920	0.7834	F2 — a fixed budget per k (256 at k=2), kept by TOTAL LSA SUPPORT -- the LSA sum over a token's successor distribution, i.e.
v011	0.1875	0.2920	0.7878	F3 — a fixed budget per k (256 at k=2), kept by SHARPNESS -- the LSA weight of a token's argmax successor less the weight of its runner-up.
p7	0.4844	0.6110	0.8354	the p7 order-1 Markov baseline

What the generation says about the keep rules

Keeping everything is the best predictor, and both prune rules cost more than they save. F3 throws away 177 of 2252 contexts at 10⁵ and gives back most of the k=2 advantage. An earlier version of this page reported the opposite ranking, because it divided a capped model by a 1 KB sample.

But the ranking is not the point, and neither is the reversal: #hutter_metrics is explicit that at k=2 a keep rule cannot be justified or refuted by compression at all — the entire model is bounded at 262 144 bytes, three orders of magnitude below the differences between variants in a. F2 and F3 are preparation for k → 64, where the table is unbounded and the pruning question is real. What they are measured on instead is trace density, coverage, hit rate, value per rule, and settling cost — the diagnostics in gen1-report.

Axes A, B and C move no byte of the archive: under D1 the trace is the causal composite pass and the settled state is never consulted, so v001 through v005 have identical class (b) and class (c) by construction. They are not identical runs — the position dumps and the settling lines differ, which is what the panel shows — and axis B is visible in class (a), where v004's stored backward LPP puts fixed at 91 656 bytes against 26 120. To make A, B or C reach the archive they must be paired with D2 or D3, which is Q4.

What this page is not

From #hutter_publication_handoff §5, which lists what must not be published as a result:

No picks. Eleven variants, six axes, nothing pruned and no axis decided. Publishing a winner would be inventing one.
No chart point. No end-to-end enwik9-scale measurement exists. The memchain interpreter reached 10⁶ only after the append-only collapse, and the compression ladder's published points came from p8-opt.c, a direct-optimization program, not from this interpreter. Nothing here goes on the progress chart.
No small-prefix rate as an estimate. See finding 2. Lower bound, not estimate.
Round-trip failure is not a bug. Step 1 does not require it; v006 and v007 fail it by construction, and the tables carry a column rather than a gate.

What it cost, and why

#hutter_run_costs: anything expensive is published with the shape that makes it expensive, not just a wall clock, because a wall clock alone does not say whether the next size up is twice the cost or a hundred times.

cost shape	because
O(M*N) shift + O(M) learn	p7 has no settling; the cost is the literal chain shift
O(min(M,dumpcap)W256)	the trace is one causal pass; all settling is the diagnostic dump
O(SCW^2256), quadratic in M	one full window settled per candidate recorded byte

The wall that mattered was not p8v2's: the memory chain was implemented as a literal shift register, O(M·N) with N = M, which is ~3 hours at 10⁷ and 10¹⁸ byte moves at 10⁹. It is now stored append-only — a step is O(1), a run is linear, and a 10⁶ run went from 213.3 s to 24.4 s. That is an optimization in the sense of #um_optimizations: it changes how the machine computes and nothing that is reported, the UM still has N cells and 256·N atomic identity patterns, --patterns still enumerates all 256 atoms, and the model file, round-trip and every reported number are byte-identical across the change — which was the condition for admitting it. p1, p3, p4, p7 and p8 still shift literally, which is why the p7 baseline rows stop at 10⁵.

The eight points that need a ruling

Q1–Q4 and Q8 stand as written on 2026-08-04; Q5–Q7 are decisions made on the programmer's behalf and are the ones most worth overruling. Stated in full in the report block; in brief:

Q1. D1. The goal says “surprising under the k=1 argmax”; taken literally the baseline's trace is p7's exactly, so the k=2 layer contributes nothing to class (c) and the model is pure added cost. Generation 1 read D1 as p7's rule shape with the COMPOSITE argmax as predictor. If the literal reading was meant, D1 is a control and the baseline moves to D2.

Q2. Eleven runs, not eight. The protocol's formula over these six axes gives 1 + (2+1+1+2+2+2) = 11; the goal says 8 in four places. If 8 was intended, three alternatives were meant to be dropped and the text does not say which.

Q3. E2's reading. “Relearn k=1 after the k=2 patterns exist” is implemented over the RESIDUAL, because the other reading is a no-op whenever the model round-trips, which it does. Is that the axis you wanted?

Q4. Where axis D goes. Under A1 the token message ASSIGNS, so wherever a k=2 rule fires it overwrites the backward message entirely; the settled argmax at a candidate is then the causal composite prediction, which is wrong there by construction, so no candidate is ever removable. Axis D cannot do anything while axis A is A1. Pair D with A2/A3, change the predicate, or close the axis until f is decided. This one changes the shape of generation 2, so it wants a ruling rather than a pick.

Q5. The message rule, changed. The frozen rule was going to be w_p − (255 − w_s), read as LSA division by the source's doubt. It annihilates every non-absolute message: a learned weight here is 1 or 2 LSA units, so one hop off a clamped position takes every message to 0 and the window settles to byte 0 (measured: 10 of 64 positions correct). It is now min(w_p, w_s) — a message is as strong as its weakest link — and the same measurement is 64 of 64. Both forms are in #omega-p8-hyper.

Q6. Size accounting for the token section, now settled by #hutter_metrics rather than by hand: class (b), capped, reported as bytes and extrapolated in log DSS. Offered because it bears on the log rule: on real enwik9 the distinct 2-gram count grows like M^0.31 over the last decade measured (227, 1074, 2252, 5800, 11845 at 10³…10⁷), which is Heaps-like and faster than log. At k=2 it does not matter — both rules stay under the cap — but at k=3 the cap stops binding and they diverge.

Q7. Settling sweeps. The goal froze W = 128 but not the sweep count. A message travels one position per sweep, so fewer than W leaves the far end unsettled (measured: at 24 sweeps over 64 positions, 10 of 64 correct). It is now W sweeps, and it is why settling a window is O(W²) and therefore why D2/D3 are quadratic in M.

Q8. SN of a token, minor. The event form is settled; the RULE form was not, and --patterns prints rules, so it drops the position clause. Also, concatenating two byte renderings is ambiguous when a byte is not printable.

Q9, turned up by generation 1 and fixed during it. The interpreter could not reach the corpus size, and it was not p8v2's fault — the memchain shift above. Fixed for the wordsv2 kind. What remains open is only the scope: whether to carry the append-only collapse across to p1/p3/p4/p7/p8, and if so whether that belongs here, to the run-ofra cleanups, or to its own push.

Sources

The measured facts and their derivation are cmpr-src's and are symlinked, not copied, so re-running the acceptance script republishes them. This page is ../hutter's, per #hutter_publication_handoff, and is rebuilt by build-p8v2-gen1, which runs no compression and imports gen1-report rather than re-deriving anything.

gen1.tsv	one row per (variant, sample), MEASURED FACTS ONLY, with a p7 baseline row per rung — the three classes as byte counts, binary size, round-trip, k=2 diagnostics, settling cost and its shape
gen1-report.txt	the derived view, verbatim: this page's tables should agree with it line for line
gen1-report	the script that derives it — the authority for a, model_b(DSS), S/U and k
gen1-pos/	per-position dumps, one per (variant, sample), kept up to 10⁴; format in §15.1 of the viz spec
p8v2-diag.py	the k=2 diagnostics, derived from (sample, model file)
acceptance-p8v2	runs the generation. `P8V2_LADDER="e64 e1k e10k" P8V2_SLOW_MAX=1000 tests/pprog/acceptance-p8v2` is about 30 s for all 11 variants
p8v2-words.md	the fixture: all named blocks, every axis alternative, `#wordsv2-v001..v011`
p8v2-viz-spec.md	§15 specifies the generation panel: the .pos format, the cell encoding, the divergence strip, six acceptance criteria

The governing blocks, dumped beside this page. They live in ../cmpr-src, a different cmpr project — cmpr --print-block from the hutter tree will not find them.

#hutter_publication_handoff	the division: cmpr-src generates facts and derives rates, ../hutter publishes; and the list of what is not a result
#hutter_metrics	the three size classes, the extrapolation, the labelling rule, and the worked example of getting it wrong
#hutter_run_costs	a cost is always reported with the shape that causes it
#um_optimizations	an optimization changes how the machine computes and never what is reported
#pprog_p8v2_gen1_report_20260804	the report this page renders
#pprog_p8v2_impl_goal_20260804	the goal: the six axes, what was frozen, scope
#variant_protocol	the method, and why agents do not prune
#pprog_p8v2_questions_20260727	what still needs defining, with MJC's answers inline

Published under #viz_standard. Data symlinked from ../cmpr-src/tests/pprog/; block dumps and this page regenerated by docs/pprog/build-p8v2-gen1. The blocks are the authority; where this page and a block disagree, the block is right.