/* #hutter_e64_viewer_prereq_20260805 @hutter_publication_handoff @pp_wordsv2 @um_optimizations @pprog_pattern_query_goal_20260706 @hutter_metrics Reply to ../hutter's #p8v2_e64_viewer_goal_20260805: its two prerequisites, plus three facts about e64 that the goal has wrong and that would produce a wrong viewer The goal blocks itself on two things it says are ours. One is ours and is trivial. The other rests on a k=0 pattern that does not exist, and the honest fix is a different thing than the one asked for. Separately, three of the goal's stated facts about e64 do not match the runs. ## 1. Retaining the e64 models -- ours, and it is two lines Confirmed: acceptance-p8v2 builds every model at $tmp/$v.$s.m under a mktemp -d with a `trap 'rm -rf "$tmp"' EXIT`, and only .pos survives, copied to gen1-pos/ under the DUMP_MAX gate. The models are discarded exactly as the goal says. The cost of keeping them is nothing. Model size is 48 + 256 + 256 + BWD + TB + G + SC, and from gen1.tsv: e64 762 bytes for ten of the eleven; 66298 for v004, which alone carries the 65536-byte backward LPP under B2. Eleven models: 74 KB. e1k 1766-1785, except v004 at 67321. e10k 8826-10850, except v004 at 76236. So retaining every model at every sample the dumps already cover costs about 350 KB for all 33. There is no reason to special-case e64: the same DUMP_MAX gate that keeps the .pos file should keep the model beside it, and then e1k and e10k are available for free if the page ever wants them. The change sits next to the existing copy at the bottom of the variant loop in acceptance-p8v2: mkdir -p "$SCRIPT_DIR/gen1-models" beside the existing mkdir for gen1-pos, and, in the same `if [ -f "$m.pos" ] && [ "$n" -le "$DUMP_MAX" ]` arm, cp "$m" "$SCRIPT_DIR/gen1-models/$v.$s.m". One question in it, below with the others: whether 350 KB of binary model files belong in the repo next to the .pos dumps, which are text. ## 2. There is no k=0 pattern, so --patterns is right not to print one The goal asks that "the k=0 background be nameable by the query layer", cites --patterns as the right surface, and calls the omission "a defect in the query layer". The omission is correct. What is actually missing is a different object with different consequences, so it is worth being exact. WHAT DOES NOT EXIST. #wordsv2's P header is memchain, mem_cell_markov_1, token_predict, token_and, top_recall. No k=0 pattern is declared, in the baseline or in any of the eleven variants. The structural line agrees; a v001 e64 run prints structural: total ESs 130, total atomic patterns 16693 (memchain 16384 + k1 table 256 + backward 0 + token 47 + top_recall 6) and the k=0 background contributes ZERO atomic patterns to it. --patterns is indexed by ES PAIR and enumerates atoms of P. There is no pair to file the background under, and manufacturing one would put atoms in the query layer that the structural line says the machine does not have. By #um_optimizations that makes the query layer wrong, not the structural line. WHAT DOES EXIST, and the goal is right that the page cannot settle without it. 256 LSA bytes at offset 48 of the model file, between the 48-byte header and the 256-byte k=1 argmax table. It is class (a) and the run says so: "class (a) fixed 26120 bytes = binary 25560 + header 48 + k0 256 + k1 table 256 + backward 0". learn_k0 builds it with one lsa_inc per input byte, so at e64 it is almost flat -- fourteen entries at 1, eleven at 2, two at 3, the rest 0 -- and the flatness is the reason the exact bytes matter: settle_argmax breaks ties to the SMALLEST INDEX, so a viewer that substitutes a uniform background gets different argmaxes at the positions no message reaches. Nothing on the causal path reads it. It is the initial state of every unclamped position in the settling window and nothing else. In UM terms that makes it an initial T learned by omega and stored, not a p_ij: a prior over the byte ES with no antecedent event. The design already names it correctly -- #f-p8 calls it the background state that "by the Markov property returns to a static distribution", i.e. the M-0 model, and p8v2-words.md's Axis A section says "the k=0 background (the M-0 byte-frequency model, 256 LSA values learned by omega)". Only the goal calls it a pattern. THREE WAYS TO DELIVER IT, and which one is the programmer's call: (a) READ IT OFF THE RETAINED MODEL at offset 48. Costs nothing once prerequisite 1 lands. It is the model's own background, so it dissolves neither boundary the goal was protecting -- it is not the builder running compressions and it is not reconstructing the background by counting bytes. What it does do is bypass the query layer, which #pprog_pattern_query_goal_20260706 says is where introspection goes, and it puts a hard-coded model offset in ../hutter's tree. (b) GIVE IT A QUERY-LAYER SURFACE THAT IS NOT --patterns. It is state, not structure, so it wants its own verb rather than an ES pair. This keeps introspection in one place and keeps the structural line honest, at the cost of a new flag and its documentation. (c) DECLARE A BACKGROUND ES with a single always-present event and make the background 256 real atomic patterns from it. This is the only option under which --patterns is the right surface and the goal's framing is correct as written. It changes #wordsv2's E and P headers, adds 256 to the structural line of every variant, and is a design change to p8v2 -- not a query-layer fix -- so it would have to go through the axes rather than be slipped in for a viewer. ## 3. Three facts about e64 the goal has wrong These are not quibbles. The goal's mockup bakes each of them into the viewer's controls, and each would make the page disagree with the dumps it claims to conform to. W IS 64 AT e64, NOT 128. The goal says "W = 128 is frozen and the sample is 64 bytes, so there is no window boundary", and its mockup reads `steps:128` and `( 14/128 )`. settle_window is called with a window (ws, we) and takes W = we - ws; #pp_dump settles in consecutive non-overlapping windows of 128 over the first D = min(M, cap) positions, so at e64 there is exactly one window, ws = 0, we = 64, and W = 64. The number of time steps is W, so e64 runs 64 of them, not 128. 128 is the architecture; 64 is what the dumps record. The conclusion the goal drew -- one window, no tiling, every position on screen -- is right; the number is not. CONV = 64 IS THE DID-NOT-CONVERGE SENTINEL AT e64. conv is initialised to W, so at e64 "never converged" reads as 64, not 128. It is not hypothetical: v004 (B2) and v005 (C2) report conv 64 at every position, and a viewer that treats 128 as the sentinel will draw those two as converging on the last step instead of not converging. THE AXES ARE MORE VISIBLE THAN THE GOAL CLAIMS, AND NOT ONLY A/B/C. The goal says D, E and F change nothing at e64. That is true of the class (c) rate -- all eleven record 14 trace bytes of 64, a = 0.2188 -- but not of settling, which is what the page draws. From gen1.tsv at e64: settled_ok v001 64 v002 27 v003 24 v004 17 v005 12 v006 64 v007 64 v008 64 v009 64 v010 64 v011 64 mean_sweeps 27, 14, 18, 64, 64, 27, 27, 59, 59, 27, 27 removed 20 for every variant except v008 and v009, which remove 57 The axis digits run A B C D E F in order, so v008 = 111121 is E2 (replay-2) and v009 = 111131 is E3 (replay-dry); v006/v007 are the D variants and v010/v011 the F variants. Sorted by axis at e64: A v002 settled_ok 27, converges at 14; v003 24 at 18. Moves everything. B v004 17, NEVER CONVERGES (conv 64 at every position). C v005 12, NEVER CONVERGES. D v006, v007 identical to v001 in every diagnostic the TSV carries. E v008, v009 still settle 64 of 64 but converge at 59 instead of 27, AND carry a DIFFERENT learned k=1 table and different token weights: replay relearns k=1 against the k=2 mask, so the table it ends with is worse on its own, which is why the diag's `removed` (positions where k=2 is right and the k=1 table is wrong) is 57 against everyone else's 20. F v010, v011 identical to v001 -- at 64 bytes every context is kept, so a prune rule has nothing to prune. So "D, E and F change nothing at e64" is right about D and F and wrong about E, and E is wrong in the most interesting way available: it is the one axis at this sample that changes what was LEARNED rather than only what settling does with it. ## What the page can already have, today 33 .pos dumps in tests/pprog/gen1-pos, eleven of them e64, with the header and column meanings in section 15.1 of p8v2-viz-spec.md. The k=2 rules in SN, once prerequisite 1 lands. The spelling is `--patterns-from token --patterns-to input`: 47 rules, 94 lines, at e64. Note `input`, not `byte` -- #byte renamed p7's #input and deliberately kept its sentence, but the ES DESIGNATOR was never renamed, so `--patterns-to byte` errors with "unknown ES designator 'byte'; valid: input, mem_cell_, token, top_mem". Whether to accept `byte` as an alias is a fourth question below. --explain byte --where "The position is N." --model , which at e64 position 12 answers with the token event at its start position and then the rule, exactly as the goal's detail pane wants. gen1.tsv, whose e64 rows carry the settling diagnostics quoted above. ## Also ours, and the goal is right to flag it p8v2-words.md line 775, in the Axis A section, still says "then 24 synchronous sweeps". The runs do W sweeps and have since Q7; v001 at e64 reports conv 27, which 24 sweeps cannot produce. The prose is stale and the runs are correct. Fixing it is a prose edit in our tree and should say W, not 128, for the reason in section 3. *"We need feedback from the programmer here." 255. The four questions, in order of how much they change: 1. k=0: (a), (b) or (c) above -- and if (b), what the verb is called. I don't understand why k=0 exists anywhere in code; I wrote it merely as a throwaway line about the k indexing, that k=0 would have a nice semantic as the prediction before any observation i.e. the base rate. The top-level memory seeds the model at known positions and all others are derived by patterns via settling. If k=0 exists anywhere else it's an error and how you handle it is the viewer I could not care less. (The correct way is to have a pattern from "true" onto each relevant atomic event, and let f run those whenever the base frequencies are used by the implementation.) In any case, it's wrong and will go away. -MJC 2. Do the retained models go into the repo as binary files next to the text dumps, or somewhere else? Make a models/ dir for them, something like p8v2 or perhaps the model version or whatever, and then by enwik9 (the dataset) and a prefix length. -MJC 3. Does F get a control on the e64 page, given that it moves settling but not the rate? Sure, let's establish a convention of having controls for all the axes. We're setting up a repeatable process here, if you can actually get some velocity going for once. -MJC 4. Should the ES designators accept `byte` as an alias for `input`, or does the query layer keep p7's spelling deliberately? We renamed it; I don't understand why we're still talking about this. -MJC ## RESOLVED 2026-08-05 -- ../hutter IS UNBLOCKED, and the k=0 prerequisite was the wrong question Answers above; what landed follows. #p8v2_e64_viewer_goal_20260805 should drop its "UNTIL BOTH LAND this goal is blocked" and read this section instead. PREREQUISITE 1, THE MODELS: DONE. acceptance-p8v2 now keeps every model under the same DUMP_MAX gate that keeps the .pos dump, laid out by program version, dataset and prefix length in bytes: tests/pprog/models/p8v2/enwik9/64/wordsv2-v001.m .. wordsv2-v011.m tests/pprog/models/p8v2/enwik9/1000/... tests/pprog/models/p8v2/enwik9/10000/... Fallback samples go under models/p8v2/fallback-not-enwik9/ so nothing there can be read as enwik9. Reach them the same way as the dumps: ../hutter's `data` symlink already points at tests/pprog. Query them with --patterns / --explain exactly as the goal's builder wants. PREREQUISITE 2, k=0: WITHDRAWN, AND THE QUERY LAYER IS NOT AT FAULT. The programmer's ruling is above: k=0 was a throwaway remark about the k indexing and the 256-byte table the implementation reads directly is an error. The correct form is a pattern from an always-present "true" event onto each atomic event, fired by f like any other -- which is option (c), and which makes it visible to --patterns and to the structural line with no special case, at which point there is nothing left for ../hutter to ask for. Recorded as a DEFECT in #pp_wordsv2 and noted in p8v2-words.md's Axis A section. Not scheduled. WHAT THE VIEWER SHOULD DO IN THE MEANTIME: read the 256 bytes at offset 48 of the retained model (option (a)) and treat them as provisional, because they will not survive. Do not build any control, label or panel around "the k=0 background" as an object -- it is not one, and the page would then have to be rewritten when the true-event patterns land. The viewer needs those bytes only as the initial state of unclamped positions, and that requirement does not change. QUESTION 3, CONTROLS FOR ALL AXES: the convention is now that every axis gets a control, D, E and F included. Section 3 above sorts the e64 evidence by axis: E has something to show (v008/v009 converge at 59 instead of 27 and carry a different learned k=1 table), and D and F genuinely change nothing at 64 bytes. A control whose alternatives coincide at this sample is still a control; the page says they coincide HERE rather than omitting them, which is also what makes the same page reusable at e1k. QUESTION 4, THE ES DESIGNATOR: FIXED. `--patterns-from`/`--patterns-to` now take `byte`; `input` is still accepted so p7 and its fixtures are untouched, and the p8v2 error message leads with `byte`. Verified against a v001 e64 model: `--patterns-from token --patterns-to byte` gives the same 47 rules, 94 lines, as the old spelling. The SN sentence stays "The input byte is X." -- #byte kept p7's wording on purpose so carried-over patterns and the query layer stay byte-identical, and only the block id and now the designator changed. THE STALE PROSE: FIXED. p8v2-words.md's Axis A section said "24 synchronous sweeps"; it now says W sweeps, says W is 128 or what is left of the sample (so 64 at e64), and says the conv sentinel moves with the window. The three e64 facts in section 3 are therefore in the spec the viewer follows, not only in this block. ## SIX THINGS THE VIEWER WILL HIT THAT THE GOAL DOES NOT MENTION Added after the answers, for a clean start rather than a discovery halfway through. 1. TWO OF THE ELEVEN CANNOT BE CONFORMANCE-CHECKED IN JS, AND THE GOAL'S CHECK MUST SAY WHICH. LSA is stochastic: lsa_inc, lsa_add and lsa_sum256 all draw from ONE global xorshift32 stream (lsa_state, seeded 0x5EED1234), and that stream is shared with learning -- by the time settling starts it sits wherever learn_k0, learn_k1 and learn_tokens left it. lsa_max, lsa_min and lsa_sub draw nothing. A1 uses lsa_max and assignment only, so its settling is DETERMINISTIC and reproducible from the model alone. Nine of the eleven are axis A = 1 -- v001 and v004..v011 -- and the goal's conformance check will pass for all nine. A2 (v002) adds each fired message entrywise with lsa_add and then lsa_sub 2; A3 (v003) adds with lsa_add and renormalises with lsa_sum256. Both consume the stream, and the draw COUNT is data-dependent (lsa_bits_zero returns without drawing when n <= 0 or n > 31), so it cannot be computed and skipped past. Reproducing v002 and v003 means implementing xorshift32 from 0x5EED1234 and making every draw the C makes, in order, including all of learning. So the page should report conformance PER VARIANT and name these two as out of reach by construction, instead of marking the whole engine UNVERIFIED. Everything the goal wants to show on axis A -- v001 settling to the true byte where v002 does not -- is still showable; it is the step-by-step frames of v002/v003 that are the viewer's own, not their endpoints, which are recorded in the dumps. 2. k=0 IS READ DURING SETTLING, NOT ONLY AT INITIALISATION. Under B1 the backward message, when the argmax table has no predecessor at all for the byte b, falls back to min(k0[x], wsrc) for every x. So the defect in section 2 is not confined to the initial state; the background is also standing in for a pattern inside a message. It does not change what the viewer must do (read the 256 bytes, treat them as provisional) but it is a second place to get right. 3. THE B2 CONTROL NEEDS A 65536-BYTE LPP AND ONLY v004 CARRIES ONE. It sits at model offset 48 + 256 + 256, length = the BWD header field, and is nonzero only for v004. At e64 the same matrix is valid for every variant that shares v004's learned k=1 -- v001..v007, v010, v011 all have byte-identical k0, k=1 table and token sections -- but NOT for v008/v009, which relearn under E and end with a different table. So B2 can be driven honestly on nine axis vectors and not on the other two, and the page should refuse rather than silently borrow. Related: --patterns prints the k=1 family from the ARGMAX TABLE at strength 1 (operational support constants, per #pprog_pattern_query_goal_20260706), so the learned LPP weights are not in the query layer even when the model stores them. Same shape of gap as k=0, but this one is a real pattern family. Not a blocker -- read it off the model -- but do not expect --patterns to show it. 4. conv IS PER-WINDOW, NOT PER-POSITION. Every row of a window carries the same conv value; the .pos column is per-position only in layout. At e64 there is one window, so all 64 rows repeat it. 5. THE Q5 MESSAGE RULE IS NOT IMPLEMENTED ANYWHERE. w_p - (255 - w_s) exists only as recorded prose in p8v2-words.md's message-rule entry, as the first form tried and rejected. It is not one of the eleven, there is no dump of it, and nothing in gen1.tsv measures it. Driving it is exactly what the goal wants, and it is legitimate -- but every frame under it is unverified BY CONSTRUCTION and carries the fence, including the endpoint, which for the eleven is the one thing that is pinned. The goal's "10 of 64 correct against 64 of 64" is in no dump and no TSV row; if it is a measurement its source needs naming, and if it is an expectation the page must not print it as a number. 6. THE BUILDER'S DATA LINK. docs/pprog/build-p8v2-gen1 symlinks named items out of cmpr-src's tests/pprog via its LINKS list rather than copying. The e64 builder wants one more entry, ("models", "models"), which reaches models/p8v2/enwik9/64/. Keeping to that mechanism is what makes "re-rendering from existing dumps costs nothing and needs no run" true for this page too. Written 2026-08-05. ## SUPERSEDED IN PART, 2026-08-06: THE k=0 BACKGROUND IS GONE The programmer's ruling above ("in any case, it's wrong and will go away") was carried out with generation 2, on the instruction in #pprog_p8v2_gen2_goal_20260806. What this block says about k=0 was right about the diagnosis and is now out of date about the remedy, so: WITHDRAWN: the whole of option (a) and the paragraph "WHAT THE VIEWER SHOULD DO IN THE MEANTIME". There are no 256 bytes at offset 48 to read. The model layout is 48 + 256 (the k=1 argmax table) + BWD + TB + G + SC, every offset past the header moved down 256, and a pre-2026-08-06 model is caught by the size identity. ../hutter's docs/pprog/build-p8v2-e64 and p8v2-e64.tpl.html are updated. WITHDRAWN: item 2 of the six-things list ("k=0 IS READ DURING SETTLING, NOT ONLY AT INITIALISATION"). Both readers are gone. An unclamped position initialises to 256 zeros, and B1's empty predecessor set now sends NO BACKWARD MESSAGE rather than the background -- no pattern has that byte as its consequent, so no pattern fires. There is nothing provisional left for the viewer to handle. NOT REPLACED BY OPTION (c) EITHER. #pprog_p8v2_gen1_choices settles it: the "true"-event pattern was a remark about how a base rate WOULD be encoded if one were needed, and it is not -- "the Markov model converges to the char frequency, so if we needed it, it would be available; however we don't need it and it adds no value." E and P are unchanged and the structural line is unchanged. STILL TRUE, and now more so: item 1. LSA is stochastic and the stream is shared with learning, so A2 and A3 cannot be conformance-checked in JS without replaying every draw. tests/pprog/p8v2-replay.py now DOES that replay, in Python, and checks itself against the retained model and the .pos dump before its output is used; it is the reference a JS engine can be checked against position by position, which the page could not have before. AND ONE NEW THING THE VIEWER MUST MATCH: the accumulating updates (A2, A3) pin the nesting of their entrywise lsa_add as MESSAGE-OUTER, ENTRY-INNER. The two nestings consume the shared stream in different orders and give different runs; generation 1's v002 and v003 had been generated with different nestings, which is now fixed in #pp_wordsv2 and in the template's runF. Item 4 (conv is per-window) and item 3 (only one model carries the LPP per generation) are unaffected. */