/* #pprog_p8v2_questions_20260727 @pprog_p8_words_20260706 @pprog_p7_markov_goal_20260704 @pprog_explain_goal_20260722 @pprog_pattern_query_goal_20260706 @pprog_p8_hapax_serialization_goal_20260706 P8v2 / words: block inventory and open questions at the P-programming level (2026-07-27) Scope of this block: the P-level shape of the p8v2 push only -- which OFRA blocks carry over, which are new, and what has to be decided before any of it can be written. Interpreter / umr (--run) implementation questions are deliberately NOT here; they come after these are settled. Source: tests/pprog/p8v2-words.md, "Immediate implementation push" (steps 1 and 2). *"We need feedback from the programmer here." 255. Feedback provided inline. -MJC ## Step 0 (prerequisite, not optional) p8v2-words.md is pure prose: no OFRA fenced blocks, so it blockizes to one anonymous block, has no stable id, and CANNOT be run -- the interpreter resolves the entry block by id and reads its E:/T:/P:/Omega: headers. Before step 1 there has to be a p8v2 program document in the p7 style (fenced ID/Type blocks), either by adding blocks to this file or by writing a sibling. This also retires the wart recorded in #root (no name for the file, :pre/:post aliases need a named neighbour). We'll solve this by importing the blocks that are carried over from p7 as-is. Then we'll add more as stubs and I'll fill them in. -MJC Precedent for "pulled in as-is" is p7-hutter-order-1-markov-v002.md: a doc that just lists the carried-over IDs under E / T / P headings. Same convention applies here. ## Carry-over inventory As-is from p7 (six blocks, byte-identical text): #input ES (see naming wart below) #mem_cell_i ES template #top_mem ES #stdin_single_byte T (clock: 200000 Hz) #stdout_single_byte T #memchain pattern ("The input byte is X." -> "... 1 time steps ago was X." 255.) Same id, REVISED text (they cannot be carried verbatim): #top_recall pattern p7 defines the trace as the cells "not predicted (under argmax) by the event in the cell to their right under mem_cell_markov_1". With k>=2 patterns the predictor is the composite, so the definition of "surprising" moves. The trace FORMAT is frozen (surprising bytes + unary gaps, per p8v2); only the predicate changes. #mem_cell_markov_1 pattern p8v2 relocates the M-1 model: no longer "applied to all mem_cell_i but learned in one place", instead an interior pattern on the input ES from one time step to the next ("it is a bit more elegant to have it be between the input ES and itself"), and it becomes k=1 of the k-indexed family. Step 2 additionally gives it strengths. Probably wants a k-indexed name rather than this one. Yes, so it's strengths, but also it's going to be spread over an area near the input space, so yeah, we'll start by stubbing this out and I'll fill it in. -MJC Dead -- nothing from p8v1 (tests/pprog/p8-words.md) carries over, consistent with calling it a false start: #word_cell_j a word-position-indexed chain; p8v2 has no separate word chain, tokens are joint events over the k byte cells at an arbitrary byte offset. #word_spelling spellings as surprise traces; p8v2 explicitly does not encode words in the trace at all. #top_recall (p8v1 version) dense word-identity recall; dead with the word-identity stream. #words the p8v1 program block. From p4: nothing that p7 does not already carry (p4's #top_recall is the dense pre-markov version). Makes sense. -MJC ## New blocks needed for step 1 program block entry, with E: / T: / P: / Omega: headers. Needs a name (see Q2). The p8 false start was already "words" so this will be "wordsv2". #joint_k ES template, parameter k: the joint event over the k most recent byte events. At k=2 this is at most 65536 atomic events; seeded with the byte ES at k=1 per "the word ES is seeded with all the values of the byte ES". Yes, so this is an ES that will grow according to \omega, so we have to specify how that works. This might be an interesting part of what's new, and it needs to be clear enough that we can determine what we should get from the observability / tracing level, all in P-programming terms, before any implementation / optimization happens. -MJC #joint_and the AND gate. p8v2: "We now introduce the first true AND gate in the UM" -- a joint event over e_1 and e_0 at the SAME time step, threshold/summation shape. This is genuinely new UM machinery and needs its own block; it is not a pattern in the existing sense. Yes, this can also be stubbed and I'll fill in the details. -MJC #joint_predict learned pattern template, From: #joint_k, To: #input. The k=2 rule that overrides the k=1 argmax where it fires. It won't be a strict override anymore, that idea is also gone, replaced by "settling". Some of the details of settling may be yet unclear. This is all part of f, the update function, which is defined (in CMP) as the specific machinery by which P is applied to T, dealing with things like timing ambiguity and so on; exactly the issues that may be underdefined here. -MJC #arbitration how competing predictions resolve into the one argmax the trace is defined against. Step 1's provisional answer is "strength 2 for k=2 against strength 1 for k=1", i.e. longer context wins. This deserves its own block because "multiple levels of predictions competing" is the stated point of the push. This is replaced by settling; see above. It means we don't need another block for it, just one, and this block defines f. So the top-level program defines E, T, P, already as comma-separated lists of blocks, and to this we now add f (and possibly Omega, which was also already included once or twice in an experimental capacity). -MJC Omega block p7 left the Omega: header as an experiment with no block behind it ("It implies that we would have another block below that has type Omega"). p8v2 needs the real thing: the rule that creates, scores, prunes and budgets the joint patterns. This is the first Type: Omega block in the corpus. Right. You can also stub this with "shape" in terms of relation to the OFRA program as a whole but leave the implementation details out. Bias towards brevity rather than speculating on implementation. -MJC hyperparameters p8v2 introduces "hyperparameter" as a term of art ("a pre-committed aspect of the shape of the total pattern"): max k, patterns kept per k, the scan length N, later the max token length. Needs a home -- own block or headers on the program block. Neither; these will be implicit or baked into omega block(s), since that is what \omega is: the generalized learning rule that can not only adjust weights but also add neurons (events) and thus determines how the model architecture grows at runtime. If we wanted headers on the program block it would be because we want to tune these at runtime or without recompilation via CLI flags, but I don't foresee either of those happening now. -MJC ## New blocks needed for step 2 strengths on k=1 revision of the k=1 pattern block: 256 antecedent bytes plus 256 strength bytes, and strengths on k>=2 patterns from the start. #arbitration revised: once strengths are learned rather than assigned by level, "k=2 beats k=1" is no longer automatic. See Q6 -- this is the sharpest collision between the two steps. Right, so we can defer some meaningful work past step 1, so it's worth keeping them separate. This is something I wanted to see. However, we'll still do the k=2 model with full weights and so there's no automatic override of k=1; we'll just let them collide and whatever happens happens, all via LSA, and then we have something interesting to introspect via the obs/tracing stuff before we finish step 2. -MJC #lsa "we will also introduce LSA, with a full complement of algebraic operations". Either a block in this doc or a shared block the doc @refs. Note the standing debt: the Hutter-archive LSA references were deferred at p7 and STILL have not been collected, so there is currently no written authority to point at. Right. I'm leaning towards redefining all of this via a short discussion in chat with Claude. It's foundational enough that discussing it fresh from time to time isn't a great burden, and having multiple redefinitions in different places (which we can compare later) is actually a good thing. -MJC ## Foreseeable before implementation is complete 1. Decoding stays sequential at k=2. The antecedent is the two PRECEDING bytes, so every antecedent is already reconstructed when the consequent is needed: predicted = k2[b(p-2)][b(p-1)] when that context is in the kept table, else k1[b(p-1)]; p=0 always surprising, p=1 falls back to k1. No settling, no abduction, no bidirectional patterns are required for step 1. Most of the p8v2 prose is about the non-causal picture (word events at unknown offsets, omni-directional patterns, rolling-window settling); that is design-ahead, and the push as written is its causal subset. This is the single biggest scoping call -- Q1. Not necessarily. This depends on how the set of surprising bytes is selected. This hasn't been specified fully yet. The selection of bytes to record in the memory trace and the settling / decoding process are dual, i.e. they must be designed and specified together. I still have some work to do here making this fully clear. If f / \omega govern the model at runtime, this may not be specified by either of them, because it is about how the memory trace is sparsified and written. This is where we will get some compression gain. Actually, on reflection, this is part of \omega, because it is about how the singular top-level memory trace is derived, which is a learning operation. So at least we've resolved where this behavior belongs. And the settling is part of f, so everything here is still clearly inside the model. -MJC 2. Size accounting. A fixed pattern budget per k means a fixed-SIZE table, so by the p7 KNOWN RED rule (data-dependent = input-size-dependent-SIZED) the k=2 table goes in the FIXED cost with the binary, the header and the 256-byte k=1 table, NOT in P. P stays gap section + trace bytes, and the zero-input P = 0 invariant survives unchanged. This is a bit densely written (I may have forgotten some context here) so let's expand this section a bit, or else discuss it. -MJC 3. The unary gap code has a ceiling, and we are not near it. Total gap bits <= M, i.e. G <= 0.125 bytes/byte, so the gap section can never cost more than 0.125 of r no matter how far the surprise density falls; the break-even the doc worries about ("once the average gap is ~8 bytes") is at a surprise density of about 1/8, far below where k=2 lands. Keeping the unary code for this push is safe; the k=2 gain is not going to be eaten by it. True. -MJC 4. Query-layer unfolding is a hard requirement, not a nicety. The standing discipline from #pprog_pattern_query_goal_20260706 is that every optimization admitted must come with its unfolding in the query layer. So: --patterns must enumerate the (joint_2, input) ES pair and print those rules in SN, and --explain must add the k=2 line to the stack for a position where the joint rule fired (anticipated verbatim in #pprog_explain_goal_20260722: "the same endpoint gains extra pattern lines in the stack and this surface does not change"). Both are P-level deliverables of this push, not follow-ups. Correct. -MJC Note, this is also largely where the f / \omega decisions become visible, because the patterns that fire are determined by the patterns that are learned per \omega, and the way they lead to the actual prediction is determined by f. -MJC 5. Structural measurables shift: ESs gain the joint ES; atomic patterns gain the kept k=2 rules; longest pattern length goes from 2 to 3. The reported breakdown lines and the fixed-cost line both change shape, and acceptance greps pin them. 6. Naming warts, two of them: the new ES needs a name (Q2), and #pprog_explain_goal_20260722 records that "input" is a poor name for an ES that on recall is the output, with a rename to "memory"/"value" anticipated "soon". A new program doc is the cheap moment to do it; doing it later churns every carried-over block and every fixture. I like "byte" for the formerly-"input" ES. Let's go with it. Also for the top-level memory event, it can literally be "enwik9" (even though, for most of our runs and trials, we won't be using the full dataset, but it'll always be at least a prefix of enwik9 anyway). Actually, a nicety that we can introduce here is (if we have clean UI around it, which I doubt) making the top-level memory event specific to the dataset, including its size, such that when models are written out, they are literally named by the top-level memory events. This deals with a minor annoyance which is that we end up re-running a lot of things because we don't have good ergonomics around serialization of models into a reasonable fs structure. -MJC 7. Determinism of strengths (step 2). Compressor and decompressor must agree bit-for-bit on the arbitration, so learned strengths cannot be IEEE floats computed by two different code paths. The model format has to carry them in an exact representation (fixed-point log, or integer counts), and LSA has to be specified over that representation. They will not be floats but single bytes. Here T is a mapping from E (all the atomic events) to [0,255]. This is all part of the CMP / umr / LSA prior art, which I'll describe afresh wherever it belongs. -MJC 8. Cache and staleness: a new kind string and a new pp_main NL branch; the p1/p3/p4/p7/p8 fixtures and their acceptance tests must stay byte-for-byte green, and cached blocks/binaries must not churn (verified the p7 way: no rewritepl, no cc on existing fixtures). Makes sense. -MJC ## Questions Q1. Is step 1 the causal subset -- k=2 joint patterns whose antecedents are strictly the preceding bytes, sequential decode, no settling, no abduction, no bidirectionalisation? I read the push that way and the rest of the doc as design-ahead. Recommendation: yes, confirm it explicitly in the fixture so the settling prose is not read as in-scope. No. Settling is very much in scope, but it's also going to be a loose settling rule that doesn't care much about correctness, as we focus more on getting this running and getting introspection. I don't even care if the model successfully roundtrips through decompression for step 1. -MJC Q2. Name for the new ES. The doc floats "word", "BPE", "token" and asks for something better. Also: rename #input at the same time, or leave the wart? Let's take #input -> #byte and stick with #token for the BPE-like thing. -MJC Q3. Serialization -- the one #root already flags as undecided and blocking. Does the joint/token model get its own file, or one archive as now? Recommendation: one archive, joint tables next to the k=1 table on the fixed-cost side; there is no evident reason for a second file. Related: #pprog_p8_hapax_serialization_goal_20260706 proposes lifting the model format into its own serialization-layer document. A new model format is required from step 1 anyway ("Also at this time, starting from step 1, we will have a new model format"), so this is the cheap moment to take that split -- or to explicitly drop it, since it was written against the p8v1 branch. Yes, new model format, everything in one file. This would be a good time to integrate the top-level memory singleton ES as having a name that identifies the dataset, as sketched above, and make some discipline around models being written to disk (even if it's just a convention for shell redirections or whatever, above the level of the binary). -MJC Q4. LSI in step 1, or deterministic count-and-argmax again? Step 1 says the joint event is "updated according to LSI"; p7 explicitly dropped LSI for counting and step 2 is where LSA is introduced in full. Recommendation: counting in step 1, LSA in step 2, which matches how the two steps are written -- but the doc's step-1 text says LSI, so this needs a ruling. Let's just do LSI and LSA in full now, put it somewhere not in p8 but in cmpr in a reasonable place. -MJC Q5. Is this the intended concrete omega? Count all (a,b) -> c triples over the scan; for each context (a,b) let c* be the argmax and score it (count[a][b][c*] - count[a][b][k1[b]]), i.e. the number of trace entries the rule removes; keep the top 256 by score; each kept rule costs 3 bytes (a, b, c*) for 768 bytes total. The doc gives the criterion informally ("the value is closely estimated by the number of errors that the word eliminates") and suggests the 256/768 budget; this is the mechanical reading of it. Let's keep the scoring rule unspecified; it's not this. We'll have \omega learning the k=2 patterns via LSA and then we'll have a scoring rule probably based on sharpness, which I'll define later. This will be a mechanical rule over the patterns, not involving counting or data replay (this is already captured sufficiently by the LSI itself). -MJC Q6. Step 2's strength collision. Once k=1 has learned strengths and k>=2 patterns have their own, is arbitration still "longer context wins" with strength as a separate axis, or does it become max-over-strength (so a confident q -> u can outvote a weak two-byte rule)? If the latter, step 1's "strength 2 against strength 1" is throwaway scaffolding and should be labelled as such in the fixture rather than looking like a decision. Yes, strength 2 doesn't even exist anymore as of step 1, because the k=2 patterns have actual strengths. However, there won't be any collision handling, all of this is subsumed under "settling" / f, which remains to be specified (but this will be done in only one place). -MJC Q7. Does step 1 stop at k=2, or land k=2 and k=3 together (256 patterns each, 768 + 1024 bytes)? The doc suggests both but the push text only commits to k=2. Recommendation: k=2 only, so the first measurement isolates one lever. Yes k=2 first. -MJC Q8. The learning scan. p7 learns over the resident chain at first fill. p8v2 says "we hold fixed the amount of space we want to keep ... and then we scan as much of the data as necessary to reach it" and separately "a tunable N input byte events". Is N a hyperparameter, or is the scan driven by budget-fill? And is the k=1 table still learned over the p7 resident chain, or over the same scan as the joint patterns? So the idea is that we replay after every change in the pattern shape, and then this will repeat. We don't actually *need* it now, but for forward-compatibility let's try to get this in. So p7 learns the k=1 model, over the zero-compression memory trace that is literally just the enwik9 joint event serialized in the obvious way. Then conceptually what we have is: the input streams into memory, p7 happens, we get a sparsification of the memory trace, then p8 step 1 happens, and we get another sparsification. Which means that p7 should be replayed going into p8. The fact is that this replay can also be parallelized / optimized into a single pass, but that's likely to not always be true in the future. Let's see how the implementation effort goes, but most likely have an actual replay of the entire joint event from p7 to p8 because it's conceptually simpler and the other thing is an optimization. -MJC Q9. The other half of the tracing ergonomics ask ("where errors occur and which patterns cause them"). --explain answers the second half but only for a position you already know. Is finding the interesting positions in this push -- a verb that lists trace entries / positions where a k=2 rule fired or was overridden -- or a separate one? Let's keep it separate; we have the data to see what's going on via --explain; we can manually advance and look for interesting cases; that's enough for now. -MJC Q10. What does p8v2 have to beat to be a real data point? I assume: strictly better r than p7 on the same samples at every M (no crossover regime, since the tables are fixed-size and cost nothing asymptotically), round-trip byte-identical, random data still round-trips, zero-input P = 0. p8v1's numbers are not a target since the chart connects to p7. Correct. Beating p7 is guaranteed by construction as long as we don't badly screw up the encoding of the k=2 patterns, which we won't. However, round-tripping at all isn't guaranteed for step 1, and I'd rather have something that matches the p8v2 implementation intent exactly and doesn't work; much rather have that than the reverse of something that works and doesn't match the intent; then we can iterate on the --explain machinery and understand the gap. -MJC I would almost say it should be expected that we don't roundtrip on the first few runs; this counts as a success, because it's the most efficient way for us to nail down the f and \omega and LSA stuff which is all landing at once and which definitely (from experience) goes beyond most prior art in the training set and hence doesn't match Claude's intuitions at all and tends to be implemented incorrectly when underspecified. -MJC */