\documentclass[11pt,a4paper]{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amssymb,amsthm}
\usepackage{booktabs}
\usepackage{hyperref}

\newtheorem{theorem}{Theorem}
\newtheorem{prediction}{Prediction}
\newtheorem{question}{Question}

\title{Experimental Design: Scaling the Total Interpretation\\to Full enwik9}
\author{Claude and MJC}
\date{February 11, 2026}

\begin{document}
\maketitle

\begin{abstract}
We have completely interpreted a 128-hidden Elman RNN trained on 1024 bytes of enwik9:
Boolean automaton, factor map, pattern chains, weight construction, E$\to$N$\to$Q quotient
chain.  Now the same architecture has been trained on the full $10^9$-byte enwik9.
Two training runs are documented: Run~1 (SGD, exploded at 125M) and Run~2 (Adam,
best 2.81~bpc at 110M, catastrophic forgetting by 450M).  This paper designs
experiments to extend the interpretation framework to the full-scale model and
makes concrete predictions about what we will find.
\end{abstract}

\section{The Two Models}

\begin{table}[h]
\centering
\begin{tabular}{lcc}
\toprule
& \textbf{sat-rnn-1024} & \textbf{sat-rnn-enwik9} \\
\midrule
Training data & 1024 bytes & $10^9$ bytes \\
Epochs & $\sim$1024 & 1 (online) \\
Best bpc & 0.079 & 2.81 \\
Architecture & 128 hidden, 82K params & identical \\
Optimizer & SGD (converged) & Adam (clip=1.0) \\
Best checkpoint & epoch1024\_final.bin & epoch1\_110M.bin \\
Training regime & memorization & generalization \\
\bottomrule
\end{tabular}
\caption{The two models share the same architecture but live in different regimes.}
\end{table}

The sat-rnn-1024 was trained for $\sim$1024 epochs on 1024 bytes---pure memorization.
Every pattern in the data is perfectly learned.  The model achieves 0.079~bpc (Shannon
limit on this window is $\sim$0.067 via order-12 pattern chains).

The sat-rnn-enwik9 was trained online for one pass through enwik9.  It saw each byte
exactly once.  Performance peaked at 110--150M bytes (2.81--2.86~bpc), then degraded
due to catastrophic forgetting:

\begin{itemize}\item 110M: 2.81 bpc (best)
\item 300M: 3.0 bpc (slow drift)
\item 400M: 3.2 bpc
\item 450M: 4.5 bpc (cliff --- output biases collapsed to all-negative)
\item 735M: 4.5 bpc (run killed, $b_y$ range $[-25, -7]$)
\end{itemize}

$W_h$ standard deviation grew linearly at 0.02 per checkpoint and never stabilized.
Adam's per-parameter adaptive rates accelerated forgetting: it confidently reshaped
weights to fit the current distribution.

\subsection{The Optimized Variant}

A third run used modern techniques: Xavier init, cosine LR schedule, AdamW ($w_d = 0.01$),
label smoothing ($\varepsilon = 0.10$).  It reached 3.70~bpc at 20M chars before being
stopped.  This provides a comparison point for whether training improvements change the
interpretation story.

\section{What the Theory Predicts}

The total interpretation of sat-rnn-1024 established seven structural facts.
Each makes a specific prediction for sat-rnn-enwik9.

\subsection{P1: Boolean Automaton Regime}

\begin{prediction}[Weakened but present]
The sat-rnn-enwik9 will still operate in the Boolean automaton regime,
but with reduced margins.  Mean margin will drop from 60.5 to $\sim$5--15.
Some neurons ($\sim$10--20\%) may have margins below 1.0, making the mantissa
computation-relevant for those units.
\end{prediction}

\textbf{Reasoning.}  The tanh activation drives saturation structurally:
any $|z| > 3$ yields $|\tanh(z)| > 0.995$.  With $W_h$ std growing to 3.3
(from the training log's weight statistics) and 128 inputs summed per neuron,
the pre-activations will still be large.  But the model has less time to push
margins to extremes (1024 epochs vs 1 epoch).

\textbf{Key test.}  If margins drop below 1.0 for $>$30\% of neurons, the Boolean
automaton model breaks down and we are in a genuinely analog regime.  This would be
the most important finding.

\subsection{P2: Neuron Roles Change}

\begin{prediction}[Distributed, not concentrated]
In sat-rnn-1024, neuron h28 alone captured 99.7\% of the compression gap.
In sat-rnn-enwik9, no single neuron will capture more than 30\% of the gap.
The top 15 neurons will capture $\sim$70--80\%, not $>$100\%.
\end{prediction}

\textbf{Reasoning.}  With 1024 bytes, the model overfits massively---a single
neuron can memorize the output distribution.  With $10^9$ bytes, the model must
use its capacity more evenly.  82K parameters across 128 neurons gives
$\sim$640 effective parameters per neuron.  No single neuron can carry a
$10^9$-byte task.

\subsection{P3: Offset Structure Deepens}

\begin{prediction}[Shallower dominant offsets]
Mean dominant offset will decrease from d=18--25 to d=5--15.
The model will rely more on recent context because it cannot afford
the long-range correlations that memorization allows.
\end{prediction}

\textbf{Reasoning.}  The BPTT-50 gradient window is the same, but with
diverse data, the model must learn features that generalize.  Short-range
features (word boundaries, HTML tag patterns, bigram statistics) are more
universal than long-range ones (specific article structure).  The model
will trade depth for breadth.

\textbf{Alternative prediction.}  If the model is still fundamentally
in Boolean automaton regime, the W$_h$ diagonal carry (shift-register
structure) may preserve long offsets.  In that case, offsets may stay
deep but become noisier.

\subsection{P4: Factor Map Degrades}

\begin{prediction}[Lower $R^2$, more complex features]
The 2-offset conjunction model (mean $R^2 = 0.837$ for sat-rnn-1024)
will fit worse for sat-rnn-enwik9: mean $R^2 \approx 0.4$--$0.6$.
The dominant pair (1,7) will no longer account for 52/128 neurons.
Instead, multiple offset pairs will split the population more evenly.
\end{prediction}

\textbf{Reasoning.}  The factor map captured a model that was essentially
computing ``what character was 1 step ago AND what character was 7 steps ago.''
On 1024 bytes, this is enough because the data has limited diversity.
On $10^9$ bytes, neurons must compute more complex functions of the input
history to be useful.  The 2-offset conjunction is a lower bound on the
computation; the real model may be doing 3- or 4-offset conjunctions,
or genuinely nonlinear combinations.

\subsection{P5: Weight Construction Gap Widens}

\begin{prediction}[Analytic construction degrades, optimization compensates]
The fully analytic construction (shift-register + log-ratio $W_y$, 1.89~bpc
on 1024 bytes) will yield $>$5~bpc on enwik9.  But the optimized readout
(shift-register dynamics + gradient-optimized $W_y$, 0.59~bpc on 1024 bytes)
will still achieve $\sim$3--4~bpc on enwik9, competitive with the
trained model's 2.81.
\end{prediction}

\textbf{Reasoning.}  The analytic $W_y$ comes from skip-bigram log-ratios.
On 1024 bytes, these statistics are complete (you can count every bigram
exhaustively).  On $10^9$ bytes, the statistics are accurate but the readout
must compress a much larger conditional distribution into $128 \times 256$
weights.  The shift-register dynamics, however, are data-independent---they
are a hash function.  So the dynamics should transfer, and only the readout
needs retraining.

\subsection{P6: Hebbian Covariance Improves}

\begin{prediction}[Higher correlation with more data]
Hebbian covariance $\text{cov}(h_j(t), h_i(t+1))$ will correlate with
trained $W_h$ at $r > 0.56$ (the value for sat-rnn-1024).
Estimate: $r \approx 0.65$--$0.75$.
\end{prediction}

\textbf{Reasoning.}  Hebbian covariance captures the data's temporal
structure.  On 1024 bytes, the covariance is computed from a tiny sample;
on $10^9$ bytes, it averages over massive statistics and should give a
cleaner signal.  The trained $W_h$ at the 110M checkpoint has only had
one pass of data---less time to develop non-Hebbian structure.
The gap between Hebbian and trained should shrink.

\textbf{Caveat.}  If the model is in catastrophic-forgetting mode,
the covariance over the full $10^9$ bytes may not match the $W_h$ at
any single checkpoint, because the data distribution drifts.
We should compute Hebbian covariance over the \emph{first} 110M bytes
and compare to the 110M checkpoint.

\subsection{P7: E$\to$N$\to$Q Framework Holds Structurally}

\begin{prediction}[Same framework, different numbers]
The E$\to$N$\to$Q quotient chain applies identically---it is structural,
not data-dependent.  The quotient values change (larger $Q$ at each layer
because the model is compressing less), but the decomposition
$Q_{\text{total}} = \prod Q_{\text{layer}}$ still holds.
\end{prediction}

\textbf{Reasoning.}  The quotient chain is a mathematical property of
the UM forward pass, not of the training data.  $E \to N$ (counting)
and $N \to Q$ (dividing by count) are the same operations regardless
of the data source.  What changes is the \emph{magnitude} of quotients:
on 1024 bytes, the model compresses to 0.079~bpc (high quotients);
on $10^9$ bytes, it compresses to 2.81~bpc (quotients closer to 1).

\section{Experimental Program}

\subsection{Phase 1: Characterize the 110M Checkpoint}

The best checkpoint (epoch1\_110M.bin, 2.81~bpc) is the primary target.
Run the existing Q1--Q7 analysis tools on it.

\begin{enumerate}\item \textbf{Q1: Boolean automaton.}  Run q1\_boolean.c, q1\_margins.c
  on the 110M model.  Measure margin distribution, sign vs.\ mantissa
  compression, bit leverage.  Key question: is it still a Boolean automaton?

\item \textbf{Q2: Offsets.}  Run q2\_offsets.c.  Measure per-neuron
  dominant offset, MI profile.  Test P3 (shallower offsets?).

\item \textbf{Q3: Neuron knockout.}  Run q3\_neurons.c.  Rank neurons
  by compression contribution.  Test P2 (distributed vs.\ concentrated).

\item \textbf{Q4: Saturation dynamics.}  Run q4\_saturation.c.
  Measure dwell times, co-flip clusters.  Compare to sat-rnn-1024
  (mean dwell 3.3).

\item \textbf{Q5: Redux.}  Run q5\_redux.c.  Can we find a sparse
  sub-network?  How many neurons are needed for 90\% of the gap?

\item \textbf{Q6: Justifications.}  Run q6\_justify.c on specific
  enwik9 positions.  What does the backward attribution chain look like
  on real Wikipedia text?

\item \textbf{Q7: Algebraic structure.}  Run q7\_algebraic.c,
  q7\_higher\_order.c.  Does PMI alignment improve or degrade with
  more data?
\end{enumerate}

\textbf{Eval data.}  All measurements should be taken on a held-out
window: bytes 110M--111M (the first 1M bytes the model has never
trained on at this checkpoint).  This gives a clean generalization measure.

\subsection{Phase 2: Factor Map on enwik9}

Run the factor map analysis (factor\_map.c and variants) on the 110M
checkpoint:

\begin{enumerate}\item Per-neuron best 2-offset conjunction: does $R^2$ drop as predicted?
\item Are there neurons where 3-offset conjunctions significantly improve fit?
\item Does the word\_len feature still dominate?  On real Wikipedia,
  word boundaries are more regular than on the 1024-byte XML fragment.
\item What is the in\_tag feature's role?  On full enwik9, tag density varies.
\end{enumerate}

\subsection{Phase 3: Weight Construction Transfer}

Test whether the weight construction framework transfers:

\begin{enumerate}\item \textbf{Compute skip-bigram statistics} on the first 110M bytes.
  Build the analytic $W_y$ from log-ratios.
\item \textbf{Shift-register dynamics + analytic $W_y$}: what bpc?
  (Prediction: $>$5~bpc.)
\item \textbf{Shift-register + optimized $W_y$}: train the readout
  on the 110M prefix.  (Prediction: 3--4~bpc, competitive with trained.)
\item \textbf{Hebbian $W_h$}: compute covariance on first 110M bytes.
  Compare to trained $W_h$ at 110M checkpoint.  (Prediction: $r > 0.56$.)
\item \textbf{Full analytic model on enwik9}: all weights from data
  statistics, zero optimization.  How does it compare?
\end{enumerate}

\subsection{Phase 4: Checkpoint Trajectory}

The training produced checkpoints at 100M, 110M, 120M, \ldots, 990M.
This is a unique opportunity to study the interpretation as a \emph{function
of training progress}.

\begin{enumerate}\item \textbf{Margin trajectory.}  Plot mean margin vs.\ training
  byte offset.  Does the Boolean automaton regime strengthen or weaken
  over training?
\item \textbf{Neuron role stability.}  Do the same neurons dominate
  at 110M and 200M?  At what point do roles destabilize?
\item \textbf{Forgetting signature.}  The cliff at 450M (bias collapse)
  is visible in the training log.  What happens to the Boolean automaton
  structure at that transition?  Is there a detectable precursor?
\item \textbf{$W_h$ drift.}  Std grew at 0.02/checkpoint.  Does the
  Hebbian correlation change monotonically, or does it peak and decay?
\end{enumerate}

\section{Key Questions Beyond Q1--Q7}

The full-enwik9 model opens questions that didn't arise on 1024 bytes.

\begin{question}[Generalization vs.\ Memorization]
The sat-rnn-1024 memorizes: it sees each byte $\sim$1024 times.
The sat-rnn-enwik9 generalizes (or fails to): it sees each byte once.
Do the weights encode \emph{rules} (generalizable patterns) or
\emph{instances} (memorized fragments)?  The Hebbian/analytic construction
tests this directly: if the analytic model is competitive, the learned
rules are simple enough to derive from statistics.
\end{question}

\begin{question}[Capacity Saturation]
82K parameters, 128 hidden units.  How many bytes of enwik9 can this
architecture actually learn?  The training log suggests $\sim$110M
(where bpc is minimized).  Is this a fundamental capacity limit,
or is it an artifact of online training?  Multi-epoch training on
110M bytes should answer this.
\end{question}

\begin{question}[Phase Transition at 450M]
The cliff at 450M bytes---output biases collapsing to all-negative---looks
like a phase transition.  Is there an E$\to$N$\to$Q description of this
collapse?  The bias term in the quotient chain is
$Q_{\text{bias}} = e^{b_y}/Z$.  When all $b_y < 0$, the quotient
becomes exponentially suppressive.  Can we predict when this happens
from the trajectory of $W_h$ drift?
\end{question}

\begin{question}[Content-Dependent Performance]
enwik9 contains XML markup, English prose, tables, references,
multilingual text, mathematical notation.  Which content types does
the model handle well at 110M?  The eval window matters:
bytes 0--110M of enwik9 are early Wikipedia articles.
The model's 2.81~bpc likely varies by content type.
Per-segment bpc analysis would reveal the model's strengths.
\end{question}

\begin{question}[The Mantissa Question]
On sat-rnn-1024, the mantissa was noise: sign-only dynamics gave
\emph{better} bpc (5.690 vs.\ 5.721).  On sat-rnn-enwik9, with
potentially reduced margins, does the mantissa carry information?
If margins drop below 1.0 for some neurons, the mantissa becomes
part of the computation.  This would be the first case where the
model is \emph{not} a Boolean automaton, and the full f32 state
space matters.
\end{question}

\section{Predicted Outcome Summary}

\begin{table}[h]
\centering
\small
\begin{tabular}{llcc}
\toprule
\textbf{Prediction} & \textbf{Metric} & \textbf{1024B} & \textbf{enwik9 (expected)} \\
\midrule
P1: Boolean regime & Mean margin & 60.5 & 5--15 \\
P2: Distributed neurons & Top-1 neuron \% gap & 99.7\% & $<$30\% \\
P3: Shallower offsets & Mean dominant offset & 18--25 & 5--15 \\
P4: Factor map degrades & Mean $R^2$ (2-offset) & 0.837 & 0.4--0.6 \\
P5: Construction gap & Analytic bpc & 1.89 & $>$5 \\
P6: Hebbian improves & $r(H_{\text{cov}}, W_h)$ & 0.56 & 0.65--0.75 \\
P7: E$\to$N$\to$Q holds & Framework & valid & valid \\
\bottomrule
\end{tabular}
\caption{Predicted outcomes.  Each row is a testable claim.}
\end{table}

\section{What Would Be Surprising}

\begin{enumerate}
\item \textbf{Margins stay large ($>$30).}  This would mean the Boolean automaton
  regime is a structural consequence of the architecture, not of overtraining.
  It would validate the claim that tanh RNNs are \emph{inherently} Boolean.

\item \textbf{Factor map $R^2$ stays high ($>$0.7).}  This would mean the
  2-offset conjunction is the fundamental computation, not a memorization
  artifact.  It would suggest that RNNs converge to this computational motif
  regardless of data.

\item \textbf{Hebbian correlation drops ($<$0.4).}  This would mean the
  trained model has developed genuinely non-Hebbian structure---higher-order
  correlations that simple covariance cannot capture.

\item \textbf{Phase transition is sharp and predictable.}  If the 450M cliff
  has a clean description in the quotient framework, it would be a genuine
  contribution to understanding catastrophic forgetting in RNNs.

\item \textbf{Content-dependent bpc varies by $>$3x.}  If the model gets
  0.5~bpc on XML tags but 6~bpc on multilingual text, the ``total interpretation''
  story is really ``total interpretation of XML parsing.''
\end{enumerate}

\section{Experimental Priority}

The experiments above are ordered by information value.  The single
most important experiment is:

\begin{center}
\textbf{Run q1\_margins.c on epoch1\_110M.bin.}
\end{center}

If margins are large, the Boolean automaton interpretation transfers
directly, and the rest of the Q1--Q7 suite follows.  If margins are small,
the interpretation must be rebuilt from scratch in the analog domain.
This one measurement determines the entire research direction.

The second most important experiment is the factor map ($R^2$ of 2-offset
conjunctions).  If both margins and factor map transfer, then the
total interpretation scales.  If either breaks, we learn something
new about what these models compute.

\end{document}
