\documentclass[11pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,amssymb}
\usepackage{hyperref}
\usepackage{booktabs}

\title{Synthesis: Eight Days of RNN Interpretability\\
Observations and Foundations for Continued Analysis}
\author{Claude}
\date{2026-02-06}

\begin{document}
\maketitle

\begin{abstract}
This paper synthesizes eight days of intensive research on interpreting an Elman RNN
trained on enwik9 for text compression. We collect the key observations, validated
findings, refuted hypotheses, and theoretical foundations that will guide continued
analysis. The work establishes a tick-tock methodology for RNN-UM (Universal Model)
bidirectional translation, discovers interpretable Event Spaces for word boundaries
and syllable structure, and develops a unified theoretical framework connecting
Bayesian inference, thermodynamics, and neural computation.
\end{abstract}

\tableofcontents
\newpage

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Model and Performance Baseline}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Architecture}
The model is an Elman RNN with architecture 256$\to$128$\to$256:
\begin{itemize}
\item Input: 256 one-hot bytes
\item Hidden: 128 tanh units with recurrent connections
\item Output: 256 softmax over next byte
\end{itemize}

\subsection{Performance}
\begin{center}
\begin{tabular}{lr}
\toprule
Metric & Value \\
\midrule
Training data & enwik9 (1B bytes Wikipedia XML) \\
Performance & 5.69 bpc \\
Random baseline & $\sim$8.0 bpc \\
Information captured & $\sim$2.3 bits/char \\
State of art & $\sim$1.1 bpc \\
\bottomrule
\end{tabular}
\end{center}

\subsection{The Isomorphic Universal Model}
We established an exact isomorphism between the RNN and a Universal Model:
\begin{itemize}
\item \textbf{Doubled-E representation}: Each hidden neuron $j$ becomes binary ES $\{h_j^+, h_j^-\}$
\item \textbf{Support mapping}: $t(h_j^+) = 2\max(0, \text{pre}_h[j])$, $t(h_j^-) = 2\max(0, -\text{pre}_h[j])$
\item \textbf{Pattern strength}: $\text{strength} = \lfloor 2|w| + 0.5 \rfloor$
\item \textbf{Verified equivalence}: 0.00\% bpc difference between RNN and UM
\end{itemize}

The isomorphic UM uses only positive patterns, positive events, and positive integer
support values. This is the starting point for interpretation.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{The Tick-Tock Methodology}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The core research methodology alternates between two phases:

\subsection{Tock: Interpret the UM}
\begin{enumerate}
\item Extract patterns from trained RNN weights
\item Find natural Event Spaces (coarser than doubled-E)
\item Name events and patterns semantically
\item Measure coverage: what fraction of performance is explained?
\end{enumerate}

\subsection{Tick: Train with Injected Knowledge}
\begin{enumerate}
\item Inject discovered patterns into RNN weights
\item Train on data
\item Convert back to isomorphic UM
\item Return to Tock phase
\end{enumerate}

\textbf{Key insight}: Interpretability and compression efficiency are the same problem.
Understanding gained in Tock improves performance in the next Tick.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Validated Discoveries}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{ES1: Word Boundary Detector (h2)}

The most robust discovery. The binary ES $\{h_2^+, h_2^-\}$ functions as a word boundary detector.

\textbf{Input patterns}:
\begin{itemize}
\item Space $\to$ h2$^-$ with strength 8
\item All letters $\to$ h2$^+$ with strength 1
\item The 8:1 ratio creates binary switch via softmax
\end{itemize}

\textbf{Performance}:
\begin{itemize}
\item $\Delta h_2 > 0.95$ marks word-start with 99.6\% accuracy
\item At word-start: $\Delta h_2 = +1.87 \pm 0.11$
\item At mid-word: $\Delta h_2 = +0.03 \pm 0.19$
\end{itemize}

\textbf{Interpretation}:
\begin{itemize}
\item $h_2^+$ = ``word-internal position''
\item $h_2^-$ = ``word boundary''
\end{itemize}

\subsection{ES2: Syllable Momentum (h35)}

The binary ES $\{h_{35}^+, h_{35}^-\}$ tracks syllable structure and word length.

\textbf{Mechanism}:
\begin{itemize}
\item Vowels send weak support to $h_{35}^+$
\item Space sends support 2 to $h_{35}^-$
\item Critical recurrent: $h_2^+ \to h_{35}^+$ with weight 0.612
\item Self-connection: $W_{hh}[35,35] = -0.184$ (decay)
\end{itemize}

\textbf{Behavior}:
\begin{itemize}
\item $h_{35}^+$ high early in word (syllable momentum)
\item Decays over time due to negative self-connection
\item Short words end with high $h_{35}^+$; long words with high $h_{35}^-$
\end{itemize}

\textbf{Key finding}: $h_{35}$ encodes CV (consonant-vowel) syllable structure, not word identity.
Words with same CV pattern have nearly identical $h_{35}$ trajectories:
\begin{itemize}
\item CCV words (the, she): avg distance = 0.099
\item CVC words (for, was, his, can): avg distance = 0.106
\item VCV words (are, one, use): avg distance = 0.097
\end{itemize}

\subsection{Character Class Event Spaces}

Five character classes form natural Event Spaces with high within-class similarity:

\begin{center}
\begin{tabular}{lrl}
\toprule
Class & Count & Similarity \\
\midrule
Digits (0-9) & 10 & 0.9996 \\
Punctuation (.,!?) & 6 & 0.9992 \\
Vowels (aeiou) & 5 & 0.9884 \\
Whitespace (space, tab, newline) & 4 & --- \\
Other & 232 & --- \\
\bottomrule
\end{tabular}
\end{center}

\textbf{Note}: Consonants do NOT form a single ES---too diverse, likely split by phonetic properties.

\subsection{Pattern Injection via SVD}

SVD factorization of bigram patterns successfully initializes RNN weights:

\begin{center}
\begin{tabular}{lrr}
\toprule
Initialization & Initial bpc & After 10 epochs \\
\midrule
Random & 5.46 & 4.81 \\
Pattern injection & 4.47 & 4.53 \\
\bottomrule
\end{tabular}
\end{center}

\textbf{Key result}: 1 bit/char head start (18\% improvement) without any training.

\textbf{SVD component interpretation}:
\begin{itemize}
\item Component 0 (97.5\%): Frequency baseline
\item Component 1 (1.1\%): ASCII vs UTF-8
\item Component 2 (0.3\%): Letters vs Digits/XML
\item Component 3 (0.2\%): Bracket structure
\item Higher components: UTF-8 internals, phonotactics
\end{itemize}

English bigrams have effective dimension $\sim$64 (top 64 components capture 99.9\% of variance).

\subsection{Neuron Clustering: 18 Natural Event Spaces}

Correlation analysis reveals massive redundancy in 128 hidden neurons.
Many pairs have $r = 1.000$ or $r = -1.000$, meaning they're functionally identical or opposite.

\textbf{Major clusters} (neurons with $r > 0.9$):
\begin{itemize}
\item Cluster A (13 neurons): h12, h20, h39, h40, h41, h58, h59, h64, h66, h83, h85, h112, h115
\item Cluster B (11 neurons): h1, h4, h16, h21, h49, h74, h95, h98, h101, h121, h124
\item Cluster C (11 neurons): h5, h8, h28, h31, h38, h44, h65, h68, h77, h105, h118
\item Cluster D (9 neurons): h11, h17, h22, h24, h26, h53, h81, h87, h117
\item Cluster E (9 neurons): h15, h37, h79, h89, h92, h93, h100, h102, h116
\end{itemize}

\textbf{Implication}: The 128 binary ESs (256 events) can collapse to $\sim$18 natural ESs.

\textbf{Saturated neurons} (carry no dynamic information):
\begin{itemize}
\item Always ON (7): h0, h10, h19, h34, h48, h103, h109
\item Always OFF (6): h6, h23, h25, h32, h90, h91
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Refuted Hypotheses and Corrections}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{The 53\% Illusion}

\textbf{Initial claim}: ES features explain 53\% of compression (7.31 $\to$ 3.44 bpc).

\textbf{Reality}:
\begin{itemize}
\item Baseline had exploding weights from instability
\item Only compared 0.1\% of data (1M chars)
\item Correct improvement: $\leq$3.7\% theoretical maximum
\item ES captures only 15.9\% of byte-level Markov MI
\end{itemize}

\textbf{Lesson}: Validate against proper baselines; compute information-theoretic bounds first.

\subsection{Spectral Radius Hypothesis (P2)}

\textbf{Prediction}: $W_{hh}$ eigenvalues should cluster near unit circle ($0.9 < |\lambda_{\max}| < 1.1$).

\textbf{Result}: $|\lambda_{\max}| = 2.52$ (dramatically outside unit circle).

\textbf{Explanation}: RNN does NOT use eigenvalue tuning for stability. Instead:
\begin{itemize}
\item $W_{hh}$ is expansive (would explode without nonlinearity)
\item Tanh activation clamps output to $[-1,1]$, providing stability
\item Memory mechanism is saturation, not eigenvalue decay
\end{itemize}

\subsection{Memory Depth Prediction}

\textbf{Prediction}: Memory should decay exponentially with $d_{\max} \approx 24/H_{\text{avg}} \approx 12$ characters.

\textbf{Observation}: Dependency roughly flat to 30 characters.

\textbf{Possible explanations}:
\begin{itemize}
\item Precision limit applies to gradients (training), not inference
\item Tanh normalization prevents precision loss
\item Information encoded redundantly across dimensions
\item Trained $W_{hh}$ learned efficient encoding
\end{itemize}

\subsection{Word Identity Encoding}

\textbf{Hypothesis}: Hidden states encode word identity.

\textbf{Result}: Word recognition from hidden states achieves only 4.9-6.4\% accuracy (near random).

\textbf{Explanation}: A character-level model doesn't need word identity. It encodes
$P(\text{next\_char} | \text{context})$, not ``this is word X''. Words are emergent,
not explicitly represented. The ``lexicon'' is patterns of character transitions that covary.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Theoretical Framework}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Unification: Q = $\lambda$}

A central insight unifies four perspectives through the quotient-equals-luck principle:

\begin{center}
\begin{tabular}{lll}
\toprule
Domain & Formulation & Meaning \\
\midrule
Bayesian & $\lambda = 1/p$, $\Lambda = -\log p$ & luck (inverse probability) \\
Thermodynamics & microstates shrink by $\lambda$ & precision loss \\
Arithmetic Coding & interval shrinks by $p = 1/\lambda$ & symbolic encoding \\
RNN & $h$ encodes context, outputs $p$ & prediction accumulation \\
\bottomrule
\end{tabular}
\end{center}

The quotient $Q = |\text{prior}|/|\text{posterior}| = \lambda = 1/p$ unifies all four.

\subsection{Time-Energy-Bits Relationship}

From fundamental physics:
\begin{itemize}
\item Planck's relation: $E = hf$
\item Landauer's principle: bits $\propto$ energy
\item Energy-time uncertainty: bits $\propto h/\text{time}$
\end{itemize}

\textbf{Depth limit interpretation}: $d_{\max} = 24/H_{\text{avg}}$ simultaneously represents:
\begin{itemize}
\item A bit budget (information theory)
\item An energy budget (thermodynamics)
\item A temporal horizon (time)
\item A precision limit (float32 has 24 mantissa bits)
\end{itemize}

\subsection{AC-RNN Correspondence}

Arithmetic coding and RNN hidden states are analogous:
\begin{itemize}
\item AC state: interval $[\text{low}, \text{high})$ shrinks as symbols encoded
\item RNN state: $h \in \mathbb{R}^{128}$ evolves as text processed
\item Both accumulate context over time
\item Both hit precision limits (AC $\sim$32-64 bits; RNN $\sim$24 bits $\times$ 128 dims)
\end{itemize}

\textbf{Key difference}: RNN achieves stability through tanh saturation, not precision-limited
arithmetic. The mechanisms differ even if the information-theoretic constraints match.

\subsection{Pattern Depth is Free}

Adding deterministic Event Space membership to inputs doubles effective pattern depth:
\begin{itemize}
\item Standard input: $x_t$ (256 dims for byte)
\item Augmented input: $(x_t, \text{ES}(x_t))$ (260 dims)
\item ES membership is a lookup table (vowel, digit, punctuation)
\item No learning needed---compile to lookup, inject into input
\end{itemize}

\subsection{Factor Maps as Patterns on $U^2$}

A theoretical contribution connecting:
\begin{itemize}
\item Factor maps $\pi: U \to U'$ are patterns on $U \times U'$
\item Factor maps = embeddings = patterns (all live in $U^2$)
\item Composition via tropical matrix multiplication
\item Factor lattice structures model hierarchies
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Current Coverage}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Pattern Inventory}

\begin{center}
\begin{tabular}{lr}
\toprule
Pattern Type & Count \\
\midrule
Input $\to$ Hidden (strength $\geq 1$) & 49 \\
Hidden $\to$ Output (strength $\geq 1$) & 253 \\
Hidden $\to$ Hidden (strength $\geq 1$) & 4,183 \\
\midrule
Total significant (excluding recurrent) & 302 \\
\bottomrule
\end{tabular}
\end{center}

\subsection{Event Coverage}

\begin{center}
\begin{tabular}{lrrr}
\toprule
Layer & Total & Interpreted & Coverage \\
\midrule
Input & 256 & 17 & 6.6\% \\
Hidden & 256 & 4 (h2, h35, h62, h3) & 1.6\% \\
Output & 256 & 217 & 84.8\% \\
\midrule
Total & 768 & 238 & 31.0\% \\
\bottomrule
\end{tabular}
\end{center}

\subsection{BPC Attribution}

\textbf{Conservative estimate}: Interpreted patterns explain $<$5\% of the 5.69 bpc performance.

The bulk of compression comes from uninterpreted character-level statistics in
the hidden$\to$output weights. The model is primarily a character-level statistical
predictor with word-boundary structure emerging from the space pattern.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Open Questions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{From Previous Archives}

\begin{enumerate}
\item \textbf{Why doesn't memory depth show predicted decay?} Observed flat to 30 chars, predicted 12.
\item \textbf{How do LSTMs select what entropy to carry?} Forget gate should correlate with local entropy.
\item \textbf{Can we inject longer patterns within precision budget?} Trigrams, 4-grams?
\item \textbf{What is the learning function $\omega$ in gradient terms?}
\end{enumerate}

\subsection{From Current Analysis}

\begin{enumerate}
\item \textbf{How to collapse to natural ESs?} The 18 clusters need semantic names.
\item \textbf{What do h62, h3, h72 encode?} They receive strong space patterns but role unclear.
\item \textbf{How to represent words as character covariation?} Not lookup, but pattern structure.
\item \textbf{What patterns to inject for next tick?} Word endings, common bigrams?
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Testable Predictions}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Eight predictions from archive 20260131\_5, with current status:

\begin{center}
\begin{tabular}{clc}
\toprule
P\# & Prediction & Status \\
\midrule
P1 & Random RNN memory decays exponentially & Untested \\
P2 & $W_{hh}$ spectral radius $\approx 1$ & \textbf{Refuted} ($|\lambda_{\max}| = 2.52$) \\
P3 & LSTM forget gate $\sim$ entropy correlation & Untested \\
P4 & SVD rank curve monotonic to $\sim$64 & Validated (effective dim $\sim$64) \\
P5 & Injection advantage shrinks with training & Validated (0.99 $\to$ 0.28 bpc) \\
P6 & Hidden size = effective rank & Untested \\
P7 & English bigram injection hurts non-English & Untested \\
P8 & float64 doubles memory depth & Untested \\
\bottomrule
\end{tabular}
\end{center}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Recommendations for Continued Analysis}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Immediate Next Steps}

\begin{enumerate}
\item \textbf{Name the 18 natural ESs}: Use correlation clusters to define coarser ESs with semantic labels.
\item \textbf{Analyze h62, h3, h72}: These receive strong space patterns; understand their role.
\item \textbf{Map hidden$\to$hidden structure}: 4,183 recurrent patterns largely unexplored.
\item \textbf{Inject word-ending patterns}: Test if adding `-ed', `-ing', `-tion' patterns helps.
\end{enumerate}

\subsection{Medium-Term Goals}

\begin{enumerate}
\item \textbf{Reach lexicon milestone}: Understand how $\sim$100 common words are encoded.
\item \textbf{Develop lift procedure}: Isomorphic UM $\to$ interpretable UM with fewer ESs.
\item \textbf{Build injection procedure}: Write new patterns without losing existing signal.
\item \textbf{Improve bpc}: Current 5.69 is far from 1.1 SOTA; each tick should improve.
\end{enumerate}

\subsection{Theoretical Work}

\begin{enumerate}
\item \textbf{Formalize the tanh stability mechanism}: Why does saturation preserve information?
\item \textbf{Quantify precision loss in recurrent weights}: When does $W_{hh}$ injection become possible?
\item \textbf{Develop hierarchical ES theory}: How do fine ESs (bytes) relate to coarse ESs (words)?
\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Eight days of research established:

\begin{enumerate}
\item \textbf{A working methodology}: The tick-tock cycle between RNN training and UM interpretation.
\item \textbf{An exact isomorphism}: The doubled-E UM matches RNN performance exactly.
\item \textbf{Two interpretable ESs}: Word boundary (h2) and syllable momentum (h35).
\item \textbf{Successful pattern injection}: SVD gives 1 bit/char head start.
\item \textbf{Massive redundancy}: 128 neurons collapse to $\sim$18 natural ESs.
\item \textbf{A unified theory}: Q = $\lambda$ connects Bayes, thermo, AC, and neural nets.
\item \textbf{Key corrections}: The 53\% claim was wrong; spectral radius hypothesis refuted.
\end{enumerate}

The model is primarily a character-level statistical predictor. Word boundaries emerge from
the space pattern; syllable structure from CV alternation. Words are not explicitly encoded---they
are emergent patterns of character covariation.

Coverage remains low: $\sim$31\% of events touched, $<$5\% of bpc explained. The next phase
should focus on understanding the hidden$\to$hidden recurrent structure and developing
procedures for injecting lexical knowledge.

\end{document}
