← Back to Archives

Archive: 2026-01-31_4

Pattern Injection: UM → RNN

Focus: Injecting UM patterns into RNN weights via SVD. Why W_hh fails.
Key Result: ~1 bit/char head start (5.46 → 4.47 bpc). Q unbounded, f32 isn't.
Builds On: 2026-01-31_3 (pattern-injection.pdf)

Figures

fig-02: RNN Memory Depth

How far back does the RNN remember? Variance method. Predicted ~12 chars, measured ~flat.

fig-01: SVD Component Interpretation

What singular values mean. ASCII vs UTF-8, letters vs digits, bracket structure, phonotactics.

Reachability Map

Archive structure visualization. Forward and back links.

Papers

RNN Memory Depth: Experiment

Predicted ~12 chars, observed ~flat. Two methods, analysis, future work.

Embeddings and the Atomic Time Step

t_{n+1} = p × t_n becomes h_{n+1} = W × h_n. Dimension d = bandwidth.

Time, Frequency, Energy, and Bits

bits ∝ h/time. Depth limit as energy budget. Fourier ↔ Bayes/Thermo duality.

Unification: Bayes, Thermo, Quotient, Factor Maps

The quotient IS the luck. Commutative diagrams. Depth limit.

Pattern Injection: UM → RNN

Theory, SVD factorization, experiments, AC analogy. Light theme.

Dashboard View

Same content, dark theme. Tables, code blocks.

Empirical Results

Pattern Injection vs Random Init

Initialization        Initial    After 10 epochs
---------------------------------------------------
Random                5.46 bpc   4.81 bpc
Pattern Injection     4.47 bpc   4.53 bpc
Gap                   0.99 bpc   0.28 bpc

SVD Factorization (H=64):
  Full bigram:        3.84 bpc
  Rank-64 approx:     3.92 bpc
  Loss from rank:     0.08 bpc