\documentclass{article}
\usepackage{amsmath,amssymb}
\usepackage{booktabs}

\title{$\omega_\infty$: The Sky-Hook Problem}
\author{Hutter RNN Project}
\date{2026-01-31}

\begin{document}
\maketitle

\section{The Universal Model}

Recall the CMP model:
\[
u = (E, T, P, f, \omega)
\]

\begin{itemize}
\item $E$ = event space (e.g., 256 bytes)
\item $T$ = thought vectors in $\mathbb{N}^{|E|}$
\item $P$ = patterns (transition weights)
\item $f$ = integration function
\item $\omega$ = \textbf{prior over patterns}
\end{itemize}

The prior $\omega$ is where the magic happens. After seeing data $D$:
\[
\omega_D = \text{posterior} = \frac{P(D|\omega_0) \cdot \omega_0}{P(D)}
\]

But what is $\omega_0$? The \textbf{prior before any data}.

\section{The Sky-Hook}

$\omega_\infty$ is the ``sky-hook'': the source of predictive power that doesn't come from data.

\subsection{The Regress}

\begin{enumerate}
\item Model learns patterns from data
\item But learning requires a prior $\omega_0$
\item Where does $\omega_0$ come from?
\item Either:
  \begin{itemize}
  \item From more data (infinite regress)
  \item From \textbf{something outside the system} (sky-hook)
  \end{itemize}
\end{enumerate}

\subsection{Sources of $\omega_\infty$}

In practice, the sky-hook is:

\begin{center}
\begin{tabular}{ll}
\toprule
Source & Example \\
\midrule
Evolution & Brain architecture, attention, memory \\
Human design & ES choice, RNN architecture, hyperparameters \\
Math & Minimum description length, Occam's razor \\
Physics & Locality, causality, symmetry \\
\bottomrule
\end{tabular}
\end{center}

\section{$\omega_\infty$ in Our System}

\subsection{What We Choose (Human Ingenuity)}

\begin{enumerate}
\item \textbf{Event space factorization:} ES = \{Digit, Punct, Vowel, Whitespace, Other\}

  This is a \textit{human choice}. We could have chosen differently.

\item \textbf{Architecture:} RNN with 512 hidden units, tanh activation

  Encodes locality bias, bounded memory, smooth functions.

\item \textbf{Training:} SGD with momentum, learning rate schedule

  Implicit regularization toward ``simple'' solutions.
\end{enumerate}

\subsection{Information Flow}

\[
\omega_\infty \xrightarrow{\text{human design}} \omega_0 \xrightarrow{\text{training}} \omega_D \xrightarrow{\text{compression}} \text{bits/char}
\]

The question: \textit{how much of final performance comes from $\omega_\infty$ vs. $D$?}

\section{Measuring the Sky-Hook}

\subsection{Definition}

The \textbf{sky-hook contribution} is:
\[
\Delta_{\omega} = H(\text{data} | \text{uniform prior}) - H(\text{data} | \omega_\infty)
\]

This is how much our prior assumptions help, independent of learning.

\subsection{Empirical Approach}

Compare:
\begin{enumerate}
\item Random weights (no $\omega_\infty$): $\sim 8$ bits/char
\item Untrained ES-augmented (ES structure only): $\sim ?$ bits/char
\item Trained on different data (transfer): $\sim ?$ bits/char
\item Trained on enwik9 (full): $\sim 1.5$ bits/char
\end{enumerate}

The gap between 1 and 4 is \textit{total} improvement.

The gap between 1 and 2 is \textit{pure architecture} ($\omega_\infty$).

\section{The ES Contribution to $\omega_\infty$}

Our ES features encode:
\begin{align}
\omega_\infty(\text{``vowels predict differently than consonants''}) &> 0 \\
\omega_\infty(\text{``whitespace follows punctuation''}) &> 0 \\
\omega_\infty(\text{``digits cluster''}) &> 0
\end{align}

These are \textbf{human-injected priors}. They came from:
\begin{itemize}
\item Linguistic knowledge (vowels vs consonants)
\item Typography conventions (spacing)
\item Domain knowledge (numbers group)
\end{itemize}

\section{The Deep Question}

\textbf{Can we discover good $\omega_\infty$ automatically?}

\begin{itemize}
\item Neural architecture search: Learn the architecture
\item Meta-learning: Learn to learn (but from what meta-prior?)
\item Compression: The ``simplest'' model that fits
\end{itemize}

But this just pushes the sky-hook up one level.

\textbf{Solomonoff induction} says: use the universal prior (Kolmogorov complexity). But this is uncomputable.

\textbf{In practice:} Human ingenuity is the sky-hook. We inject $\omega_\infty$ through our choices.

\section{Implications}

\begin{enumerate}
\item \textbf{Interpretability is about $\omega_\infty$:} We understand what we designed.
\item \textbf{The Hutter Prize rewards $\omega_\infty$:} Better priors = better compression.
\item \textbf{ES features are explicit $\omega_\infty$:} We can audit them.
\end{enumerate}

The goal: Make $\omega_\infty$ visible, measurable, and improvable.

\end{document}
