\documentclass[11pt]{article}
\usepackage[margin=1in]{geometry}
\usepackage{amsmath,booktabs}
\title{H3 Pipeline Integration:\\Conviction-Weighted Meta-Combination\\and the Scale Convergence}
\author{Vector}
\date{2026-03-12}
\begin{document}
\maketitle

\begin{abstract}
We show that blending per-order Kneser--Ney chain probabilities with
$2^{g/H}$ weights (H3 meta-combination) beats standard KN-6 interpolation
on enwik9 at 100K ($-0.079$ bpc) and 1M ($-0.045$ bpc), but
\emph{converges to parity at 10M} ($+0.000$ bpc). Three integration
architectures were tested; only meta-combination of complete KN chains
succeeds. The advantage is scale-dependent: H3 adds value when orders
differ in quality (small data), but vanishes when all orders are
well-calibrated (large data). This suggests H3 is a \emph{cold-start
correction} rather than a fundamental improvement to KN.
\end{abstract}

\section{Background}

The H3 combination rule $w_k = 2^{g_k / H_k}$ was discovered via
ablation experiments on the pure ngram UM model (\S20260312 papers).
There, it beat KN interpolation at orders $\geq 6$ but trailed at
orders $\leq 4$. The open question: does it transfer to the real
KN-6 pipeline?

\section{Three Integration Attempts}

We tested three architectures for integrating H3 into the KN-6
pipeline. All use the same hash table and learning.

\subsection{Flat blend of raw distributions (FAILED)}

Materialize the raw KN distribution at each order $k$:
$p_k(b) = \max(c_{kb} - D, 0) / tc_k$, normalize, compute
$g_k$ and $H_k$, weight by $2^{g_k/H_k}$, average.

\textbf{Result: +0.713 bpc at 100K} (catastrophic).
The unigram's weight $w_0 \approx 1$ dilutes everything.

\subsection{Conviction-modulated backoff (FAILED)}

Replace KN's fixed $\lambda = D \cdot ty / tc$ with
$\lambda_{H3} = \lambda_{KN} / (1 + \text{conviction})$.

\textbf{Result: +0.486 bpc at 1M}. Reducing $\lambda$ removes
probability mass that KN needs for smoothing.

\subsection{Meta-combination of KN chains (SUCCESS at small scale)}

Compute $p_k^{\text{KN}}(\text{actual})$ = standard KN interpolation
from order 0 up to order $k$. Then blend across orders:
\[
p_{H3} = \frac{\sum_{k=0}^{6} w_k \cdot p_k^{\text{KN}}}{\sum_{k=0}^{6} w_k},
\quad w_k = 2^{\lfloor g_k / H_k \rfloor}
\]

\section{Results}

\begin{table}[h]
\centering
\begin{tabular}{lccc}
\toprule
\textbf{Method} & \textbf{100K} & \textbf{1M} & \textbf{10M} \\
\midrule
KN-6 interpolation & 2.719 & 2.398 & 2.178 \\
H3 flat blend & 3.433 & --- & --- \\
H3 modulated backoff & --- & 2.883 & --- \\
\textbf{H3 meta-combination} & \textbf{2.641} & \textbf{2.353} & 2.178 \\
\midrule
$\Delta$ (H3 $-$ KN-6) & $-0.079$ & $-0.045$ & $+0.000$ \\
\bottomrule
\end{tabular}
\caption{H3 advantage shrinks with scale and vanishes at 10M.}
\end{table}

The full pipeline at 10M (KN-6 + sparse + match) reaches 1.954 bpc.
H3 provides no additional gain when combined with sparse and match.

\section{The Scale Convergence}

The advantage curve $(-0.079, -0.045, +0.000)$ is monotonically
decreasing. This is not a sampling artifact---it reflects a real
structural phenomenon:

\begin{itemize}
\item \textbf{Small data}: orders differ sharply in quality. Order 6
  may have 2 observations for a context while order 3 has 200.
  H3 correctly downweights the sparse high-order prediction.
  KN's fixed $\lambda$ chain cannot adapt to this.
\item \textbf{Large data}: all orders are well-populated. KN's
  interpolation chain, tuned by the discount $D$, is near-optimal.
  H3's conviction weights become approximately uniform, adding no
  information. The $256\times$ compute overhead buys nothing.
\end{itemize}

\section{Interpretation: Cold-Start Correction}

H3 meta-combination is a \emph{cold-start correction}: it helps
when the model hasn't seen enough data for KN's fixed discount to
work well. As data grows, the correction vanishes.

This is consistent with the ablation results. The ablation tested
at 100K--1M, where H3 won. The crossover between KN and H3
happens between orders 4 and 6 at 100K---exactly where context
sparsity becomes severe.

\section{Architectural Lessons}

\begin{enumerate}
\item \textbf{Don't modify KN's internals.} The interpolation chain
  is not a trust knob. $\lambda$ carries discounted mass; reducing
  it creates coverage holes.
\item \textbf{Compose complete predictors.} Meta-combination of KN
  chains works; internal modification doesn't.
\item \textbf{Conviction is a cold-start signal.} Useful when
  predictors differ in quality; irrelevant when all are mature.
\item \textbf{The discount IS the combination rule.} At scale, KN's
  $D = 0.9$ already implements near-optimal order combination.
  H3 is redundant.
\end{enumerate}

\section{Implications}

The path to beating KN-6+sparse+match on enwik9 does not go through
H3 meta-combination. The sparse and match models already provide
independent signals that compose well with KN's interpolation.
H3 provides a smoothing benefit at small scale that KN's chain
already provides at large scale.

The combination problem (\S20260312) remains open. The tropical tax
is real (2.9 bpc max-min vs.\ KN interpolation), and H3 does not
solve it at scale---it only masks it during cold-start.

\end{document}


