Order Scaling: The KN–H3 Crossover

6-policy ablation at orders 2–8 on 1M bytes of enwik9. The right combination rule changes with order.

Scaling Curves

Each line shows how a policy's bpc changes as n-gram order increases. The crossover between KN (blue) and H3 (coral) occurs between orders 4 and 6.

max-min

sharpest

KN-interp

ent-blend

gap-blend

2^(g/H)

The crossover: KN gets worse from order 6→8 (+0.147 bpc) because recursive backoff cascades put 99.9% weight on unigram. H3 gets better (−0.050 bpc) because 2^(gap/H) concentrates on the rare confident high-order contexts. The gap grows monotonically after order 4.

Marginal Gain per Order Step

How much does adding the next order help (negative = better)? KN's marginal gain turns negative after order 4. H3 stays positive throughout.

Full Table

Order	max-min	sharpest	KN	ent	gap	2^(g/H)
2	5.288	4.597	3.232	3.332	3.514	3.506
3	5.247	4.555	2.655	2.744	2.914	2.890
4	5.285	4.806	2.391	2.427	2.512	2.484
6	5.347	5.100	2.418	2.337	2.244	2.239
8	5.543	4.701	2.565	2.379	2.294	2.189

Full Paper (PDF) Hybrid Blend Paper Oracle Correlation Conviction Explainer Ablation Explainer Archive Index