How Words Build a World

1. Two people walk into the same sentence

Read this sentence slowly: He went to the bank. Where did he go?

You probably pictured a building with a counter, an ATM, and somebody waiting to deposit money. Or perhaps you pictured the edge of a river. The sentence allows both. Now let me add a few words, and watch what happens to the picture in your head.

My question is about the machinery that holds those distinctions together. Could a language machine become more economical if it explicitly maintained the situation, the open contexts, and the people or objects being discussed? The paper develops a precise version of that question. It contains mathematical results for carefully defined systems and hypotheses that still require experiments on real language.

The phrase context lift means turning a history of observations into an explicit structured state. It does not mean travelling to a higher physical dimension. Imagine keeping a small, continuously updated scene notebook while reading a story. The notebook records whatever could matter next.

2. What the machine is allowed to know

A prediction has a deadline. If I ask you to guess the ending of a sentence, you may use the words already spoken. You may not secretly read the ending first. A model that builds its “current context” from a complete novel and then predicts earlier passages is cheating at this particular task, even if its answers sound impressive.

Equation guide 01 · From the original paperBuild the current state from what has arrived

$$X_t=\Phi_t(H_t), \qquad \sigma(X_t)\subseteq\sigma(H_t).$$

Hₜ is the observed history (W₁,…,Wₜ); W can be a word or token, and t marks the current position. Φₜ is a rule that turns that history into the state Xₜ. The σ symbols describe the information available from each variable. The inclusion means the state cannot contain evidence from observations that have not arrived. In Appendix C this same requirement is written Xₜ = Φ(W≤ₜ).

Two sentence examples

• ‘Ravi put the key under the mat.’ After hearing this, the state can record the key's location without waiting for another sentence.
• ‘Tomorrow I will tell you where I hid it.’ At this point, the state may record that the location is unknown; it cannot use tomorrow's disclosure today.

This is causality as an information-access restriction. It does not require the state to be a perfect interpretation or to be cheap to compute.

Find this expression in Part II →

Now suppose two people keep different notes about the same conversation. One writes every word; the other writes “Ravi owes Maya a blue pen and will return it on Friday.” For some questions, the shorter note may be enough. For others, it may fail.

Equation guide 02 · From the original paperWhen the saved state is enough

$$F_t\perp H_t\mid X_t,$$

Fₜ is the future target. The perpendicular sign means conditional independence: once Xₜ is known, consulting Hₜ adds no information about Fₜ. The equivalent probability statement is P(Fₜ | Hₜ) = P(Fₜ | Xₜ), except on zero-probability cases. The vertical bar means ‘given’. This is the definition of predictive sufficiency.

Two sentence examples

• ‘The door is locked, and Ravi has its only key.’ If a toy task asks only who can unlock it, a state retaining those two facts can be sufficient under that task's rules.
• ‘Maya ordered the blue notebook.’ A state retaining only ‘Maya ordered stationery’ is insufficient for a question about which colour will arrive.

A state can be sufficient for one target and insufficient for another. An exact quote and an event prediction need not require the same memory.

Find this expression in Part II →

Predictive sufficiency is the exact version of “my notes retain everything relevant to what comes next.” It is a demanding property. Keeping a generally useful summary is easier than proving that the complete history could never improve the required prediction.

3. Organising information does not create information

This is the first important boundary of the idea. If useful information was never present in what you observed, rearranging your notes cannot manufacture it. If it was present and you delete it, the future may become harder to predict. A well-designed representation tries to retain the relevant information in a form that is easier to use.

Equation guide 03 · From the original paperUseful information equals what you kept plus what you missed

$$\mathrm{I}(H_t;F_t)=\mathrm{I}(X_t;F_t)+\mathrm{I}(H_t;F_t\mid X_t).$$

I measures mutual information: how much one variable tells us about another, averaged across possible cases. The history's information about the future equals the state's information plus the extra information still in the history after the state is known. The last term cannot be negative, so I(Xₜ;Fₜ) ≤ I(Hₜ;Fₜ). Equality holds when that leftover term is zero, under the paper's sufficiency conditions.

Two sentence examples

• ‘The meeting is at school at four.’ A note saying ‘at school’ retains location while leaving useful timing information in the original sentence.
• ‘The meeting is at school at four.’ A note preserving both relevant facts can eliminate that leftover advantage for a task concerned only with place and time.

The identity does not say every detail must survive. It says no genuinely future-relevant information can be discarded without a corresponding loss.

Find this expression in Part II →

Equation guide 04 · From the original paperPutting your own summary beside the original adds no new evidence

$$\mathrm{I}(H_t;F_t)=\mathrm{I}(H_t,X_t;F_t).$$

The pair (Hₜ,Xₜ) means the full history together with a state deterministically computed from it. Because that state is already fixed by the history, the pair carries exactly the same information about the future as the history alone. This is the first intermediate step in Proposition 2.2's proof.

Two sentence examples

• ‘The parcel arrives Friday.’ Copying ‘Friday delivery’ onto a sticky note creates a convenient reminder, not an independent second delivery forecast.
• ‘Ravi scored twice.’ Writing ‘Ravi: 2’ beneath the same sentence does not tell you something absent from the sentence.

The summary may make a limited reader's job easier. The equality concerns available information, not effort.

Find this expression in Part II →

Equation guide 05 · From the original paperRead the summary first, then ask what the transcript adds

$$\mathrm{I}(H_t,X_t;F_t)=\mathrm{I}(X_t;F_t)+\mathrm{I}(H_t;F_t\mid X_t).$$

The chain rule splits the information in the pair into two parts: what Xₜ tells us about Fₜ, and what Hₜ adds after we know Xₜ. Combining this step with the previous equation gives the no-free-information identity. The order is deliberate: first the proposed state, then the residual history.

Two sentence examples

• ‘Sita bought a red bicycle yesterday.’ A summary saying ‘Sita owns a bicycle’ helps with ownership; the original still helps with a question about colour.
• ‘The red team won 2–1.’ If the state already retains the teams, winner, and score, rereading the same statement adds nothing for a toy task limited to those facts.

Conditional information means additional evidence after something else is already known; it avoids counting the same clue twice.

Find this expression in Part II →

The proof is bookkeeping. A note computed entirely from a transcript adds no new evidence when placed beside that transcript. Then we split the transcript’s predictive information into the part accessible through the note and the part still useful after reading the note. The second part is zero only when the note is sufficient for the target.

Equation guide 06 · From the original paperFind the cheapest acceptable machine

$$C_{\mathcal{M}}(P,\varepsilon;c) = \inf\left\{c(M): M\in\mathcal{M},\;R_P(M)\leq R_P^*+\varepsilon\right\}.$$

P is the task, M is a model, and ℳ is the allowed family of models. Rₚ(M) is its average prediction loss; Rₚ* is the best theoretically attainable loss. ε is the extra error we are willing to tolerate. Among models that meet that accuracy condition, take the infimum—the greatest lower bound—of the declared cost c(M). If no eligible finite-cost model exists, the cost can be infinite.

Two sentence examples

• ‘Tell me who borrowed the book.’ We could compare machines that achieve the same answer accuracy and ask which uses less memory, including the memory needed to identify the borrower.
• ‘Complete this sentence naturally.’ A tiny memory with a huge output generator may be expensive once both parts are charged to the same system.

Cost must be declared in advance: for example, bytes, time, or a specified combination. A better result under one measure need not win under another.

Find this expression in Part II →

Here realization simply means building a machine that performs the prediction. The cheapest realization depends on which tools that machine is allowed to use. A stack, an entity register, or a learned network can change that calculation. None receives a free pass on memory, running time, or the cost of constructing its inputs.

4. A small working world: what, inside what, involving whom?

The proposed state has three parts. Local state is what is currently going on in the prediction. Context frames keep track of enclosing situations. Referent registers keep track of the particular people and objects that remain relevant. A referent is just the thing a word or phrase points to.

Equation guide 07 · From the original paperThe three-part working state

$$X_t=(q_t,\Gamma_t,R_t),$$

qₜ is local control or predictive activity. Γₜ = (γ₁,…,γdₜ) is the stack of open typed frames; dₜ is its depth. Rₜ holds live referents and their equality, types, roles, salience, and finite status features. CALL opens a frame, RETURN closes one, and INTERNAL updates within the current frame. These are distinct operational roles, not three spatial directions.

Two sentence examples

• ‘At the café, Maya handed Ravi the bill.’ The local event is handing; the context concerns a café; the register distinguishes Maya, Ravi, and the bill.
• ‘In Ravi's dream, the dog could talk.’ The dream frame contains the talking-dog claim, while the system can retain Ravi as the person having the dream.

The theory proposes this decomposition. It does not establish that every thought fits it or that each component has a unique biological location.

Find this expression in Part II →

The symbols are a proposed design, not a scan of three compartments in the human brain. A machine could realize these roles using many different mechanisms. Giving a number array three dimensions would not, by itself, make those dimensions acquire the intended meanings.

Equation guide 08 · From the original paperA route through sporting context

$$\mathrm{World}\triangleright\mathrm{Event}\triangleright\mathrm{Sport} \triangleright\mathrm{Football}\triangleright\mathrm{Match},$$

Read the triangle as an illustrative ‘within’ relation: a match inside football, football inside sport, and sport inside an event in the world. These frame labels make background expectations explicit. This chain helps explain why the same utterance can receive a different interpretation in different settings.

Two sentence examples

• ‘The captain raised the cup after the final whistle.’ The match context supports reading the object as a trophy.
• ‘The substitute waited by the touchline.’ The football frame helps interpret both the person's role and the location.

These semantic layers are illustrative. A learned parser still has to decide how and when a real discourse opens and closes frames.

Find this expression in Part II →

Equation guide 09 · From the original paperA route through shopping context

$$\mathrm{World}\triangleright\mathrm{Commerce}\triangleright\mathrm{Shopping} \triangleright\mathrm{DepartmentStore}\triangleright\mathrm{Ceramics}.$$

Here the active setting narrows through commerce, shopping, a department store, and ceramics. The words can now trigger expectations about buying, prices, shelves, and breakable drinking vessels. The point is the change in predictive setting, not a claim that this is the only correct classification tree.

Two sentence examples

• ‘He chose the cup with the blue handle.’ In the ceramics aisle, a drinking vessel is a natural interpretation.
• ‘She asked whether the bowl was dishwasher-safe.’ The shopping and ceramics context helps predict a product question rather than a sporting result.

A trophy could still be sold in a shop, and a football player could still drink tea. Context changes probabilities.

Find this expression in Part II →

Equation guide 10 · From the original paperKeep uncertainty about both the person and the meaning

$$\mathbb{P}(A,S\mid U,X_t).$$

A denotes the antecedent—the person or thing a referring expression points back to. S denotes the sense of an ambiguous expression. U is the local utterance. P(A,S | U,Xₜ) is a joint distribution over those two answers given the words and current state; the paper also packages them as Y = (A,S).

Two sentence examples

• ‘Ravi won the final. He lifted the cup.’ The state can favour Ravi as the antecedent and trophy as the sense together.
• ‘Ravi and Nikhil entered the kitchen. He picked up the cup.’ Drinking vessel may be clear while the identity of ‘he’ remains uncertain.

A distribution can retain several possibilities. Assigning slots does not justify pretending an ambiguous sentence has only one interpretation.

Find this expression in Part II →

5. How much does a clue actually help?

Consider prediction as assigning confidence to possible answers. A machine that gives the correct answer a tiny probability receives a large penalty under logarithmic loss. A well-calibrated machine tries to assign probabilities that match how often outcomes really happen. The paper asks how much the best achievable average penalty falls when context becomes available.

Equation guide 11 · From the original paperThe average value of available context

$$R_U^*-R_{U,C}^*=\mathrm{I}(Y;C\mid U)\geq0.$$

Rᵤ* is the best average log-loss risk when only the utterance U is known. Rᵤ,꜀* is the best risk when context C is also known. Their difference is I(Y;C | U), the extra information context gives about the interpretation. The proof writes these optimal risks as H(Y | U) and H(Y | U,C): uncertainty before and after context.

Two sentence examples

• ‘She stood at the crease.’ Knowing the conversation concerns cricket can reduce uncertainty about what ‘crease’ means.
• ‘She stood at the crease.’ Knowing an unrelated neighbour owns a bicycle may provide no relevant information, giving zero improvement in an idealized task.

Nonnegativity is an average optimal-prediction statement. A particular clue can lower confidence in a candidate; a limited real model can also use context badly.

Find this expression in Part II →

The Bayes-optimal predictor is an ideal benchmark that knows the true relevant probabilities. The theorem concerns that benchmark, averaged over the task’s distribution. It does not promise that adding text will improve every real model or every individual prediction.

Equation guide 12 · From the original paperCredit each layer only for its new contribution

$$\mathrm{I}(Y;C\mid U) = \sum_{j=1}^{d}\mathrm{I}(Y;F_j\mid U,F_{<j}).$$

C = (F₁,…,Fd) is a stack of d context layers. The sum adds what layer Fⱼ contributes after the utterance and all earlier layers F<ⱼ are known. These Fⱼ symbols are frames in this section; Fₜ elsewhere denotes a future target. The total contribution is exactly the information supplied by the whole stack.

Two sentence examples

• ‘He received the cup.’ ‘Sport’ may help, then ‘football final’ may add a more specific clue beyond it.
• ‘She bought a mouse.’ ‘Shopping’ supplies one setting; ‘computer accessories’ adds the distinction that points towards an electronic device.

The assignment to individual layers depends on the order of conditioning. Repeated or redundant information is not counted twice.

Find this expression in Part II →

Equation guide 13 · From the original paperHow one layer changes confidence in one answer

$$g_j(y)= \log \frac{\mathbb{P}(y\mid U,F_{\leq j})} {\mathbb{P}(y\mid U,F_{<j})}.$$

Choose a candidate interpretation y. Divide its probability after the new layer by its probability before that layer, then take a logarithm. A ratio above one gives a positive signal; below one gives a negative signal; one gives zero. Both probabilities must be nonzero for this expression. Using base-two logs makes a doubling worth one bit.

Two sentence examples

• ‘He got the cup.’ In an invented example, a football clue raises the trophy probability from 0.25 to 0.5, producing log₂(2) = 1.
• ‘It was full of tea.’ If that clue lowers the trophy probability from 0.5 to 0.125, the corresponding signal is log₂(0.25) = −2.

These numbers illustrate the formula; they are not measurements from the paper. The signal concerns a chosen answer, not a physical wave.

Find this expression in Part II →

Equation guide 14 · From the original paperAll the little confidence changes add up

$$\sum_{j=1}^{d}g_j(y) = \log \frac{\mathbb{P}(y\mid U,C)}{\mathbb{P}(y\mid U)}.$$

Adding the per-layer log ratios gives the log ratio between final confidence and starting confidence. Intermediate probabilities cancel: multiplying 0.4/0.2 by 0.8/0.4 leaves 0.8/0.2. Taking logarithms turns that product into a sum. This is the telescoping step in the proof.

Two sentence examples

• ‘She served an ace.’ Suppose successive sporting clues move the tennis interpretation from 0.2 to 0.4 to 0.8; the total base-two signal is 1 + 1 = 2.
• ‘He got the cup.’ If later tea-related evidence exactly reverses an earlier probability increase, the two signals sum to zero and confidence returns to its starting value.

The identity describes successive conditional probabilities, not a promise that useful interpretations always become more certain.

Find this expression in Part II →

Equation guide 15 · From the original paperA compressed clue cannot contain more evidence than its source

$$\mathrm{I}(Y;Z\mid U)\leq\mathrm{I}(Y;C\mid U).$$

Z = φ(C) is a summary computed from context C. Its conditional information about Y cannot exceed the context's. Equality holds precisely when Y is conditionally independent of the full context given (U,Z): the summary leaves no additional context information useful for that target. Here Z is a compressed context, not the discrete coordinate set also named Z in Section 6.

Two sentence examples

• ‘Maya ordered vegetarian momo; Ravi ordered chicken.’ A summary preserving the two assignments can retain the plate-delivery distinction.
• ‘Maya ordered vegetarian momo; Ravi ordered chicken.’ A summary saying only ‘food ordered’ throws that distinction away.

This is a test of retained predictive content. Low dimension, smoothness, or a pleasing visualization is insufficient evidence.

Find this expression in Part II →

6. Stories inside stories

Some situations open inside other situations. You pause one task, do a smaller task, and return. Computer scientists call the most recently opened, first closed organization a stack. Think of keeping your place in several nested activities.

The next section uses brackets because brackets give us a tiny world in which the rules are perfectly clear. Real people do not helpfully announce “opening belief frame” every time they speak. Discovering that structure from ordinary sentences is part of the work a future implementation must perform.

7. Why brackets can require different kinds of machinery

A Hankel matrix is a big compatibility table. Rows contain possible beginnings, columns contain possible endings, and each cell asks whether joining its beginning and ending gives an acceptable whole. Rank measures how many independent row or column patterns the table needs over the chosen number system. It is not a synonym for model parameter count.

Equation guide 16 · From the original paperA table of beginnings and endings

$$H_f(u,v)=f(uv), \qquad u,v\in\Sigma^*.$$

u is a beginning and v is an ending; uv means join them. Σ* is the set of all finite strings over the allowed symbol alphabet. Dₘ contains properly nested strings with m bracket types, and f = 1Dₘ outputs 1 for membership and 0 otherwise. The ordinary Hankel entry Hf(u,v) records the joined string's answer.

Two sentence examples

• ‘Open a round bracket, then close a round bracket.’ With u = ‘(’ and v = ‘)’, the entry is 1.
• ‘Open a round bracket, then close a square bracket.’ With u = ‘(’ and v = ‘]’, the entry is 0.

This is an exact artificial language, deliberately chosen so that correctness is unambiguous.

Find this expression in Part II →

Each opening has one matching reversed ending

Opening ↓ / Ending →	))	])	)]	]]
((	1	0	0	0
([	0	1	0	0
[(	0	0	1	0
[[	0	0	0	1

Equation guide 17 · From the original paperThere are always more independent nesting patterns

$$\operatorname{rank}(H_f)\geq m^k\quad\text{for every }k,$$

With m opening types and depth k, the proof finds an mᵏ-by-mᵏ identity submatrix: diagonal ones, zeros elsewhere. Its independent rows force rank(Hf) to be at least mᵏ. Because k can grow without bound and m ≥ 2, the paper concludes rank(Hf) = ∞. A finite sampled matrix, by contrast, always has finite dimensions and rank.

Two sentence examples

• ‘Open two brackets, choosing round or square each time.’ There are 2² = 4 opening-type sequences, each requiring its own matching reverse ending.
• ‘Open three brackets, choosing round or square each time.’ There are 2³ = 8 such sequences; increasing the permitted depth keeps expanding the independent subtable.

Infinite rank is a property of the idealized unbounded language, not a measured number for a finite collection of English sentences.

Find this expression in Part II →

Equation guide 18 · From the original paperWrite down the opening sequence

$$u_{\bm a}=c_{a_1}\cdots c_{a_k}.$$

The bold a is a list (a₁,…,aₖ) of k type choices, each from [m] = {1,…,m}. cₐ₁ through cₐₖ are the corresponding opening symbols. Writing them together defines a row prefix uₐ. Different lists give different outstanding obligations about how the sequence must eventually close.

Two sentence examples

• ‘Open red, then blue.’ For a = (red,blue), the prefix contains those two openings in that order.
• ‘Open blue, then red.’ The reversed choice gives a different prefix, even though the same two colours occur.

The colour sentences explain the bracket types. The formal symbols carry no real-world colour semantics.

Find this expression in Part II →

Equation guide 19 · From the original paperClose those openings in reverse order

$$v_{\bm b}=r_{b_k}\cdots r_{b_1}.$$

The suffix vᵦ closes the type list b from its last element back to its first. The joined sequence uₐvᵦ is well matched exactly when a = b. That is why the selected table is an identity matrix Iₘᵏ: one correct partner per row and column, with all other pairings failing.

Two sentence examples

• ‘Open red, open blue, close blue, close red.’ This pairs the opening sequence with its matching reversed ending.
• ‘Open red, open blue, close red, close blue.’ This closes the outer type too soon and fails the matching test.

The intermediate construction makes the rank claim concrete rather than relying only on a verbal appeal to complexity.

Find this expression in Part II →

The proof pairs every length-$k$ opening sequence with exactly one reversed matching closing sequence. Different pairs demand different answers. The resulting subtable has ones down its diagonal and zeros elsewhere. At two types and depth two, it contains four independent patterns. At depth three, eight. Since the language allows arbitrarily deep nesting, there is no finite upper ceiling on the ordinary rank.

Equation guide 20 · From the original paperThe permitted memory operations change what a finite description can do

$$C_{\mathrm{WFA}}(f,0)=\infty, \qquad C_{\mathrm{VPA}}(f,0)<\infty,$$

For exact recognition of this f, no finite weighted word automaton (WFA) suffices under the stated cost convention. A visibly-pushdown automaton (VPA) does have a finite controller and finite stack alphabet. Its stack can still grow. The two cost statements therefore distinguish realization classes, not an infinite-memory task magically becoming fixed-memory.

Two sentence examples

• ‘Check any depth of correctly matched round and square brackets.’ A fixed finite weighted-word realization cannot represent the constructed function exactly.
• ‘On opening, push the type; on closing, check and pop it.’ This finite stack procedure handles increasing depth by storing more stack entries.

This is not a theorem separating the proposal from all neural networks, and it does not give an end-to-end language benchmark result.

Find this expression in Part II →

The paper formalises that last example as a history $c_Ix^L$: an opening type $I$, followed by $L$ identical filler observations, followed by a required matching return $r_I$. “Remember the password, then hear ten identical beeps” is another instance. “Remember the requested drink, then hear twenty identical announcements” is a third. The limited observer sees the same suffix in every case; the saved state retains the differentiating fact.

The separation is between the specified computational classes. It does not show that attention-based or recurrent neural systems cannot remember nesting. It also does not prove that English behaves like brackets, or that the theoretical stack will run faster on real hardware.

8. Who matters more than the spelling of a name

Equation guide 21 · From the original paperRename labels while respecting their types

$$\mathcal{G}=\prod_{\tau\in\mathcal{T}}\mathrm{Sym}(\mathcal N_\tau)$$

𝒯 is the finite set of types. 𝒩τ is the name set for type τ. Sym(𝒩τ) is the collection of one-to-one, reversible renamings within that set. The product combines one permitted renaming for each type, forming the group 𝒢. People stay people; locations stay locations. Equality relationships survive a bijection.

Two sentence examples

• ‘Maya visited Pokhara.’ Renaming Maya to Tara and Pokhara to another place label preserves the person–place relation in the formal story.
• ‘Maya called Maya.’ Consistently replacing Maya by Tara produces ‘Tara called Tara,’ preserving the repeated identity.

Collapsing two different names into one is not a permutation. Neither is exchanging a person with an object of another declared type.

Find this expression in Part II →

Equation guide 22 · From the original paperRename first or update first: the structure agrees

$$T(g\cdot x,g\cdot a)=g\cdot T(x,a), \qquad O(g\cdot x)=O(x).$$

T updates a state x using an observation a. g·x means rename x consistently. The first equality says renaming both input and state before an update has the same effect as renaming the updated state afterwards. The second says the structural output O stays unchanged. Name-valued answers instead change consistently with the renaming.

Two sentence examples

• ‘Maya lent Ravi a book.’ Whether we rename them before or after recording the loan, the same lender–borrower relation should result.
• ‘Who borrowed the book?’ A structural answer ‘the second person’ stays the same; an answer using the literal name must change from Ravi to the replacement label.

For probabilistic transitions, the renaming acts on the resulting probability distribution. Exact canonical-trace claims require the corresponding specified allocation/update discipline.

Find this expression in Part II →

The formal symmetry treats names as arbitrary labels within their declared types. In ordinary life, a famous name can bring relevant knowledge: replacing “Ronaldo” with a person who has never played football changes what we know. The mathematical comparison must preserve relevant attributes, roles, and facts rather than silently swapping biographies. It tests dependence on arbitrary spelling, not independence from all identity-related knowledge.

Equation guide 23 · From the original paperAlready known, or new to the live memory?

$$\kappa_R(n)= \begin{cases} \mathrm{EQ}_{[i]}, & n\text{ is equal to the name in occupied slot }i,\\ \mathrm{NEW}_\tau, & n\text{ is unequal to every live register name}. \end{cases}$$

κR turns a name into a finite code using the current register. EQ[i] means it equals the name in live slot i, or that slot's equality block if duplicates are allowed. NEWτ means no current live name matches, and the incoming name has type τ. Allocation and eviction are additional specified actions. This is causal because it only checks the current register.

Two sentence examples

• ‘Maya entered. Maya waved.’ If the first mention occupies slot 1, the next name receives an equality-to-slot-1 code.
• ‘Maya entered. Ravi waved.’ If Ravi is absent from every live slot, his name receives a typed-new code.

NEW means new relative to live memory, not necessarily never encountered in the entire past. Coreference between different expressions still needs to be resolved.

Find this expression in Part II →

The canonicalization proof follows the two renamed stories together, one observation at a time. A one-to-one renaming preserves both equality and inequality. If the same finite allocation rule is used, both stories produce the same new-versus-existing codes. Induction means repeating that argument after every step. This proof assumes the specified equivariant update discipline; it does not solve the harder problem of learning perfect entity recognition from raw text.

9. A finite set of relationship patterns

Equation guide 24 · From the original paperCount shapes of register memory rather than literal names

$$N_{B,\mathcal{T},A} \leq 4^B\operatorname{Bell}(B)(|\mathcal{T}|A)^B.$$

N counts canonical register arrangements with B slots, |𝒯| types, and A finite attribute combinations. The generous bound includes occupancy, pin flags, equality partitions counted by Bell(B), types, and attributes. Renaming labels leaves these arrangements unchanged. The finite number can nevertheless be extremely large.

Two sentence examples

• ‘Maya and Ravi are both present.’ Two occupied slots containing distinct people form a different pattern from two slots that point to the same person.
• ‘Maya is still needed in an open quotation.’ A pinned slot and an otherwise identical unpinned slot have different allowed eviction behaviour.

The bound intentionally overcounts. It proves a finite upper limit under finite assumptions, not an efficient implementation. Appendix A expands the factors.

Find this expression in Part II →

Equation guide 25 · From the original paperCombine the finite choices into one controller

$$n\leq |Q|\,|Z|\,N_{B,\mathcal{T},A}$$

|Q| counts local-control settings, |Z| counts discrete predictive-coordinate settings, and N counts canonical register arrangements. Their product bounds the combined canonical controller's states n. Finite frame labels and finite slot pointers give a finite stack alphabet. Under the theorem's assumptions, the constructed weighted VPA has a nested Hankel rank at most n².

Two sentence examples

• ‘The toy assistant is waiting or answering.’ With two local settings, three discrete predictive settings, and four register patterns, the product bound is 2 × 3 × 4 = 24.
• ‘The toy assistant adds one more waiting mode.’ Raising the local count from two to three, while keeping the other counts fixed, raises that bound to 36.

The numerical machines are illustrations. The paper does not establish these counts for English, cheap context extraction, or a bound on unbounded stack height.

Find this expression in Part II →

To understand the main structured-realization theorem, imagine choosing one local-control setting, one discrete predictive setting, and one canonical register arrangement. Those choices give a finite combined controller. Context frames also use finite labels and pointers. After the input has been causally labelled as call, return, or internal, this can be organized as a weighted visibly-pushdown machine under the paper’s assumptions.

The accompanying nested-rank statement is a bound of $n^2$, where $n$ counts canonical control states. For a hypothetical two-state realization, the claimed bound is four; for a hypothetical three-state realization, it is nine. These are conditional mathematical counts, not measured neural-network sizes. The stack itself may still become arbitrarily deep unless a separate depth budget is imposed.

The assumptions matter: finite local and discrete predictive settings; finite structural alphabets; visible operations after the lift; a global live-name budget with protected pointers; symmetry-respecting transitions and structural outputs; and finite transition/output weights. A continuous coordinate with unlimited precision cannot simply be counted as one finite discrete setting. A huge or expensive context encoder cannot be ignored either.

10. What happens when the notebook fills up?

Equation guide 26 · From the original paperGive the approximation four explicit budgets

$$X_t^{B,K,D,R}$$

This notation names a bounded approximation to the ideal state: B live entities, K retained ontology items, D explicit nesting depth, and R continuous predictive dimensions. Ωₜᴮ,ᴷ,ᴰ denotes the event that a required discrete resource exceeds its budget. R here is a dimension budget, while Rₜ earlier was a register; q in the following equations denotes a forecast distribution rather than local control.

Two sentence examples

• ‘Six players are involved in this exchange.’ A four-entity state may overflow even when the conversation is only one frame deep.
• ‘Asha thinks Ravi believes Maya suspects Nima is lying.’ A depth-limited state may overflow even if it has enough entity slots.

A resource setting identifies the approximation being tested. It does not certify that the predictor has learned the right interpretation.

Find this expression in Part II →

Equation guide 27 · From the original paperCap the loss of a very confident mistake

$$\ell_\alpha(q,y)=-\log\max\{q(y),\alpha\}, \qquad 0<\alpha<1.$$

q(y) is the probability assigned to the outcome that actually occurs. The maximum with α replaces probabilities below α by the floor for scoring. Minus the logarithm gives clipped loss, with 0 < α < 1. Its range is from zero to Lα = log(1/α). The cap makes the later worst-case bound possible.

Two sentence examples

• ‘The next word is red.’ With an illustrative base-two floor α = 1/16, predicting the actual word with probability 1/2 costs one bit.
• ‘The next word is blue.’ Assigning that actual word probability 1/1024 still costs only four bits under the same clipped score, because the floor replaces it by 1/16.

Clipping changes the scoring rule and is not ordinary unbounded log loss. The bound should not be advertised as an automatic guarantee for standard perplexity.

Find this expression in Part II →

Equation guide 28 · From the original paperBound the damage if mistakes occur only on overflow

$$R_\alpha(q^{B,K,D,R})-R_\alpha(p^*) \leq L_\alpha\mathbb{P}(\Omega_t^{B,K,D}).$$

Rα is expected clipped loss; p* is the ideal forecast. If the bounded predictor agrees with the ideal whenever no overflow occurs and the continuous state is exact, their loss difference can arise only on overflow. Its average is at most the maximum loss range Lα times the overflow probability.

Two sentence examples

• ‘The story needs a fifth live person.’ In a toy setting with Lα = 4 and overflow probability 0.02, the upper bound on the average excess clipped loss is 0.08.
• ‘Every story fits the notebook.’ If overflow never occurs and the agreement assumption really holds, this particular bound gives zero extra loss.

The agreement assumption is strong. Learning errors or a faulty context lift can violate it even with plenty of unused memory. This is an upper bound, not an observed error rate.

Find this expression in Part II →

Equation guide 29 · From the original paperAdd the three overflow risks without assuming independence

$$\begin{aligned} R_\alpha(q)-R_\alpha(p^*) \leq L_\alpha\big[&\mathbb{P}(N_{\mathrm{live}}>B) +\mathbb{P}(d_t>D)\\ &+\mathbb{P}(o_t\notin\mathcal{O}_K)\big]. \end{aligned}$$

Nlive counts live entities, dₜ is context depth, and oₜ is the needed ontology item. 𝒪K is the retained K-item ontology. The chance of at least one overflow is no greater than the sum of the three individual chances. Multiplying by Lα converts that union bound into a risk envelope under the previous assumptions.

Two sentence examples

• ‘This passage mentions too many people and nests too many beliefs.’ It can be counted in both component risks, so their sum can overestimate the chance of any failure.
• ‘Entity overflow occurs in 2%, depth overflow in 1%, and ontology overflow in 1% of toy cases.’ The combined overflow chance is at most 4%, without an independence assumption.

These percentages are hypothetical. Overlapping events explain why the bound may be loose; it is not an exact decomposition of error.

Find this expression in Part II →

Equation guide 30 · From the original paperAdd the error from a coarser predictive coordinate

$$R_\alpha(q)-R_\alpha(p^*) \leq L_\alpha\mathbb{P}(\Omega_t^{B,K,D})+\frac{2\eta_R}{\alpha}.$$

ηR bounds the expected total-variation difference between the ideal and approximated forecasts outside overflow. Total variation measures how far probability distributions differ. Clipped loss changes by at most the probability change divided by α; using the paper's L1 bound gives ∥p*−q∥₁/α = 2 TV(p*,q)/α. Averaging adds the term 2ηR/α to the overflow contribution.

Two sentence examples

• ‘It may rain or remain dry.’ An approximated forecast close to the ideal still has a nonzero projection term even when all discrete memory budgets fit.
• ‘Will the parcel arrive today?’ With hypothetical ηR = 0.001 and α = 0.1, the additional upper-bound term is 0.02 in the formula's log-loss units.

The Lipschitz constant 1/α is for natural logarithms; another log base changes it by a constant factor. A smaller floor can make the bound much looser.

Find this expression in Part II →

The strongest assumption here is easy to miss: outside the stated overflow and approximation errors, the bounded predictor must agree with the ideal predictor. A real model can still misread a sentence, learn the wrong rule, or use a bad ontology while every memory slot is available. Those errors are not automatically covered by the overflow theorem. A practical evaluation must measure them too.

11. Knowing the event still leaves the words to choose

Equation guide 31 · From the original paperFirst predict the category, then the word inside it

$$\mathrm{H}(W\mid H) = \mathrm{H}(S\mid H)+\mathrm{H}(W\mid S,H).$$

H in upright information notation measures entropy, while the conditioning H denotes the observed history. W is the next token and S = g(W) is its deterministic coarser category. The total uncertainty equals uncertainty about the category plus the remaining uncertainty about the token once category and history are known.

Two sentence examples

• ‘She chose the…’ In a controlled vocabulary, knowing the next token is one of four equally likely colour words still leaves two bits to choose which colour.
• ‘He whispered, “…”’ Knowing that the next token belongs to a greeting category can still leave a choice among hello, hi, and hey.

The category must satisfy the deterministic-coarsening assumption. A spelling alone does not deterministically identify an ambiguous word's semantic sense.

Find this expression in Part II →

Equation guide 32 · From the original paperWhy counting the word and its fixed category does not double the uncertainty

$$\mathrm{H}(W,S\mid H)=\mathrm{H}(S\mid H)+\mathrm{H}(W\mid S,H)$$

The entropy chain rule first counts uncertainty about S, then about W given S and history. Because S is fixed by W, H(S | W,H) = 0 and H(W,S | H) = H(W | H). Substituting that fact yields the preceding structural-versus-lexical identity. There is no additional unknown category after a token and its fixed classification are known.

Two sentence examples

• ‘The chosen word is blue.’ If a toy dictionary always classifies blue as a colour word, learning blue also reveals its category without another guess.
• ‘The chosen word is hello.’ If hello always belongs to the toy greeting class, reporting both the word and that class adds no independent uncertainty.

If the category is not determined even after word and history are known, the simple decomposition requires subtracting the remaining H(S | W,H) term.

Find this expression in Part II →

For the stated identity, the category must be a deterministic coarsening of the predicted token, or of the token together with fixed history. A word like “bank” does not carry a uniquely determined sense by spelling alone. A toy dictionary that assigns every word one fixed category satisfies the simple assumption; real semantic annotations need more care. If the structural variable is independently uncertain, the simple two-term expression needs an additional correction term.

The implication for thinking machines is practical. A compact account of who did what may be valuable while fluent wording remains expensive. That would be an interesting partial result. It would not demonstrate that all language generation can be compressed into a tiny state.

12. Three hypotheses, three separate chances to fail

In everyday language, H1 asks whether explicit nested memory saves resources for some long-range dependencies. H2 asks how much entity-related prediction depends on equality, role, and type rather than arbitrary name spelling. H3 asks whether the complete machine actually achieves a better prediction–cost trade-off than appropriate comparison systems. These remain hypotheses about English and implemented systems.

The paper draws on established work: discourse focus spaces, discourse referents, frame semantics, predictive states, information theory, and structured automata. Its proposed contribution is how those pieces are assembled around bounded resources and testable costs. The references in Part II retain the original bibliography. This article does not claim that stacks, symmetry, or the information identities were invented here.

13. The appendices: count the patterns, then measure what survives

Appendix A expands the state-counting argument. You count which slots are occupied, which occupied slots refer to the same entity, the allowed types and attributes, and which slots are pinned. Multiplying generous upper bounds proves that the total is finite under finite budgets. It can substantially overcount impossible combinations.

Equation guide 33 · From the original paperOpen up the counting bound

$$N_{B,\mathcal{T},A} \leq 2^B\cdot\operatorname{Bell}(B)\cdot(|\mathcal{T}|A)^B\cdot2^B =4^B\operatorname{Bell}(B)(|\mathcal{T}|A)^B.$$

The first 2ᴮ allows for occupied versus empty slots; Bell(B) overcounts equality partitions; (|𝒯|A)ᴮ allows types and finite attributes; the final 2ᴮ allows pin flags. Multiplying the two powers of two gives 4ᴮ. Appendix A notes that forbidding duplicate slot values removes the Bell factor and that a sum over occupied counts can tighten the bound.

Two sentence examples

• ‘Only Maya is currently tracked.’ With two available slots, the occupancy pattern differs from one in which Maya and Ravi both occupy slots.
• ‘Both saved references point to Maya.’ Allowing this equality pattern requires counting same-versus-different arrangements; forbidding duplicates removes that choice.

Many counted combinations may be impossible, such as inconsistent types inside an equality class. An overcount is acceptable for an upper bound.

Find this expression in Part II →

Equation guide 34 · From the original paperCompare the state's observed gain with the full-history gain

$$\widehat\rho = \frac{\mathrm{CE}(F_t\mid L_t)-\mathrm{CE}(F_t\mid X_t,L_t)} {\mathrm{CE}(F_t\mid L_t)-\mathrm{CE}(F_t\mid H_t,L_t)}.$$

CE is measured cross-entropy: smaller is better. Lₜ is a local baseline, Xₜ the structured state, and Hₜ the full raw history. The numerator measures the improvement from giving the local predictor the state. The denominator measures the improvement from giving it full history. Their ratio is an operational capture metric; only Bayes-optimal models justify reading it as an information ratio.

Two sentence examples

• ‘The answer depends on an earlier promise.’ With illustrative losses 5 locally, 3.5 with the state, and 3 with full history, the ratio is (5−3.5)/(5−3) = 0.75.
• ‘All useful clues are already in the local words.’ If local and full-history losses are identical, the denominator is zero and the ratio is undefined.

Learned-model ratios can fall outside [0,1], and a near-zero denominator is unstable. Report the underlying losses and uncertainty instead of treating the ratio as a literal percentage of understanding.

Find this expression in Part II →

Equation guide 35 · From the original paperAsk what the original history still contributes

$$\Delta_{\mathrm{hist}} = \mathrm{CE}(F_t\mid X_t)-\mathrm{CE}(F_t\mid X_t,H_t).$$

Subtract the loss of a predictor receiving both state and full history from the loss of a state-only predictor. A positive Δhist means raw history still gives an observed gain. Under ideal log-loss prediction, this corresponds to residual conditional information. With actual fitted models, a small gain can also reflect a history model that failed to use the extra evidence.

Two sentence examples

• ‘Ravi promised to return the red book.’ If the state forgot the colour, adding the transcript may improve prediction of the returned object's description.
• ‘The state already preserved the relevant promise.’ If a strong transcript-assisted predictor gains almost nothing across careful tests, that supports—but does not by itself prove—the sufficiency of the retained state.

The paper's Appendix B uses state-only and state-plus-history losses here. This metric is distinct from the local-baseline capture ratio above.

Find this expression in Part II →

The ratios are estimator-dependent measurements. They become exact information quantities only under the stated optimal-prediction conditions. A tiny or zero denominator makes the capture ratio unstable or undefined. Trained models can yield values outside zero to one, so the ratio should be accompanied by its underlying losses and uncertainty, not presented as a universal percentage of understanding.

14. How to give the idea an honest examination

Appendix C asks for strictly online states; complete encoder, update, and decoder costs; resource curves over entity, ontology, depth, and residual budgets; matched baselines; name permutation, context shuffling, lexical substitution, and nesting destruction; separate structural and lexical results; author-level book splits; and a final test opened only after the code and metrics are frozen.

Those are requirements for a future evaluation, not completed experiments. This theory paper contains no demonstrated large-scale win on English. A full evaluation should also expose uncertainty in frame boundaries and referent assignment, quantify overflow, and test whether a large unrestricted local state has silently learned to do everything while the named structures sit unused.

15. The philosophy beneath the equations

When I describe a useful state as a small working world, I mean an organized collection of distinctions that changes as evidence arrives. Somebody remains the same person across mentions. A claim remains inside the right person’s belief. A promise remains unfinished until the relevant event resolves it. A new observation changes what the machine expects next.

This suggests a way to think about understanding: preserve relationships that continue to matter while allowing irrelevant details to fall away. But “irrelevant” belongs to a task. A detective may need the exact spelling of a name. A poet may care about the rhythm of a sentence. A legal quotation may require the exact words. There is no universal permission to forget the surface of language.

The paper calls its deeper object a causal quotient. In plain English, histories may be treated alike when they lead to the same required future-prediction distribution. Imagine two differently worded accounts that leave precisely the same relevant expectations. A good representation would not need to store their superficial differences merely to answer those same questions. Whether a small, learnable approximation exists for broad language tasks is the research question.

The three-part state is one proposed anatomy for that approximation. Its success would depend on learning reliable distinctions and using them economically. It is not proof that these are the only parts of thought, that the brain literally contains this stack, or that a machine with these fields necessarily understands in every philosophical sense.

What the mathematics establishes is narrower and useful: for certain structured problems, exposing the right operations changes the kind of machinery needed; bounded reference can be canonicalized under specified symmetries; and finite-resource errors can be related to explicit failure conditions. What remains open is how far those benefits survive the messiness of language and the full cost of implementation.

My question, after all the notation, is this: What would a machine need to carry forward so that the next sentence changes the right understanding of the situation? Building that machine, measuring its costs, and examining its failures is how this proposal can earn its place.

Reading the original

Part II reproduces the complete Theory Version 1.0 manuscript: the abstract, all thirteen sections, proofs, three appendices, and references. The web edition corrects cross-reference labels and retains the supplied PDF unchanged for comparison. The 35 equation-guide labels above are added for this article; the source paper’s displayed equations were unnumbered.

Continue to Part II: Context-Lifted Predictive States · Download the original PDF