Generalization Is a Test, Not a Promise
A model's behavior beyond familiar examples tells us more than its ability to reproduce them.
The geography of unfamiliar examples
A familiar test set can tell us something important and still leave the intended use unresolved. If its examples resemble the training examples, success demonstrates performance in a similar environment. We should welcome that result. We should also ask what changes when the model encounters a different setting, a longer sequence, or a combination absent from its records. Generalization is always generalization across some specified change.
The language of support helps locate this issue. A distribution has regions where examples can occur; a finite dataset samples those regions imperfectly. In practice, familiarity is rarely a clean boundary drawn on a plot. An example can be ordinary in one respect and novel in another. A new patient, a new instrument, or a changed background may preserve many features while altering a relationship the model relied on.
This is why a single label such as “out of distribution” needs explanation. What changed, and why should that change matter to the proposed use? Randomly withholding nearby examples asks a different question from withholding an entire context. Neither is automatically the correct evaluation. The claim determines the test. Without that connection, we can accumulate benchmarks that look demanding while leaving the scientifically important uncertainty untouched.
High dimensions change the picture
With thousands of coordinates, visual intuition becomes unreliable. For familiar random distributions, distances between points can become tightly concentrated as the number of dimensions increases. In a uniformly filled ball, a large fraction of the volume lies near its outer surface. The illustration below makes these effects visible. They depend on the distribution and the geometry being considered; they do not mean all real observations are equally far apart.
The useful implication is that raw dimensionality and usable information are different things. A list with thousands of entries may vary mainly along a much smaller set of patterns. The manifold hypothesis pictures meaningful data as occupying a lower-dimensional region within a larger space. It is a helpful model of structure, rather than a guarantee that every dataset lives on one clean sheet.
A learning system can exploit that structure to generalize efficiently. But learning the shape of common observations does not automatically determine what happens under every departure from them. Nor does the volume near the surface of an abstract ball prove that the model will fail in its interior. Geometry helps us ask where evidence is dense and where assumptions do more work. Actual reliability still requires evaluation on the changes that concern us.
When the rule is known
Artificial tasks offer an advantage that much scientific work lacks: we know the target rule. If a system must reverse a string or recognize a specified pattern, we can generate cases and check answers exactly. Training on shorter examples and testing on longer ones reveals whether its success transfers when memory or computational demands change. A rule that works over many lengths provides a stronger result than success only within the lengths used for learning.
Delétang and colleagues' 2023 study, Neural Networks and the Chomsky Hierarchy, examined architecture-dependent generalization on such tasks. Dziri and colleagues' 2023 Faith and Fate investigated compositional problems, including multi-digit multiplication, using computation graphs and intermediate subproblems. These studies document important limits in the systems and settings examined. They also demonstrate the value of tests organized around known structure.
The interpretation requires restraint. A model that fails a longer example may lack the relevant representation, fail to learn an available representation, or encounter a limitation of the evaluation setup. Success may be systematic over the tested range without establishing correctness for arbitrary lengths. Artificial tasks make diagnosis easier, but they do not abolish the difference between empirical evidence and a theorem about every possible model.
What depth does and does not establish
A transformer has a fixed number of layers in a single forward pass. This constrains how its operations are organized. It does not make the total amount of work independent of input length: longer inputs can require substantially more calculation even when the layer count stays fixed. We should distinguish the depth of dependent computation from the quantity of parallel work.
Merrill and Sabharwal's 2023 analysis shows that specified log-precision transformers can be simulated by constant-depth threshold circuits. The result is valuable precisely because its precision and architecture assumptions are explicit. Turning it into a broad statement that shallow models cannot do arithmetic loses the mathematics. Integer addition and multiplication have highly parallel formulations. A sequential carry procedure is one way to obtain an answer, not proof that every procedure must follow the same sequence.
Related complexity-class separations can also depend on unresolved conjectures. A theoretical restriction is strongest when we name its conditions and the exact target problem. It is weaker when it becomes a metaphor for every difficult task. The empirical fact that particular language models make arithmetic errors remains important. What those errors establish about representation, learning, and execution should be argued from the evidence, rather than supplied by the appearance of a long written algorithm.
Order and regrouping are different
Consider a sequence of card shuffles. Doing shuffle A and then shuffle B generally differs from doing B and then A. This is noncommutativity: order matters. Yet composing permutations is associative. Grouping A with B first, or B with C first, gives the same overall composition as long as the original order is preserved. These properties are different, and the distinction matters for parallel computation.
Associativity can support a tree of combinations. Combine neighboring pieces, combine those results, and continue. A standard parallel scan uses depth growing logarithmically with sequence length, rather than the depth of a straightforward one-step-after-another execution. That does not make the scan automatically constant-depth. Results about a particular state-space architecture need the assumptions of that result, not a general inference that anything parallel must have the same power.
The wider lesson is constructive. The demand for serial work depends partly on the representation and the operations available. Finding a useful summary or a different organization can change the computation. A model may struggle to learn that organization from examples even when it exists. We should ask both whether a solution is expressible and whether the training system reliably finds it. Conflating those questions hides the place where progress could occur.
Giving a system room to work
Intermediate outputs, external memory, repeated passes, and tool use can change the resources available during inference. A scratchpad lets later computations refer to earlier results instead of asking one pass to compress the entire procedure into its internal activations. Such arrangements can improve reliability. They also introduce new opportunities for mistakes, since a wrong intermediate result can shape everything that follows.
A long explanation is therefore not a certificate that a correct procedure was executed. We need to inspect the result and, where possible, intermediate checks. Puzzle evaluations similarly need to distinguish a wrong answer from an output limit, an ambiguous task, or an unsolvable instance. These details are part of the claim, not inconveniences to be discarded from the story.
Generalization becomes a productive question when we state the change, the resources, and the standard of success. Does the method transfer to longer inputs? To a new context? Does it preserve relationships known to matter? Are failures detectable? These questions tell us more than a global declaration that a model “reasons” or “only interpolates.” They connect the observed capability to its practical boundary and show which modification might move that boundary.