The Many Worlds Behind the Same Data
Identifiability asks whether the evidence can distinguish the explanation we want from alternatives that look equally convincing.
The blindfolded downhill walk
Training a neural network is often described as teaching it, but a more concrete picture is a blindfolded walk down a hillside. Each setting of the network's weights corresponds to a location. The height of the land is its error, measured by a loss function. The training procedure calculates which small changes would reduce that loss and takes a step downhill. Repeating this process can transform poor initial guesses into a highly capable predictor.
The hillside has one direction for every adjustable weight, so we cannot picture the actual landscape. The analogy still captures something useful: the walker responds to the shape of the objective. It does not independently decide what question ought to have been asked. If the loss rewards reconstruction, reducing it improves reconstruction. If the loss rewards a particular kind of forecast, reducing it improves that forecast on the examples used for training.
Real training uses approximations, batches of examples, and procedures more elaborate than a plain downhill walk. These affect which solution is found. None changes the first question: what counts as a good answer in the landscape we built? A loss function is a scientific choice as well as an engineering one. It converts an ambition into a score, and some parts of the ambition may disappear in that conversion.
The objective chooses the question
For a language model, training may reward assigning high probability to the next token given its context. For another model, it may reward predicting a hidden measurement. These tasks encourage learning regularities in the records. They can produce representations that are useful far beyond the original objective. But the reward is still attached to specified predictions, rather than to a general obligation to discover a uniquely correct explanation of the world.
The phrase “learns the distribution” also needs care. A probabilistic objective can estimate conditional distributions; squared prediction error instead favors a conditional mean. Different losses preserve different aspects of uncertainty. A mean prediction may be excellent by its assigned score while smoothing over several distinct possibilities. We should therefore ask which properties the objective rewards, rather than assume every form of prediction training reconstructs the same statistical object.
This is where shortcuts become relevant. If one outcome is overwhelmingly common, predicting it repeatedly can achieve a strong score. A model may also exploit a genuine relationship that is useful in the training environment but unstable elsewhere. Neither requires malicious behavior or a defective optimizer. The procedure can be doing exactly what the score asks. Scientific evaluation must decide whether what the score asks is a faithful proxy for what we want to know.
Many rules, one score
Suppose we observe six points and permit a very flexible family of curves. Many members can fit all six. Reaching a small training error tells us the model has found a member of that family; it does not tell us that the member is unique. Researchers sometimes call a collection of similarly successful models a Rashomon set, borrowing the image of different accounts of the same events.
There are two uncertainties to separate. With limited samples, even a statistical relationship that is identifiable in principle may be estimated poorly. More data can help. A different problem arises when two explanations imply the same complete distribution of available observations. More samples from that distribution can describe it more accurately while leaving the explanations tied. The ambiguity concerns what is observed, not simply how many observations we have.
Identifiability gives this distinction a formal name. A target is identifiable when the permitted models cannot disagree about it while agreeing about the available evidence. The word refers to a target, a model class, a kind of data, and assumptions considered together. It is not a permanent label attached to an entire field or an architecture. Changing any of those ingredients can change whether the question has a unique answer.
Newton from photographs
Imagine trying to infer the laws of motion from still photographs. A picture tells you where something was, but ordinarily leaves its velocity and acceleration unresolved. Many trajectories can pass through the pictured position. If all you possess is a distribution of positions with no temporal organization, a description of where objects tend to be need not determine how they move. The photograph can be accurate while the proposed dynamics remain uncertain.
This is the force of the Newton-from-photographs analogy. It is not a claim that images or observational records never reveal dynamics. A sequence with known timing, additional measurements, and physical restrictions can supply much more information. The lesson is that the relevant restrictions must enter somewhere. We cannot infer a unique motion merely because our account of the positions is detailed.
A collection of cell states raises a related issue. It can map recurring patterns without uniquely determining the processes that move cells between them. Historical information or observations across contexts may help; evidence from a deliberate change may help further. The right question is which alternatives still reproduce the records. If those alternatives imply different futures, then a model's answer about the future includes commitments that were not settled by its descriptive fit alone.
What the search finds easy
When several solutions fit, the procedure has preferences. Architecture, initialization, optimization, and regularization affect what is easy to learn. Some neural systems tend to learn broad, slowly varying patterns before finer detail. Such tendencies are often discussed as spectral or simplicity bias. They can support generalization when the target has suitable structure. They can also favor an approximation that loses a sharp distinction important to the task.
A smooth dose–response curve, for instance, might describe the observed range while leaving behavior near an unmeasured threshold uncertain. The issue is not that life is always sharp and learning is always smooth. Both biological processes and learned models can have many kinds of structure. The issue is whether the preferred solution preserves the relationships needed for the question we ask next.
Nor does a preference establish impossibility. Networks can sometimes acquire more systematic generalization late in training, and changing their representations or objectives can alter their behavior. Inductive bias is therefore a resource to examine and design, not merely a flaw to denounce. A helpful restriction reduces the set of explanations in a scientifically defensible way. An unexamined restriction can reduce it just as decisively while quietly choosing the wrong answer.
A world model has to be tested
A navigation model offers a vivid test. Predicting the next turn on familiar routes does not require every inferred connection between streets to form a coherent map. Local answers can look plausible while the implied global structure contains contradictions. To assess the map, we need questions about consistency, legal transitions, and routes that differ from those commonly observed.
Vafa and colleagues' 2024 study, Evaluating the World Model Implicit in a Generative Model, developed evaluations of world-model recovery across controlled domains. Models could perform well on existing diagnostics while their implied worlds were substantially less coherent. The result challenges the inference from prediction to faithful recovery. It leaves open which training conditions and evaluation demands might support better recovery.
For scientific use, the same discipline is valuable. Decide what “understanding” would require operationally. Specify a test that alternative explanations would answer differently. Then examine what the model does. A descriptive model need not be dismissed because it has not passed a mechanistic test it was never meant to address. But a mechanistic claim should not borrow its credibility from a descriptive success. Naming the distinction makes both the achievement and the remaining uncertainty easier to see.
