What does it mean for a physical system to compute? Not in the hand-wavy sense that everything is information processing, but in the precise sense: when does a lump of matter reliably track an abstract mathematical relationship?
Susan Stepney gives the sharpest answer I know. Computation, she says, is when the physical dynamics of a system predict the abstract dynamics of a formal model. The physical system evolves according to its own laws. The abstract system evolves according to its own rules. Computation happens when these two evolutions stay in correspondence—when ε, the tracking error between them, remains small.
Reservoir computing makes this definition visceral. You take a high-dimensional nonlinear dynamical system—a “reservoir” of randomly connected nodes—and you do not train it. The reservoir’s internal weights are fixed. Random. You only train a simple linear readout that watches the reservoir’s state and extracts the answer. The reservoir itself is just physics, churning. The readout is where physics becomes computation.
The first equation is the reservoir: a leaky integrator with nonlinear activation. The second is the readout: a linear map. All the complexity lives in the first line. All the learning lives in the second. This separation is the point.
Below is a live reservoir. Choose a task, adjust the parameters, and watch how random dynamics solve problems they were never designed for.
The reservoir’s magic is separation. Inputs that are similar in a low-dimensional space become distinguishable in the reservoir’s high-dimensional state. Below, the reservoir’s internal state is projected onto its first two principal components. Watch how different inputs trace different trajectories—this geometric separation is the computation.
The top trace shows the input signal (white) and the target output (gold). The bottom trace shows the reservoir’s actual output (cyan). When ε is small, the cyan tracks the gold. When ε is large, the physical dynamics have lost contact with the abstract truth.
Why should random dynamics compute anything at all? The answer lies in what “high-dimensional nonlinear projection” means geometrically. A reservoir with N nodes lives in an N-dimensional state space. Each input drives the state along a trajectory through this space. The nonlinearity (tanh) means the trajectory curves, folds, and wraps. The recurrent connections mean the trajectory depends on history—the reservoir has memory.
This combination—high dimension, nonlinearity, memory—is remarkably powerful. It means that almost any “interesting” random reservoir will separate different input histories into different regions of state space. And once they’re separated, a linear readout can draw a hyperplane between them. Classification becomes trivial when the features are right. The reservoir provides the features; the readout provides the decision.
This is exactly Stepney’s framework: the physical system (reservoir dynamics) creates a rich enough representation that a simple map (linear readout) can extract the abstract relationship (the desired computation). The reservoir doesn’t “know” what it’s computing. It just evolves. The computation is in the correspondence between the physical evolution and the abstract target.
The spectral radius of the reservoir weight matrix W controls everything. It is the largest absolute eigenvalue—the factor by which small perturbations grow or shrink. Set it too low and the reservoir is a damped system: all inputs fade to the same fixed point, memory is short, separation is poor. Set it too high and the reservoir is chaotic: tiny differences in input explode, the state is dominated by internal noise, and the readout sees only turbulence.
The sweet spot is near 1.0—the edge of chaos. Here, the reservoir has maximum memory capacity, maximum separation power, and maximum computational richness. Information from the past fades slowly enough to be useful but fast enough not to overwhelm the present. Try it above: drag the spectral radius slider and watch the metrics respond. At 0.9–1.0, the reservoir is at its most computationally powerful.
Stepney introduces the idea of a semantic gap: the distance between the complexity of the physical substrate and the complexity of the abstract computation. A reservoir of 100 random nonlinear nodes is a very complex physical system. XOR is a very simple abstract operation. The semantic gap between them is enormous. Yet the reservoir computes XOR, because all that complexity creates the right kind of separation.
This is profound. The physical system is far more complex than the computation it performs. Most of the reservoir’s dynamics are irrelevant—noise, from the perspective of the abstract task. The readout’s job is to ignore the noise and extract the signal. Wout is a projection from a 100-dimensional space down to a 1-dimensional answer. It discards 99% of the physical dynamics. That 99% is the semantic gap, made literal.
But here is the key insight: you need that excess complexity. Without it, the reservoir cannot separate the inputs. The noise is not waste—it is the medium in which the signal becomes distinguishable. You cannot compute XOR in a 1-dimensional system. You need the higher dimensions to create the geometry that makes the answer linearly readable. The semantic gap is not a bug. It is the mechanism.
Stepney’s ε is the tracking error: how well the physical dynamics predict the abstract dynamics. In a reservoir, ε is literally the readout error—the difference between the reservoir’s output and the target computation. Watch the signal trace above. When ε hovers near zero, the reservoir is computing. When ε spikes, the physical system has lost contact with the abstract truth.
What makes ε grow? Several things. If the spectral radius is too low, the reservoir lacks the memory to track temporal dependencies—ε grows on tasks that require history. If the spectral radius is too high, the reservoir’s internal chaos overwhelms the input signal—ε grows because the physics is too noisy. If the input scaling is wrong, the reservoir either saturates (all nodes at ±1) or underresponds (all nodes near 0)—in either case, ε grows because the physical dynamics are not in the right regime.
ε is Stepney’s deepest concept. It is the quality of the engineering—how well the physical system has been tuned (even by random chance) to track the abstract target. A well-engineered reservoir has small ε. A poorly-engineered one has large ε. And the question “what is computation?” reduces to: “when is ε small enough to be useful?”
The reservoir makes Stepney’s definition tangible. Computation is not a property of a system in isolation. It is a relationship between a physical system and an abstract system. The reservoir alone is just dynamics—nodes activating, states evolving, attractors forming. It computes nothing by itself. It only computes when we pair it with a target: “predict the next sine value,” “compute XOR,” “echo the input with a delay.” The computation lives in the pairing, not the physics.
This means that the same physical system can compute many things simultaneously. The same reservoir state can be read out in different ways to produce different answers. One readout extracts XOR. Another extracts a delayed echo. A third predicts a sine wave. The physics is the same. The computation changes with the readout. What changes is which aspect of the physical dynamics is declared to “correspond” to which abstract relationship.
And this, I think, is the deepest lesson of reservoir computing: computation is a lens, not a substance. It is something we find in physics when we look at it the right way, with the right readout, against the right abstract target. The reservoir does not know it is computing. But the correspondence is real, the ε is measurable, and the answers are correct. That is enough.