A reservoir computer learns a dynamical input/output relation with a recurrent network whose internal weights are fixed once and never trained; only a linear readout on the state is fitted. The method works in practice — it is a standard tool for learning chaotic dynamics — but its justification requires an approximation theorem: the family of reservoirs used must be rich enough to reach any reasonable target system.
The target class is fixed by fading memory, the continuity notion Boyd and Chua introduced in 1985 for the approximation of nonlinear operators: a filter has fading memory when inputs that agree on the recent past produce nearby present outputs, however much they differ long ago. The question is then which reservoir families are dense in that class.
Non-homogeneous state-affine systems are the family that answers it. They are affine in the state, with coefficients depending polynomially on the input, and the density result proved for them is what every later universality theorem for reservoir computing rests on — including the one for echo state networks, whose proof approximates a target filter by a state-affine system first and only then by a network.
Timeline.
Time is indexed by the nonpositive integers, so an input has an infinite past and a present. Inputs are real-valued and bounded by one: the set of sequences with .
A non-homogeneous state-affine system is the reservoir
where is a polynomial with matrix coefficients, a polynomial with -vector coefficients, and the linear readout. Writing , the system is affine in the state and polynomial in the input.
Two constants govern it: and . When the state map contracts, the system has the echo state property — exactly one bounded state sequence per input — and the states obey . The induced map from input history to present output is the SAS functional .
Any fading memory filter on uniformly bounded scalar inputs is approximated, uniformly over all such inputs, by a state-affine system read out linearly.
The result itself. It is the density theorem of reservoir computing. Without it, nothing guarantees that a reservoir family can represent the system one is trying to learn, and the practice of fitting only a linear readout has no theoretical backing. It is also the input to the universality theorem for echo state networks: that proof replaces the target filter by a state-affine system before replacing it by a network, so the present result is a prerequisite rather than a parallel statement.
Formalizing it. The supporting target is a specialization of a result already published on this platform: a state-affine system is a contracting reservoir map, so its echo state property follows from the abstract contraction theorem rather than from a new argument. What this mission adds beyond that is the density statement itself, which is of a different nature — an approximation theorem in a function space, not a fixed point argument.
The obvious approach to the goal is to exhibit an approximating system directly, and it fails: the target is an arbitrary fading memory filter, given by no formula, so no construction can be read off it. The proof is not constructive in that sense. It proceeds instead by showing that the family of SAS functionals is a polynomial algebra which separates points and contains the constants, and by applying a Stone-Weierstrass argument on a space of input sequences made compact by the weighted topology.
Two points resist. The compactness is not that of the supremum norm — the space of uniformly bounded sequences is not compact for it — but of the weighted norm, and it is that topology in which the approximation is obtained. And the algebra property is delicate: the product of two SAS functionals must again be one, which is what forces the non-homogeneous form. The corresponding statement fails for linear reservoirs, whose products leave the family.
Time is indexed by , index denoting the instant steps into the past and the present; the system equation reads . This is a relabelling of the source's indexing, not a weakening.
Inputs are scalar, as in the source's Section 3, where the restriction is made explicit and the multidimensional extension deferred to a remark. Polynomials are given by their coefficient families, and evaluated as ; the bounds and are stated as explicit operator and norm bounds valid on rather than through a maximum, so that any valid bound may be supplied.
The fading memory property of the target is the one already published on this platform, stated for a functional rather than a filter: the two are in linear bijection, so nothing is lost and causality and time-invariance need not be formalized separately.
One trivialization is ruled out. The goal quantifies over state sequences satisfying the system equation, and the supporting target is what guarantees such a sequence exists and is unique under the stated bounds; without it, the approximation claim could be read as vacuous.
A complete development needs the Stone-Weierstrass theorem, available in Mathlib, together with compactness of the weighted sequence space, which is not and has to be built. Contributions are welcome on both targets.
namespace ReservoirSAS
theorem sas_universal
(ε : ℝ) (hε0 : 0 < ε) (hε1 : ε < 1) (w : ℕ → ℝ) (hw : IsWeighting w)
(H : (ℕ → ℝ) → ℝ) (hH : FunctionalFMP H 1 w) :
∃ (N r s : ℕ) (P : Fin r → Matrix (Fin N) (Fin N) ℝ)
(Q : Fin s → EuclideanSpace ℝ (Fin N)) (W : EuclideanSpace ℝ (Fin N)),
MatPolyOpBound P (1 - ε) ∧ VecPolyBound Q (1 - ε) ∧
∀ z : ℕ → ℝ, (∀ k, z k ∈ Set.Icc (-1 : ℝ) 1) →
∀ x : ℕ → EuclideanSpace ℝ (Fin N), IsSASSolution P Q z x →
(∀ k, ‖x k‖ ≤ (1 - ε) / ε) →
|H z - inner ℝ W (x 0)| < ε := by sorry
end ReservoirSASLet be a functional on scalar input sequences bounded by one, with the fading memory property. The theorem asserts that for every there exist polynomials , and a readout vector , with both polynomial bounds below , such that the associated state-affine system approximates uniformly:
This is Theorem 3.12 of the source. It is the density result of reservoir computing: the family of state-affine systems with linear readouts is dense in the fading memory category. Every later universality theorem for reservoir families rests on it, including the one for echo state networks, whose proof replaces the target filter by a state-affine system before replacing it by a network.
The proof is not constructive in the naive sense: the target is an arbitrary fading memory filter given by no formula. It proceeds by showing that the SAS functionals form a polynomial algebra separating points and containing the constants, then applying a Stone-Weierstrass argument on a space of inputs made compact by the weighted topology.
Formalization Note A single controls both the approximation accuracy and the polynomial bounds , exactly as in the source; it is not an independent tolerance. The state ceiling appearing in the conclusion is what the supporting target delivers for those bounds, and it is that target which guarantees such a state sequence exists and is unique, so the approximation clause is not vacuous. The conclusion is stated for every bounded solution rather than for a distinguished one, which is stronger. Inputs are scalar, as in Section 3 of the source. The fading memory property of the target is the one already published on the platform, stated for a functional rather than a filter; the two are in linear bijection.
No open leaves. Every sub-goal is proved or awaiting decomposition.