A recurrent network, a nonlinear filter, a physical transducer: all are operators carrying an input signal to an output signal. Approximating such an operator — not a function on , but a map between signal spaces — is the question behind every claim that a recurrent architecture is "universal".
In 1985 Boyd and Chua gave the answer that still underpins the field. They isolated fading memory as the exact continuity notion required, and proved that a time-invariant operator with fading memory can be approximated by a finite Volterra series — uniformly over an infinite time horizon and over a noncompact set of signals. Both italicised words mark the break with what was available before: the classical Volterra approximation theorems hold only on a finite interval and only on a compact set of inputs, which rules out most signals of engineering interest.
This is the result reservoir computing inherits. Every modern universality theorem for echo state networks, state-affine systems, or linear-dynamics-plus-polynomial-readout architectures proceeds by showing the architecture realises enough of these operators, then invokes Boyd–Chua. Formalising it turns the foundation of those arguments into machine-checked mathematics.
Timeline.
Following the convention of the published ReservoirESN definitions, the index counts steps into the past: (discrete) or (continuous) is the value of the signal steps, or units of time, before the present. A sequence or function is therefore the complete history of a signal up to now. Under this convention every operator below is causal.
A weighting is a map decreasing to zero with values in , and the weighted norm is — the distant past is discounted.
An operator is time-invariant when its value at any instant is its present-time value applied to the shifted history, with . It has fading memory on a set when its present-time functional is -continuous on . The quantifier order is taken verbatim from the source: may depend on the input as well as on , so this is pointwise continuity — formally weaker than the uniform version ReservoirESN.FunctionalFMP already on the platform.
In continuous time the admissible inputs are the bounded, slew-limited signals
and the approximating functionals are the convolutions with .
Let and let be any time-invariant operator with fading memory on . Then there are finitely many admissible kernels and a polynomial such that
That is: a bank of linear filters followed by a polynomial readout — the architecture of Fig. 3 of the paper, and, recognisably, the architecture of a reservoir computer. The goal is stated in this form rather than as an explicit Volterra kernel expansion; expanding a polynomial in convolutions into Volterra kernels is a purely algebraic restatement, and formalising Volterra kernels would add bookkeeping without adding mathematical content. The approximation is uniform over all of at once and over all time at once, and is not compact in the sup norm — the whole point of fading memory is that it makes behave as though it were.
The decomposition follows the paper: the discrete-time chain first (Section VI and Appendix A2), then the two continuous-time lemmas the goal rests on (Section IV and Appendix A1).
1 — Damping, and compactness of the discrete ball. On the ball, closeness over a finite horizon already forces closeness in weighted norm; consequently the weighted topology and the product topology coincide there, and the ball is compact by Tychonoff. This is the discrete analogue of Lemma A1, and the only place where is used.
2 — Theorem 4, the discrete NLMA approximation. Fading memory becomes topological continuity on that compact ball; the delay functionals separate points; Stone–Weierstrass then yields a nonlinear moving average — a polynomial read from a finite window, — approximating uniformly. Boyd and Chua note this implies Theorem 3, the discrete finite-Volterra statement.
3 — Lemma 1: compactness in continuous time. The bounded slew-limited set is compact for the weighted norm. The source proves it by Arzelà–Ascoli on each interval followed by a diagonal extraction; the slew limit is exactly the equicontinuity that makes this work, and it is required here — the discrete case needs no analogue, as the paper remarks.
4 — Lemma 2: the convolution functionals separate points. Admissible kernels give -continuous functionals, and they separate: for the kernel is admissible and .
Goal. Stone–Weierstrass on the compact set of milestone 3 with the separating family of milestone 4, then time-invariance to transport the estimate to every instant.
The companion missions on reservoir computing have published, machine-checked, the definitions reused here — UnifBdd, WeightedBound, IsWeighting, FunctionalFMP — along with WeightedCompact.unifBdd_tendsto_subseq, a weighted sequential-compactness result for the discrete ball. Solvers can build on those rather than restate them.
Mathlib has Stone–Weierstrass in the form needed (ContinuousMap.subalgebra_topologicalClosure_eq_top_of_separatesPoints, which asks only for CompactSpace), and it has Arzelà–Ascoli in an abstract uniform-space form. What it has nothing about is fading memory, weighted norms on signal spaces, Volterra series, Laguerre systems, or moving-average operators. The work is the bridge: turning a weighted-norm continuity hypothesis into a topological statement Mathlib's Stone–Weierstrass will accept, extracting a genuine MvPolynomial from an abstract density result, and — for the goal — assembling a compactness proof in continuous time from Mathlib's Ascoli machinery.
Two modelling points are load-bearing and stated plainly rather than buried. The discrete ball must be taken in scalar (or finite-dimensional) signals: for infinite-dimensional values it is not compact and the theorem fails. And in continuous time the slew limit cannot be dropped: without equicontinuity the set is not compact in any topology that makes the convolution functionals continuous.
S. Boyd and L. O. Chua, Fading memory and the problem of approximating nonlinear operators with Volterra series, IEEE Transactions on Circuits and Systems, vol. CAS-32, no. 11, pp. 1150–1161, November 1985.
namespace BoydChua
theorem solution (M₁ M₂ : ℝ) (hM₁ : 0 < M₁) (hM₂ : 0 < M₂)
(w : ℝ → ℝ) (hw : IsWeightingC w)
(N : (ℝ → ℝ) → (ℝ → ℝ)) (hTI : IsTimeInvariantC N)
(hFM : PointwiseFMPC (fun u => N u 0) M₁ M₂ w) (ε : ℝ) (hε : 0 < ε) :
∃ (m : ℕ) (g : Fin m → (ℝ → ℝ)) (p : MvPolynomial (Fin m) ℝ),
(∀ i, AdmissibleKernel w (g i)) ∧
∀ u, SlewBdd M₁ M₂ u → ∀ r ≥ (0:ℝ),
|N u r - MvPolynomial.eval (fun i => Gconv (g i) (ShiftC r u)) p| ≤ ε := by sorry
end BoydChuaTheorem 1 of the source, the headline result. Let a tolerance be given and let N be any time-invariant operator with fading memory on the class of amplitude- and slew-bounded signals. Then there are finitely many admissible kernels and a polynomial such that the operator sending a signal to the polynomial evaluated on its convolutions against those kernels approximates N to within the tolerance — simultaneously for every admissible signal and at every instant.
That is a bank of linear filters followed by a polynomial readout: the architecture of Fig. 3 of the source, and recognisably the architecture of a reservoir computer.
The statement is given in this form rather than as an explicit Volterra kernel expansion. Expanding a polynomial in convolutions into Volterra kernels is a purely algebraic restatement — the source performs it in its Section V — and formalising Volterra kernels would add bookkeeping without adding mathematical content.
What makes the result non-classical is the domain of uniformity. The approximation holds over the whole infinite time horizon and over the entire admissible class at once, and that class is not compact for the supremum norm. The proof supplied by the source is Stone-Weierstrass applied on the weighted-norm compact set of Lemma 1, with the separating family of Lemma 2, followed by time invariance to carry the present-time estimate to every instant.
No open leaves. Every sub-goal is proved or awaiting decomposition.