Geometric Deep Learning VIII: An RNN Seeded at a Fixed Point h⁽⁰⁾ = R(0, h⁽⁰⁾) Is Equivariant to Left Shifts of a Zero-Padded Input SequenceTextbook
Why sequence boundaries matter
A recurrent neural network (RNN) processes an ordered stream while retaining a summary of earlier inputs. The same update rule is applied at every time step, so it is natural to ask whether shifting the input stream simply shifts the resulting sequence of summaries. The answer depends on how the run begins. The first summary is supplied as an initial state, while the first input encountered after a shift can be different. Bronstein, Bruna, Cohen and Veličković use this boundary effect to explain a precise form of temporal translation equivariance for RNNs in §5.7 of Geometric Deep Learning (arXiv:2104.13478v2, pp. 90–93).
The claim matters when the same sequential model is applied to streams presented with different starting times. Without a specified boundary convention, the shared update rule alone does not guarantee that corresponding hidden states agree. The chapter's zero-padding construction makes that convention explicit. Its condition on the initial state can then be stated independently of a particular architecture or training procedure. The book numbers equations but no theorems or lemmas here; the mission's items are cited by section, equation, page, and, for a marginal statement, “side-note.”
RNN runs and padded sequences
Let be the input at time , let be its hidden summary, and let be a shared update map. Given a seed , equation (38) defines a run by
The first processed input is ; is unused. This is the book's time convention, including the named state (Bronstein et al., §5.7, p. 90, equation (38)). The update map is arbitrary: the results concern the recurrence itself, so they do not require the affine layer and activation used as one example in equation (39).
A left shift by steps replaces the input at time with the old input at time . A left-padded input of length is
Here is the zero vector in . The original input begins at time . The two runs being compared use the same update map and the same seed. This is the exact padding convention displayed on p. 93 of the book (Bronstein et al., §5.7, p. 93).
Formalization targets
The capstone is the chapter's one-step equivariance claim. For every update map, input sequence, and seed fixed by the zero-input update, one or more zero-padding steps suffice:
The multi-step side note extends the conclusion to every with :
The milestone list also records the book's unpadded boundary counterexample and its observation that a contraction has a unique attracting fixed point. The latter is represented by an already proved, reusable Banach fixed-point theorem rather than a second draft of the same result (Bronstein et al., §5.7, pp. 92–93, equations (40)–(42) and side-note).
What the result supplies
The equivariance theorem identifies a checkable condition under which the hidden-state sequence tracks a left-shifted input exactly, including at the first state. It also marks the limit of the informal idea that sharing the same update map across time automatically creates shift equivariance: the unpadded run can fail already at time . The fixed-point condition is therefore part of the mathematical claim, rather than an implementation preference (Bronstein et al., §5.7, pp. 92–93).
The RNN shift statements are targets for machine-checked proofs. The contraction milestone already has a machine-checked proof in the referenced CursoEDO.contraction_fixed_point theorem. A completed development would make the boundary convention and the arbitrary-update-map quantifiers available for later statements about sequential models. The recurrence and padding definitions are reusable whenever a discrete-time state update is compared before and after a shift.
The boundary difficulty
The tempting comparison takes two runs that begin with the same seed, then tries to pair the shifted run's state at with the original run's state at . At the first step, those states have encountered different input histories. Equations (40)–(41) display the mismatch: the original run has applied to its first input, while the shifted run still uses the supplied seed at the corresponding update. For an arbitrary and arbitrary first input, the states need not agree (Bronstein et al., §5.7, p. 92). A shift statement that omits its padding length or seed condition would therefore assert something the chapter explicitly rejects.
Formalization scope
Lean represents and as functions on Fin k and Fin m; time and padding lengths are natural numbers. The run uses and consumes at its next update. The padding function returns the zero vector through index and then the correspondingly indexed original input. These definitions cover every finite dimension, including zero-dimensional edge cases, while the unpadded counterexample uses one input and one hidden coordinate.
The target ranges over all update maps and all input sequences; it does not impose a special neural-network parameterization. The seed is any fixed point of , not necessarily the zero state, and the zero-input update need not be the identity map. The formal statements include time and the endpoint . No contraction assumption is needed for shift equivariance itself; contraction enters only the separate existence-and-convergence milestone. The contraction theorem is stated for a nonempty complete metric space, which includes the finite-dimensional real hidden-state spaces used here.
The work requires a small discrete recurrence interface, an exact padding definition, and proofs for the boundary counterexample and shift identities. Those two definitions and the shift results can also support variants with a nonzero padding vector. The LSTM equations of §5.8 have a separate gated state model and are outside this mission's formalization scope. Contributions that keep the full fixed-point hypothesis and the stated boundary cases are welcome.
Selected references
- M. M. Bronstein, J. Bruna, T. Cohen and P. Veličković, Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges, arXiv preprint, 2021, arXiv:2104.13478v2, §5.7, pp. 90–93.