The innovation is orthogonal to past data

Throughout, we work in the Hilbert space of random variables: zero-mean random variables with finite second moments, under the inner product $\langle a, b\rangle = E[ab]$. In this space, orthogonality is uncorrelatedness, and the best linear estimate of a variable given some data is its orthogonal projection onto the subspace the data generates.

Let $S$ be the subspace generated by past data, let $\beta$ be a random $n$-vector, and let $\hat\beta$ be its componentwise orthogonal projection onto $S$ — the best estimate of $\beta$ from the past. Let new measurements arrive as

with $W$ a known $m \times n$ matrix and each noise component $\varepsilon_i$ orthogonal to $S$. Then the innovation

is orthogonal to $S$: every component of $\tilde y$ is uncorrelated with every element of the past-data subspace.

The innovation is the part of the new measurement that could not have been anticipated from the old data. Its orthogonality to $S$ is what makes recursive estimation possible: updating an estimate requires only the innovation, never a recomputation over the whole data history.

Formalization Note. The action of a matrix on a random vector is componentwise, $(W\beta)_i = \sum_j W_{ij}\,\beta_j$ with scalar multiplication in the abstract space. Zero means are implicit in this representation, so no expectation operator appears — only inner products.