Finite Neural Networks, Gaussian Initialization, and the Limiting NTK
DefinitionJGH_NTK_ModelMathematical model
Notation and probability model
Let be the input and output dimensions, the number of hidden layers, , , and a Lipschitz activation with a nonnegative Lipschitz constant . For widths , , and with , the probability space is the finite real parameter space with every weight and bias coordinate independently . Its law is . The network has the recursion
with output . The full kernel, including all weights and biases, is
For a centered Gaussian pair with covariance induced by on , put
All kernel products in the last expression are pointwise. Local index in
covarianceKernel and limitingNTK denotes paper depth .
The dataset is any fixed finite family; repetitions and
are allowed. is the Kronecker delta.
The limit takes to infinity first and last. More precisely, for any required error tolerance, the width condition is ; each inner threshold may depend on the fixed outer widths. For the filter is concentrated on the unique empty width vector, so the statements require the exact affine base case. This is not a simultaneous-width or whole-input-space uniform limit.
Formalization note: Gaussian measures are concrete Mathlib measures, including singular covariance. The covariance-validity milestone establishes their covariance interpretation; it is not a hypothesis of either convergence target. The activation assumption is only Lipschitz. Derivatives take Mathlib's zero value at points without derivatives, and proofs must justify the null exceptional set under positive Gaussian bias. Native convergence in distribution includes almost-everywhere measurability and weak convergence of probability laws. Primary source conventions: Jacot–Gabriel–Hongler, Section 2, PDF pp. 2–3; Section 4.1, PDF p. 5, Proposition 1, Theorem 1 and Remarks 2–3; Appendix A opening paragraphs, PDF p. 11, and Appendix A.1, PDF pp. 11–13. The relevant displays have no equation numbers.
import Mathlib.Analysis.Calculus.FDeriv.Basic
import Mathlib.Analysis.Calculus.Deriv.Basic
import Mathlib.Probability.Distributions.Gaussian.Multivariate
import Mathlib.MeasureTheory.Function.ConvergenceInDistribution
import Mathlib.Data.Fin.Tuple.Basic
/-!
Finite networks and their initialization laws for Jacot--Gabriel--Hongler,
arXiv:1806.07572v4, Sections 2 and 4.1, PDF pp. 2--5; Appendix A, pp. 11--13.
The referenced architecture, kernel recurrences, and limit conventions are unnumbered displays.
-/
noncomputable section
open MeasureTheory ProbabilityTheory Filter
open scoped BigOperators Topology ENNReal
namespace JGH
/-- Layer `l`, destination neuron, and source neuron (`none` denotes its bias). -/
abbrev ParameterIndex (L : ℕ) (width : ℕ → ℕ) :=
Σ l : Fin L, Fin (width (l.val + 1)) × Option (Fin (width l.val))
abbrev Parameters (L : ℕ) (width : ℕ → ℕ) := ParameterIndex L width → ℝ
/-- Independent standard normal law on every weight and bias coordinate. -/
def initialization (L : ℕ) (width : ℕ → ℕ) : Measure (Parameters L width) :=
Measure.pi (fun _ ↦ gaussianReal 0 1)
instance (L : ℕ) (width : ℕ → ℕ) : IsProbabilityMeasure (initialization L width) := by
unfold initialization
infer_instance
/-- Preactivations, with the input at layer zero. Values beyond `L` are unused. -/
def preactivation (L : ℕ) (width : ℕ → ℕ) (σ : ℝ → ℝ) (β : ℝ)
(θ : Parameters L width) (x : Fin (width 0) → ℝ) : (l : ℕ) → Fin (width l) → ℝ
| 0 => x
| l + 1 => fun j ↦
if hl : l < L then
(∑ i : Fin (width l), θ ⟨⟨l, hl⟩, j, some i⟩ *
(if l = 0 then preactivation L width σ β θ x l i
else σ (preactivation L width σ β θ x l i))) / Real.sqrt (width l) +
β * θ ⟨⟨l, hl⟩, j, none⟩
else 0
/-- The final affine preactivation, without applying the nonlinearity to the output. -/
def network (L : ℕ) (width : ℕ → ℕ) (σ : ℝ → ℝ) (β : ℝ)
(θ : Parameters L width) (x : Fin (width 0) → ℝ) : Fin (width L) → ℝ :=
preactivation L width σ β θ x L
/-- Parameter derivative in a coordinate direction; totalized by Mathlib off differentiability. -/
def parameterDerivative (L : ℕ) (width : ℕ → ℕ) (σ : ℝ → ℝ) (β : ℝ)
(θ : Parameters L width) (x : Fin (width 0) → ℝ)
(k : Fin (width L)) (p : ParameterIndex L width) : ℝ := by
classical
exact fderiv ℝ (fun η ↦ network L width σ β η x k) θ (Pi.single p 1)
/-- The full weight-and-bias neural tangent kernel, including cross-output entries. -/
def finiteNTK (L : ℕ) (width : ℕ → ℕ) (σ : ℝ → ℝ) (β : ℝ)
(θ : Parameters L width) (x y : Fin (width 0) → ℝ)
(k k' : Fin (width L)) : ℝ :=
∑ p, parameterDerivative L width σ β θ x k p *
parameterDerivative L width σ β θ y k' p
abbrev Input (d : ℕ) := Fin d → ℝ
abbrev Kernel (d : ℕ) := Input d → Input d → ℝ
/-- Centered Gaussian pair, allowing singular covariance (in particular, repeated inputs). -/
def gaussianPair {d : ℕ} (K : Kernel d) (x y : Input d) :
Measure (EuclideanSpace ℝ (Fin 2)) :=
multivariateGaussian 0 (fun i j ↦ K (![x, y] i) (![x, y] j))
def gaussianProductExpectation {d : ℕ} (K : Kernel d) (φ : ℝ → ℝ)
(x y : Input d) : ℝ :=
∫ z, φ (z 0) * φ (z 1) ∂gaussianPair K x y
/-- Index `h` means network depth `h + 1`, so `h = 0` is the affine base case. -/
def covarianceKernel (d : ℕ) (σ : ℝ → ℝ) (β : ℝ) : ℕ → Kernel d
| 0 => fun x y ↦ (∑ i, x i * y i) / d + β ^ 2
| h + 1 => fun x y ↦ gaussianProductExpectation (covarianceKernel d σ β h) σ x y + β ^ 2
/-- The deterministic NTK recurrence in Theorem 1, PDF p. 5. -/
def limitingNTK (d : ℕ) (σ : ℝ → ℝ) (β : ℝ) : ℕ → Kernel d
| 0 => covarianceKernel d σ β 0
| h + 1 => fun x y ↦ limitingNTK d σ β h x y *
gaussianProductExpectation (covarianceKernel d σ β h) (deriv σ) x y +
covarianceKernel d σ β (h + 1) x y
/-- Positive hidden width `w i + 1`; fixed input and output widths `d` and `q`. -/
def widths {h : ℕ} (d q : ℕ) (w : Fin h → ℕ) (l : ℕ) : ℕ :=
if l = 0 then d else if hl : l - 1 < h then w ⟨l - 1, hl⟩ + 1 else q
/-- Earlier hidden widths tend to infinity first; the last width is outermost. -/
def sequentialWidths : (h : ℕ) → Filter (Fin h → ℕ)
| 0 => pure Fin.elim0
| h + 1 => (atTop : Filter ℕ).bind
(fun last ↦ (sequentialWidths h).map (fun earlier ↦ Fin.snoc earlier last))
def inputForWidths {h : ℕ} (d q : ℕ) (w : Fin h → ℕ) (x : Input d) :
Fin (widths d q w 0) → ℝ :=
fun i ↦ x ⟨i.val, by simpa only [widths, if_pos rfl] using i.isLt⟩
def outputForWidths {h : ℕ} (d q : ℕ) (w : Fin h → ℕ) (k : Fin q) :
Fin (widths d q w (h + 1)) :=
⟨k.val, by simpa only [widths, Nat.add_one_ne_zero, if_false, Nat.add_sub_cancel,
Nat.lt_irrefl, dite_false] using k.isLt⟩
def initializedOutput {h N : ℕ} (d q : ℕ) (σ : ℝ → ℝ) (β : ℝ)
(X : Fin N → Input d) (w : Fin h → ℕ)
(θ : Parameters (h + 1) (widths d q w)) : EuclideanSpace ℝ (Fin N × Fin q) :=
WithLp.toLp 2 (fun a ↦ network (h + 1) (widths d q w) σ β θ
(inputForWidths d q w (X a.1)) (outputForWidths d q w a.2))
/-- Joint limit law on all dataset and output coordinates: independent output GPs. -/
def outputGaussian {N : ℕ} (d q : ℕ) (σ : ℝ → ℝ) (β : ℝ)
(h : ℕ) (X : Fin N → Input d) : Measure (EuclideanSpace ℝ (Fin N × Fin q)) :=
multivariateGaussian 0 (fun a b ↦
if a.2 = b.2 then covarianceKernel d σ β h (X a.1) (X b.1) else 0)
instance {N : ℕ} (d q : ℕ) (σ : ℝ → ℝ) (β : ℝ) (h : ℕ) (X : Fin N → Input d) :
IsProbabilityMeasure (outputGaussian d q σ β h X) := by
unfold outputGaussian
infer_instance
/-- Probability that some entry on the fixed finite dataset exceeds the tolerance. -/
def ntkBadProbability {h N : ℕ} (d q : ℕ) (σ : ℝ → ℝ) (β : ℝ)
(X : Fin N → Input d) (ε : ℝ) (w : Fin h → ℕ) : ℝ≥0∞ :=
initialization (h + 1) (widths d q w) {θ | ∃ i j : Fin N, ∃ k k' : Fin q,
ε < |finiteNTK (h + 1) (widths d q w) σ β θ
(inputForWidths d q w (X i)) (inputForWidths d q w (X j))
(outputForWidths d q w k) (outputForWidths d q w k') -
(if k = k' then limitingNTK d σ β h (X i) (X j) else 0)|}
end JGH
Read-back
What the Lean code literally says, in plain math · gpt-6
ParameterIndex. For every natural number and width function , this is the finite set
.
The distinguished index represents a bias parameter; the other choices represent incoming weight parameters. Its cardinality is . No positivity condition is imposed. When , it is empty. A layer with target width zero contributes no parameters; a layer with source width zero still contributes one bias parameter per target coordinate.
Parameters. For every and width function , this is the space of all real assignments to the indices with , , and either a bias symbol or an incoming index . There are no restrictions or identifications among the assigned values. When there are no parameter indices, including , this function space contains exactly one element: the empty assignment.
initialization. For every and width function , this is the finite product probability measure on the parameter space, where the indices comprise every weight and bias with , , and . All these coordinates are independent standard real Gaussians, with mean zero and variance one. No width normalization or bias coefficient appears in this measure. For an empty parameter index set, it is the probability measure concentrated on the unique empty parameter assignment.
Probability instance for initialization. For every natural depth and every natural-valued width function , the product measure assigning independent coordinates to all weights and biases for the layers has total mass one. This also applies when the parameter index set is empty, since the resulting function space is a singleton. No positive-depth or positive-width assumption is required.
preactivation. Given , widths , an arbitrary function , a real number , parameters and indexed by , , and , and an input , this defines vectors for every natural . It sets ; for , it sets
,
where and for ; for , it instead sets . Widths in square roots are interpreted as real numbers. Empty sums are zero and real division by zero is zero, so an active layer with source width zero has value . A zero target width gives the empty vector. The input is not activated before the first affine map, and the recursive step does not activate its resulting vector. No regularity of or restriction on is assumed. All positive layer indices beyond are assigned zero vectors.
network. For natural , widths , arbitrary , real , parameters comprising all layer weights and biases, and input , this returns , where and
for , with and for . Empty sums and division by zero are zero; a zero source width therefore leaves only the bias term, and a zero output width gives an empty output vector. For , the output is exactly , independently of the unique empty parameter assignment. No activation is applied after the last affine map. No continuity, measurability, or differentiability of is assumed.
parameterDerivative. Given natural , widths , arbitrary , real , parameters , input , an output index , and a parameter index , this is
.
Here contains all weights and biases with , , and ; equals one at and zero elsewhere; and is determined by and the affine recursion
,
using and for . The derivative is the real Fréchet derivative of the entire scalar output function on the parameter space. If that scalar function is not Fréchet differentiable at , its derivative is defined to be the zero linear map, so the declared value is zero. Merely having a derivative along coordinate is not the condition tested. Empty sums and division by zero are zero. If no parameter index or output index exists, there are no corresponding admissible arguments or .
finiteNTK. For every natural , width function , arbitrary , real , parameters , inputs , and output indices , this is
,
with no further normalization. The index set contains every layer weight and bias, and is the coordinate unit vector. For either input , is given by and
,
where and for . Each full scalar-output Fréchet derivative is set to zero if it does not exist. Consequently the displayed sum is zero if either scalar output function is not differentiable at . Empty sums and division by zero are zero. In particular, this kernel is zero at , although the network then returns its input. If , there are no output indices .
Input. For each natural number , this is the space , represented by real-valued functions on the indices , with no restrictions on their values. For , it is a singleton containing the empty function, rather than an empty set of inputs.
Kernel. For each natural number , this is the collection of all functions . No symmetry, positivity, positive semidefiniteness, continuity, or measurability is required. For , there is one possible input pair, so a kernel is determined by one real value.
gaussianPair. Given any natural , any function , and inputs , form
.
This defines a measure on the Euclidean plane as the law of , where is a standard two-dimensional Gaussian and the mean is zero. If is positive semidefinite, its positive semidefinite square root is used, and the resulting Gaussian has covariance , including singular cases. If is not positive semidefinite, the matrix-square-root operation returns zero and the measure is the point mass at . No positive-semidefiniteness or symmetry assumption on is present. The definition also applies when , with the unique empty input.
gaussianProductExpectation. For any natural , arbitrary functions and , and inputs , this is the real Bochner integral
.
The measure is centered Gaussian with covariance
if that matrix is positive semidefinite, and is the point mass at zero otherwise. No measurability or integrability assumption on is included. If the integrand is not Bochner integrable, including failure of the requisite almost-everywhere measurability, the integral is defined to be zero. In the point-mass fallback case it equals . Zero input dimension and singular positive-semidefinite covariance matrices are permitted.
covarianceKernel. For each natural , arbitrary , and real , this defines real functions on , for all , by
and
.
The measure is centered Gaussian with covariance
when that matrix is positive semidefinite, and the point mass at zero otherwise. Every integral is a real Bochner integral set to zero when its integrand is not integrable. Thus a nonintegrable product contributes zero, while a non-positive-semidefinite matrix contributes through the point-mass convention. No regularity assumptions on occur. For , the initial empty sum and its quotient by zero are zero, so on the unique input pair. The definition contains no separate assertion that these functions are positive semidefinite or are actual covariances of random network outputs.
limitingNTK. For each natural , arbitrary , and real , this defines functions for every . Set
and recursively
,
where is centered Gaussian with matrix entries , or the point mass at zero when this matrix is not positive semidefinite. Then
and
.
Here is the ordinary real derivative when it exists and zero otherwise. All integrals are real Bochner integrals and are zero for nonintegrable integrands. No regularity, integrability, or positive-dimension assumptions are imposed. At , the normalized empty sum is zero, giving . This definition contains no convergence assertion.
widths. Given natural numbers , a vector , and a natural layer index , this defines
,
for , and
for .
The subtraction in is truncated natural-number subtraction, but the first branch already handles . All hidden widths are therefore positive, even if some . The input width and output width may be zero. When , is the unique empty vector, , and every positive layer index has width .
sequentialWidths. For every natural , this defines a filter on . For , it is the point filter at the unique empty vector, so an eventual predicate must hold at that vector. Recursively, a predicate is eventual for precisely when there exists such that for every last coordinate , the predicate is eventual for . For positive , the complete quantifier order is
.
Each may depend on the already chosen later coordinates . In particular, for the order is exactly
;
for , it is the ordinary eventual condition at infinity on . The last coordinate has the outermost eventual quantifier, and coordinate zero has the innermost one. No single bound uniform in all coordinates is required.
inputForWidths. Given natural , hidden-width parameters , and , define widths , for , and thereafter. This returns the element of with coordinate equal to . It only changes the indexing type using ; no coordinate value changes. At , it sends the unique empty input vector to the unique empty vector.
outputForWidths. Given natural , hidden-width parameters , and , define , for , and for . This returns the same integer , now indexed below . It neither permutes nor numerically changes the output index. If , no admissible argument exists.
initializedOutput. For all natural , arbitrary , real , inputs for , hidden-width parameters , and parameters for affine layers with widths , for , and , this returns a vector in the Euclidean space indexed by pairs , , . Its coordinate is , where and , using and for . Each weight and bias is a separate real parameter. Conversion to Euclidean-space representation preserves the coordinates. The function does not itself sample or assert a distribution, measurability, or differentiability. For , it evaluates one affine layer with no activation; for , the first sum and its quotient by zero vanish. If or , the result is the unique empty-coordinate vector. Inputs may repeat and is unrestricted.
outputGaussian. For all natural , arbitrary , real , and inputs for , this is a centered Gaussian measure on the Euclidean coordinates , , , constructed from the matrix when , and zero otherwise. The scalar functions satisfy and , where is centered Gaussian with covariance if positive semidefinite, and the point mass at zero otherwise; nonintegrable products have integral zero. The output measure is the law of , for standard Gaussian , when is positive semidefinite. If is not positive semidefinite, it is instead the point mass at zero. Thus identifying as its covariance requires positive semidefiniteness. No regularity of , distinctness of inputs, or positive dimensions are assumed. For , the initial normalized sum is zero. For or , the measure is concentrated on the unique empty-coordinate vector.
Probability instance for outputGaussian. For all natural , arbitrary , real , and inputs , the measure on coordinates , , , constructed with zero mean and matrix , has total mass one. Here , and is plus the Bochner integral of under the centered pair Gaussian with entries . Non-positive-semidefinite pair matrices give point masses at zero; nonintegrable products give zero integrals. The output measure likewise uses the square root of applied to a standard Gaussian when positive semidefinite, and a point mass otherwise. Consequently this probability assertion requires no separate positive-semidefiniteness or regularity hypothesis. Empty coordinate sets still give total mass one on a singleton; zero input dimension uses the division-by-zero value zero.
ntkBadProbability. For all natural , arbitrary , real , inputs for , and , define widths , for , and . Give all weights and biases , for , , and , independent distributions, and call their index set . For parameters , define by and , where and for . Put , where is the coordinate unit vector and a nonexistent full scalar-output Fréchet derivative is zero. Define the deterministic comparison by , , , and . Here is the centered pair Gaussian with matrix , replaced by a point mass at zero if this matrix is not positive semidefinite; is the derivative set to zero where nonexistent; and nonintegrable Bochner integrals are zero. The declared value is the extended nonnegative real measure, under the independent Gaussian parameter law, of . The inequality is strict, all input and output pairs are included, and no indices must be distinct. No measurable-event assumption, regularity assumption on , or positivity assumption on is supplied; for arbitrary sets this uses the measure’s canonical outer-measure extension. The value lies between zero and one. If or , the existential event is empty and the value is zero. If and , the event is the whole parameter space and the value is one. Empty sums and division by zero are zero, and permits the unique empty width vector. No limiting assertion is included.
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.