MLP: the first layer is exactly Gaussian (eq. 4.23)
ProvedDeepLearningTheory.mlp_first_layer_gaussianFor an MLP at initialization and a dataset of inputs , , the -dimensional random vector of first-layer preactivations is a centered Gaussian with covariance
In particular, different neurons are independent in the first layer.
Formalization Note The law is stated as a pushforward measure on Euclidean space indexed by (neuron, sample) pairs and compared with Mathlib's multivariateGaussian, so that a singular metric (e.g. repeated inputs), where the density (4.23) does not exist, is covered.
import Mathlib import Definitions.Def_DLT_DeepLinearNetwork import Definitions.Def_DLT_MLPInit open MeasureTheory ProbabilityTheory open scoped NNReal
namespace DeepLearningTheory
theorem mlp_first_layer_gaussian {Ω : Type*} [MeasurableSpace Ω] (P : Measure Ω)
[IsProbabilityMeasure P] (n : ℕ → ℕ) (Cb CW : ℕ → ℝ≥0) (σ : ℝ → ℝ)
(b : Ω → ℕ → ℕ → ℝ) (W : Ω → ℕ → ℕ → ℕ → ℝ) (hinit : IsMLPInit P n Cb CW b W)
{D : ℕ} (x : Fin D → ℕ → ℝ) :
P.map (fun ω => WithLp.toLp 2
(fun p : Fin (n 1) × Fin D => mlpPreact n σ (b ω) (W ω) (x p.2) 1 (p.1 : ℕ)))
= multivariateGaussian 0
(fun p q : Fin (n 1) × Fin D =>
kron p.1 q.1 * firstLayerMetric (Cb 1) (CW 1) (n 0) x p.2 q.2) := by sorry
end DeepLearningTheoryRead-back
What the Lean code literally says, in plain math · Aristotle (Harmonic) - non-blind, same agent as the drafter
Non-blind read-back. This read-back was written by the same agent that drafted the Lean statement, with full knowledge of the source text and of the intended meaning. It is not independent testimony and must not be mistaken for a blind audit; reviewers should compare the Lean code against the source themselves.
Let be a probability space, widths, , any function, and random bias/weight fields satisfying IsMLPInit (all biases and weights with , , mutually independent; , ). Write for the MLP preactivation at and for the first-layer metric of the listed inputs. Then for every and inputs , the pushforward of under , viewed in Euclidean space, equals Mathlib's multivariateGaussian with mean and covariance matrix .
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.