ReLU Networks, Reciprocal Layer Scaling, and the Euclidean Hessian
DefinitionDinhSharpness_ModelNotation and network conventions
Let be the input dimension and hidden width. The parameter consists of and , with the Euclidean norm on all entries. The scalar-output network is
There are no biases and no output activation. For any real-valued functional on prediction functions, . In particular, losses with additional parameter-dependent penalties are not included unless they also admit this representation. The positive rescaling is
Observational equivalence means equality of predictions on every input.
The local regularity condition means that is Fréchet differentiable at every point of some neighborhood of , and the map is Fréchet differentiable at . Write , a continuous bilinear form. Its norm is
Under Euclidean/Riesz identification, this is the spectral operator norm of the Hessian matrix. A local minimum uses the usual Euclidean neighborhood; it need not be isolated or global. No probability model is assumed: the claim is deterministic and compares the same prediction function.
Formalization note: the network and scaling directly encode Section 3, Definition 3 (PDF p. 3), Theorem 1 and Definition 5 (PDF p. 4). The function-based continuous-loss convention is Section 2, PDF p. 2. For the Hessian targets, the local regularity condition makes the source's implicit second differentiability explicit without requiring global smoothness or continuity of second derivatives. The model defines actual Fréchet derivatives, not an arbitrary matrix constrained by desired conclusions. Relevant displayed formulas have no equation numbers.
Source: Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio, Sharp Minima Can Generalize For Deep Nets, ICML 2017, arXiv:1703.04933v2, https://arxiv.org/abs/1703.04933v2; Section 2, PDF p. 2; Section 3, PDF p. 4, Theorem 1 and Definition 5; Section 4.2, PDF p. 5, Theorems 3–4. Displays are unnumbered.
import Mathlib.Analysis.InnerProductSpace.PiL2
import Mathlib.Analysis.Calculus.FDeriv.Basic
import Mathlib.Topology.Order.LocalExtr
/-!
Parameter geometry for Dinh--Pascanu--Bengio--Bengio, arXiv:1703.04933v2.
Source: Section 2, PDF p. 2; Section 3, PDF pp. 3--4, Definitions 3--5;
Section 4.2, PDF p. 5, Theorems 3--4 (unnumbered displays).
-/
noncomputable section
open scoped Topology
namespace DinhSharpness
/-- Input vectors with their Euclidean norm. -/
abbrev Input (d : ℕ) := EuclideanSpace ℝ (Fin d)
/-- First-layer matrix entries followed by the scalar output layer's weights. -/
abbrev Coordinate (d h : ℕ) := (Fin d × Fin h) ⊕ Fin h
/-- The Euclidean norm here is essential to the spectral-norm interpretation. -/
abbrev Parameter (d h : ℕ) := EuclideanSpace ℝ (Coordinate d h)
/-- The paper's rectified activation. -/
def relu (t : ℝ) : ℝ := max t 0
/-- One hidden layer, no biases, and a scalar linear output. -/
def prediction {d h : ℕ} (θ : Parameter d h) (x : Input d) : ℝ :=
∑ j : Fin h, relu (∑ i : Fin d, x i * θ (Sum.inl (i, j))) * θ (Sum.inr j)
/-- The parameter loss is computed from the complete prediction function. -/
def parameterLoss {d h : ℕ} (ℓ : (Input d → ℝ) → ℝ) (θ : Parameter d h) : ℝ :=
ℓ (prediction θ)
/-- Positive reciprocal layer rescaling; theorems explicitly require `0 < α`. -/
def scale {d h : ℕ} (α : ℝ) (θ : Parameter d h) : Parameter d h :=
WithLp.toLp 2 (fun k ↦ match k with
| Sum.inl ij => α * θ (Sum.inl ij)
| Sum.inr j => α⁻¹ * θ (Sum.inr j))
/-- Explicit local regularity: an actual derivative nearby and its derivative at the point. -/
def TwiceDifferentiableAt {d h : ℕ} (L : Parameter d h → ℝ)
(θ : Parameter d h) : Prop :=
(∀ᶠ z in 𝓝 θ, DifferentiableAt ℝ L z) ∧ DifferentiableAt ℝ (fderiv ℝ L) θ
/-- The actual second Fréchet derivative, as a continuous bilinear form. -/
def hessian {d h : ℕ} (L : Parameter d h → ℝ) (θ : Parameter d h) :
Parameter d h →L[ℝ] Parameter d h →L[ℝ] ℝ :=
fderiv ℝ (fderiv ℝ L) θ
end DinhSharpness
Read-back
What the Lean code literally says, in plain math · gpt-6
DinhSharpness.Input. For every natural number , including , the input space is the real Euclidean space of functions from to , with norm . When , the index set is empty and this space consists of the single zero vector.
DinhSharpness.Coordinate. For every pair of natural numbers , the parameter-coordinate index set is the disjoint union . Thus it has one tagged coordinate for each pair with and , and a separate tagged coordinate for each , for a total of coordinates. When , both parts are empty; when , the pair-indexed part is empty and the second part still has coordinates.
DinhSharpness.Parameter. For all natural numbers , the parameter space is the real Euclidean space of functions on the disjoint union . Writing a parameter as the coordinate collection , where is its coordinate in the first part and its coordinate in the second part, its norm is . No positivity condition is imposed on or . For the space consists of the single zero vector; for it consists only of the second-part coordinates.
DinhSharpness.relu. For every real number , the function is defined by ; hence its value is when , and when , with value at .
DinhSharpness.prediction. For arbitrary natural numbers , a real Euclidean parameter vector with coordinates for and separate coordinates for , and an input with its Euclidean norm, the real prediction is defined to be . All parameter and input coordinates may be arbitrary real numbers. Empty sums are zero: for , the prediction is zero for every input, and for , every inner sum and every maximum in this expression is zero, so the prediction is zero at the sole input, regardless of and of the parameter.
DinhSharpness.parameterLoss. For arbitrary natural numbers , an arbitrary functional on all real-valued functions on the Euclidean input space, and a real Euclidean parameter vector with coordinates for and for , the parameter loss is . There are no regularity or sign assumptions on . If or , the function passed to is the identically zero function on the corresponding input space, and the resulting loss is independent of .
DinhSharpness.scale. For arbitrary natural numbers , any real number , and any real Euclidean parameter vector with coordinates for and separate coordinates for , the scaled parameter is the vector in the same Euclidean space whose first-part coordinates are and whose second-part coordinates are . The real reciprocal here is total, with ; consequently sends every parameter to the zero parameter. Negative values of are included. For there are only second-part coordinates to scale, and for the result is the sole vector of the zero-dimensional parameter space.
DinhSharpness.TwiceDifferentiableAt. For arbitrary natural numbers , let be the real Euclidean space with coordinates indexed by the disjoint union . For any function and any , the stated predicate means the conjunction of the following two conditions: there is a neighborhood of on which is real Fréchet differentiable at every point; and the map , taking values in the normed space of continuous real-linear maps from to , is real Fréchet differentiable at . Here denotes the real Fréchet derivative when it exists and is defined to be the zero continuous linear map otherwise. The neighborhood condition includes itself. Zero dimensions are allowed: when , has coordinates; when , is the singleton zero-dimensional space and both differentiability conditions hold for every .
DinhSharpness.hessian. For arbitrary natural numbers , let be the real Euclidean space with coordinates indexed by . For every function and every , its defined Hessian is , an element of the space of continuous real-linear maps from into the space of continuous real-linear maps from to . Thus is a real number for directions , with the first direction belonging to the outer derivative. Both derivatives are totalized: is zero wherever is not differentiable at , and is zero if the resulting derivative-valued map is not differentiable at . No differentiability predicate is required as an argument. The definition includes , for which has coordinates, and , for which is a singleton and the Hessian is the zero map.
Confirmed by the mission captain (proposal self-audit).
Confirmed by the moderator at approval.