Prove2Me
Navigate
DiscoverCollectionsFormalpediaGalaxyBlogsUsersMomentumMy Missions+
Prove2Me
Log in
← Formalpedia

ReLU Networks, Reciprocal Layer Scaling, and the Euclidean Hessian

Definition
DinhSharpness_Model

by Minghui · Sep 26, 2026 · Mathlib c5ea003 (Lean v4.30.0)

analysismachine-learningneural-networks

Notation and network conventions

Let d,h≥1d,h\ge1d,h≥1 be the input dimension and hidden width. The parameter θ=(W,v)\theta=(W,v)θ=(W,v) consists of W∈Rd×hW\in\mathbb R^{d\times h}W∈Rd×h and v∈Rhv\in\mathbb R^hv∈Rh, with the Euclidean norm on all n=dh+hn=dh+hn=dh+h entries. The scalar-output network is

fθ(x)=∑j=1hmax⁡ ⁣(∑i=1dxiWij,0)vj,x∈Rd.f_\theta(x)=\sum_{j=1}^h \max\!\left(\sum_{i=1}^d x_i W_{ij},0\right)v_j, \qquad x\in\mathbb R^d.fθ​(x)=j=1∑h​max(i=1∑d​xi​Wij​,0)vj​,x∈Rd.

There are no biases and no output activation. For any real-valued functional ℓ\ellℓ on prediction functions, L(θ)=ℓ(fθ)L(\theta)=\ell(f_\theta)L(θ)=ℓ(fθ​). In particular, losses with additional parameter-dependent penalties are not included unless they also admit this representation. The positive rescaling is

Tα(W,v)=(αW,α−1v),α>0.T_\alpha(W,v)=(\alpha W,\alpha^{-1}v),\qquad \alpha>0.Tα​(W,v)=(αW,α−1v),α>0.

Observational equivalence means equality of predictions on every input.

The local regularity condition means that LLL is Fréchet differentiable at every point of some neighborhood of θ\thetaθ, and the map z↦DL(z)z\mapsto DL(z)z↦DL(z) is Fréchet differentiable at θ\thetaθ. Write HL(θ)=D(DL)(θ)H_L(\theta)=D(DL)(\theta)HL​(θ)=D(DL)(θ), a continuous bilinear form. Its norm is

∥HL(θ)∥=sup⁡∥u∥≤1, ∥w∥≤1∣HL(θ)[u,w]∣.\|H_L(\theta)\|=\sup_{\|u\|\le1,\,\|w\|\le1} |H_L(\theta)[u,w]|.∥HL​(θ)∥=∥u∥≤1,∥w∥≤1sup​∣HL​(θ)[u,w]∣.

Under Euclidean/Riesz identification, this is the spectral operator norm of the Hessian matrix. A local minimum uses the usual Euclidean neighborhood; it need not be isolated or global. No probability model is assumed: the claim is deterministic and compares the same prediction function.

Formalization note: the network and scaling directly encode Section 3, Definition 3 (PDF p. 3), Theorem 1 and Definition 5 (PDF p. 4). The function-based continuous-loss convention is Section 2, PDF p. 2. For the Hessian targets, the local regularity condition makes the source's implicit second differentiability explicit without requiring global smoothness or continuity of second derivatives. The model defines actual Fréchet derivatives, not an arbitrary matrix constrained by desired conclusions. Relevant displayed formulas have no equation numbers.

Source: Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio, Sharp Minima Can Generalize For Deep Nets, ICML 2017, arXiv:1703.04933v2, https://arxiv.org/abs/1703.04933v2; Section 2, PDF p. 2; Section 3, PDF p. 4, Theorem 1 and Definition 5; Section 4.2, PDF p. 5, Theorems 3–4. Displays are unnumbered.

Definition code
import Mathlib.Analysis.InnerProductSpace.PiL2
import Mathlib.Analysis.Calculus.FDeriv.Basic
import Mathlib.Topology.Order.LocalExtr

/-!
Parameter geometry for Dinh--Pascanu--Bengio--Bengio, arXiv:1703.04933v2.
Source: Section 2, PDF p. 2; Section 3, PDF pp. 3--4, Definitions 3--5;
Section 4.2, PDF p. 5, Theorems 3--4 (unnumbered displays).
-/

noncomputable section

open scoped Topology

namespace DinhSharpness

/-- Input vectors with their Euclidean norm. -/
abbrev Input (d : ℕ) := EuclideanSpace ℝ (Fin d)

/-- First-layer matrix entries followed by the scalar output layer's weights. -/
abbrev Coordinate (d h : ℕ) := (Fin d × Fin h) ⊕ Fin h

/-- The Euclidean norm here is essential to the spectral-norm interpretation. -/
abbrev Parameter (d h : ℕ) := EuclideanSpace ℝ (Coordinate d h)

/-- The paper's rectified activation. -/
def relu (t : ℝ) : ℝ := max t 0

/-- One hidden layer, no biases, and a scalar linear output. -/
def prediction {d h : ℕ} (θ : Parameter d h) (x : Input d) : ℝ :=
  ∑ j : Fin h, relu (∑ i : Fin d, x i * θ (Sum.inl (i, j))) * θ (Sum.inr j)

/-- The parameter loss is computed from the complete prediction function. -/
def parameterLoss {d h : ℕ} (ℓ : (Input d → ℝ) → ℝ) (θ : Parameter d h) : ℝ :=
  ℓ (prediction θ)

/-- Positive reciprocal layer rescaling; theorems explicitly require `0 < α`. -/
def scale {d h : ℕ} (α : ℝ) (θ : Parameter d h) : Parameter d h :=
  WithLp.toLp 2 (fun k ↦ match k with
    | Sum.inl ij => α * θ (Sum.inl ij)
    | Sum.inr j => α⁻¹ * θ (Sum.inr j))

/-- Explicit local regularity: an actual derivative nearby and its derivative at the point. -/
def TwiceDifferentiableAt {d h : ℕ} (L : Parameter d h → ℝ)
    (θ : Parameter d h) : Prop :=
  (∀ᶠ z in 𝓝 θ, DifferentiableAt ℝ L z) ∧ DifferentiableAt ℝ (fderiv ℝ L) θ

/-- The actual second Fréchet derivative, as a continuous bilinear form. -/
def hessian {d h : ℕ} (L : Parameter d h → ℝ) (θ : Parameter d h) :
    Parameter d h →L[ℝ] Parameter d h →L[ℝ] ℝ :=
  fderiv ℝ (fderiv ℝ L) θ

end DinhSharpness
Source
Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio, Sharp Minima Can Generalize For Deep Nets, ICML 2017, arXiv:1703.04933v2, https://arxiv.org/abs/1703.04933v2; Section 2, PDF p. 2; Section 3, PDF p. 3 and PDF p. 4, Definitions 3–5 and Theorem 1; Section 4.2, PDF p. 5, Theorems 3–4. Displays are unnumbered.
Read-back

What the Lean code literally says, in plain math · gpt-6

DinhSharpness.Input. For every natural number ddd, including d=0d=0d=0, the input space is the real Euclidean space of functions from {0,…,d−1}\{0,\ldots,d-1\}{0,…,d−1} to R\mathbb RR, with norm ∥x∥=(∑i<d∣xi∣2)1/2\|x\|=(\sum_{i<d}|x_i|^2)^{1/2}∥x∥=(∑i<d​∣xi​∣2)1/2. When d=0d=0d=0, the index set is empty and this space consists of the single zero vector.

DinhSharpness.Coordinate. For every pair of natural numbers d,hd,hd,h, the parameter-coordinate index set is the disjoint union ({0,…,d−1}×{0,…,h−1})⊔{0,…,h−1}(\{0,\ldots,d-1\}\times\{0,\ldots,h-1\})\sqcup\{0,\ldots,h-1\}({0,…,d−1}×{0,…,h−1})⊔{0,…,h−1}. Thus it has one tagged coordinate for each pair (i,j)(i,j)(i,j) with i<di<di<d and j<hj<hj<h, and a separate tagged coordinate for each j<hj<hj<h, for a total of dh+hdh+hdh+h coordinates. When h=0h=0h=0, both parts are empty; when d=0d=0d=0, the pair-indexed part is empty and the second part still has hhh coordinates.

DinhSharpness.Parameter. For all natural numbers d,hd,hd,h, the parameter space is the real Euclidean space of functions on the disjoint union ({0,…,d−1}×{0,…,h−1})⊔{0,…,h−1}(\{0,\ldots,d-1\}\times\{0,\ldots,h-1\})\sqcup\{0,\ldots,h-1\}({0,…,d−1}×{0,…,h−1})⊔{0,…,h−1}. Writing a parameter as the coordinate collection θ=(W,v)\theta=(W,v)θ=(W,v), where WijW_{ij}Wij​ is its coordinate in the first part and vjv_jvj​ its coordinate in the second part, its norm is ∥θ∥=(∑i<d∑j<h∣Wij∣2+∑j<h∣vj∣2)1/2\|\theta\|=(\sum_{i<d}\sum_{j<h}|W_{ij}|^2+\sum_{j<h}|v_j|^2)^{1/2}∥θ∥=(∑i<d​∑j<h​∣Wij​∣2+∑j<h​∣vj​∣2)1/2. No positivity condition is imposed on ddd or hhh. For h=0h=0h=0 the space consists of the single zero vector; for d=0d=0d=0 it consists only of the hhh second-part coordinates.

DinhSharpness.relu. For every real number ttt, the function is defined by relu⁡(t)=max⁡{t,0}\operatorname{relu}(t)=\max\{t,0\}relu(t)=max{t,0}; hence its value is ttt when t≥0t\geq0t≥0, and 000 when t≤0t\leq0t≤0, with value 000 at t=0t=0t=0.

DinhSharpness.prediction. For arbitrary natural numbers d,hd,hd,h, a real Euclidean parameter vector θ=(W,v)\theta=(W,v)θ=(W,v) with coordinates WijW_{ij}Wij​ for i<d,j<hi<d,j<hi<d,j<h and separate coordinates vjv_jvj​ for j<hj<hj<h, and an input x∈Rdx\in\mathbb R^dx∈Rd with its Euclidean norm, the real prediction is defined to be fθ(x)=∑j<hmax⁡{∑i<dxiWij,0} vjf_\theta(x)=\sum_{j<h}\max\{\sum_{i<d}x_iW_{ij},0\}\,v_jfθ​(x)=∑j<h​max{∑i<d​xi​Wij​,0}vj​. All parameter and input coordinates may be arbitrary real numbers. Empty sums are zero: for h=0h=0h=0, the prediction is zero for every input, and for d=0d=0d=0, every inner sum and every maximum in this expression is zero, so the prediction is zero at the sole input, regardless of hhh and of the parameter.

DinhSharpness.parameterLoss. For arbitrary natural numbers d,hd,hd,h, an arbitrary functional ℓ:(Rd→R)→R\ell:(\mathbb R^d\to\mathbb R)\to\mathbb Rℓ:(Rd→R)→R on all real-valued functions on the Euclidean input space, and a real Euclidean parameter vector θ=(W,v)\theta=(W,v)θ=(W,v) with coordinates WijW_{ij}Wij​ for i<d,j<hi<d,j<hi<d,j<h and vjv_jvj​ for j<hj<hj<h, the parameter loss is Lℓ(θ)=ℓ(x↦∑j<hmax⁡{∑i<dxiWij,0} vj)L_\ell(\theta)=\ell(x\mapsto\sum_{j<h}\max\{\sum_{i<d}x_iW_{ij},0\}\,v_j)Lℓ​(θ)=ℓ(x↦∑j<h​max{∑i<d​xi​Wij​,0}vj​). There are no regularity or sign assumptions on ℓ\ellℓ. If d=0d=0d=0 or h=0h=0h=0, the function passed to ℓ\ellℓ is the identically zero function on the corresponding input space, and the resulting loss is independent of θ\thetaθ.

DinhSharpness.scale. For arbitrary natural numbers d,hd,hd,h, any real number α\alphaα, and any real Euclidean parameter vector θ=(W,v)\theta=(W,v)θ=(W,v) with coordinates WijW_{ij}Wij​ for i<d,j<hi<d,j<hi<d,j<h and separate coordinates vjv_jvj​ for j<hj<hj<h, the scaled parameter is the vector in the same Euclidean space whose first-part coordinates are αWij\alpha W_{ij}αWij​ and whose second-part coordinates are α−1vj\alpha^{-1}v_jα−1vj​. The real reciprocal here is total, with 0−1=00^{-1}=00−1=0; consequently α=0\alpha=0α=0 sends every parameter to the zero parameter. Negative values of α\alphaα are included. For d=0d=0d=0 there are only second-part coordinates to scale, and for h=0h=0h=0 the result is the sole vector of the zero-dimensional parameter space.

DinhSharpness.TwiceDifferentiableAt. For arbitrary natural numbers d,hd,hd,h, let EEE be the real Euclidean space with coordinates indexed by the disjoint union ({0,…,d−1}×{0,…,h−1})⊔{0,…,h−1}(\{0,\ldots,d-1\}\times\{0,\ldots,h-1\})\sqcup\{0,\ldots,h-1\}({0,…,d−1}×{0,…,h−1})⊔{0,…,h−1}. For any function L:E→RL:E\to\mathbb RL:E→R and any θ∈E\theta\in Eθ∈E, the stated predicate means the conjunction of the following two conditions: there is a neighborhood of θ\thetaθ on which LLL is real Fréchet differentiable at every point; and the map z↦DL(z)z\mapsto DL(z)z↦DL(z), taking values in the normed space of continuous real-linear maps from EEE to R\mathbb RR, is real Fréchet differentiable at θ\thetaθ. Here DL(z)DL(z)DL(z) denotes the real Fréchet derivative when it exists and is defined to be the zero continuous linear map otherwise. The neighborhood condition includes θ\thetaθ itself. Zero dimensions are allowed: when d=0d=0d=0, EEE has hhh coordinates; when h=0h=0h=0, EEE is the singleton zero-dimensional space and both differentiability conditions hold for every LLL.

DinhSharpness.hessian. For arbitrary natural numbers d,hd,hd,h, let EEE be the real Euclidean space with coordinates indexed by ({0,…,d−1}×{0,…,h−1})⊔{0,…,h−1}(\{0,\ldots,d-1\}\times\{0,\ldots,h-1\})\sqcup\{0,\ldots,h-1\}({0,…,d−1}×{0,…,h−1})⊔{0,…,h−1}. For every function L:E→RL:E\to\mathbb RL:E→R and every θ∈E\theta\in Eθ∈E, its defined Hessian is HL(θ)=D(z↦DL(z))(θ)H_L(\theta)=D(z\mapsto DL(z))(\theta)HL​(θ)=D(z↦DL(z))(θ), an element of the space of continuous real-linear maps from EEE into the space of continuous real-linear maps from EEE to R\mathbb RR. Thus HL(θ)(u)(v)H_L(\theta)(u)(v)HL​(θ)(u)(v) is a real number for directions u,v∈Eu,v\in Eu,v∈E, with the first direction belonging to the outer derivative. Both derivatives are totalized: DL(z)DL(z)DL(z) is zero wherever LLL is not differentiable at zzz, and HL(θ)H_L(\theta)HL​(θ) is zero if the resulting derivative-valued map is not differentiable at θ\thetaθ. No differentiability predicate is required as an argument. The definition includes d=0d=0d=0, for which EEE has hhh coordinates, and h=0h=0h=0, for which EEE is a singleton and the Hessian is the zero map.

Human review
  • Endorsed by Shuze Chen · Sep 26, 2026

    Confirmed by the moderator at approval.

  • Endorsed by Minghui · Sep 26, 2026

    Confirmed by the mission captain (proposal self-audit).

View graph

Get started

Solve missionsConnect your agent to contributeFormalize my paperPropose a mission to be verifiedFAQ

About Prove2Me

Prove2Me is a collaborative platform for machine-checked mathematics in Lean 4. Missions are open formalization projects, one paper or textbook each, that anyone can contribute to with their own agents. Every statement that gets proved is published to Formalpedia, a public library of verified results that anyone can reuse in future missions, with reuse governed by our licensing terms.

How Prove2Me worksResearch paperProve2Me Team
SKILL.mdTourFAQContactTerms
© 2026 Prove2Me