A Self-Generated Floor on Adam's $\varepsilon$: Evidence from Power-Law Matrix Sensing
Lngel Kyo
Abstract
In nonconvex Burer--Monteiro low-rank matrix sensing of $U^\ast \in \mathbb{R}^{V \times D}$ under power-law covariance $Σ= \mathrm{diag}(j^{-s})_{j=1}^V$, the precision-wall approximation predicts that Adam stalls when the smallest non-gauge Hessian eigenvalue drops below its denominator floor, giving $s^\ast(V,D) = [\log(1/\varepsilon_{\mathrm{eff}}) - \log V]/\log(VD) + δ_{\mathrm{Adam}}$. If $\varepsilon_{\mathrm{eff}}$ were set by $\varepsilon_0$, lowering $\varepsilon_0$ from $10^{-10}$ to $10^{-12}$ would raise $s^\ast$ by $0.95\text{--}1.33$. Instead, across six values of $\varepsilon_0$ two decades apart, $10^{-2}$ to $10^{-12}$, the boundary follows the $\varepsilon_0$ slope down to $10^{-6}$, bends near $10^{-8}$, and is flat below $10^{-8}$, while warm-started L-BFGS and Gauss--Newton reach machine precision on stalled instances. We trace this saturation to a self-generated dynamical floor $\varepsilon_{\mathrm{dyn}} \propto α$: high-curvature modes oscillate persistently near the adaptive edge of stability with amplitude $\propto α(1-β_1)/(1+β_1)$ and leak gradient fluctuations of scale $κα$ into rare modes via measurement coupling, pinning $\sqrt{\hat{v}_V} \approx κα\gg \varepsilon_0$. Fitting a single threshold-crossing wall parameter $\varepsilon_{\mathrm{dyn}} = 4.76\times 10^{-9}$ (with the detection lag frozen at its companion calibration) predicts the saturated failure boundary for $D \ge 4$ across five $(V,D)$ scales to RMS $0.098$ (breaking down at $D = 2$), while shrinking $\varepsilon_0$ below $10^{-8}$ is ineffective and lowering the step size $α$ reduces the floor directly.