A look at Generalization and Overparameterized Networks

\(\newcommand{\round}[1]{\left( #1 \right)} \newcommand{\curly}[1]{\left\lbrace #1 \right\rbrace} \newcommand{\squarebrack}[1]{\left\lbrack #1 \right\rbrack} \newcommand{\sumi}[2]{\sum\limits_{i=#1}^{#2}} \newcommand{\sumj}[2]{\sum\limits_{j=#1}^{#2}} \newcommand{\sumk}[2]{\sum\limits_{k=#1}^{#2}} \newcommand{\sump}[2]{\sum\limits_{p=#1}^{#2}} \newcommand{\suml}[2]{\sum\limits_{l=#1}^{#2}} \newcommand{\sumn}[2]{\sum\limits_{n=#1}^{#2}} \newcommand{\summ}[2]{\sum\limits_{m=#1}^{#2}} \newcommand{\sumt}[2]{\sum\limits_{t=#1}^{#2}} \newcommand{\Sum}{\sum_{i = 1}^{n}} \newcommand{\Sumi}[1]{\sum\limits_{i = 1}^{#1}} \newcommand{\Sumt}[1]{\sum\limits_{t = 1}^{#1}} \newcommand{\abs}[1]{\left\lvert #1 \right\rvert} \newcommand{\norm}[2]{\left\lVert#2\right\rVert_{#1}} \newcommand{\esqnorm}[1]{\left\lVert#1\right\rVert_2^2} \newcommand{\enorm}[1]{\left\lVert#1\right\rVert_2} \newcommand{\infnorm}[1]{\left\lVert#1\right\rVert_\infty} \newcommand{\opnorm}[1]{\left\lVert#1\right\rVert_\text{op}} \newcommand{\normF}[1]{\left\lVert#1\right\rVert_{\text{F}}} \newcommand{\inner}[1]{\left\langle#1\right\rangle} \newcommand{\ceil}[1]{\left\lceil#1\right\rceil} \newcommand{\floor}[1]{\left\lfloor#1\right\rfloor} \newcommand{\zero}{\mathbf{0}} \newcommand{\one}{\mathbf{1}} \newcommand{\avec}{\mathbf{a}} \newcommand{\bvec}{\mathbf{b}} \newcommand{\cvec}{\mathbf{c}} \newcommand{\dvec}{\mathbf{d}} \newcommand{\e}{\mathbf{e}} \newcommand{\f}{\mathbf{f}} \newcommand{\g}{\mathbf{g}} \newcommand{\h}{\mathbf{h}} \newcommand{\ivec}{\mathbf{i}} \newcommand{\jvec}{\mathbf{j}} \newcommand{\kvec}{\mathbf{k}} \newcommand{\lvec}{\mathbf{l}} \newcommand{\m}{\mathbf{m}} \newcommand{\n}{\mathbf{n}} \newcommand{\ovec}{\mathbf{o}} \newcommand{\p}{\mathbf{p}} \newcommand{\q}{\mathbf{q}} \newcommand{\rvec}{\mathbf{r}} \newcommand{\s}{\mathbf{s}} \newcommand{\tvec}{\mathbf{t}} \newcommand{\uvec}{\mathbf{u}} \newcommand{\vvec}{\mathbf{v}} \newcommand{\w}{\mathbf{w}} \newcommand{\x}{\mathbf{x}} \newcommand{\y}{\mathbf{y}} \newcommand{\z}{\mathbf{z}} \newcommand{\A}{\mathbf{A}} \newcommand{\B}{\mathbf{B}} \newcommand{\C}{\mathbf{C}} \newcommand{\D}{\mathbf{D}} \newcommand{\Emat}{\mathbf{E}} \newcommand{\F}{\mathbf{F}} \newcommand{\G}{\mathbf{G}} \newcommand{\Hmat}{\mathbf{H}} \newcommand{\I}{\mathbf{I}} \newcommand{\J}{\mathbf{J}} \newcommand{\K}{\mathbf{K}} \newcommand{\Lmat}{\mathbf{L}} \newcommand{\M}{\mathbf{M}} \newcommand{\N}{\mathbf{N}} \newcommand{\Omat}{\mathbf{O}} \newcommand{\Pmat}{\mathbf{P}} \newcommand{\Q}{\mathbf{Q}} \newcommand{\Rmat}{\mathbf{R}} \newcommand{\Smat}{\mathbf{S}} \newcommand{\T}{\mathbf{T}} \newcommand{\U}{\mathbf{U}} \newcommand{\V}{\mathbf{V}} \newcommand{\W}{\mathbf{W}} \newcommand{\X}{\mathbf{X}} \newcommand{\Y}{\mathbf{Y}} \newcommand{\Z}{\mathbf{Z}} \newcommand{\SIGMA}{\mathbf{\Sigma}} \newcommand{\LAMBDA}{\mathbf{\Lambda}} \newcommand{\Acal}{\mathcal{A}} \newcommand{\Bcal}{\mathcal{B}} \newcommand{\Ccal}{\mathcal{C}} \newcommand{\Dcal}{\mathcal{D}} \newcommand{\Ecal}{\mathcal{E}} \newcommand{\Fcal}{\mathcal{F}} \newcommand{\Gcal}{\mathcal{G}} \newcommand{\Hcal}{\mathcal{H}} \newcommand{\Ical}{\mathcal{I}} \newcommand{\Jcal}{\mathcal{J}} \newcommand{\Kcal}{\mathcal{K}} \newcommand{\Lcal}{\mathcal{L}} \newcommand{\Mcal}{\mathcal{M}} \newcommand{\Ncal}{\mathcal{N}} \newcommand{\Ocal}{\mathcal{O}} \newcommand{\Pcal}{\mathcal{P}} \newcommand{\Qcal}{\mathcal{Q}} \newcommand{\Rcal}{\mathcal{R}} \newcommand{\Scal}{\mathcal{S}} \newcommand{\Tcal}{\mathcal{T}} \newcommand{\Ucal}{\mathcal{U}} \newcommand{\Vcal}{\mathcal{V}} \newcommand{\Wcal}{\mathcal{W}} \newcommand{\Xcal}{\mathcal{X}} \newcommand{\Ycal}{\mathcal{Y}} \newcommand{\Zcal}{\mathcal{Z}} \newcommand{\alphavec}{\boldsymbol{\alpha}} \newcommand{\betavec}{\boldsymbol{\beta}} \newcommand{\gammavec}{\boldsymbol{\gamma}} \newcommand{\deltavec}{\boldsymbol{\delta}} \newcommand{\epsvec}{\boldsymbol{\epsilon}} \newcommand{\etavec}{\boldsymbol{\eta}} \newcommand{\nuvec}{\boldsymbol{\nu}} \newcommand{\tauvec}{\boldsymbol{\tau}} \newcommand{\rhovec}{\boldsymbol{\rho}} \newcommand{\lmbda}{\boldsymbol{\lambda}} \newcommand{\muvec}{\boldsymbol{\mu}} \newcommand{\thetavec}{\boldsymbol{\theta}} \newcommand{\BigO}[1]{\mathcal{O}\round{#1}} \newcommand{\BigOmega}[1]{\Omega\round{#1}} \newcommand{\R}{\mathbb{R}} \newcommand{\Rd}[1]{\mathbb{R}^{#1}} \newcommand{\Natural}{\mathbb{N}} \newcommand{\Complex}{\mathbb{C}} \newcommand{\Integer}{\mathbb{Z}} \newcommand{\Rational}{\mathbb{Q}} \newcommand{\Field}{\mathbb{F}} \newcommand{\E}[1]{\mathbb{E}\squarebrack{#1}} \newcommand{\Exp}[2]{\mathbb{E}_{#1}\squarebrack{#2}} \newcommand{\Prob}[1]{P\curly{#1}} \newcommand{\Var}[1]{\mathrm{Var}\squarebrack{#1}} \newcommand{\inv}[1]{\frac{1}{#1}} \newcommand{\indicator}[2]{\mathbbm{1}_{#1}\squarebrack{#2}} \newcommand{\Tr}[1]{\text{Tr}\squarebrack{#1}} \newcommand{\BOX}[1]{\fbox{\parbox{\linewidth}{\centering#1}}} \newcommand{\textequal}[1]{\stackrel{#1}{=}} \newcommand{\textleq}[1]{\stackrel{#1}{\leq}} \newcommand{\textgeq}[1]{\stackrel{#1}{\geq}} \newcommand{\dd}[2]{\frac{d #1}{d #2}} \newcommand{\ddn}[3]{\frac{d^{#1} #2}{d #3^{#1}}} \newcommand{\dodo}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\dodon}[3]{\frac{\partial^{#1} #2}{\partial {#3}^{#1}}}\)

Blog post coming soon…