\section{Introduction}

Understanding and modelling climate extremes such as floods, heatwaves and heavy precipitation, is of paramount importance because they often lead to severe impacts on our socio-economic system \citep{IPCC2012}. Many impacts are associated with compounding drivers, for instance multiple hazards occurring at the same time at the same or different locations affecting the same system \citep{leonard2014compound,zscheischler2018future}. Ignoring potential dependencies between multiple hazards can lead to severe miss-specification of the associated risk \citep{zscheischler2017dependence,Hillier2020}. However, estimating dependencies between extremes is a challenging task, requiring large amounts of data and/or suitable approaches to model  the phenomena of interest in a computationally feasible time.

A particularly challenging class of events are spatially compounding events \citep{Zscheischler2020}. Spatially compounding events occur when multiple connected locations are affected by the same or different hazards within a limited time window, thereby causing an impact. The compounding is established via a system capable of spatial integration, which accumulates hazard impacts in spatially distant locations. For instance, the global food systems are vulnerable to multiple co-occurring droughts and heatwaves in key crop-producing regions of the world \citep{Anderson2019,Mehrabi2019}. Similarly, spatially extensive floods \citep{Jongman2014} or sequences of cyclones \citep{Raymond2020} can deplete emergency response funds. Climate extremes can be correlated over very large distances due to teleconnections \citep{Boers2019} but modelling such dependencies is challenging.

One approach to tackle the challenge of spatially correlated extremes is to create a lot of data by running very long simulations with state-of-the-art climate models, which have the physical spatial dependencies built into them. For instance, over the recent years for a number of climate models large ensembles have been created \citep{deser2020insights}. However, these simulations typically have rather coarse resolution, are usually not stationary in time and are very expensive to run.

Extreme value theory provides mathematically justified models for the tail region of a multivariate distribution $(X_1,\dots, X_d)$, $d\geq 2$. This enables the extrapolation beyond the range of the data and the accurate estimation of the small probabilities of rare (multivariate) events. Statistical methods building on this theory are popular in a broad range of domains such as meteorology \citep{le2018dependence}, climate science \citep{naveau2020statistical} and finance \citep{poon2003extreme}. Applications have so far been limited to low-dimensional settings for several reasons.
On the one hand, even for moderately large dimensions $d$, the fitting and simulation of parametric models is computationally intensive because it requires computing complex likelihoods \citep{dom2016a}. On the other hand, the extremal dependence structures in applications are difficult to model and the existing approaches are either simplistic or over-parametrised. In spatial applications, for instance, Euclidean distance is used to parametrise the isotropic dependence of stationary max-stable random fields \citep{bla2011}. Most real world datasets that cover larger spatial domains do, however, feature non-stationarities in space that cannot be captured by the stationarity assumed in current geostatistical models; see \cite{eng2021} for a review on recent extreme value methods in higher dimensions.

Methods from machine learning are well-suited for complex and non stationary data sets. Their loss functions are, however, typically designed with the purpose to predict well in the bulk of the distribution. It is therefore a difficult problem to construct approaches with an accurate performance outside the range of the training data. In prediction problems, one possibility is to adapt the loss function to make the algorithm more sensitive to extreme values, as for instance done in quantile regression forests \citep{meinshausen2006quantile} or extreme gradient boosting \citep{velthoen2021gradient} for the prediction of high conditional quantiles. \citet{jalalzai2018binary} discuss this problem for classification in extreme regions and propose a new notion of asymptotic risk that helps to define classifiers with good generalization capacity beyond the observations range in the predictor space.

Rather than predicting an unknown response, in this work we are interested in a generative model that is able to learn a high-dimensional distribution $(X_1,\dots, X_d)$ both in the bulk and in the extremes. We concentrate on Generative Adversarial Networks (GANs), which are known to be competitive models for multivariate density estimation. While classical applications of GANs are often in the field of image analysis \citep{zhang2017stackgan,karras2017progressive,choi2018stargan}, they have been used much more broadly in domains such as finance \citep{efimov2020using}, fraud detection \citep{zheng2019one}, speech recognition \citep{sahu2019modeling} and medicine \citep{schlegl2017unsupervised}.

There are two different aspects that characterize the extremal properties of multivariate data: the univariate tail behaviour of each margin and the extremal dependence between the largest observations.
For GANs, it has recently been shown \citep{wie2019, hus2021} that the marginal distributions of the generated samples are either bounded or light-tailed if the input noise is uniform or Gaussian, respectively. \citet{hus2021} propose to use heavy-tailed input to overcome this issue.
\cite{bhatia2020exgan} use conditional GANs to perform importance sampling of extreme scenarios.

Here we concentrate on modeling the extremal properties of both the marginal distributions and the dependence structure in a realistic way even for high-dimensional and spatially non-stationary data. Our model combines the asymptotic theory of extremes with the flexibility of GANs to overcomes the limitations of classical statistical approaches to low dimensional and stationary data. We use a stationary 2000-year climate simulation to test and validate our approach.

\section{Methodology}
\label{sec:meth}
In the climate community there is a great need for methods able to efficiently provide an empirical description of the climate system, including extreme events, starting from as few ensemble runs as possible \citep{castruccio2019reproducing,deser2020insights}. Emulation techniques that tackle this challenge have been proposed in the last few years, usually focusing on the spatial correlation properties of the phenomena of interest \citep{mckinnon2017observational,mckinnon2018internal,link2019fldgen,Brunner2021}. In this work, our aim is to build an emulator specifically designed for extreme events that is able to reproduce the spatial tail dependencies of the data and extrapolate outside the range of the training samples. Our method does not require to discard any simulations and is therefore highly efficient. In this section we first recall some background on extreme value theory and generative adversarial networks, and then describe our algorithm called evtGAN. Similar to a copula approach \citep[e.g.][]{Brunner2021}, it relies on the idea to disentangle the modeling of the marginal distributions and the extremal dependence structure in order to use extreme value theory for former and a flexible machine learning technique called generative adversarial network for the latter.

\subsection{Extreme value theory}
\label{sec:evt}

Extreme value theory provides mathematically justified methods to analyze the tail of a $d$-dimensional random vector $\mathbf X = (X_1,\dots, X_d)$. This branch of statistical theory has been widely applied in climate science, for example to study droughts \citep{burke2010extreme}, floods \citep{asadi2018optimal}, heatwaves \citep{tanarhte2015heat} and extreme rainfalls \citep{le2018dependence}. We use extreme value theory in conjunction with generative adverserial networks to develop a flexible way of simulating rare climate events.

For multivariate data, there are two different aspects that determine the quality of an extrapolation method: the univariate tail behaviour of each margins and the extremal dependence between the largest observations. We explain how to measure and model those based on componentwise maxima.
Let $\mathbf X_i = (X_{i1}, \dots, X_{id})$, $i=1,\dots, n$, be a sample of $\mathbf X$ and let
$\mathbf M_n = (M_{n1}, \dots, M_{nd})$, where $M_{nj} = \max(X_{1j}, \dots, X_{nj})$ is the maximum over the $j$th margin.

\subsubsection{Univariate theory}

The Fisher--Tippett--Gnedenko theorem \citep[e.g.,][]{coles2001introduction} describes the limit behavior of the univariate maximum $M_{nj}$. It states that if there exist sequences $a_{nj}\in\mathbb R$, $b_{nj} > 0$ such that
\begin{align}\label{GEV_conv}
 \lim_{n\to\infty} \mathbb P\left(\frac{M_{nj}-a_{nj}}{b_{nj}} \le z\right) =   G_j(z), \qquad n\rightarrow \infty,
  \end{align}
then if $G_j$ is the distribution function of a non-degenerate random variable $Z_j$, then is in the class of generalized extreme value (GEV) distributions with
$$ G_j(z)=\exp \left[ -\left\{ 1+\xi_j\left(\frac{z-\mu_j}{\sigma_j}\right)\right\}_+^{-1/\xi_j} \right], \quad z\in \mathbb R, $$
where $x_+$ denotes the positive part of a real number $x$, and $\xi_j \in \mathbb R$, $\mu_j \in \mathbb R$ and $\sigma_j>0$ are the shape, location and scale parameters, respectively. The shape parameter is the most important parameter since it indicates whether the $j$th margin is heavy-tailed ($\xi_j > 0$), light-tailed ($\xi =0$) or whether it has a finite uper end-point ($\xi<0$).

It can be shown that under mild conditions on the distribution of margin $X_j$, appropriate sequences exist for \eqref{GEV_conv} to hold. The above theorem therefore suggests to fit a generalized extreme value distribution $\widehat G_j$ to maxima taken over blocks of the same lengths, which is common practice when modelling yearly maxima. The shape, location and scale parameters can then be estimated in different ways, including moment-based estimators \cite{hosking1985algorithm}, Bayesian methods \citep{yoon2010full} and maximum likelihood estimation \citep{hosking1985estimation}. We use the latter in this work. This allows extrapolation in the direction of the $j$th marginal since the size of a $T$-year event can be approximated by the $(1-1/T)$-quantile of the distribution $\widehat G_j$, even if $T$ is larger than the length of the data record.

\subsubsection{Multivariate theory}
For multivariate data, the correct extrapolation of the tail not only depends on correct models for the marginal extremes, but also whether the dependence between marginally large values is well captured.
For two components $X_i$ and $X_j$ with limits $Z_i$ and $Z_j$ of the corresponding maxima in \eqref{GEV_conv}, this dependence is summarized by the extremal correlation $\chi_{ij}\in[0,1]$ \citep{sch2003}
\begin{equation*}
  \chi_{ij} = \lim_{q\to1} \mathbb P(F_i(X_i) >  q \mid F_j(X_j)  > q) \in [0,1].
\end{equation*}
The larger $\chi_{ij}$, the stronger the dependence in the extremes between the two components, that is, between $Z_i$ and $Z_j$. If
$\chi_{ij}>0$ we speak of asymptotic dependence, and otherwise of asymptotic independence. While the extremal correlation is a useful summary statistic, it does not reflect the complete extremal dependence structure between $X_i$ and $X_j$. Under asymptotic dependence, the latter can be characterized by the so-called spectral distribution \citep{deh1977}.
Let $\tilde X_j = -1/\log F_j(X_j)$, $j=1,\dots, d$, be the data normalized to standard Fr\'echet margins. The spectral distribution describes the extremal angle of $(X_i, X_j)$, given that the radius exceeds a high threshold, that is,
$$H(x) = \lim_{u\to \infty} \mathbb P( \tilde X_i / (\tilde X_i + \tilde X_j) \leq x   \mid  \tilde X_i + \tilde X_j > u ), \quad x \in [0,1].$$
Under strong extremal dependence, the spectral distribution centers around $1/2$; under weak extremal independence, it has mass close to the boundary points $0$ and $1$.

In multivariate extreme value theory, a popular way to ensure a correct extrapolation of tail dependence under asymptotic dependence is by modeling the joint distribution $\mathbf Z = (Z_1,\dots, Z_d)$ of the limits in \eqref{GEV_conv} as a max-stable distribution with multivariate distribution function
\begin{equation}\label{mevd}
  \mathbb P(G_1(Z_1)\leq z_1,\dots,G_d(Z_d)\leq z_d) = \exp\{-\ell(z_1,\dots,z_d)\}, \quad z_1,\dots,z_d \in [0,1],
\end{equation}
where $\ell$ is the so-called stable tail dependence function \citep[e.g.,][]{de2007extreme}. This is a copula approach in the sense that the right-hand side is independent of the original marginal distributions $G_j$ and only describes the dependence structure. In practice, the $G_j$ are replaced by a estimates $\hat G_j$ that are fitted to the data first. The right-hand side of~\eqref{mevd} is also called an extreme value copula \citep[e.g.,][]{seg2010}.

A natural extension of max-stable distributions to the spatial setting is given by max-stable processes $\{Z(t) : t \in \mathbb R^d\}$ \citep[e.g.,][]{davison2012statistical}, where the maxima $Z_i = Z(t_i)$ are observed at spatial locations $t_i$, $i=1,\dots, d$. This type of models have been widely applied in many different areas such flood risk \citep{eng2014b}, heat waves \citep{eng2017a} and extreme precipitation \citep{buishand2008spatial}.
A popular parametric model for a max-stable process $Z$ is the Brown--Resnick process \citep{kab2009, eng2014}. This model class is parameterized by a variogram function $\gamma$ on $\mathbb R^d$. In this case, each bivariate distribution is in the so-called H\"usler--Reiss family \citep{hue1989}, and the corresponding dependence parameter is the variogram function evaluated at the distance between the two locations,
$$\lambda_{ij} = \gamma( \| t_i - t_j \|),\qquad i,j =1,\dots, d.$$
One can show that the extremal correlation coefficient for this model is given by $\chi_{ij} = 2 - 2\Phi(\sqrt{\lambda_{ij}} /2)$, where $\Phi$ is the standard normal distribution function. A popular parametric family is the class of fractal variograms that can be written as  $\gamma_{\alpha, s}(h) = h^\alpha/s$, $\alpha \in (0,2]$, $s>0$.
While these methods enjoy many interesting properties, they are often quite complex to fit \citep{dom2016a} and impose strong assumptions on the spatial phenomena of interest, such as spatial stationarity, isotropic behaviour and asymptotic dependence between all pairs of locations.

\subsection{Generative adversarial network}
Generative adversarial networks \citep{goodfellow2014generative} are attracting great interest thanks to their ability of learning complex multivariate distributions that are possibly supported on lower-dimensional manifolds. These methods are generative since they allow the generation of new realistic samples from the learnt distribution that can be very different from the training samples.

Given observations $\mathbf U_1, \dots, \mathbf U_n$ from some $d$-dimensional unknown target distribution $p_{data}$,
generative adversarial networks can be seen as a game opposing two agents, a discriminator $D$ and a generator $G$.
The generator $G$ takes as input a random vector $\mathbf Y= (Y_1,\dots, Y_m)$ from an $m$-dimensional latent space and transforms it to a new sample $\mathbf U^* = G(\mathbf Y)$. The components $Y_j$, $j=1,\dots, m$, are independent and typically have a standard uniform or Gaussian distribution.

The discriminator $D$ has to decide whether a new sample $\mathbf U^*$ generated by $G$ is fake and following the distribution $p_G$, or coming from the real observations with distribution $p_{data}$. The discriminator expresses its guesses with a value between 0 and 1 corresponding to its predicted probability of the sample combing from the real data distribution $p_{data}$. Both the generator and the discriminator become better during the game, in which $D$ is trained to maximize the probability of correctly labelling samples from both sources, and $G$ is trained to minimize $D$’s performance and thus learns to generate more realistic samples.

Mathematically, the optimization problem is a two-player minimax game with cross-entropy objective function \citep[see,][]{goodfellow2014generative}:
\begin{equation}
\min\limits_{G}\max\limits_{D} \mathbb{E}_{\mathbf U \sim p_{data}} [\log(D(\mathbf U))] + \mathbb{E}_{\mathbf Y\sim p_{\mathbf Y}} [\log(1-D(G(\mathbf Y))], \label{eq:goodfel}
\end{equation}
where $\mathbf Y$ is a random vector sampled from a latent space. In equilibrium, the optimal generator satisfies $p_G = p_{data}$ \citep{goodfellow2014generative}.

In practice, the discriminator and the generator are modeled through feed-forward neural networks. While the equilibrium with $p_G = p_{data}$ is only guaranteed thanks to the convex-concave property in the non-parametric formulation~\eqref{eq:goodfel}, the results remain very good as long as suitable adjustments to the losses, training algorithm and overall architecture are made to improve on the stability and convergence of training.

The architecture of the generator $G$ and the discriminator $D$ can be specifically designed for the structure of the data under consideration. In this direction, \citet{radford2015unsupervised} introduces the use of the convolutional neural network as an add-on feature to the GAN yielding a deep convolutional generative adversarial network model (DCGAN). It is considered the preferred standard for dealing with image data and more generally any high dimensional vector of observations describing a complex underlying dependence structure, showing superior performance in the representation and recognition fields.

\subsection{evtGAN}

Let $\mathbf Z_1, \dots, \mathbf Z_n$ be independent observations obtained as approximations of componentwise maxima as described in Section~\ref{sec:evt}.
The evtGAN algorithm (Algorithm~\ref{alg:evt}) below follows a copula approach where marginal distributions and dependence structure are treated separately. In particular, this allows us to impose different amounts of parametric assumptions on margins and dependence.

It is known that classical GANs that are trained with bounded or light tailed noise input distributions in the latent space will also generate bounded or light-tailed samples, respectively \citep[see][]{wie2019, hus2021}. That means that they are not well-suited for extrapolating in a realistic way outside the range of the data. For this reason, we suppose that the approximation of the margins by generalized extreme value distributions $G_j$ as in \eqref{GEV_conv} holds, which allows for extrapolation beyond the data range.
On the other hand, we do not make explicit use of the assumption of a multivariate max-stable distribution as in \eqref{mevd}. This has two reasons.  First, while the univariate limit \eqref{GEV_conv} holds under very weak assumptions, the multivariate max-stability as in \eqref{mevd} requires much stronger assumptions \citep[see][]{res2008} and may be too restrictive for cases with asymptotic independence \citep[e.g.,][]{wadsworth2012dependence}. Second, even if the data follow a multivariate max-stable distribution \eqref{mevd}, the probabilistic structure is difficult to impose on a flexible machine learning model such as a GAN.

\begin{algorithm}
\caption{evtGAN}
\textbf{Input:} Observations $\mathbf Z_i = (Z_{i1}, \dots, Z_{id})$, $i=1,\dots, n$.
\begin{algorithmic}[1]
    \STATE For $j=1,\dots, d$, fit a GEV distribution $\widehat G_j$ to the data $Z_{1j},\dots, Z_{nj}$ with estimated parameters $(\hat \mu_j, \hat \sigma_j, \hat \xi_j)$.
    \STATE Normalize all margins empirically to a standard uniform distribution to obtain pseudo-observations
    $$ \mathbf{U}_i = (\widehat F_1(Z_{i1}), \dots, \widehat F_d(Z_{id})), \quad i=1,\dots, n,$$
    where $\widehat F_j$ is the empircal distribution function of the $Z_{1j},\dots, Z_{nj}$.
    \STATE Train a DCGAN $G$ on the normalized data $\mathbf{U}_1,\dots, \mathbf{U}_n$ based on the loss in equation \eqref{eq:goodfel}.
    \STATE Generate $n^*$ new data points $\mathbf{U}_1^*,\dots, \mathbf{U}_n^*$ from $G$ with uniform margins, $i=1,\dots, n^*$.
    \STATE Normalize back to the scale of the original observations
    $$\mathbf{Z}_i^* = (\widehat G_1^{-1}(U_{i1}^*), \dots, \widehat G_d^{-1}(U_{id}^*)),\quad i=1,\dots, n^*.$$
\end{algorithmic}
\textbf{Output:} Set of new generated observations $\mathbf Z^*_i = (Z_{i1}, \dots, Z^*_{id})$, $i=1,\dots, n^*$.
\label{alg:evt}
\end{algorithm}

The margins are first fitted by a parametric GEV distribution (line 1 in Algorithm~\ref{alg:evt}). They are then normalized to (approximate) pseud-observations by applying the empirical distributions functions to each margin (line 2). This standardization to uniform margins stabilizes the fitting of the GAN. Alternatively, we could use the fitted GEV distributions for normalization but this seems to give slightly worse results in practice.
The pseudo-observations contain the extremal dependence structure of the original observations. Since this dependence structure is very complex in general and our focus is to reproduce gridded spatial fields, we do not rely on restrictive parametric assumptions but rather learn it by a highly flexible DCGAN (line 3). From the fitted model we can efficiently simulate any number of new pseudo-observations that have the correct extremal dependence (line 4). Finally, we transform the generated pseudo-observations back to the original scale to have realistic properties of the new samples also in terms of the margins (line 5).

\section{Application}
\label{sec:appl}

We asses the ability of the proposed method to correctly model the spatial tail dependence structure between climate extremes. Because observational records are relatively short and contain temporal trends, here we rely on 2000 years of climate model output that is representative of present-day weather over western Europe. In particular, we apply our approach to summer temperature and winter precipitation maxima, yielding 2000 and 1999 maxima for temperature and precipitations, respectively. We then compare the performance of our algorithm to the Brown--Resnick model, a state-of-the-art statistical model for spatial extremes.
In order to retrieve a non-biased estimate of the error that each of the method makes, we divide the data into a training set of 50 observations, and the rest is taken as a test set. Both methods are fitted using only the training dataset and the evaluation is made considering the information contained in the test set as ground truth. When not stated otherwise, the results from the Brown--Resnick model are analytical, while the results from evtGAN are obtained simulating $10'000$ data points

\subsection{Data}
The model experiment, which uses large ensemble simulations with the EC-Earth global climate model \citep[v2.3,][]{hazeleger2012} and has been used in climate impact studies \citep{van2019meteorological,kempen2021,Tschumi2021,Vogel2021,vanderwiel2021}, was originally designed to investigate the influence of natural variability and climate extremes. A detailed description of these climate simulations is provided in \citet{vanderWiel2019}, here we only provide a short overview.
The large ensemble contains 2000~years of daily weather data, representative of the present-day climate. Present-day was defined by the observed value of global mean surface temperature (GMST) over the period 2011-2015 \citep[HadCRUT4 data,][]{morice2012}; the 5-year time slice in long transient model simulations (forced by historical and Representative Concentration Pathway (RCP) 8.5 scenarios) with the same GMST was simulated repeatedly. To create the large ensemble, at the start of the time slice, twenty-five ensemble members were branched off from the sixteen transient runs. Each ensemble member was integrated for the five year time slice period. Differences between ensemble members were forced by choosing different seeds in the atmospheric stochastic perturbations \citep{buizza1999}. This resulted in a total of $16 \times 25 \times 5 = 2000$~years of meteorological data, at T159 horizontal resolution (approximately 1$^\circ$), among which we selected temperature (Kelvin) and precipitations (meter/day) for this paper.

We choose an area such that it is big enough to be relevant for climate application while being small enough to ensure a fair comparison of our approach and the classical approach in statistics to model spatial tail dependence, the Brown--Resnick process. Our analysis thus focuses on a total of $18\times 22$ grid points covering most of western Europe.
For that area we compute for each grid point the summer temperature maxima and winter precipitation maxima.

\subsection{Architecture and hyper-parameters}

In our application, we consider the non-heuristic saturating loss \citep{goodfellow2014generative} for the generator, and the standard empirical cross-entropy loss for the discriminator.
Training is done iteratively, where the discriminator is allowed to train two times longer than the generator.
We incorporate an exponential moving average scheme \citep{gidel2018variational} to the training algorithm as it has demonstrated more stability, a far better convergence and improved results.

Since we focus on reproducing gridded spatial fields, we make use of the DCGAN model \citep{radford2015unsupervised} as an architectural design to take advantage of convolution \citep{fukushima1980neocognitron}.
We use classical regularisation tools such as drop-out \citep{srivastava2014dropout}, batch-normalisation \citep{ioffe2015batch} and the Adam optimizer \citep{kingma2014adam} with a learning rate of $2\times10^{-4}$ and batch size of $100$ to train the neural network, and a total of $30'000$ training epochs.
The value of the exponential moving average parameter in the training algorithm is set to $\alpha=0.9$. The values of the hyper-parameters are the results of an extensive heuristic tuning.

\subsection{Results}
A particularity of evtGAN is that it decomposes the full high-dimensional distribution of the data into its marginal distributions and their dependence structure and processes them separately. We thus first report estimated key parameters of the marginal distributions.
For temperature, the location parameter $\mu$ is typically higher over land than over oceans (Fig.~\ref{fig:grid}a). Furthermore, there is a trend towards lower values of $\mu$ for more northern regions, illustrating the well-known latitudinal temperature gradient from south to north. The scale parameter $\sigma$, representing the width of the distribution, is also higher over land than over the ocean, with a fairly homogeneous distribution over land (Fig.~\ref{fig:grid}b). The shape parameter $\xi$ is well below 0 for most regions except some areas in the Atlantic north of Spain and in the Mediterranean at the north African coast, indicating a bounded tail (Fig.~\ref{fig:grid}c).

\begin{figure}
    \centering
        \includegraphics[width=.81\textwidth]{New_figures/grid.pdf}
        \caption{The generalized extreme value distribution parameters estimated for each grid point for temperature (a-c) and precipitation (d-f) extremes. On the left (a, d) the mean parameter $\mu$, in the middle (b, e) the scale parameter $\sigma$ and on the right (c, f) the shape parameter $\xi$.}
    \label{fig:grid}
\end{figure}

For precipitation, the location parameter $\mu$ is more similar between land and ocean compared to temperature but is a little bit more heterogeneous in space (Fig.~\ref{fig:grid}d), illustrating orographic effects on precipitation extremes. For the scale parameter $\sigma$ there is also no clear land-sea contrast but areas with relatively much higher values (corresponding to higher interannual variability in extremes) occur over the Western part of the Iberian peninsula and along a coastal band at the northern Mediterranean coast (Fig.~\ref{fig:grid}e). The shape parameter $\xi$ is spatially quite heterogeneous, with values ranging from -0.15 to 0.25 (Fig.~\ref{fig:grid}f). Overall, the parameter of the extreme value distributions of precipitation extremes show much higher spatial heterogeneity than the one for temperature extremes, suggesting that it might be more difficult to learn a model that represents well all tail dependencies for the entire region.

Next we look at examples of bivariate scatterplots of simulated temperature extremes from the different approaches for three cases with varying tail dependence (Fig.~\ref{fig:bivtemp}). The three rows correspond to three pairs of grid points with weak, mild and strong tail dependence, respectively. The train sets (Fig.~\ref{fig:bivtemp}a, f, k) illustrate the distributions from which the models are tasked to learn while the test sets (Fig.~\ref{fig:bivtemp}b, g, l) are used for validation. As can be clearly seen, extremes are much less likely to co-occur between locations that are characterized by weak tail dependence (Fig.~\ref{fig:bivtemp}b) compared to locations that are characterized by strong tail dependence (Fig.~\ref{fig:bivtemp}l). Purely based on visual judgement, it seems that evtGAN is able to characterize the different tail dependencies relatively well and can simulate samples outside of the range of the train set (Fig.~\ref{fig:bivtemp}c, h, m) whereas DCGAN only simulates distributions bounded to the range of the train set (Fig.~\ref{fig:bivtemp}d, i, n) and Brown--Resnick tends to overestimate tail dependence in cases where tail dependence is weak (Fig.~\ref{fig:bivtemp}e, j).

\begin{figure}[h]
    \includegraphics[width=\textwidth]{New_figures/biv_temp.pdf}
    \caption{Bivariate plots of temperature extremes based on train and test set and simulated through the different presented methods. Shown are selected pairs of locations with varying tail dependence. Columns from left to right: train, test, evtGAN, DCGAN and Brown--Resnick. From top to bottom: weak tail dependence (a-e), mild tail dependence (f-j), strong tail dependence (k-o)}
    \label{fig:bivtemp}
\end{figure}

The corresponding figure for precipitation extremes is shown in Fig.~\ref{fig:bivrain}. Conclusions are similar as for temperature extremes. The evtGAN simulates distributions with different tail dependencies and samples that are far outside the range of the train set (Fig.~\ref{fig:bivrain}c, h, m). DCGAN simulates distributions that are bounded to the train set range (Fig.~\ref{fig:bivrain}d, i, n). On the other hand, Brown--Resnick overestimates tail dependence in the case of mild tail dependence (Fig.~\ref{fig:bivrain}j), and underestimates tail dependence when tail dependence is strong (Fig.~\ref{fig:bivrain}o).

\begin{figure}[h]
    \includegraphics[width=\textwidth]{New_figures/biv_rain.pdf}
    \caption{Bivariate plots of precipitation extremes based on train and test set and simulated through the different presented methods. Shown are selected pairs of locations with varying tail dependence. Columns from left to right: train, test, evtGAN, DCGAN and Brown--Resnick. From top to bottom: weak tail dependence (a-e), mild tail dependence (f-j), strong tail dependence (k-o)}
    \label{fig:bivrain}
\end{figure}

A scatterplot of bivariate extremal correlations between 100 randomly selected locations estimated from the train set, evtGAN and Brown--Resnick, respectively, against estimates based on the test set (1950 samples) is shown in Fig.~\ref{fig:eval}. The estimates derived directly from the train sets (Fig.~\ref{fig:eval}a, d) are the benchmark, and by design better performance is not possible. Clearly, pairs of locations with stronger tail dependence are much more likely for temperature (Fig.~\ref{fig:eval}a) than for precipitation (Fig.~\ref{fig:eval}d), confirming the impression obtained from the spatial homogeneity of the parameters in the extreme value distributions (Fig.~\ref{fig:grid}). Furthermore, evtGAN seems to better capture the observed relationship. Brown--Resnick has difficulties in particular with pairs that have weak or no tail dependence (extremal correlation equals 0, lower left in the figures), which is consistent with Fig.~\ref{fig:bivtemp} and Fig.~\ref{fig:bivrain}.

\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{New_figures/qqplots50.pdf}
    \caption{Scatter plots of the extremal correlations for temperature extremes (a-c) and precipitation extremes (d-f) between $100$ randomly selected locations. The x-axes always show the estimates based on the test set. Estimates on the y-axes are based on the train set (a, d), on the output of evtGAN (b, d) and on the Brown--Resnick model}
\label{fig:eval}
\end{figure}

To further explore the behaviour of evtGAN and Brown--Resnick in dealing with different levels of dependence, Fig.~\ref{fig:spectral} shows a comparison of the estimated spectral distribution for the same three pairs of chosen grid points characterized by weak (a, d), mild (b, e) and strong tail dependence (c, f), respectively. The magenta bars show the estimates based on the 1950 sample in the test set. The estimates of evtGAN are very close to the ground truth for all cases, i.e., weak, mild and strong dependence (red lines in Fig.~\ref{fig:spectral}), except for some bias in the case of mild dependence in precipitation extremes (Fig.~\ref{fig:spectral}e). In contrast, the performance of the Brown--Resnick model (black lines in Fig.~\ref{fig:spectral}) is much more variable. It captures relatively well the two pairs with weak tail dependence (Fig.~\ref{fig:spectral}a, d) and does a decent job for strong dependence in temperature extremes (Fig.~\ref{fig:spectral}c) but fails completely for the remaining three cases (Fig.~\ref{fig:spectral}b,e, f). Furthermore, in contrast to Brown--Resnick, evtGAN is able to represent asymmetric dependence structures, as is evident from Fig.~\ref{fig:spectral}.

\begin{figure}[h]
\centering
  \includegraphics[width=0.91\linewidth]{New_figures/spec50.pdf}
\caption{Spectral distributions for a threshold of $0.95$ for selected pairs of locations with varying tail dependence for temperature (a-c) and precipitation (d-f). (a, d) weak tail dependence, (b, e) mild tail dependence, (c, f) strong tail dependence. In red the kernel density estimate of the evtGAN, in blue the Brown--Resnick model and in magenta bars the ground truth}
\label{fig:spectral}
\end{figure}

We finally present a sensitivity analysis of the performance of evtGAN for different realistic samples sizes, namely 30, 50 and 100, while learning for up to 30,000 epochs (Fig.~\ref{fig:conv}). Overall the error tends to decrease the longer the models learn and it does not seem to matter whether we evaluate the performance on the train set or the test set (difference between black and red lines in Fig.~\ref{fig:conv} is small). The results for the best performance with respect to the train sets (minima in the red lines, $min(C_{tr})$, in Fig.~\ref{fig:conv}c, d) are shown in the figures presented in the results section above. An improvement with increasing sample size is clearly visible (compare Fig.~\ref{fig:conv}a, b with Fig.~\ref{fig:conv}e, f). Furthermore, the model errors in temperature extremes are smaller than the ones for precipitation extremes (left versus right column in Fig.~\ref{fig:conv}).

\begin{figure}[h]
    \centering
        \includegraphics[width=.91\textwidth]{New_figures/accuracy.pdf}
        \caption{Model performance versus training epochs for different sample sizes in evtGAN for temperature (a, c, e) and precipitation extremes (b, d, f). A number of observations equal to $n_{test}=2000-n_{train}$ was sampled from evtGAN. The mean $l^2$ norms for train (black) and test set (red) are defined as $C_{te}= ||\chi_{evtG}-\chi_{te}||_2$, $C_{tr}= ||\chi_{evtG}-\chi_{tr}||_2$, where $\chi_{evtG}$, $\chi_{te}$ and $\chi_{tr}$ denote the vectors of extremal correlations calculated on the samples from evtGAN, the test and the train set, respectively. The epochs at which the minima are reached in the train set are marked with vertical bars}
    \label{fig:conv}
\end{figure}

\section{Discussion}

The use of evtGAN combines the best of two worlds: correct extrapolation based on extreme value theory on the one hand, and flexible dependence modeling through GANs on the other hand.
GANs are an excellent tool to model complex dependencies in high-dimensional spaces. However, they are typically not tailored to model extremes in the marginals well. Indeed, for a standard DCGAN implementation where the marginals are not estimated by GEV distributions but empirically transformed, Figs.~\ref{fig:bivtemp}d, i, n and Figs.~\ref{fig:bivrain}d, i, n show that the generated samples are bounded by the range of the training sample. For an accurate extrapolation that resembles the marginal distributions of the test set, extreme value theory is required (panels c, e, h, j, m and o of Figs.~\ref{fig:bivtemp} and \ref{fig:bivrain}).

On the other hand, classical methods of spatial extreme value theory such as the Brown--Resnick process have accurate extrapolation properties for the marginal distributions. However, for an application to a spatially heterogeneous data set on a large domain (Fig.~\ref{fig:grid}) their parametric assumption may be too restrictive. Indeed, Figs.~\ref{fig:eval}c and f show a clear bias of the Brown--Resnick model in terms of bivariate extremal correlations, which is particularly visible for pairs with weak extremal dependence. Another indication for this bias can be seen in Figs.~\ref{fig:bivrain}j and o where the distributions of the Brown--Resnick samples differ strongly from the test set distributions. The evtGAN does not make prior assumptions on spatial stationarity or isotropy and it does therefore not exhibit a bias (points in Figs.~\ref{fig:eval}b and e are centered around diagonal). This is particularly noteworthy since modeling complex non-stationarities for extremes is very difficult with parametric models \citep{hus2016,engelke2020graphical}.

Considering the fitted bivariate distributions of evtGAN and Brown--Resnick underlines this point. The spectral distributions of the Brown--Resnick model are restricted to a parametric class of functions, which, for instance, are symmetric about $1/2$. The blue lines in Fig.~\ref{fig:spectral} show that this is too restrictive for our data since the strength of dependence is not correctly modelled (Fig.~\ref{fig:spectral}b) or the asymmetry is not captured (Fig.~\ref{fig:spectral}c,f). The evtGAN (red lines) on the other hand can model weak and strong dependence, and it even adapts to possible asymmetries. This is also evident from the scatterplots in Fig.~\ref{fig:bivtemp} and Fig.~\ref{fig:bivrain}, where the shape of the Brown--Resnick samples are restricted to a parametric sub-class of distributions, the so-called H\"usler--Reiss family \citep{hue1989}.

A further restriction of classical spatial max-stable processes is the fact that they are limited to modeling asymptotic dependence. For precipitation, it can be seen in Fig.~\ref{fig:eval}d-f that most of the test extremal correlations are close to zero, indicating asymptotic independence. While the evtGAN is able to capture this fairly well (Fig.~\ref{fig:eval}e), the Brown--Ressnick model always has positive extremal correlations, explaining the bias in the bottom left corner of Fig.~\ref{fig:eval}f. A spatial asymptotically independent model \citep[e.g.,][]{wadsworth2012dependence} would be a possible remedy for this, but such processes would still suffer from the limitation in terms of non-stationarity and asymmetry described above.

Overall, evtGAN tends to perform better in capturing dependencies between temperature extremes than precipitation extremes (Fig.~\ref{fig:conv}). This is likely related to the fact that extremes in temperature are more spatially coherent \citep{Keggenhoff2014} (Fig.~\ref{fig:grid}).

\section{Conclusions}
\label{sec:conc}

Understanding and evaluating the risk associated with extreme events is of primal importance for our society, as recently emphasized in the 6th Assessment Report of the Intergovernmental Panel on Climate Change \citep{Seneviratne2021}. Extreme event analysis and impact assessments are often limited by the available sample size in observations. Furthermore, simulations with complex climate models are very expensive. Here we combine a machine learning approach with extreme value theory to model complex spatial dependencies between extreme events in temperature and precipitation across Europe based on a limited sample size. We demonstrate that this hybrid approach outperforms the typically used approach in multivariate extreme value theory and can well represent the marginal distributions and extremal dependencies across spatially distributed climate extremes. The approach can be easily adapted to other types of extremes and used to create large sample sizes that are often required for climate risk assessments.

\paragraph{Impact Statement}
Spatially co-occurring climate extremes such as heavy precipitation events or temperature extremes can have devastating impacts on human and natural systems. Modelling complex spatial dependencies between climate extremes in different locations is notoriously difficult and traditional approaches from the field of extreme value theory are relatively inflexible. We show that combining extreme value theory with a deep learning architecture (Generative Adversarial Networks) can well represent complex spatial dependencies between extremes. Hence, instead of running expensive climate models, the approach can be used to sample many instances of spatially co-occurring extremes with realistic dependence structure, which may be used for climate risk modelling and stress testing of climate-sensitive systems.

\paragraph{Funding Statement}
This work was funded by the Swiss National Science Foundation (grant nos. 179876, 189908 and 186858) and the Helmholtz Initiative and Networking Fund (Young Investigator Group COMPOUNDX; grant agreement no. VH-NG-1537).

\paragraph{Competing Interests}
None

\paragraph{Data Availability Statement} The temperature and precipitation maxima used in this study are available at https://doi.org/10.5281/zenodo.5554105.

\paragraph{Author Contributions}
Conceptualization: Y.B.; E.V.; S.E. Methodology: Y.B; E.V.; S.E. Formal analysis: Y.B. Software: Y.B.; E.V. Data curation: K.v.d.W. Data visualisation: Y.B. Supervision: J.Z.; S.E. Writing original draft: J.Z.; E.V.; S.E. All authors approved the final submitted draft.

\bibliographystyle{apalike}


\section{Accuracy}\label{apx:accuracy}

\begin{figure*}[t!]
  \subfigure[VGGNet ($B=64$)]{
    \includegraphics[width=.48\hsize]{imgs/{VGG-10-64-0.01-accuracy}.pdf}
  }
  \subfigure[VGGNet ($B=4096$)]{
    \includegraphics[width=.48\hsize]{imgs/{VGG-10-4096-0.1-accuracy}.pdf}
  }
  \subfigure[NIN ($B=64$)]{
    \includegraphics[width=.48\hsize]{imgs/{NIN-64-0.1-accuracy}.pdf}
  }
  \subfigure[NIN ($B=4096$)]{
    \includegraphics[width=.48\hsize]{imgs/NIN-4096-1-accuracy.pdf}
  }
  \subfigure[DenseNet on STL-10 ($B=64$)]{
    \includegraphics[width=.48\hsize]{imgs/{DenseNet-STL-64-0.01-accuracy}.pdf}
  }
  \subfigure[DenseNet on STL-10 ($B=4096$)]{
    \includegraphics[width=.48\hsize]{imgs/{DenseNet-STL-4096-0.1-accuracy}.pdf}
  }
  \caption{Accuracy. Solid, dashed, and dotted lines indicate testing accuracy, training accuracy, and the generalization gap, respectively.}\label{fig:accuracy-appendix}
\end{figure*}

The training curves for the VGGNet, NIN, and DenseNet models on the STL-10 dataset are shown in Figure~\ref{fig:accuracy-appendix}.
We can observe that, in every setting, \textsf{spectral} shows the smallest generalization gap or the best test accuracy, which demonstrates that \textsf{spectral} can effectively reduce the generalization gap without suppressing the model complexity significantly.

\section{Sensitivity to the perturbation of the input}\label{apx:sensitivity}

\begin{figure*}
  \centering
  \subfigure[VGGNet (Train)]{
    \includegraphics[width=.475\hsize]{./imgs/scattered-VGG-10-train-crop.pdf}
  }
  \subfigure[VGGNet (Test)]{
    \includegraphics[width=.475\hsize]{./imgs/scattered-VGG-10-test-crop.pdf}
  }

  \subfigure[NIN (Train)]{
    \includegraphics[width=.475\hsize]{./imgs/scattered-NIN-train-crop.pdf}
  }
  \subfigure[NIN (Test)]{
    \includegraphics[width=.475\hsize]{./imgs/scattered-NIN-test-crop.pdf}
  }

  \subfigure[DenseNet on STL-10 (Train)]{
    \includegraphics[width=.475\hsize]{./imgs/scattered-DenseNet-STL-train-crop.pdf}
  }
  \subfigure[DenseNet on STL-10 (Test)]{
    \includegraphics[width=.475\hsize]{./imgs/scattered-DenseNet-STL-test-crop.pdf}
  }

  \caption{Relation between the generalization gap and the $\ell_2$-norm of the gradient. The solid and dashed lines indicate the results for the small-batch and large-batch regimes, respectively.}\label{fig:grad-transition-appendix}
\end{figure*}

Figure~\ref{fig:grad-transition-appendix} shows the transition of the $\ell_2$-norm of the gradient of the loss function, defined with the training data or test data, with respect to the input.
For every setting, the $\ell_2$-norm of the gradient with respect to the test data is well-correlated with the generalization gap.
On the contrary, for the VGGNet and NIN models, the $\ell_2$-norm of the gradient with respect to the training data does not predict generalization gap well.

\section{Conclusions}\label{sec:conclusions}
In this work, we hypothesized that a high sensitivity to the perturbation of the input data degrades the performance of the data.
In order to reduce the sensitivity to the perturbation of the test data, we proposed the spectral norm regularization method, and confirmed that it exhibits a better generalizability than other baseline methods through experiments.
Experimental comparison with other methods indicated that the insensitivity to the perturbation of the test data plays a crucial role in determining the generalizability.

There are several interesting future directions to pursue.
It is known that weight decay can be seen as a regularization in MAP estimation derived from a Gaussian prior to the model parameters.
Is it possible to understand spectral norm regularization as a derivation of a prior?
We also need a theoretical understanding of the effect of spectral norm regularization on generalization.
It is known that, in some ideal cases, weight decay improves generalizability by preventing neural networks from fitting noises~\cite{Krogh:1991uo}.
Can we extend this argument to spectral norm regularization?

\section{Experiments}\label{sec:experiments}

In this section, we experimentally demonstrate the effectiveness of spectral norm regularization on classification tasks over other regularization techniques, and confirm that the insensitivity to test data perturbation is an important factor for generalization.

All the training methods discussed here are based on stochastic gradient descent (SGD).
We consider two regimes for the choice of the mini-batch size.
In the small-batch regime, we set the mini-batch size to $B=64$, and in the large-batch regime, we set it to $B=4096$.
In our experiments, we compared the following four problems:
\begin{itemize}
\itemsep=0pt
\item Vanilla problem (\textsf{vanilla}):
  As a vanilla problem, we considered empirical risk minimization without any regularization, that is, $\mathop{\text{minimize}}_{\Theta} \frac{1}{K}\sum_{i=1}^K L(f_\Theta({\bm x}_i),{\bm y}_i)$, where $L$ is the cross entropy.
\item Weight decay (\textsf{decay}): We considered the problem~\eqref{eq:weight-decay}, where $L$ is the cross entropy.
We selected the regularization factor $\lambda=10^{-4}$.
\item Adversarial training (\textsf{adversarial}): We considered the problem~\eqref{eq:adversarial}, where $L$ is the cross entropy.
We selected $\alpha=0.5$  and $\epsilon=1$, as suggested in~\cite{Goodfellow:2015tl}.
\item Spectral norm regularization (\textsf{spectral}): We considered the problem~\eqref{eq:spectral}, where $L$ is the cross entropy.
We selected the regularization factor $\lambda =0.01$.
\end{itemize}
We trained models using Nesterov's accelerated gradient descent~\cite{Bengio:2013fn} with momentum 0.9.
We decreased the learning rate by a factor of $1/10$ when the half and the three quarters of the training process have passed.
We optimized the hyper-parameters through a grid search and selected those that showed a reasonably good performance for every choice of neural network and mini-batch size.

In our experiments, we used the following four settings on the model and dataset.
\begin{itemize}
\itemsep=0pt
\item The VGG network (VGGNet for short)~\cite{Simonyan:2014ws} on the CIFAR-10 dataset~\cite{Krizhevsky:2009tr}.
\item The network in network (NIN) model ~\cite{Lin:2014wb} on the CIFAR-100 dataset~\cite{Krizhevsky:2009tr}.
\item The densely connected convolutional network (DenseNet) ~\cite{Huang:2016wa} having a depth of 40 on the CIFAR-100 dataset.
\item DenseNet having a depth of 22 on the STL-10 dataset~\cite{Coates:2011wo}.
  The depth was decreased because of the memory consumption issue.
\end{itemize}
We preprocessed all the datasets using global contrast normalization.
We further applied data augmentation with cropping and horizontal flip on STL-10 because the number of training data samples is only 5,000, which is small considering the large mini-batch size of 4096.
The learning rate of SGD was initialized to $0.1$ in the small-batch regime and $1.0$ in the large-batch regime for the NIN and DenseNet models on the CIFAR-100 dataset, and was initialized to $0.01$ in the small-batch regime and $0.1$ in the large-batch regime for the VGGNet and DenseNet models on the CIFAR-10 dataset.

\subsection{Accuracy and Generalization Gap}\label{subsec:accuracy}

\begin{table}
  \centering
  \caption{Test accuracy and generalization gap (best values in bold).}\label{tab:accuracy}
  {
  \small
  \tabcolsep=1mm
  \begin{tabular}{|c|r||rrrr||r|rrrr|}
    \hline
    & & \multicolumn{4}{c||}{Test accuracy} & \multicolumn{5}{c|}{Generalization gap} \\
    Model & $B$ & \textsf{vanilla} & \textsf{decay} & \textsf{adver.} & \textsf{spectral} & \multicolumn{1}{c}{$\alpha$} & \textsf{vanilla} & \textsf{decay} & \textsf{adver.} & \textsf{spectral}  \\ \hline \hline
    \multirow{2}{*}{VGGNet} & 64 & 0.898 & 0.897 & 0.884 & \textbf{0.904} & 0.88 & 0.079 & 0.074 & 0.109 & \textbf{0.068} \\
    & 4096 & 0.858 & 0.863 & 0.870 & \textbf{0.885} & 0.85 & 0.092 & 0.064 & 0.064 & \textbf{0.045} \\ \hline
    \multirow{2}{*}{NIN} & 64 & 0.626 & \textbf{0.672} & 0.627 & 0.669 & 0.62 & 0.231 & 0.120 & 0.253 & \textbf{0.090} \\
    & 4096 & 0.597 & 0.618 & 0.607 & \textbf{0.640} & 0.59 & 0.205 & 0.119 & 0.196 & \textbf{0.090} \\ \hline
    DenseNet & 64 & 0.675 & \textbf{0.718} & 0.675 & 0.709 & 0.67 & 0.317 & 0.080 & 0.299 & \textbf{0.095}\\
    (CIFAR100) & 4096 & 0.639 & 0.671 & 0.649 & \textbf{0.697} & 0.63 & 0.235 & 0.111 & 0.110 & \textbf{0.051}  \\ \hline
    DenseNet  & 64 & 0.724 & 0.723 & 0.707 & \textbf{0.735} & 0.70 & \textbf{0.063} & 0.073 & 0.069 & 0.068 \\
    (STL-10) & 4096 & 0.686 & 0.689 & 0.676 & \textbf{0.697} & 0.67 & 0.096 & 0.057 & \textbf{0.015} & 0.042 \\
    \hline
  \end{tabular}
  }
\end{table}

\begin{figure*}[t!]
  \subfigure[$B=64$]{
    \includegraphics[width=.48\hsize]{imgs/{DenseNet-64-0.1-accuracy}.pdf}
  }
  \subfigure[$B=4096$]{
    \includegraphics[width=.48\hsize]{imgs/DenseNet-4096-1-accuracy.pdf}
  }
  \caption{Accuracy of the DenseNet model on the CIFAR-100 dataset. The solid, dashed, and dotted lines indicate the test accuracy, training accuracy, and generalization gap, respectively.}\label{fig:accuracy-DenseNet}
\end{figure*}

First, we look at the test accuracy obtained by each method, which is summarized in the left columns of Table~\ref{tab:accuracy}.
In the small-batch regime, \textsf{decay} and \textsf{spectral} show better test accuracies than the other two methods.
In the large-batch regime, \textsf{spectral} clearly achieves the best test accuracy for every model.
Although the test accuracy decreases as the mini-batch size increases, as reported in~\cite{Keskar:2017tz}, the decrease in the test accuracy of \textsf{spectral} is less significant than those of the other three methods.

Next, we look at the generalization gap.
We define the generalization gap at a threshold $\alpha$ as the minimum difference between the training and test accuracies when the test accuracy exceeds $\alpha$.
The generalization gap of each method is summarized in the right columns of Table~\ref{tab:accuracy}.
For each setting, we selected the threshold $\alpha$ so that every method achieves a test accuracy that (slightly) exceeds this threshold.
For each of the settings, except for the DenseNet model on the STL-10 dataset, \textsf{spectral} clearly shows the smallest generalization gap followed by \textsf{decay}, which validates the effectiveness of \textsf{spectral}.

Figure~\ref{fig:accuracy-DenseNet} shows the training curve of the DenseNet model on the CIFAR-100 dataset.
The results for other settings are given in Appendix~\ref{apx:accuracy}.
As we can observe, in both the small-batch and large-batch regimes, \textsf{spectral} shows the smallest generalization gaps.
The generalization gap of \textsf{decay} increases as the mini-batch size increases, whereas that of \textsf{spectral} does not increase significantly.
Investigating the reason behind this phenomena is an interesting future work.

The choice of the mini-batch size and the training method is not important to obtain a model with a good training accuracy; all of them exceeds 95\%.
However, obtaining a model with a good test accuracy, or a small generalization gap, is important.
In the subsequent sections, we investigate which property of a trained model determines its generalization gap.

To summarize, \textsf{spectral} consistently achieves the small generalization gap and shows the best test accuracy, especially in the large-batch regime.

\subsection{Sensitivity to the perturbation of the input}

\begin{figure}
  \centering
  \subfigure[Train]{
    \label{fig:grad-transition-train}
    \includegraphics[width=.475\hsize]{./imgs/scattered-DenseNet-train-crop.pdf}
  }
  \subfigure[Test]{
    \label{fig:grad-transition-test}
    \includegraphics[width=.475\hsize]{./imgs/scattered-DenseNet-test-crop.pdf}
  }
  \caption{Relation between the generalization gap and the $\ell_2$-norm of the gradient of the DenseNet model on the CIFAR-100 dataset. The solid and dashed lines indicate the results for the small-batch and large-batch regimes, respectively.}\label{fig:grad-transition}
\end{figure}

To understand the relation between the sensitivity to the input perturbation and the generalization gap of the trained model, we look at the gradient of the loss function, defined with the training data or the test data, with respect to the input.
Figure~\ref{fig:grad-transition} shows the transition of the $\ell_2$-norm when training the DenseNet model on the CIFAR-100 dataset.
The results for other settings are given in Appendix~\ref{apx:sensitivity}.

We can observe in Figure~\ref{fig:grad-transition-test} that the $\ell_2$-norm of the gradient with respect to the test data is well  correlated with the generalization gap.
In particular, the $\ell_2$-norm of the gradient gradually increases as training proceeds, which matches the behavior of the generalization gap, which also increases as training proceeds.

On the contrary, as shown in Figure~\ref{fig:grad-transition-train}, although the $\ell_2$-norm of the gradient gradually decreases as training proceeds, the generalization gap actually increases.
Hence, the $\ell_2$-norm of the gradient with respect to the training data does not predict the generalization gap well.

These results motivate us to reduce the $\ell_2$-norm of the gradient with respect to the test data to obtain a smaller generalization gap.
As we do not know the test data a priori, the spectral norm regularization method is reasonable to achieve this goal because it reduces the gradient at any point.

The superiority of \textsf{spectral} to \textsf{decay} can be elucidated from this experiment.
In order to achieve a better test accuracy, we must make the training accuracy high and the generalization gap small.
To achieve the latter, we have to penalize the $\ell_2$-norm of the gradient with respect to the test data.
To this end, \textsf{decay} suppresses all the weights, which decreases model complexity, and hence, we cannot fit the model to the training data well.
On the other hand, \textsf{spectral} only suppresses the spectral norm, and hence, we can achieve a greater model complexity than \textsf{decay} and fit the model to the training data better.

\subsection{Maximum eigenvalue of the Hessian with respect to the model parameters}

In~\cite{Keskar:2017tz}, it is claimed that the maximum eigenvalue of the Hessian of the loss function defined with the training data predicts the generalization gap well.
To confirm this claim, we computed the maximum eigenvalue of the DenseNet model trained on the CIFAR-100 dataset, shown in Figure~\ref{fig:hessian}.
As it is computationally expensive to compute the Hessian, we approximated its maximum eigenvalue by using the power iteration method because we can calculate the Hessian-vector product without explicitly calculating the Hessian~\cite{Martens:2010vo}.
We also computed the maximum eigenvalue of the Hessian of the loss function defined with the test data.

We can observe that, for \textsf{vanilla}, larger eigenvalues (in both the training and test data) are obtained if the mini-batch size is increased, which confirms the claim of~\cite{Keskar:2017tz}.
However, the models trained with regularizations tend to have larger eigenvalues, although they have better generalizability.
In particular, the models trained by \textsf{spectral} have the largest eigenvalues, although they have the best generalizability as we have seen in Section~\ref{subsec:accuracy}.

\begin{figure}
  \centering
  \begin{minipage}{.45\hsize}
  \subfigure[Train]{
    \includegraphics[width=0.46\hsize]{./imgs/hessian-train.pdf}
  }
  \subfigure[Test]{
    \includegraphics[width=0.46\hsize]{./imgs/hessian-test.pdf}
  }
  \caption{Maximum eigenvalue of the Hessian of the DenseNet on CIFAR100}\label{fig:hessian}
  \end{minipage}
  \quad
  \begin{minipage}{.45\hsize}
  \includegraphics[width=\hsize]{imgs/spectral-DenseNet-200-trans3_conv-crop.pdf}
  \caption{Singular values of weight matrices in the DenseNet on CIFAR100. Solid and dashed lines indicate the results for the small-batch and large-batch regimes, respectively.}\label{fig:spectrum}
  \end{minipage}
\end{figure}

This phenomena can be understood as follows:
If we train a model without regularization, then a small perturbation does not significantly affect the Frobenius and spectral norms of weight matrices because they are already large.
However, if we train a model with regularization, then because of these small norms a small perturbation may significantly affect those norms, which may cause a significant change in the output.

To summarize, this experiment indicates that the maximum eigenvalue of the Hessian of the loss function is not a suggestive measure for predicting generalizability.

\subsection{Singular values of weight matrices}

Finally, we look at the singular values of weight matrices in the models trained by each method.
Figures~\ref{fig:spectrum} shows the singular values of a weight matrix taken from the DenseNet model on the CIFAR-100 dataset.
The matrix is selected arbitrarily because all matrices showed similar spectra.

We can observe that the spectrum of \textsf{vanilla} is highly skewed, and \textsf{adversarial} and \textsf{decay} shrink the spectrum while maintaining the skewed shape.
In contrast, the spectrum of \textsf{spectral} is flat.
This behavior is as expected because the spectral norm regularization tries to reduce the largest singular value.
Because the maximum singular value obtained by \textsf{spectral} is low, we obtain less sensitivity to the perturbation of the input.

\section{Introduction}

Deep learning has been successfully applied to several machine learning tasks including visual object classification~\cite{he2016identity,Krizhevsky:2012wl}, speech recognition~\cite{Hinton:2012is}, and natural language processing~\cite{Collobert:2008kg, jozefowicz2016exploring}.
A well-known method of training deep neural networks is stochastic gradient descent (SGD).
SGD can reach a local minimum with high probability over the selection of the starting point~\cite{Lee:2016vd}, and all local minima attain similar loss values~\cite{Choromanska:2015ui,Dauphin:2014tn,Kawaguchi:2016ub}.
However, the performance on test data, that is, \emph{generalizability}, can be significantly different among these local minima.
Despite the success of deep learning applications, we lack the understanding of generalizability, even though there has been progress ~\cite{Keskar:2017tz,Zhang:2017te}.

While understanding the generalizability of deep learning is an interesting topic, it is important from a practical point of view.
For example, suppose that we are training a deep neural network using SGD and want to parallelize the process using multiple GPUs or nodes to accelerate the training.
A well-known method for achieving this is synchronous SGD~\cite{das2016distributed, chen2016revisiting}, which requires a large minibatch to effectively exploit parallel computation on multiple GPUs or nodes.
However, it is reported that models trained through synchronous SGD with large minibatches exhibit poor generalization~\cite{Keskar:2017tz}, and a method is required to resolve this problem.

In this study, we consider the generalizability of deep learning from the perspective of sensitivity to input perturbation.
Intuitively, if a trained model is insensitive or sensitive to the perturbation of an input, then the model is confident or not confident about the output, respectively.
As the performance on test data is important, models that are insensitive to the perturbation of \emph{test} data are required.
Note that adversarial training~\cite{Szegedy:2014vw,Goodfellow:2015tl} is designed to achieve insensitivity to the perturbation of \emph{training} data, and it is not always effective for achieving insensitivity to the perturbation of test data.

To obtain insensitivity to the perturbation of test data, we propose a simple and effective regularization method,  referred to as spectral norm regularization.
As the name suggests, spectral norm regularization prevents the weight matrices used in neural networks  from having large spectral norms.
Through this, even though test data are not known in advance, a trained model is ensured to exhibit slight sensitivity to the perturbation of test data.

Using several real-world datasets, we experimentally confirm that models trained using spectral norm regularization exhibit better generalizability than  models trained using other baseline methods.
It is claimed in~\cite{Keskar:2017tz} that the maximum eigenvalue of the Hessian predicts the generalizability of a trained model.
However, we show that the insensitivity to the perturbation of test data is a more important factor for predicting generalizability, which further motivates the use of spectral norm regularization.
Finally, we show that spectral norm regularization effectively reduces the spectral norms of weight matrices.

The rest of this paper is organized as follows:
We review related works in Section~\ref{sec:related}.
In Section~\ref{sec:method}, we explain spectral norm regularization and compare it with other regularizing techniques .
Experimental results are provided in Section~\ref{sec:experiments}, and conclusions are stated in Section~\ref{sec:conclusions}.

\section*{Acknowledgement}
The authors thank Takuya Akiba and Seiya Tokui for helpful discussions.

\bibliographystyle{abbrvurl}
{
  \small

\section{Spectral Norm Regularization}\label{sec:method}

In this section, we explain spectral norm regularization and how it reduces the sensitivity to test data perturbation.

\subsection{General idea}\label{subsec:intuition}
We consider feed-forward neural networks as a simple example to explain the intuition behind spectral norm regularization.
A feed-forward neural network can be represented as cascaded computations, ${\bm x}^\ell = f^\ell(W^\ell{\bm x}^{\ell-1} + {\bm b}^\ell)$ for $\ell = 1,\ldots,L$ for some $L$, where ${\bm x}^{\ell-1} \in \mathbb{R}^{n_{\ell-1}}$ is the input feature of the $\ell$-th layer, $f^\ell:\mathbb{R}^{n_\ell}  \to \mathbb{R}^{n_\ell}$ is a (non-linear) activation function, and $W^\ell \in \mathbb{R}^{n_{\ell} \times n_{\ell-1}}$ and $b^\ell \in \mathbb{R}^{n_\ell}$ are the layer-wise weight matrix and bias vector, respectively.
For a set of parameters, $\Theta = \set{W^\ell,{\bm b}^\ell}_{\ell=1}^L$, we denote by $f_\Theta:\mathbb{R}^{n_0} \to \mathbb{R}^{n_L}$ the function defined as $f_\Theta({\bm x}^0) = {\bm x}^L$.
Given training data, $({\bm x}_i,{\bm y}_i)_{i=1}^K$, where ${\bm x}_i \in \mathbb{R}^{n_0}$ and ${\bm y}_i \in \mathbb{R}^{n_L}$, the loss function is defined as $\frac{1}{K}\sum_{i=1}^K L(f_\Theta({\bm x}_i),{\bm y}_i)$, where $L$ is frequently selected to be cross entropy and the squared $\ell_2$-distance for classification and regression tasks, respectively.
The model parameter to be learned is $\Theta$.

Let us consider how we can obtain a model that is insensitive to the perturbation of the input.
Our goal is to obtain a model, $\Theta$, such that the $\ell_2$-norm of $f({\bm x} + {\bm \xi}) - f({\bm x})$ is small, where ${\bm x} \in \mathbb{R}^{n_0}$ is an arbitrary vector and ${\bm \xi} \in \mathbb{R}^{n_0}$ is a perturbation vector with a small $\ell_2$-norm.
A key observation is that most practically used neural networks exhibit nonlinearity only because they use piecewise linear functions, such as ReLU , maxout~\cite{Goodfellow:2013tf}, and maxpooling~\cite{Ranzato:2007eb}, as activation functions.
In such a case, function $f_\Theta$ is a piecewise linear function.
Hence, if we consider a small neighborhood of ${\bm x}$, we can regard $f_\Theta$ as a linear function. In other words, we can represent it by an affine map, ${\bm x} \mapsto W_{\Theta,{\bm x}}{\bm x} + {\bm b}_{\Theta,{\bm x}}$, using a matrix, $W_{\Theta,{\bm x}}  \in \mathbb{R}^{n_0\times n_L}$, and a vector, ${\bm b}_{\Theta,{\bm x}} \in \mathbb{R}^{n_L}$, which depend on $\Theta$ and ${\bm x}$.
Then, for a small perturbation, ${\bm \xi} \in \mathbb{R}^{n_0}$, we have
\begin{align*}
  \frac{\|f_\Theta({\bm x} + {\bm \xi}) - f({\bm x})\|_2}{\|{\bm \xi}\|_2}
  = \frac{\|(W_{\Theta,{\bm x}}({\bm x} + {\bm \xi}) + {\bm b}_{\Theta,{\bm x}}) - (W_{\Theta,{\bm x}}{\bm x} + {\bm b}_{\Theta,{\bm x}}) \|_2 }{\|{\bm \xi}\|_2}
  = \frac{\|W_{\Theta,{\bm x}}{\bm \xi}\|_2}{\|{\bm \xi}\|_2} \leq \sigma(W_{\Theta,{\bm x}}),
\end{align*}
where $\sigma(W_{\Theta,{\bm \xi}})$ is the spectral norm of $W_{\Theta,{\bm \xi}}$.
The \emph{spectral norm} of a matrix $A \in \mathbb{R}^{m \times n}$ is defined as
\[
  \sigma(A) = \max_{{\bm \xi} \in \mathbb{R}^n,{\bm \xi} \neq {\bm 0}}\frac{\|A {\bm \xi}\|_2}{\|{\bm \xi}\|_2},
\]
which corresponds to the largest singular value of $A$.
Hence, the function $f_\Theta$ is insensitive to the perturbation of ${\bm x}$ if the spectral norm of $W_{\Theta,{\bm x}}$ is small.

The abovementioned argument suggests that model parameter $\Theta$ should be trained so that the spectral norm of $W_{\Theta,{\bm x}}$ is small for any ${\bm x}$.
To further investigate the property of $W_{\Theta,{\bm x}}$, let us assume that each activation function, $f^\ell$, is an element-wise ReLU (the argument can be easily generalized to other piecewise linear functions).
Note that, for a given vector, ${\bm x}$, $f^\ell$ acts as a diagonal matrix, $D^\ell_{\Theta,{\bm x}} \in \mathbb{R}^{n_\ell \times n_{\ell}}$, where an element in the diagonal is equal to one if the corresponding element in ${\bm x}^{\ell-1}$ is positive; otherwise, it is equal to zero.
Then, we can rewrite $W_{\Theta,{\bm x}}$ as $W_{\Theta,{\bm x}} = D^{L}_{\Theta,{\bm x}}W^{L}D^{L-1}_{\Theta,{\bm x}} W^{L-1} \cdots D^1_{\Theta,{\bm x}} W^1$.
Note that $\sigma(D^\ell_{\Theta,{\bm x}}) \leq 1$ for every $\ell \in \set{1,\ldots,L}$.
Hence, we have
\[
  \sigma(W_{\Theta,{\bm x}})
  \leq
  \sigma(D^{L}_{\Theta,{\bm x}})\sigma(W^{L})\sigma(D^{L-1}_{\Theta,{\bm x}}) \sigma(W^{L-1}) \cdots \sigma(D^{1}_{\Theta,{\bm x}}) \sigma(W^1)
  \leq
  \prod_{\ell=1}^L \sigma(W^\ell).
\]
It follows that, to bound the spectral norm of $W_{\Theta,{\bm x}}$, it suffices to bound the spectral norm of $W^\ell$ for each $\ell \in \set{1,\ldots,L}$.
This motivates us to consider spectral norm regularization, which is described in the next section.

\subsection{Details of spectral norm regularization}\label{subsec:proposed}

In this subsection, we explain spectral norm regularization.
The notations are the same as those used in Section~\ref{subsec:intuition}.
To bound the spectral norm of each weight matrix, $W^\ell$, we consider the following empirical risk minimization problem:
\begin{align}
  \mathop{\text{minimize}}_{\Theta} \frac{1}{K}\sum_{i=1}^K L(f_\Theta({\bm x}_i),{\bm y}_i) + \frac{\lambda}{2} \sum_{\ell=1}^L \sigma(W^\ell)^2,
  \label{eq:spectral}
\end{align}
where $\lambda \in \mathbb{R}_+$ is a regularization factor.
We refer to the second term as the \emph{spectral norm regularizer}. It decreases the spectral norms of the weight matrices.

When performing SGD, we need to calculate the gradient of the spectral norm regularizer.
To this end, let us consider the gradient of $\sigma(W^\ell)^2_2/2$ for a particular $\ell \in \set{1,2,\ldots,L}$.
Let $\sigma_1 = \sigma(W^\ell)$ and $\sigma_2$ be the first and second singular values, respectively.
If $\sigma_1 > \sigma_2$, then the gradient of $\sigma(W^\ell)^2/2$ is $\sigma_1 {\bm u}_1{\bm v}_1^\top$, where ${\bm u}_1$ and ${\bm v}_1$ are the first left and right singular vectors, respectively.
If $\sigma_1 = \sigma_2$, then $\sigma(W^\ell)^2_2$ is not differentiable.
However, for practical purposes, we can assume that this case never occurs because numerical errors prevent $\sigma_1 $ and $\sigma_2$ from being exactly equal.

As it is computationally expensive to compute $\sigma_1$, ${\bm u}_1$, and ${\bm v}_1$, we approximate them using the power iteration method.
Starting with a randomly initialized ${\bm v} \in \mathbb{R}^{n_{\ell-1}}$, we iteratively perform the following procedure a sufficient number of times: ${\bm u} \leftarrow W^\ell {\bm v}$ and ${\bm v} \leftarrow (W^\ell)^\top {\bm u}$, and $\sigma \leftarrow \|{\bm u}\|_2/\|{\bm v}\|_2$.
Then, $\sigma$, ${\bm u}$, and ${\bm v}$ converge to $\sigma_1$, ${\bm u}_1$, and ${\bm v}_1$, respectively (if $\sigma_1 > \sigma_2$).
To approximate $\sigma_1$, ${\bm u}_1$, and ${\bm v}_1$ in the next iteration of SGD, we can reuse ${\bm v}$ as the initial vector.
in our experiments, which are explained in Section~\ref{sec:experiments}, we performed only one iteration because it was adequate for obtaining a sufficiently good approximation.
A pseudocode is provided in Algorithm~\ref{alg:spectral}.

\begin{algorithm}[t!]
  \caption{SGD with spectral norm regularization}\label{alg:spectral}
  \begin{algorithmic}[1]
    \For{$\ell=1$ to $L$}
      \State ${\bm v}^\ell \leftarrow$ a random Gaussian vector.
    \EndFor
    \For{each iteration of SGD}
      \State Consider a minibatch, $\set{({\bm x}_{i_1},y_{i_1}),\ldots,({\bm x}_{i_k},y_{i_k})}$, from training data.
      \State Compute the gradient of $\frac{1}{k}\sum_{i=1}^k L(f_{\Theta}({\bm x}_{i_j}),y_{i_j})$ with respect to $\Theta$.
      \For{$\ell=1$ to $L$}
        \For{a sufficient number of times} \Comment{One iteration was adequate in our experiments}
          \State ${\bm u}^\ell \leftarrow W^\ell {\bm v}^\ell$, ${\bm v}^\ell \leftarrow (W^\ell)^\top {\bm u}^\ell$, $\sigma^\ell \leftarrow \|{\bm u}^\ell\|/\|{\bm v}^\ell\|$
          \State Add $\lambda \sigma^\ell {\bm u}^\ell ({\bm v}^\ell)^\top$ to the gradient of $W^\ell$.
        \EndFor
      \EndFor
      \State Update $\Theta$ using the gradient.
    \EndFor
  \end{algorithmic}
\end{algorithm}

\paragraph{Convolutions.}
We describe how to handle convolutions because they are used widely in recent applications of  deep neural networks.
Consider a convolutional layer with $a$ input channels, $b$ output channels, and a $k_w \times k_h$-sized kernel.
This implies that the convolution has $abk_wk_h$ parameters.
Note that a value in an output channel is determined using $ak_wk_h$ values in the input channels.
Hence, we align the parameters as a matrix of size $b \times ak_wk_h$ and apply the abovementioned power iteration method to the matrix to calculate its spectral norm and gradient.

\subsection{Comparison with other regularization methods}
We now compare the spectral norm regularization with other regularization techniques.
\paragraph{Weight decay.}

\emph{Weight decay}, or the \emph{Frobenius norm regularization}, is a well-known regularization technique for deep learning.
It considers the following problem:
\begin{align}
  \mathop{\text{minimize}}_{\Theta} \frac{1}{K}\sum_{i=1}^K L(f_\Theta({\bm x}_i),{\bm y}_i) + \frac{\lambda}{2}\sum_{\ell=1}^L\|W^\ell\|_F^2,
  \label{eq:weight-decay}
\end{align}
where $\lambda \in \mathbb{R}_+$ is a regularization factor.
We note that $\|W^\ell\|_F^2 = \sum_{i=1}^{\min\set{n_{\ell-1},n_\ell}}\sigma_i(W^\ell)^2$, where $\sigma_i(W^\ell)$ is the $i$-th singular value of $W^\ell$.
Hence, the Frobenius norm regularization reduces the sum of squared singular values.
Even though it will be effective to train models that are  insensitive to input perturbation, a trained model may lose important information about the input because each trained weight matrix, $W^\ell$, acts as an operator that shrinks in all directions.
In contrast, spectral norm regularization focuses only on the first singular value, and each trained weight matrix, $W^\ell$, does not shrink significantly in the directions orthogonal to the first right singular vector.

\paragraph{Adversarial training.}

Adversarial training~\cite{Goodfellow:2015tl} considers the following problem:
\begin{align}
  \mathop{\text{minimize}}_{\Theta} \alpha \cdot \frac{1}{K}\sum_{i=1}^K L(f_\Theta({\bm x}_i),{\bm y}_i) + (1-\alpha) \cdot \frac{1}{K}\sum_{i=1}^K L(f_\Theta({\bm x}_i + {\bm \eta}_i), {\bm y}_i)\label{eq:adversarial},
\end{align}
where
\[
  {\bm \eta}_i = \epsilon \cdot \frac{{\bm g}_i}{\|{\bm g}_i\|_2} \quad \text{and} \quad {\bm g}_i = \nabla_{{\bm x}} L(f_{\Theta}({\bm x}),{\bm y}_i)|_{{\bm x} = {\bm x}_i},
\]
and $\alpha$ and $\epsilon \in \mathbb{R}_+$ are hyperparameters.
It considers the perturbation toward the direction that increases the loss function the most.
Hence, a trained model is insensitive to the adversarial perturbation of training data.
In contrast, spectral norm regularization automatically trains a model that is insensitive to the perturbation of training data and test data.

\paragraph{Jacobian regularization.}
Another method of reducing the sensitivity of a model against input perturbation is penalizing the $\ell_2$-norm of the derivative of the output with respect to the input.
Let us denote the Jacobian matrix of ${\bm y}$ with respect to ${\bm x}$ by $\partial {\bm y} / \partial {\bm x}$.
The \emph{Jacobian regularization} considers the following regularization term:
\begin{align*}
  \frac{1}{K}\sum_{i=1}^K \left\|\frac{\partial {\bm y}}{\partial {\bm x}}\Big|_{{\bm x}={\bm x}_i}\right\|_F^2.
\end{align*}
The Jacobian regularization promotes the smoothness of a model against input perturbation.
However, this regularization is impractical because calculating the derivative of a Jacobian with respect to parameters is computationally expensive.
To resolve the issue, Gu~\emph{et~al.}~\cite{gu2014towards} proposed an alternative method that regularizes layer-wise Jacobians:
\[
  \frac{1}{K}\sum_{i=1}^K \sum_{\ell=1}^{L} \left\|\frac{\partial {\bm x}^{\ell}}{\partial {\bm x}^{\ell-1}}\Big|_{{\bm x}^{\ell-1}={\bm x}^{\ell-1}_i}\right\|_F^2,
\]
where ${\bm x}^\ell_i$ is the input to the $\ell$-th layer calculated using ${\bm x}_i$.
Note that, if we neglect the effect of activation functions between layers, this regularization coincides with weight decay.
Hence, we exclude this regularization from our experiments.

\section{Related Works}\label{sec:related}

A conventional method of understanding the generalizability of a trained model is the notion of the flatness/sharpness of a local minimum~\cite{Hochreiter:1997wf}.
A local minimum is (informally) referred to as \emph{flat} if its loss value does not increase significantly when it is perturbed; otherwise, it is referred to as \emph{sharp}.
In general, the high sensitivity of a training function at a sharp local minimizer negatively affects the generalizability of the trained model.
In~\cite{Hochreiter:1997wf}, this is explained in more detail through  the minimum description length  theory, which states that statistical models that require fewer bits to describe generalize better~\cite{Rissanen:1983fr}.

It is known that SGD with a large minibatch size leads to a model that does not generalize well~\cite{lecun2012efficient}.
In~\cite{Keskar:2017tz}, this problem is studied based on the flatness/sharpness of the obtained local minima.
They formulated a flat local minimum as a local minimum at which all eigenvalues of the Hessian are small (note that all eigenvalues are non-negative at a local minimum). Using a proxy measure, they experimentally showed that SGD with a smaller minibatch size tends to converge to a flatter minimum.

The notion of flat/sharp local minima considers the sensitivity of a loss function against the perturbation of model parameters.
However, it is natural to consider the sensitivity of a loss function against the perturbation of input data, as we discuss in this paper.
In~\cite{Szegedy:2014vw}, the perturbation to training data that increases the loss function the most is considered, and the resulting perturbed training data are referred to as \emph{adversarial examples}.
It is reported in~\cite{Goodfellow:2015tl} that training using adversarial examples improves test accuracy.

Recently,~\cite{Zhang:2017te} showed that the  classical notions of Rademacher complexity and the VC  dimension are not adequate for understanding the generalizability of deep neural networks.

Note that the spectral norm of a matrix is equal to its largest singular value.
Singular values have attracted attention in the context of training recurrent neural networks (RNN).
In~\cite{Arjovsky:2016tb,Wisdom:2016vk}, it is shown that by restricting the weight matrices in RNN to be unitary or orthogonal, that is, matrices with all singular values equal to one, the problem of diminishing and exploding gradients can be prevented and better performance  can be obtained.