\section{Electronic Submission}
\label{submission}

Submission to ICML 2023 will be entirely electronic, via a web site
(not email). Information about the submission process and \LaTeX\ templates
are available on the conference web site at:
\begin{center}
\textbf{\texttt{http://icml.cc/}}
\end{center}

The guidelines below will be enforced for initial submissions and
camera-ready copies. Here is a brief summary:
\begin{itemize}
\item Submissions must be in PDF\@.
\item \textbf{New to this year}: If your paper has appendices, submit the appendix together with the main body and the references \textbf{as a single file}. Reviewers will not look for appendices as a separate PDF file. So if you submit such an extra file, reviewers will very likely miss it.
\item Page limit: The main body of the paper has to be fitted to 8 pages, excluding references and appendices; the space for the latter two is not limited. For the final version of the paper, authors can add one extra page to the main body.
\item \textbf{Do not include author information or acknowledgements} in your
    initial submission.
\item Your paper should be in \textbf{10 point Times font}.
\item Make sure your PDF file only uses Type-1 fonts.
\item Place figure captions \emph{under} the figure (and omit titles from inside
    the graphic file itself). Place table captions \emph{over} the table.
\item References must include page numbers whenever possible and be as complete
    as possible. Place multiple citations in chronological order.
\item Do not alter the style template; in particular, do not compress the paper
    format by reducing the vertical spaces.
\item Keep your abstract brief and self-contained, one paragraph and roughly
    4--6 sentences. Gross violations will require correction at the
    camera-ready phase. The title should have content words capitalized.
\end{itemize}

\subsection{Submitting Papers}

\textbf{Paper Deadline:} The deadline for paper submission that is
advertised on the conference website is strict. If your full,
anonymized, submission does not reach us on time, it will not be
considered for publication.

\textbf{Anonymous Submission:} ICML uses double-blind review: no identifying
author information may appear on the title page or in the paper
itself. \cref{author info} gives further details.

\textbf{Simultaneous Submission:} ICML will not accept any paper which,
at the time of submission, is under review for another conference or
has already been published. This policy also applies to papers that
overlap substantially in technical content with conference papers
under review or previously published. ICML submissions must not be
submitted to other conferences and journals during ICML's review
period.
Informal publications, such as technical
reports or papers in workshop proceedings which do not appear in
print, do not fall under these restrictions.

\medskip

Authors must provide their manuscripts in \textbf{PDF} format.
Furthermore, please make sure that files contain only embedded Type-1 fonts
(e.g.,~using the program \texttt{pdffonts} in linux or using
File/DocumentProperties/Fonts in Acrobat). Other fonts (like Type-3)
might come from graphics files imported into the document.

Authors using \textbf{Word} must convert their document to PDF\@. Most
of the latest versions of Word have the facility to do this
automatically. Submissions will not be accepted in Word format or any
format other than PDF\@. Really. We're not joking. Don't send Word.

Those who use \textbf{\LaTeX} should avoid including Type-3 fonts.
Those using \texttt{latex} and \texttt{dvips} may need the following
two commands:

{\footnotesize
\begin{verbatim}
dvips -Ppdf -tletter -G0 -o paper.ps paper.dvi
ps2pdf paper.ps
\end{verbatim}}
It is a zero following the ``-G'', which tells dvips to use
the config.pdf file. Newer \TeX\ distributions don't always need this
option.

Using \texttt{pdflatex} rather than \texttt{latex}, often gives better
results. This program avoids the Type-3 font problem, and supports more
advanced features in the \texttt{microtype} package.

\textbf{Graphics files} should be a reasonable size, and included from
an appropriate format. Use vector formats (.eps/.pdf) for plots,
lossless bitmap formats (.png) for raster graphics with sharp lines, and
jpeg for photo-like images.

The style file uses the \texttt{hyperref} package to make clickable
links in documents. If this causes problems for you, add
\texttt{nohyperref} as one of the options to the \texttt{icml2023}
usepackage statement.

\subsection{Submitting Final Camera-Ready Copy}

The final versions of papers accepted for publication should follow the
same format and naming convention as initial submissions, except that
author information (names and affiliations) should be given. See
\cref{final author} for formatting instructions.

The footnote, ``Preliminary work. Under review by the International
Conference on Machine Learning (ICML). Do not distribute.'' must be
modified to ``\textit{Proceedings of the
$\mathit{39}^{th}$ International Conference on Machine Learning},
Honolulu, Hawaii, USA, PMLR 202, 2023.
Copyright 2023 by the author(s).''

For those using the \textbf{\LaTeX} style file, this change (and others) is
handled automatically by simply changing
$\mathtt{\backslash usepackage\{icml2023\}}$ to
$$\mathtt{\backslash usepackage[accepted]\{icml2023\}}$$
Authors using \textbf{Word} must edit the
footnote on the first page of the document themselves.

Camera-ready copies should have the title of the paper as running head
on each page except the first one. The running title consists of a
single line centered above a horizontal rule which is $1$~point thick.
The running head should be centered, bold and in $9$~point type. The
rule should be $10$~points above the main text. For those using the
\textbf{\LaTeX} style file, the original title is automatically set as running
head using the \texttt{fancyhdr} package which is included in the ICML
2023 style file package. In case that the original title exceeds the
size restrictions, a shorter form can be supplied by using

\verb|\icmltitlerunning{...}|

just before $\mathtt{\backslash begin\{document\}}$.
Authors using \textbf{Word} must edit the header of the document themselves.

\section{Format of the Paper}

All submissions must follow the specified format.

\subsection{Dimensions}

The text of the paper should be formatted in two columns, with an
overall width of 6.75~inches, height of 9.0~inches, and 0.25~inches
between the columns. The left margin should be 0.75~inches and the top
margin 1.0~inch (2.54~cm). The right and bottom margins will depend on
whether you print on US letter or A4 paper, but all final versions
must be produced for US letter size.
Do not write anything on the margins.

The paper body should be set in 10~point type with a vertical spacing
of 11~points. Please use Times typeface throughout the text.

\subsection{Title}

The paper title should be set in 14~point bold type and centered
between two horizontal rules that are 1~point thick, with 1.0~inch
between the top rule and the top edge of the page. Capitalize the
first letter of content words and put the rest of the title in lower
case.

\subsection{Author Information for Submission}
\label{author info}

ICML uses double-blind review, so author information must not appear. If
you are using \LaTeX\/ and the \texttt{icml2023.sty} file, use
\verb+\icmlauthor{...}+ to specify authors and \verb+\icmlaffiliation{...}+ to specify affiliations. (Read the TeX code used to produce this document for an example usage.) The author information
will not be printed unless \texttt{accepted} is passed as an argument to the
style file.
Submissions that include the author information will not
be reviewed.

\subsubsection{Self-Citations}

If you are citing published papers for which you are an author, refer
to yourself in the third person. In particular, do not use phrases
that reveal your identity (e.g., ``in previous work \cite{langley00}, we
have shown \ldots'').

Do not anonymize citations in the reference section. The only exception are manuscripts that are
not yet published (e.g., under submission). If you choose to refer to
such unpublished manuscripts \cite{anonymous}, anonymized copies have
to be submitted
as Supplementary Material via CMT\@. However, keep in mind that an ICML
paper should be self contained and should contain sufficient detail
for the reviewers to evaluate the work. In particular, reviewers are
not required to look at the Supplementary Material when writing their
review (they are not required to look at more than the first $8$ pages of the submitted document).

\subsubsection{Camera-Ready Author Information}
\label{final author}

If a paper is accepted, a final camera-ready copy must be prepared.
For camera-ready papers, author information should start 0.3~inches below the
bottom rule surrounding the title. The authors' names should appear in 10~point
bold type, in a row, separated by white space, and centered. Author names should
not be broken across lines. Unbolded superscripted numbers, starting 1, should
be used to refer to affiliations.

Affiliations should be numbered in the order of appearance. A single footnote
block of text should be used to list all the affiliations. (Academic
affiliations should list Department, University, City, State/Region, Country.
Similarly for industrial affiliations.)

Each distinct affiliations should be listed once. If an author has multiple
affiliations, multiple superscripts should be placed after the name, separated
by thin spaces. If the authors would like to highlight equal contribution by
multiple first authors, those authors should have an asterisk placed after their
name in superscript, and the term ``\textsuperscript{*}Equal contribution"
should be placed in the footnote block ahead of the list of affiliations. A
list of corresponding authors and their emails (in the format Full Name
\textless{}email@domain.com\textgreater{}) can follow the list of affiliations.
Ideally only one or two names should be listed.

A sample file with author names is included in the ICML2023 style file
package. Turn on the \texttt{[accepted]} option to the stylefile to
see the names rendered. All of the guidelines above are implemented
by the \LaTeX\ style file.

\subsection{Abstract}

The paper abstract should begin in the left column, 0.4~inches below the final
address. The heading `Abstract' should be centered, bold, and in 11~point type.
The abstract body should use 10~point type, with a vertical spacing of
11~points, and should be indented 0.25~inches more than normal on left-hand and
right-hand margins. Insert 0.4~inches of blank space after the body. Keep your
abstract brief and self-contained, limiting it to one paragraph and roughly 4--6
sentences. Gross violations will require correction at the camera-ready phase.

\subsection{Partitioning the Text}

You should organize your paper into sections and paragraphs to help
readers place a structure on the material and understand its
contributions.

\subsubsection{Sections and Subsections}

Section headings should be numbered, flush left, and set in 11~pt bold
type with the content words capitalized. Leave 0.25~inches of space
before the heading and 0.15~inches after the heading.

Similarly, subsection headings should be numbered, flush left, and set
in 10~pt bold type with the content words capitalized. Leave
0.2~inches of space before the heading and 0.13~inches afterward.

Finally, subsubsection headings should be numbered, flush left, and
set in 10~pt small caps with the content words capitalized. Leave
0.18~inches of space before the heading and 0.1~inches after the
heading.

Please use no more than three levels of headings.

\subsubsection{Paragraphs and Footnotes}

Within each section or subsection, you should further partition the
paper into paragraphs. Do not indent the first line of a given
paragraph, but insert a blank line between succeeding ones.

You can use footnotes\footnote{Footnotes
should be complete sentences.} to provide readers with additional
information about a topic without interrupting the flow of the paper.
Indicate footnotes with a number in the text where the point is most
relevant. Place the footnote in 9~point type at the bottom of the
column in which it appears. Precede the first footnote in a column
with a horizontal rule of 0.8~inches.\footnote{Multiple footnotes can
appear in each column, in the same order as they appear in the text,
but spread them across columns and pages if possible.}

\begin{figure}[ht]
\vskip 0.2in
\begin{center}
\centerline{\includegraphics[width=\columnwidth]{icml_numpapers}}
\caption{Historical locations and number of accepted papers for International
Machine Learning Conferences (ICML 1993 -- ICML 2008) and International
Workshops on Machine Learning (ML 1988 -- ML 1992). At the time this figure was
produced, the number of accepted papers for ICML 2008 was unknown and instead
estimated.}
\label{icml-historical}
\end{center}
\vskip -0.2in
\end{figure}

\subsection{Figures}

You may want to include figures in the paper to illustrate
your approach and results. Such artwork should be centered,
legible, and separated from the text. Lines should be dark and at
least 0.5~points thick for purposes of reproduction, and text should
not appear on a gray background.

Label all distinct components of each figure. If the figure takes the
form of a graph, then give a name for each axis and include a legend
that briefly describes each curve. Do not include a title inside the
figure; instead, the caption should serve this function.

Number figures sequentially, placing the figure number and caption
\emph{after} the graphics, with at least 0.1~inches of space before
the caption and 0.1~inches after it, as in
\cref{icml-historical}. The figure caption should be set in
9~point type and centered unless it runs two or more lines, in which
case it should be flush left. You may float figures to the top or
bottom of a column, and you may set wide figures across both columns
(use the environment \texttt{figure*} in \LaTeX). Always place
two-column figures at the top or bottom of the page.

\subsection{Algorithms}

If you are using \LaTeX, please use the ``algorithm'' and ``algorithmic''
environments to format pseudocode. These require
the corresponding stylefiles, algorithm.sty and
algorithmic.sty, which are supplied with this package.
\cref{alg:example} shows an example.

\begin{algorithm}[tb]
   \caption{Bubble Sort}
   \label{alg:example}
\begin{algorithmic}
   \STATE {\bfseries Input:} data $x_i$, size $m$
   \REPEAT
   \STATE Initialize $noChange = true$.
   \FOR{$i=1$ {\bfseries to} $m-1$}
   \IF{$x_i > x_{i+1}$}
   \STATE Swap $x_i$ and $x_{i+1}$
   \STATE $noChange = false$
   \ENDIF
   \ENDFOR
   \UNTIL{$noChange$ is $true$}
\end{algorithmic}
\end{algorithm}

\subsection{Tables}

You may also want to include tables that summarize material. Like
figures, these should be centered, legible, and numbered consecutively.
However, place the title \emph{above} the table with at least
0.1~inches of space before the title and the same after it, as in
\cref{sample-table}. The table title should be set in 9~point
type and centered unless it runs two or more lines, in which case it
should be flush left.

\begin{table}[t]
\caption{Classification accuracies for naive Bayes and flexible
Bayes on various data sets.}
\label{sample-table}
\vskip 0.15in
\begin{center}
\begin{small}
\begin{sc}
\begin{tabular}{lcccr}
\toprule
Data set & Naive & Flexible & Better? \\
\midrule
Breast    & 95.9$\pm$ 0.2& 96.7$\pm$ 0.2& $\surd$ \\
Cleveland & 83.3$\pm$ 0.6& 80.0$\pm$ 0.6& $\times$\\
Glass2    & 61.9$\pm$ 1.4& 83.8$\pm$ 0.7& $\surd$ \\
Credit    & 74.8$\pm$ 0.5& 78.3$\pm$ 0.6&         \\
Horse     & 73.3$\pm$ 0.9& 69.7$\pm$ 1.0& $\times$\\
Meta      & 67.1$\pm$ 0.6& 76.5$\pm$ 0.5& $\surd$ \\
Pima      & 75.1$\pm$ 0.6& 73.9$\pm$ 0.5&         \\
Vehicle   & 44.9$\pm$ 0.6& 61.5$\pm$ 0.4& $\surd$ \\
\bottomrule
\end{tabular}
\end{sc}
\end{small}
\end{center}
\vskip -0.1in
\end{table}

Tables contain textual material, whereas figures contain graphical material.
Specify the contents of each row and column in the table's topmost
row. Again, you may float tables to a column's top or bottom, and set
wide tables across both columns. Place two-column tables at the
top or bottom of the page.

\subsection{Theorems and such}
The preferred way is to number definitions, propositions, lemmas, etc. consecutively, within sections, as shown below.
\begin{definition}
\label{def:inj}
A function $f:X \to Y$ is injective if for any $x,y\in X$ different, $f(x)\ne f(y)$.
\end{definition}
Using \cref{def:inj} we immediate get the following result:
\begin{proposition}
If $f$ is injective mapping a set $X$ to another set $Y$,
the cardinality of $Y$ is at least as large as that of $X$
\end{proposition}
\begin{proof}
Left as an exercise to the reader.
\end{proof}
\cref{lem:usefullemma} stated next will prove to be useful.
\begin{lemma}
\label{lem:usefullemma}
For any $f:X \to Y$ and $g:Y\to Z$ injective functions, $f \circ g$ is injective.
\end{lemma}
\begin{theorem}
\label{thm:bigtheorem}
If $f:X\to Y$ is bijective, the cardinality of $X$ and $Y$ are the same.
\end{theorem}
An easy corollary of \cref{thm:bigtheorem} is the following:
\begin{corollary}
If $f:X\to Y$ is bijective,
the cardinality of $X$ is at least as large as that of $Y$.
\end{corollary}
\begin{assumption}
The set $X$ is finite.
\label{ass:xfinite}
\end{assumption}
\begin{remark}
According to some, it is only the finite case (cf. \cref{ass:xfinite}) that is interesting.
\end{remark}

\subsection{Citations and References}

Please use APA reference format regardless of your formatter
or word processor. If you rely on the \LaTeX\/ bibliographic
facility, use \texttt{natbib.sty} and \texttt{icml2023.bst}
included in the style-file package to obtain this format.

Citations within the text should include the authors' last names and
year. If the authors' names are included in the sentence, place only
the year in parentheses, for example when referencing Arthur Samuel's
pioneering work \yrcite{Samuel59}. Otherwise place the entire
reference in parentheses with the authors and year separated by a
comma \cite{Samuel59}. List multiple references separated by
semicolons \cite{kearns89,Samuel59,mitchell80}. Use the `et~al.'
construct only for citations with three or more authors or after
listing all authors to a publication in an earlier reference \cite{MachineLearningI}.

Authors should cite their own work in the third person
in the initial version of their paper submitted for blind review.
Please refer to \cref{author info} for detailed instructions on how to
cite your own papers.

Use an unnumbered first-level section heading for the references, and use a
hanging indent style, with the first line of the reference flush against the
left margin and subsequent lines indented by 10 points. The references at the
end of this document give examples for journal articles \cite{Samuel59},
conference publications \cite{langley00}, book chapters \cite{Newell81}, books
\cite{DudaHart2nd}, edited volumes \cite{MachineLearningI}, technical reports
\cite{mitchell80}, and dissertations \cite{kearns89}.

Alphabetize references by the surnames of the first authors, with
single author entries preceding multiple author entries. Order
references for the same authors by year of publication, with the
earliest first. Make sure that each reference includes all relevant
information (e.g., page numbers).

Please put some effort into making references complete, presentable, and
consistent, e.g. use the actual current name of authors.
If using bibtex, please protect capital letters of names and
abbreviations in titles, for example, use \{B\}ayesian or \{L\}ipschitz
in your .bib file.

\section*{Accessibility}
Authors are kindly asked to make their submissions as accessible as possible for everyone including people with disabilities and sensory or neurological differences.
Tips of how to achieve this and what to pay attention to will be provided on the conference website \url{http://icml.cc/}.

\section*{Software and Data}

If a paper is accepted, we strongly encourage the publication of software and data with the
camera-ready version of the paper whenever appropriate. This can be
done by including a URL in the camera-ready copy. However, \textbf{do not}
include URLs that reveal your institution or identity in your
submission for review. Instead, provide an anonymous URL or upload
the material as ``Supplementary Material'' into the CMT reviewing
system. Note that reviewers are not required to look at this material
when writing their review.

\section*{Acknowledgements}

\textbf{Do not} include acknowledgements in the initial version of
the paper submitted for blind review.

If a paper is accepted, the final camera-ready version can (and
probably should) include acknowledgements. In this case, please
place such acknowledgements in an unnumbered section at the
end of the paper. Typically, this will include thanks to reviewers
who gave useful comments, to colleagues who contributed to the ideas,
and to funding agencies and corporate sponsors that provided financial
support.

\nocite{langley00}

\section{Introduction}

In a series of influential papers, Amari and Wu proposed that one could improve the generalization performance of support vector machine (SVM) classifiers through data-dependent transformations of the kernel to expand the Riemannian volume element near decision boundaries \cite{amari1999improving,wu2002conformal,williams2007geometrical}. This proposal was based on the idea that this local magnification of areas improves class discriminability \cite{cho2011analysis,amari1999improving,burges1999geometry}.

Over the past decade, SVMs have largely been eclipsed by neural networks, whose ability to flexibly learn features from data is believed to underlie their superior generalization performance \cite{lecun2015deep,zhang2021understanding}. Previous works have explored some aspects of the geometry induced by neural networks feature maps with random parameters \cite{poole2016exponential,amari2019statistical,cho2009kernel,cho2011analysis,zv2022capacity,hauser2017principles,benfenati2023singular}, but have not characterized data-dependent changes in representational geometry over training.

In this work, we explore the possibility that neural networks learn to enhance local input disciminability automatically over the course of training. Our primary contributions are:
\begin{itemize}[leftmargin=*]
    \item
    In \S \ref{sec:fixedweights}, we study general properties of the metric induced by shallow fully-connected neural networks. Next, in \S \ref{sec:nngp}, we compute the volume element and curvature of the metric induced by infinitely wide shallow networks with Gaussian weights and smooth activation functions, showing that it is spherically symmetric.

    \item
    In \S \ref{sec:experiments}, we empirically show that training shallow networks on simple classification tasks expands the volume element along decision boundaries, consistent with the hand-engineered modifications proposed by Amari and Wu. In \S \ref{sec:deep}, we provide evidence that deep residual networks trained on more complex tasks behave similarly.

\end{itemize}
In total, our results provide a preliminary picture of how feature learning shapes local input discriminability.

\section{Preliminaries}\label{sec:preliminaries}

We begin by introducing the basic idea of the Riemannian geometry of feature space representations. Our setup and notation largely follow \citet{burges1999geometry}, which in turn follows the conventions of \citet{dodson1991tensor}.

\subsection{Feature embeddings as Riemannian manifolds}

Consider $d$-dimensional data living in some submanifold $\mathcal{D} \subseteq \mathbb{R}^{d}$. Let the \emph{feature map} $\mathbf{\Phi}: \mathbb{R}^{d} \to \mathcal{H}$ be a map from $\mathbb{R}^{d}$ to some separable Hilbert space $\mathcal{H}$ of possibly infinite dimension $n$, with $\mathbf{\Phi}(\mathcal{D}) = \mathcal{M} \subseteq \mathcal{H}$. We index input space dimensions by Greek letters $\mu,\nu,\rho,\ldots \in [d]$ and feature space dimensions by Latin letters $i,j,k, \ldots \in [n]$. We use the Einstein summation convention; summation over all repeated indices is implied.

Assume that $\mathbf{\Phi}$ is $\mathcal{C}^{k}$ for $k \geq 3$, and is everywhere of rank $r=\min\{d,n\}$. If $r = d$, then $\mathcal{M}$ is a $d$-dimensional $\mathcal{C}^{k}$ manifold immersed in $\mathcal{H}$. If $k = \infty$, then $\mathcal{M}$ is a smooth manifold. In contrast, if $r < d$, then $\mathcal{M}$ is a $d$-dimensional $\mathcal{C}^{k}$ manifold submersed in $\mathcal{H}$. The flat metric on $\mathcal{H}$ can then be pulled back to $\mathcal{M}$, with components
\begin{align} \label{eqn:general_metric}
    g_{\mu\nu} = \partial_{\mu} \Phi_{i} \partial_{\nu} \Phi_{i},
\end{align}
where we write $\partial_{\mu} \equiv \partial/\partial x^{\mu}$. If $r=d$ and the pullback metric $g_{\mu\nu}$ is full rank, then $(\mathcal{M},g)$ is a $d$-dimensional Riemannian manifold \cite{dodson1991tensor,burges1999geometry}. However, if the pullback $g_{\mu\nu}$ is a degenerate metric, as must be the case if $r < d$, then $(\mathcal{M},g)$ is a singular semi-Riemannian manifold \cite{benfenati2023singular,kupeli2013singular}. In this case, if we let $\sim$ be the equivalence relation defined by identifying points with vanishing pseudodistance, the quotient $(\mathcal{M}/\sim,g)$ is a Riemannian manifold \cite{benfenati2023singular}. Unless noted otherwise, our results will focus on the non-singular case. We denote the matrix inverse of the metric tensor by $g^{\mu\nu}$, and we raise and lower input space indices using the metric.

If we define the feature kernel $k(\mathbf{x},\mathbf{y}) = \Phi_{i}(\mathbf{x}) \Phi_{i}(\mathbf{y})$ for $\mathbf{x},\mathbf{y} \in \mathcal{D}$, then the resulting metric can be written in terms of the kernel as $g_{\mu\nu} = (1/2) \partial_{x_\mu} \partial_{x_\nu} k(\mathbf{x},\mathbf{x}) - [\partial_{y_{\mu}} \partial_{y_{\nu}} k(\mathbf{x},\mathbf{y})]_{\mathbf{y} = \mathbf{x}}$.
This formula applies even if $n = \infty$, giving the metric induced by the feature embedding associated to a suitable Mercer kernel \cite{burges1999geometry}.

\subsection{Volume element and curvature}

With this setup, $(\mathcal{M},g)$ is a Riemannian manifold, hence we have at our disposal a powerful toolkit with which we may study its geometry. We will focus on two geometric properties of $(\mathcal{M},g)$. First, the volume element is given by
\begin{align}
    dV = \sqrt{\det g}\,  d^{d}x,
\end{align}
where the factor $\sqrt{\det g}$ measures how local areas in input space are magnified by the feature map \cite{dodson1991tensor,amari1999improving,burges1999geometry}. Second, we consider the intrinsic curvature of the manifold, which is characterized by the Riemann tensor $R^{\mu}_{\nu\alpha\beta}$ \cite{dodson1991tensor}.
If $R^{\mu}_{\nu\alpha\beta}= 0$, then the manifold is intrinsically flat. As a tractable measure, we focus on the Ricci curvature scalar $R = g^{\beta \nu} R^{\alpha}_{\nu \alpha \beta}$, which measures the deviation of the volume of an infinitesimal geodesic ball in the manifold from that in flat space \cite{dodson1991tensor}.

In the singular case, we can compute the volume element on $\mathcal{M} / \sim $ at a given point by taking the square root of the product of the non-zero eigenvalues of the degenerate metric $g_{\mu\nu}$ at that point \cite{benfenati2023singular}. However, the curvature in this case is generally not straightforward to compute; we will therefore leave this issue for future work.

\subsection{Shallow neural network feature maps}

In this work, we consider a particular class of feature maps: those given by the hidden layer representations of neural networks \cite{williams1997computing,neal1996priors,lecun2015deep,cho2009kernel,cho2011analysis,lee2018deep,matthews2018gaussian,hauser2017principles,benfenati2023singular}. We will mostly focus on shallow fully-connected neural networks, i.e., those with only a single hidden layer followed by readout. Concretely, such a feature map is of the form
\begin{align} \label{eqn:shallow_featuremap}
    \Phi_{j}(\mathbf{x}) = n^{-1/2} \phi(\mathbf{w}_{j} \cdot \mathbf{x} + b_{j})
\end{align}
for weights $\mathbf{w}_{j}$, biases $b_{j}$, and an activation function $\phi$. For convenience, we abbreviate the Euclidean inner product on feature or input space by $\cdot$, e.g., $\mathbf{w} \cdot \mathbf{x} = w_{\mu} x_{\mu}$. In this case, the feature space dimension $n$ is equal to the number of hidden units, and is referred to as the \emph{width} of the hidden layer. In \eqref{eqn:shallow_featuremap}, we scale the components of the feature map by $n^{-1/2}$ such that the associated kernel $k(\mathbf{x},\mathbf{y}) = \Phi_{i}(\mathbf{x}) \Phi_{i}(\mathbf{y})$ and metric \eqref{eqn:shallow_metric} have the form of averages over hidden units, and therefore should be well-behaved at large widths \cite{neal1996priors,williams1997computing}.

We will assume that $\phi$ is $\mathcal{C}^{k}$ for $k \geq 3$, so that this feature map satisfies the smoothness conditions required in the setup above. We will also assume that the activation function and weight vectors are such that the Jacobian $\partial_{\mu} \Phi_{j}$ is full-rank, i.e., is of rank $\min\{d,n\}$. Then, the shallow network feature map satisfies the required conditions for the feature embedding to be a (possibly singular) Riemannian manifold. These conditions extend directly to deep fully-connected networks formed by composing feature maps of the form \eqref{eqn:shallow_featuremap} \cite{hauser2017principles,benfenati2023singular}.

\section{Related works} \label{sec:related}

Having established the geometric preliminaries of \S\ref{sec:preliminaries}, we can give a more complete overview of related works. As introduced above, our hypothesis for how the Riemannian geometry of neural network representations changes during training is directly inspired by the work of \citet{amari1999improving}. In that and subsequent works \cite{amari1999improving,wu2002conformal,williams2007geometrical}, they proposed to modify the kernel of an SVM as $\tilde{k}(\mathbf{x},\mathbf{y}) = h(\mathbf{x}) h(\mathbf{y}) k(\mathbf{x},\mathbf{y})$
for some positive scalar function $h(\mathbf{x})$ chosen such that the magnification factor $\sqrt{\det g}$ is large near the SVM's decision boundary. Concretely, they proposed to fit an SVM with some base kernel $k$, choose $h(\mathbf{x}) = \sum_{\mathbf{v} \in \textrm{SV}(k)} \exp[-\Vert \mathbf{x}-\mathbf{v}\Vert^2/2 \tau^2]$
for $\tau$ a bandwidth parameter and $\textrm{SV}(k)$ the set of support vectors for $k$, and then fit an SVM with the modified kernel $\tilde{k}$. Here, $\Vert \cdot \Vert$ denotes the Euclidean norm. This process could then be iterated, yielding a sequence of modified kernels. They found that this method can improve generalization performance relative to the original kernel \cite{amari1999improving,wu2002conformal,williams2007geometrical}. This approach is a hand-designed form of iterative feature learning.

The geometry induced by common kernels was investigated in detail by \citet{burges1999geometry}, who established a broad range of technical results. He showed that translation-invariant kernels of the form $k(\mathbf{x},\mathbf{y}) = k(\Vert \mathbf{x}-\mathbf{y} \Vert^2)$ yield flat, constant metrics,\footnote{It is interesting to note that this holds for the simplest form of a method for learning data-adaptive  kernels recently proposed by \citet{radhakrishnan2022recursive}; see Appendix \ref{app:invariant}.} and gave a detailed characterization of polynomial kernels $k(\mathbf{x},\mathbf{y}) = (\mathbf{x} \cdot \mathbf{y})^{q}$. \citet{cho2011analysis} subsequently analyzed the geometry induced by arc-cosine kernels, i.e., the feature kernels of infinitely-wide shallow neural networks with threshold-power law activation functions $\phi(x) = \max\{0,x\}^q$ and random parameters \cite{cho2009kernel}. Our results on infinitely-wide networks for general smooth activation functions build on these works.

The representational geometry of deep networks with random Gaussian parameters in the limit of large width and depth was studied by \citet{poole2016exponential}, and in later work by \citet{amari2019statistical}.
These works tie into a broader line of research on infinite-width limits of deep neural networks in which inference and prediction is captured by a kernel machine \cite{neal1996priors,williams1997computing,daniely2016deeper,lee2018deep,matthews2018gaussian,yang2019scaling,yang2021feature,zv2021asymptotics,zv2022capacity,bordelon2022selfconsistent}. Our results on the representational geometry of wide shallow networks with smooth activation functions build on these ideas, particularly those relating activation function derivatives to input discriminability \cite{poole2016exponential,daniely2016deeper,zv2021activation,zv2022capacity}.

Particularly closely related to our work are several recent papers that aim to study the curvature of neural network representations. \citet{hauser2017principles,benfenati2023singular} discuss formal principles of Riemannian geometry in deep neural networks, but do not characterize how training shapes the geometry. \citet{kaul2020riemannian} aimed to study the curvature of metrics induced by the outputs of pretrained classifiers. However, their work is limited by the fact that they estimate input-space derivatives using inexact finite differences under the strong assumption that the input data is confined to a \emph{known} smooth submanifold of $\mathbb{R}^{d}$. Recent works by \citet{shao2018riemannian,kuhnel2018latent,wang2021generative} have studied the Riemannian geometry of the latent representations of deep generative models. Finally, in very recent work \citet{benfenati2023reconstruction} have used the geometry induced by the full input-output mapping to reconstruct iso-response curves of deep networks. In contrast, our work focuses on hidden representations, and seeks to characterize the representational manifolds themselves.

\section{Representational geometry of shallow neural network feature maps}\label{sec:fixedweights}

We begin by studying general properties of the Riemannian metrics induced by shallow neural networks feature maps.

\subsection{Finite-width networks}

We first consider finite-width networks with fixed weights, assuming that $n \geq d$.  Writing $z_{j} = \mathbf{w}_{j} \cdot \mathbf{x} + b_{j}$ for the preactivation of the $j$-th hidden unit, the general formula \eqref{eqn:general_metric} for the metric yields
\begin{align} \label{eqn:shallow_metric}
    g_{\mu\nu} = \frac{1}{n} \phi'(z_{j})^{2} w_{j \mu} w_{j \nu} .
\end{align}
This metric has the useful property that $\partial_{\alpha} g_{\mu\nu}$ is symmetric under permutation of its indices, hence the formula for the Riemann tensor simplifies substantially (Appendix \ref{app:riemann}).

Then, using the Leibniz formula for determinants, we show in Appendix \ref{app:fixedweights} that the determinant of the metric can be expanded as a sum over $d$-tuples of hidden units:
\begin{align} \label{eqn:finite_volume}
    \det g = \frac{1}{n^{d} d!} M_{j_{1}\cdots j_{d}}^{2} \phi'(z_{j_{1}})^2 \cdots \phi'(z_{j_{d}})^2  ,
\end{align}
where
\begin{align}
    M_{j_{1}\cdots j_{d}} = \det
    \begin{pmatrix}
        w_{j_{1} 1} & \cdots & w_{j_{1} d} \\
        \vdots & \ddots & \vdots \\
        w_{j_{d} 1} & \cdots & w_{j_{d} d}
    \end{pmatrix}
\end{align}
is the minor of the weight matrix obtained by selecting units $j_{1}, \ldots, j_{d}$.
For the error function $\phi(x) = \erf(x/\sqrt{2})$, $\det g$ expands as a superposition of Gaussian bump functions, one for each tuple of hidden units \eqref{eqn:fixed_erf}. This is reminiscent of Amari and Wu's approach, which yields a Gaussian contribution to $\sqrt{\det g}$ from each support vector (\S \ref{sec:related}).

We can also derive similar expansions for the Riemann tensor and Ricci scalar. The resulting expressions are rather unwieldy, so we give their general forms only in Appendix \ref{app:fixedweight_riemann}. However, in two dimensions the situation simplifies, as the Riemann tensor is completely determined by the Ricci scalar \cite{dodson1991tensor,mtw2017gravitation} (Appendix \ref{app:2d}). In this case, we have the compact expression
\begin{align} \label{eqn:finite_ricci}
     (\det g)^2 R &=  - \frac{3}{n^3} M_{jk}^{2} M_{ij} M_{ik} \nonumber\\&\quad \times \phi'(z_{i})^{2} \phi'(z_{j}) \phi'(z_{k}) \phi''(z_{j}) \phi''(z_{k}) .
\end{align}
This shows that in $d=2$ the curvature acquires contributions from each triple of distinct hidden units, hence if $n=2$ we have $R=0$. This follows from the fact that the feature map is in this case a change of coordinates on the two-dimensional manifold \cite{dodson1991tensor}.

\subsection{Geometry of infinite shallow networks}\label{sec:nngp}

We now characterize the metric induced by infinite-width networks ($n \to \infty$) with Gaussian weights and biases
\begin{align}
    \mathbf{w}_{j} \sim_{\textrm{i.i.d.}} \mathcal{N}(\mathbf{0},\sigma^2 \mathbf{I}_{d}); \quad
    b_{j} \sim_{\textrm{i.i.d.}} \mathcal{N}(0,\zeta^2) ,
\end{align}
as commonly chosen at initialization \cite{lecun2015deep,lee2018deep,matthews2018gaussian,yang2019scaling,yang2021feature,poole2016exponential}. For such networks, the hidden layer representation is described by the neural network Gaussian process (NNGP) kernel \cite{neal1996priors,williams1997computing,matthews2018gaussian,lee2018deep}:
\begin{align}
    k(\mathbf{x},\mathbf{y}) &= \lim_{n \to \infty} n^{-1} \bm{\Phi}(\mathbf{x}) \cdot \bm{\Phi}(\mathbf{y}) \nonumber \\
    &= \mathbb{E}_{\mathbf{w},b}[\phi(\mathbf{w} \cdot \mathbf{x} + b) \phi(\mathbf{w} \cdot \mathbf{y} + b) ] .
\end{align}
This kernel also completely describes the representation after training for networks in the lazy regime \cite{yang2021feature,bordelon2022selfconsistent}.

In Appendix \ref{app:shallow_nngp}, we show that the metric associated with the NNGP kernel, $g_{\mu\nu} = \mathbb{E}_{\mathbf{w},b}[\phi'(\mathbf{w} \cdot \mathbf{x} + b)^2 w_{\mu} w_{\nu}]$, can be written more illuminatingly as
\begin{align}
    g_{\mu\nu} = e^{\Omega(\Vert \mathbf{x} \Vert^2)} [\delta_{\mu\nu} + 2 \Omega'(\Vert \mathbf{x} \Vert^2) x_{\mu} x_{\nu} ] ,
\end{align}
where the function $\Omega(\Vert\mathbf{x}\Vert^2)$ is defined via
\begin{align}
    e^{\Omega(\Vert \mathbf{x} \Vert^2)} = \sigma^{2} \mathbb{E}_{z \sim \mathcal{N}(0,\sigma^2 \Vert \mathbf{x} \Vert^2 + \zeta^2)}[\phi'(z)^2] .
\end{align}
For these results, we must also assume that $\phi$ and its (weak) derivatives satisfy suitable boundedness assumptions for $\Omega$ to be twice-differentiable \cite{daniely2016deeper}. Therefore, like the metrics induced by other dot-product kernels, the NNGP metric has the form of a projection \cite{burges1999geometry}. Such metrics have determinant
\begin{align}
    \det g = e^{\Omega d} (1 + 2 \Vert \mathbf{x} \Vert^2 \Omega')
\end{align}
and Ricci scalar
\begin{align}
    R &= - \frac{3 (d-1) e^{-\Omega} (\Omega')^2 \Vert \mathbf{x} \Vert^2}{(1 + 2 \Vert \mathbf{x} \Vert^2 \Omega')^2} \nonumber\\&\qquad \times \left[ d + 2 + 2 \Vert \mathbf{x} \Vert^2 \left( (d-2) \Omega' + 2 \frac{\Omega''}{\Omega'} \right)  \right] .
\end{align}

Thus, all geometric quantities are spherically symmetric, depending only on $\Vert \mathbf{x} \Vert^2$. Thanks to the assumption of independent Gaussian weights, the geometric quantities associated to the shallow Neural Tangent Kernel and to the deep NNGP will share this spherical symmetry (Appendix \ref{app:ntk}) \cite{lee2018deep,matthews2018gaussian,yang2019scaling,yang2021feature}. This generalizes the results of \citet{cho2011analysis} for threshold-power law functions to arbitrary smooth activation functions.
The relation between Gaussian norms of $\phi'$ and input discriminability indicated by this result is consistent with previous studies \cite{poole2016exponential,daniely2016deeper,zv2021activation,zv2022capacity}. In short, unless the task depends only on the input norm, the geometry of infinite-width networks will not be linked to the task structure.

\begin{figure}[t]
    \centering
    \includegraphics[width=\columnwidth]{TheoryFigures/paper_curvature_2d_convergence_to_nngp_v0.pdf}
    \vspace{-2em}
    \caption{Convergence of geometric quantities for finite-width networks with Gaussian random parameters to the infinite-width limit. \textbf{a}. The magnification factor $\sqrt{\det g}$ (\emph{left}) and Ricci scalar $R$ (\emph{right}) as functions of the input norm $\Vert \mathbf{x} \Vert$ for networks with $\phi(x) = \erf(x/\sqrt{2})$. Empirical results for finite networks, computed using \eqref{eqn:finite_volume} and \eqref{eqn:2dricci} are shown in blue, with solid lines showing the mean and shaded patches the standard deviation over 25 realizations of random Gaussian parameters. In all cases, $\sigma = \zeta = 1$. The infinite-width result is shown as a black dashed line. \textbf{b}. As in \textbf{a}, but for normalized quadratic activation functions $\phi(x) = x^2/\sqrt{3}$. }
    \label{fig:convergence_to_nngp}
    \vspace{-1em}
\end{figure}

\subsection{Examples}

In Appendix \ref{sec:nngp_examples}, we evaluate the geometric quantities of the NNGP for certain analytically tractable activation functions. The resulting expressions for $\sqrt{\det g}$ and $R$ are rather lengthy, so we discuss only their qualitative behavior here. For the error function $\phi(x) = \erf(x/\sqrt{2})$, $R$ is negative for all $d>1$, and both $R$ and $\sqrt{\det g}$ are monotonically decreasing functions of $\Vert \mathbf{x} \Vert$ for all $\zeta$ and $\sigma$. For monomials $\phi(x) \propto x^{q}$ for integer $q > 1$, $\sqrt{ \det g}$ is a monotonically increasing function of $\Vert \mathbf{x} \Vert^2$, while $R$ is again non-positive. However, in this case the behavior of $R$ depends on whether or not bias terms are present: if $\zeta = 0$, then $R$ is a non-decreasing function of $\Vert \mathbf{x} \Vert^2$ that diverges towards $-\infty$ as $\Vert \mathbf{x} \Vert^2 \downarrow 0$, while if $\zeta > 0$, $R$ may be non-monotonic in $\Vert \mathbf{x} \Vert^2$. In Figure \ref{fig:convergence_to_nngp}, we illustrate this behavior, and show convergence of the empirical geometry of finite networks with random Gaussian parameters to the infinite-width results.

\begin{figure*}[t]
    \centering
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/expansion_xor_w5_nlSigmoid_lr0.1_wd0.0_mom0.9_sindata_epochs10000_seed405_e200.pdf}
    \end{subfigure}
    \hfill
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/expansion_xor_w5_nlSigmoid_lr0.1_wd0.0_mom0.9_sindata_epochs10000_seed405_e5000.pdf}
    \end{subfigure}
    \hfill
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/expansion_xor_w5_nlSigmoid_lr0.1_wd0.0_mom0.9_sindata_epochs10000_seed405_e10000.pdf}
    \end{subfigure} \\

    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/expansion_xor_w20_nlSigmoid_lr0.1_wd0.0_mom0.9_sindata_epochs10000_seed401_e200.pdf}
    \end{subfigure}
    \hfill
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/expansion_xor_w20_nlSigmoid_lr0.1_wd0.0_mom0.9_sindata_epochs10000_seed401_e5000.pdf}
    \end{subfigure}
    \hfill
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/expansion_xor_w20_nlSigmoid_lr0.1_wd0.0_mom0.9_sindata_epochs10000_seed401_e10000.pdf}
    \end{subfigure} \\

    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/expansion_xor_w250_nlSigmoid_lr0.1_wd0.0_mom0.9_sindata_epochs10000_seed401_e200.pdf}
    \end{subfigure}
    \hfill
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/expansion_xor_w250_nlSigmoid_lr0.1_wd0.0_mom0.9_sindata_epochs10000_seed401_e5000.pdf}
    \end{subfigure}
    \hfill
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/expansion_xor_w250_nlSigmoid_lr0.1_wd0.0_mom0.9_sindata_epochs10000_seed401_e10000.pdf}
    \end{subfigure} \\
    \caption{Evolution of the volume element over training in a network trained to classify points separated by a sinusoidal boundary $y=\frac{3}{5}\sin(7x - 1)$ (single hidden layer with 5 hidden units (top), 20 hidden units (mid), and 250 hidden units (bottom)). Red lines indicate the decision boundaries of the network. See Appendix \ref{app:xor} for experimental details. More hidden units offer better approximation to the sinusoid curve. }
    \label{fig:sinusoid}
\end{figure*}

\section{Changes in shallow network geometry during training}\label{sec:experiments}

We now consider how the geometry of the pullback metric changes during training. Changes in the volume element and curvature during gradient descent training are challenging to study analytically, because models for which the learning dynamics are solvable---deep linear networks \cite{saxe2013exact}---trivially yield flat, constant metrics. More generally, the dependence of the metric on the instantaneous configuration of parameters makes it difficult to gain intuition for its evolution over training, even for two-dimensional inputs.

\subsection{Wide Bayesian neural networks}

We can make slightly more analytical progress for Bayesian neural networks at large but finite width. This setting is convenient because there is a fixed parameter posterior; one does not need to solve kernel or metric dynamics through time \cite{bordelon2022selfconsistent}. In Appendix \ref{app:bayesian}, we use recent results on perturbative feature-learning corrections to the NNGP kernel \cite{zv2021asymptotics,roberts2022principles} to compute corrections to the posterior mean of the volume element. In general, it is not possible to evaluate these corrections in closed form \cite{zv2021asymptotics}. For networks with monomial activation functions, no bias terms, and linear readout constrained to interpolate a single training example $(\mathbf{x}_{a},\mathbf{y}_{a})$, we can show that the correction to $\sqrt{\det g}$ is maximal for $\mathbf{x} \parallel \mathbf{x}_{a}$, and minimal for $\mathbf{x} \perp \mathbf{x}_{a}$ \eqref{eqn:bayesian_monomial_detg}. The sign of the correction is positive or negative depending on whether the second moment of the prior predictive is greater than or less than the norm of the output, respectively. For example, if we train on a single point from the XOR task, $(1,1) \mapsto 0$, $\sqrt{\det g}$ will be contracted maximally along $x_1 = x_2$. This simple case shows how interpolating a single point shapes the network's global representational geometry.

\begin{figure*}[t]
    \centering
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/effvols_mnist_w2000_nlSigmoid_lr0.001_wd0.0001_mom0.9_mnist_epochs200_seed404_7_6_e0.pdf}
    \end{subfigure}
    \hfill
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/effvols_mnist_w2000_nlSigmoid_lr0.001_wd0.0001_mom0.9_mnist_epochs200_seed404_7_6_e60.pdf}
    \end{subfigure}
    \hfill
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/effvols_mnist_w2000_nlSigmoid_lr0.001_wd0.0001_mom0.9_mnist_epochs200_seed404_7_6_e180.pdf}
    \end{subfigure} \\

    \begin{subfigure}
    \centering
        \includegraphics[height=2.5in]{Figures/effvols_mnist_plane_w2000_nlSigmoid_lr0.001_wd0.0001_mom0.9_mnist_epochs200_seed404_7_6_1_ternary_e200.pdf}
    \end{subfigure} \\

    \caption{\emph{Top panel}: $\log_{10}(\sqrt{\det g})$ induced at interpolated images between 7 and 6 by a single-hidden-layer fully-connected network trained to classify MNIST digits. \emph{Bottom panel}: Digit class predictions and $\log_{10}(\sqrt{\det g})$ for the plane spanned by MNIST digits 7, 6, and 1 at the final training epoch (200) . Sample images are visualized at the endpoints and midpoint for each set. Each line is colored by its prediction at the interpolated region and end points. As training progresses, the volume elements bulge in the middle (near the decision boundary) and taper off when travelling towards endpoints. See Appendix \ref{app:mnist} for experimental details and Figure \ref{fig:more_mnist} for images interpolated between other digits.}
    \label{fig:mnist}
\end{figure*}

\subsection{Changes in representational geometry for networks trained on two-dimensional toy tasks}

Thus, given the intractability of studying changes in geometry analytically, we resort to numerical experiments. For details of our numerical methods, see Appendix \ref{app:numerics}.

To build intuition, we first consider networks trained on simple two-dimensional tasks, for which we can directly visualize the input space. We first consider a simple two-dimensional binary classification task with sinusoidal boundary, inspired by that considered in the original work of \citet{amari1999improving}. We train networks with sigmoidal activation functions of varying widths to perform this task, and visualize the resulting geometry over the course of training in Figure \ref{fig:sinusoid}. At initialization, the peaks in the volume element lack a clear relation to the structure of the task, with approximate rotational symmetry at large widths as we would expect from \S\ref{sec:nngp}. As the network's decision boundary is gradually molded to conform to the true boundary, the volume element develops peaks in the same vicinity. At all widths, the final volume elements are largest near the peaks of the sinusoidal decision boundary. At small widths, the shape of the sinusoidal curve is not well-resolved, but at large widths there is a clear peak in the close neighborhood of the decision boundary. This result is consistent with the proposal of \citet{amari1999improving}.

In Appendix \ref{app:numerics}, Figure \ref{fig:sinusoid_ricci}, we plot the Ricci curvature for these trained networks. Even for these small networks, the curvature computation is computationally expensive and numerically challenging. Over training, it evolves dynamically, with task-adapted structure visible at the end of training. However, the patterns here are harder to interpret than those in the volume element.

\subsection{Changes in representational geometry for shallow networks trained to classify MNIST digits}

We now provide evidence that a similar phenomenon is present in networks trained to classify MNIST images. We give details of these networks in Appendix \ref{app:mnist}; note that all reach above 95\% train and test accuracy within 200 epochs.

In Figure \ref{fig:mnist}, we plot the induced volume element at synthetic images generated by linearly interpolating between two input images (see Appendix \ref{app:numerics} for details). We emphasize that linear interpolation in pixel space of course does not respect the structure of the image data, and results in unrealistic images. However, this approach has the advantage of being straightforward, and also illustrates how small Euclidean perturbations are expanded by the feature map \cite{novak2018sensitivity}. At initialization, the volume element varies without clear structure along the interpolated path. However, as training progresses, areas near the center of the path, which roughly aligns with the decision boundary, are expanded, while those near the endpoints defined by true training examples remain relatively small. This is again consistent with the proposal of \citet{amari1999improving}. We provide additional visualizations of this behavior in Appendix \ref{app:mnist}.

To gain an understanding of the structure of the volume element beyond one-dimensional slices, in Figure \ref{fig:mnist} we also plot its value in the plane spanned by three randomly-selected example images, at points interpolated linearly within their convex hull. Here, we only show the end of training; in Appendix \ref{app:mnist} we show how the volume element in this plane changes over the course of training. The edges of the resulting ternary plot are one-dimensional slices like those shown in the top row of Figure \ref{fig:mnist}, and we observe consistent expansion of the volume element along these paths. The volume element becomes large near the centroid of the triangle, where multiple decision boundaries intersect.

Because of the computational complexity of estimating the curvature---the Riemann tensor has $d^2(d^2-1)/12$ independent components \cite{mtw2017gravitation,dodson1991tensor}---and its numerical sensitivity (Appendix \ref{app:xor}), we do not attempt to estimate it for this high-dimensional task.

\begin{figure*}[t]
    \centering
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/eff_vol_e0.pdf}
    \end{subfigure}
    \hfill
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/eff_vol_e50.pdf}
    \end{subfigure}
    \hfill
    \begin{subfigure}
        \centering
        \includegraphics[width=0.32\textwidth]{Figures/eff_vol_e200.pdf}
    \end{subfigure} \\

    \begin{subfigure}
        \centering
        \includegraphics[height=2.5in]{Figures/effvols_cifar10_plane_w34_nlGELU_lr0.01_wd0.0001_mom0.9_cifar10_epochs200_seed501_7_6_1_ternary_e200.pdf}
    \end{subfigure}

    \caption{\emph{Top panel}: $\log_{10}(\sqrt{\det g})$ induced at interpolated images between a horse and a frog by ResNet34 trained to classify CIFAR-10 digits.  \emph{Bottom panel}: Digits classification of a horse, a frog, and a car. The volume element is the largest at the intersection of several binary decision boundaries, and smallest within each of the decision region. The one-dimensional slices along the edges of each ternary plot are consistent with the top panel. See Appendix \ref{app:resnet} for experimental details, Figure \ref{fig:more_cifar} for linear interpolation and plane spanned by other classes, and how the plane evolves during training.}
    \label{fig:resnet}
\end{figure*}

\section{Extensions to deep networks}\label{sec:deep}

Thus far, we have focused on the geometry of the feature maps of single-hidden-layer neural networks. However, these analyses can also be applied to deeper networks, regarding the representation at each hidden layer as defining a feature map, and study how the geometry changes with depth \cite{hauser2017principles,benfenati2023singular}. As a simple version of this, in Figure \ref{fig:deep_sinusoid_full} we consider a network with three fully-connected hidden layers trained on the sinusoid task. The metrics induced by the feature maps of all three hidden layers all show the same qualitative behavior as we observed in the shallow case in Figure \ref{fig:sinusoid}: areas near the decision boundary are magnified. As one progresses deeper into the network, the contrast between regions of low and high magnification factor increases.

As a more realistic example, we consider deep residual networks (ResNets) \cite{he2016residual} trained to classify the CIFAR-10 image dataset \cite{krizhevsky2009cifar}. To make the feature map differentiable, we replace the rectified linear unit (ReLU) activation functions used in standard ResNets with Gaussian error linear units (GELUs) \cite{hendrycks2016gelu}. With this modification, we achieve comparable test accuracy (92\%) with a ResNet-34---the largest model we can consider given computational constraints---to that obtained with ReLUs (Appendix \ref{app:resnet}). Importantly, the feature map defined by the input-to-final-hidden-layer mapping of a ResNet-34 gives a submersion of CIFAR-10, as the input images have $32 \times 32 \times 3 = 3072$ pixels, while the final hidden layer has 512 units. Empirically, we find that the Jacobian of this mapping is full-rank (Appendix \ref{app:resnet}); we therefore consider the volume element on $(\mathcal{M}/\sim, g)$ defined by the product of the non-zero eigenvalues of the degenerate pullback metric (\S\ref{sec:preliminaries}, Appendix \ref{app:resnet}).

In Figures \ref{fig:resnet}, we visualize the resulting geometry in the same way we did for networks trained on MNIST, along 1-D interpolated slices and in a 2-D interpolated plane (see Appendix \ref{app:resnet} for details and additional figures). In the 1-D slices, we see a clear trend of large volume elements near decision boundaries, as we observed for shallow networks. However, in two dimensions the picture is less clear. The decision boundaries are more complicated than for MNIST, reflecting the more complex structure of the task. This also highlights the deficiency of our approach of linear interpolation in pixel space, which we further discuss and illustrate in Appendix \ref{app:resnet}. We observe some magnification of areas in the vicinity of decision boundaries, though here it is harder to interpret all forms of structure that are present. Thus, even in this more realistic setting, we observe shaping of geometry over training that appears consistent with the proposal of \citet{amari1999improving}.

\section{Discussion}\label{sec:discussion}

To conclude, we have shown that training on simple tasks shapes the Riemannian geometry induced by neural network representations by magnifying areas along decision boundaries, consistent with the proposal of Amari and Wu for geometrically-inspired kernel learning \cite{amari1999improving,wu2002conformal,williams2007geometrical}. Our results on the geometry induced by the NNGP kernel provide a preliminary picture of the geometric priors of neural networks, and our experimental results begin to show how representational geometry is shaped over the course of training. These results are relevant to the broad goal of leveraging non-Euclidean geometry in deep learning \cite{bronstein2021geometric,weber2020hyperbolic}. We now discuss the limitations of our work, as well as directions for future study.

Perhaps the most important limitation of our work is the fact that we focus either on toy tasks with two-dimensional input domains, or on low-dimensional slices through high-dimensional domains. This is a fundamental limitation of how we have attempted to visualize the geometry. We are also restricted by computational constraints (see in particular Appendix \ref{app:resnet}); to characterize the geometry of state-of-the-art network architectures, more efficient and numerically stable algorithms for computing these quantities must be developed.

An important question that we leave open for future work is whether expanding areas near decision boundaries generically improves generalization in deep neural networks, consistent with \citet{amari1999improving}'s original motivations. Indeed, it is easy to imagine a scenario in which the geometry is overfit, and the trained network becomes too sensitive to small changes in the input. This possibility is consistent with prior work on the sensitivity of deep networks \cite{novak2018sensitivity}, and with the related phenomenon of adversarial vulnerability \cite{szegedy2013intriguing,goodfellow2014explaining}. Investigating these links will be an important objective for future work.

Because of the smoothness conditions required by the definition of the pullback metric and the requirement that $(\mathcal{M},g)$ be a differentiable manifold \cite{hauser2017principles,benfenati2023singular}, the approach pursued in this work does not apply directly to networks with ReLU activation functions, which are not differentiable. Deep ReLU networks are continuous piecewise-linear maps, with many distinct activation regions \cite{hanin2019complexity,hanin2019few}. Within each region, the corresponding linear feature map will induce a flat metric on the input space, but the magnification factor will vary from region to region. It will be interesting to investigate the resulting geometry in future work.

One possible area of application of our results is to the general problem of how to analyze and compare neural network representations \cite{kornblith2019similarity,williams2021generalized}. Importantly, one could compute and plot the volume element induced by a feature map even when one does not have access to explicit class labels. This could allow one to study networks trained with self-supervised learning, or even differentiable approximations to biological neural networks \cite{wang2022tuning}. Exploring the rich geometry induced by these networks is an exciting avenue for future investigation.

\section*{Acknowledgements}

We thank Alexander Atanasov and Blake Bordelon for helpful comments on our manuscript. JAZV, CP and this research were supported by a Google Faculty Research Award and NSF DMS-2134157. The computations in this paper were run on the FASRC Cannon cluster supported by the FAS Division of Science Research Computing Group at Harvard University.

\newpage