text:
# Hilbert–Schmidt volume of the set of mixed quantum states

## I Introduction

Although the notion of a density matrix is one of the fundamental concepts discussed in the elementary courses of quantum mechanics, the structure of the set $`_N`$ of all density matrices of size $`N`$ is not easy to characterize Bl76 ; Ha78 ; ACH93 . The only exception is the case $`N=2`$, for which $`_2`$ embedded in $`^3`$ has an appealing form of the Bloch ball. Its boundary, $`_2`$, consists of pure states and forms the Bloch sphere. For larger number of states the dimensionality of $`_N`$ grows quadratically with $`N`$, which makes its analysis involved. In particular, for $`N>2`$ the set of pure states forms a $`2N2`$ dimensional manifold, of measure zero in the $`N^22`$ dimensional boundary $`_N`$.

In this work we compute the volume of $`_N`$ with respect to the Hilbert–Schmidt (HS) measure. The HS measure is defined by the HS metric which is distinguished by the fact that it induces the flat, Euclidean geometry into the set of mixed states. The (hyper)area of the boundary of the space of the density matrices, $`_N`$, is also computed, as well as the area of (hyper)edges of this set - the HS volume of the subspace of density matrices of an arbitrary rank $`k<N`$. In the special case of $`k=1`$ we obtain a well–known formula for the volume of the space of pure states, equivalent to the complex projective manifold $`P^{N1}`$.

A similar analysis is also performed for the set of real density matrices. To calculate the volume of the set of complex (real) mixed states we use the volume of the unitary (orthogonal) groups and the volume of the complex (real) flag manifolds - these results are described in the appendix.

A motivation for such a study is twofold. On one hand, the complex structure of the set of mixed quantum states is interesting for itself. It is well–known that for $`N>2`$ the $`D=N^21`$ dimensional set $`_N`$ is neither a $`D`$–ball nor a polytope, but, how it looks like? More like a ball or more like a polytope? Instead of using techniques of differential geometry and computing the average curvature on the boundary of the set $`_N`$, we compute the volume of its boundary and compare it with the volume of the $`D1`$ sphere, which surrounds the ball of the same volume as $`_N`$. Such a comparison shows us, to what extend the shape of the body of mixed quantum states differs from the ball, in a sense that more (hyper)area of the surface is needed to cover the same volume.

A complementary information characterizing the structure of a given set is obtained by calculating the ratio between the area of its boundary and its volume. Among all $`D`$-dimensional bodies of a fixed volume, such a ratio is smallest for the $`D`$–ball. Hence computing such a ratio for the $`D`$–dimensional body of mixed quantum states we may compare it with similar ratios obtained for $`D`$–balls, $`D`$–cubes and $`D`$–simplices.

On the other hand our investigations might be useful in characterizing the absolute volume of the subset of mixed states distinguished by a certain attribute. For instance, if $`\varrho `$ describes a composite system, one may ask, what is the volume of the set of separable (entangled) mixed states ZHSL98 ; Zy99 . Furthermore, assume we are given a concrete mixed quantum state $`\varrho `$. It is natural to ask, wether $`\varrho `$ is in some sense typical, e.g. wether its von Neumann entropy is close to the average taken over the entire set $`_N`$ with respect to the HS measure. To compute such averages (see e.g. ZS01 ) it is usefull to know the volume of $`_N`$ and to make use of integrals developed for such a calculation.

## II Geometry of $`_N`$ with respect to the Hilbert-Schmidt metric

The set of mixed quantum states $`_N`$ consists of Hermitian, positive matrices of size $`N`$, normalized by the trace condition

$$_N:=\{\varrho :\varrho =\varrho ^{};\varrho 0;\mathrm{tr}\varrho =1;\mathrm{dim}(\rho )=N\}.$$
(1)
It is a compact convex set of dimensionality $`D=N^21`$. Any density matrix may be diagonalized by a unitary rotation,

$$\varrho =U\mathrm{\Lambda }U^1,$$
(2)
where $`\mathrm{\Lambda }`$ is a diagonal matrix of eigenvalues $`\mathrm{\Lambda }_i`$. Due to the trace condition they satisfy $`_{i=1}^N\mathrm{\Lambda }_i=1`$, so the space of spectra is isomorphic with a $`(N1)`$–dimensional simplex $`\mathrm{\Delta }_{N1}`$.

Let $`B`$ be a diagonal unitary matrix. Since $`\varrho =UB\mathrm{\Lambda }B^{}U^{}`$, in the generic case of a non degenerate spectrum the unitary matrix $`U`$ is determined up to $`N`$ arbitrary phases entering $`B`$. To specify uniquely the unitary matrix of eigenvectors $`U`$ it is thus sufficient to select a point on the coset space $`Fl_{}^{(N)}:=U(N)/[U(1)]^N`$, called complex flag manifold. The generic density matrix is thus determined by $`(N1)`$ parameters determining eigenvalues and $`N^2N`$ parameters related to eigenvectors, which sum up to the dimensionality $`D`$ of $`_N`$. Although for degenerated spectra the dimension of the flag manifold decreases (see e.g. ACH93 ; ZSlo01 ), these cases of measure zero do not influence the estimation of the volume of the entire set of density matrices. Several different distances may be introduced into the set $`_N`$, (see for instance PS96 ; ZSlo01 ). In this work we shall use the Hilbert-Schmidt metric, which induces the flat geometry.

The Hilbert-Schmidt distance between any two density operators is defined as the Hilbert-Schmidt (Frobenius) norm of their difference,

$$D_{\mathrm{HS}}(\varrho _1,\varrho _2)=\varrho _1\varrho _2_{\mathrm{HS}}=\sqrt{\mathrm{Tr}[(\varrho _1\varrho _2)^2]}.$$
(3)
The set of all mixed states of size two acquires under this metric the geometry of the Bloch ball $`𝐁^3`$ embedded in $`^3`$. Its boundary, $`𝐁^3=𝐒^2`$ contains all pure states and is called Bloch sphere. To show this let us use the Bloch representation of a $`N=2`$ density matrix

$$\varrho =\frac{𝕀}{N}+\stackrel{}{\tau }\stackrel{}{\lambda },$$
(4)
where $`\stackrel{}{\lambda }`$ denotes the vector of three rescaled traceless Pauli matrices $`\{\sigma _x,\sigma _y,\sigma _z\}/\sqrt{2}`$. They are normalized according to tr$`\lambda _i^2=1`$. The three dimensional Bloch vector $`\stackrel{}{\tau }`$ is real due to Hermiticity of $`\varrho `$. Positivity requires tr$`\varrho ^21`$ and this implies $`|\stackrel{}{\tau }|1/\sqrt{2}=:R_2`$. Demanding equality one distinguishes the set of all pure states, $`\varrho ^2=\varrho `$, which form the Bloch sphere of radius $`R_2`$. Consider two arbitrary density matrices and express their difference $`\varrho _1\varrho _2`$ in the representation (4). The entries of this difference consists of the differences between components of both Bloch vectors $`\stackrel{}{\tau }_1`$ and $`\stackrel{}{\tau }_2`$. Therefore

$$D_{\mathrm{HS}}(\varrho _{\stackrel{}{\tau }_1},\varrho _{\stackrel{}{\tau }_2})=D_E(\stackrel{}{\tau }_1,\stackrel{}{\tau }_2),$$
(5)
where $`D_E`$ is the Euclidean distance between both Bloch vectors in $`^3`$. This proves that with respect to the HS metric the set $`_2`$ possesses the geometry of a ball $`𝐁^3`$ . The unitary rotations of a density matrix $`\varrho U\varrho U^{}`$ correspond to the rotations of $`\stackrel{}{\tau }`$ in $`^3`$. This is due to the fact that the adjoint representation of $`SU(2)`$ is isomorphic with $`SO(3)`$.

The Hilbert–Schmidt metric induces a flat geometry inside $`_N`$ for arbitrary $`N`$. Any state $`\varrho `$ may be represented by (4), but now the $`\stackrel{}{\lambda }`$ represents an operator valued vector which consists of $`D=N^21`$ traceless Hermitian generators of $`SU(N)`$, which fulfill tr$`\lambda _i\lambda _j=\delta _{ij}`$. This generalized Bloch representation of density matrices for arbitrary $`N`$ was introduced by Hioe and Eberly HE81 , and recently used in RMNDMC01 . The case $`N=3`$, related to the Gell-Mann matrices, is discussed in detail in the paper by Arvind et al. AMM97 . The generalized Bloch vector $`\stackrel{}{\tau }`$ (also called coherence vector) is $`D`$ dimensional. In the general case of an arbitrary $`N`$ the right hand side of (5) denotes the Euclidean distance between two Bloch vectors in $`^{N^21}`$. Positivity of $`\rho `$ implies the bound for its length

$$|\stackrel{}{\tau }|D_{\mathrm{HS}}(𝕀/N,|\psi \psi |)=\sqrt{\frac{N1}{N}}=:R_N.$$
(6)
In contrast to the Bloch sphere, the complex projective space $`𝐏^{N1}`$ which contains all pure states, forms for $`N>2`$ only a measure zero, simply connected $`2(N1)`$-dimensional subset of the $`N^22`$ dimensional sphere of radius $`R_N`$ embedded in $`^{N^21}`$. Thus not every vector $`\stackrel{}{\tau }`$ of the maximal length $`R_N`$ represents a quantum state. This is related with the fact that for $`N3`$ the adjoint representation of $`SU(N)`$ forms only a subset of $`SO(N^21)`$, (see e.g. Ma68 ). Sufficient and necessary conditions for a Bloch vector to represent a pure state were given in AMM97  for $`N=3`$, and in JS01  for an arbitrary $`N`$. Furthermore, by far not all vectors of length shorter then $`R_N`$ represent a quantum state, as not all the points inside a hyper-sphere belong to the simplex inscribed inside it. Necessary conditions for a Bloch vector to represent quantum mixed state were recently provided by Kimura Ki03 . On the other hand, there exists a smaller sphere inscribed inside the set $`_N`$. Its radius reads Ha78

$$r_N=D_{\mathrm{HS}}(𝕀/N,\varrho _{N1})\frac{1}{\sqrt{N(N1)}}=\frac{R_N}{N1},$$
(7)
where $`\varrho _{N1}`$ denotes any state with the spectrum $`(\frac{1}{N1},\mathrm{},\frac{1}{N1},0)`$.

## III Hilbert-Schmidt measure

Any metric in the space of mixed quantum states generates a measure, inasmuch as one can assume that drawing random density matrices from each ball of a fixed radius is equally likely. The balls are understood with respect to a given metric. In this work we investigate the measure induced by the Hilbert-Schmidt distance (3). The infinitesimal distance takes a particularly simple form

$$(\mathrm{d}s_{\mathrm{HS}})^2=\mathrm{Tr}[(\mathrm{d}\varrho )^2]$$
(8)
valid for any dimension $`N`$. Making use of the diagonal form $`\rho =U\mathrm{\Lambda }U^1`$ we may write

$$\mathrm{d}\varrho =U[\mathrm{d}\mathrm{\Lambda }+U^1\mathrm{dU}\mathrm{\Lambda }\mathrm{\Lambda }U^1\mathrm{dU}]U^1.$$
(9)
Thus (8) can be rewritten as

$$(\mathrm{d}s_{\mathrm{HS}})^2=\underset{i=1}{\overset{N}{}}(\mathrm{d}\mathrm{\Lambda }_i)^2+2\underset{i<j}{\overset{N}{}}(\mathrm{\Lambda }_i\mathrm{\Lambda }_j)^2|(U^1\mathrm{d}U)_{ij}|^2.$$
(10)
Since the density matrices are normalized, $`_{i=1}^N\mathrm{\Lambda }_i=1`$, thus $`_{i=1}^N\mathrm{d}\mathrm{\Lambda }_i=0`$. Hence one may consider the variation of the $`N`$-th eigenvalue as a dependent one, $`\mathrm{d}\mathrm{\Lambda }_N=_{i=1}^{N1}\mathrm{d}\mathrm{\Lambda }_i`$, which implies

$$\underset{i=1}{\overset{N}{}}(\mathrm{d}\mathrm{\Lambda }_i)^2=\underset{i=1}{\overset{N1}{}}(\mathrm{d}\mathrm{\Lambda }_i)^2+\left(\underset{i=1}{\overset{N1}{}}\mathrm{d}\mathrm{\Lambda }_i\right)^2=\underset{i,j=1}{\overset{N1}{}}\mathrm{d}\mathrm{\Lambda }_ig_{ij}\mathrm{d}\mathrm{\Lambda }_j.$$
(11)
The corresponding volume element gains a factor $`\sqrt{\mathrm{det}g}`$, where $`g`$ is the metric in the $`(N1)`$ dimensional simplex $`\mathrm{\Delta }_{N1}`$ of eigenvalues. From (11) one may read out the explicit form of the metric $`g_{ij}`$

$$g=\left[\begin{array}{ccc}1& & 0\\ & \mathrm{}& \\ 0& & 1\end{array}\right]+\left[\begin{array}{ccc}1& \mathrm{}& 1\\ \mathrm{}& \mathrm{}& \mathrm{}\\ 1& \mathrm{}& 1\end{array}\right].$$
(12)
It is easy to check that the spectrum of the $`N1`$ dimensional matrix $`g`$ consists of one eigenvalue equal to $`N`$ and remaining $`N2`$ eigenvalues equal to unity, so that det$`g=N`$. Thus the Hilbert-Schmidt volume element is given by

$$\mathrm{d}V_{\mathrm{HS}}=\sqrt{N}\underset{j=1}{\overset{N1}{}}\mathrm{d}\mathrm{\Lambda }_j\underset{j<k}{\overset{1\mathrm{}N}{}}(\mathrm{\Lambda }_j\mathrm{\Lambda }_k)^2|\underset{j<k}{\overset{1\mathrm{}N}{}}2\mathrm{R}\mathrm{e}(U^1\mathrm{d}U)_{jk}\mathrm{Im}(U^1\mathrm{d}U)_{jk}|.$$
(13)
and has the following product form

$$\mathrm{d}V=\mathrm{d}\mu (\mathrm{\Lambda }_1,\mathrm{\Lambda }_2,\mathrm{},\mathrm{\Lambda }_N)\times \mathrm{d}\nu _{\mathrm{Haar}}.$$
(14)
The first factor depends only on the eigenvalues $`\mathrm{\Lambda }_i`$, while the latter on the eigenvectors of $`\varrho `$ which compose the unitary matrix $`U`$.

Any unitary matrix may be considered as an element of the Hilbert-Schmidt space of operators with the scalar product $`A|B=\mathrm{Tr}A^{}B`$. This suggests the following definition of an invariant metric of the unitary group $`U(N)`$,

$$(\mathrm{d}s)^2:=\mathrm{Tr}(U^1\mathrm{d}U)^2=\underset{jk=1}{\overset{N}{}}|(U^1\mathrm{d}U)_{jk}|^2=\underset{j=1}{\overset{N}{}}|(U^1\mathrm{d}U)_{jj}|^2+2\underset{j<k=1}{\overset{N}{}}|(U^1\mathrm{d}U)_{jk}|^2.$$
(15)
This metric induces the unique Haar measure $`\nu _{Haar}`$ on $`U(N)`$, invariant with respect to unitary transformations, $`\nu _{\mathrm{Haar}}(W)=\nu _{\mathrm{Haar}}(UW)`$, where $`W`$ denotes an arbitrary measurable subset of $`U(N)`$. Integrating the volume element corresponding to (15) over the unitary group we obtain the volume

$$\mathrm{Vol}\left[U(N)\right]=\frac{(2\pi )^{N(N+1)/2}}{1!2!\mathrm{}(N1)!}.$$
(16)
Integrating the volume element with the diagonal terms in (15) omitted (in that case the diagonal elements of $`U`$ are fixed by $`U_{ii}0`$) we obtain the volume of the complex flag manifold, $`Fl_{}^{(N)}:=U(N)/[U(1)^N]`$,

$$\mathrm{Vol}\left[Fl_{}^{(N)}\right]=\frac{\mathrm{Vol}\left[U(N)\right]}{(2\pi )^N}=\frac{(2\pi )^{N(N1)/2}}{1!2!\mathrm{}(N1)!}.$$
(17)
Both results are known in the literature for almost fifty years Hu63 . However, since many different conventions in defining the volume of the unitary group are in use Ma81 ; Tu85 ; Fu01 ; TBS02 ; BST02 ; TS02 ; Ca02  we sketch a derivation of the above expressions in the appendix and provide a list of related results.

Comparing formulae (13) and (15) we recognize that the measure $`\nu `$, responsible for the choice of eigenvectors of $`\varrho `$, is the natural measure on the complex flag manifold $`Fl_{}^{(N)}=U(N)/[U(1)^N]`$ induced by the Haar measure on $`U(N)`$. Since the trace is unitarily invariant, it follows directly from the definition (8), that the volume element with respect to the HS measure is invariant with respect to the group of unitary rotations, $`\mathrm{d}V_{\mathrm{HS}}(\varrho )=\mathrm{d}V_{\mathrm{HS}}(U\varrho U^{})`$. Such a property is characteristic to any product measure of the form (14). Several product measures with different choices of $`\mu `$ were examined in Zy99 ; Sl99a ; ZS01 ; Ca02 .

Integrating the volume element (13) with respect to the eigenvectors of $`\varrho `$ distributed according to the Haar measure one obtains the probability distribution in the simplex of eigenvalues

$$P_{\mathrm{HS}}^{(2)}(\mathrm{\Lambda }_1,\mathrm{},\mathrm{\Lambda }_N)=C_N^{\mathrm{HS}}\delta (1\underset{j=1}{\overset{N}{}}\mathrm{\Lambda }_j)\underset{j<k}{\overset{N}{}}(\mathrm{\Lambda }_j\mathrm{\Lambda }_k)^2,$$
(18)
where for future convenience we have decorated the symbol $`P`$ with the superscript (2) consistent with the exponent in the last factor. As discussed in the following section the normalization constant $`C_N^{\mathrm{HS}}`$ may be expressed ZS01  in terms of the Euler Gamma function $`\mathrm{\Gamma }(x)`$ SO87

$$C_N^{\mathrm{HS}}=\frac{\mathrm{\Gamma }(N^2)}{_{j=0}^{N1}\mathrm{\Gamma }(Nj)\mathrm{\Gamma }(Nj+1)}.$$
(19)
The above joint probability distribution, derived by Hall Ha98 , defines the measure $`\mu _{\mathrm{HS}}`$ in the space of diagonal matrices, and the Hilbert-Schmidt measure (13) in the space of density matrices $`_N`$.

Interestingly, the very same measure may be generated by drawing random pure states $`|\varphi _1_2`$ of a composite $`N\times N`$ system according to the Fubini-Study measure on $`𝐏^{N^21}`$. Then the density matrices of size $`N`$ obtained by partial trace, $`\varrho =\mathrm{tr}_2(|\varphi \varphi |)`$, are distributed according to the HS measure Br96 ; Ha98 ; ZS01 . Alternatively, one may generate a random matrix $`A`$ of the Ginibre ensemble, (non-Hermitian complex matrix with all entries independent Gaussian variables with zero mean and a fixed variance) and obtain a HS distributed random density matrix by a projection $`\varrho =A^{}A/\mathrm{tr}A^{}A`$ ZS01 . A similar approach was recently advocated by Tucci Tu02 , who used the name ’uniform ensemble’ just for ensemble of density matrices generated according to the HS measure.

## IV Volume of the set of mixed states

For later convenience let us introduce generalized normalization constants

$$\frac{1}{C_N^{(\alpha ,\beta )}}:=_0^{\mathrm{}}d\mathrm{\Lambda }_1\mathrm{}d\mathrm{\Lambda }_N\delta (\underset{i=1}{\overset{N}{}}\mathrm{\Lambda }_i1)\underset{i=1}{\overset{N}{}}\mathrm{\Lambda }_i^{\alpha 1}\underset{i<j}{}|\mathrm{\Lambda }_i\mathrm{\Lambda }_j|^\beta $$
(20)
with $`\alpha ,\beta >0`$. These constants may be calculated using the formula for the Laguerre ensemble, discussed in the book of Mehta Me91 ,

$$_0^{\mathrm{}}d\mathrm{\Lambda }_1\mathrm{}d\mathrm{\Lambda }_N\mathrm{exp}\left(\underset{i=1}{\overset{N}{}}\mathrm{\Lambda }_i\right)\underset{i=1}{\overset{N}{}}\mathrm{\Lambda }_i^{\alpha 1}\underset{i<j}{}|\mathrm{\Lambda }_i\mathrm{\Lambda }_j|^\beta =\underset{j=1}{\overset{N}{}}\left[\frac{\mathrm{\Gamma }[1+j\beta /2]\mathrm{\Gamma }[\alpha +(j1)\beta /2]}{\mathrm{\Gamma }[1+\beta /2]}\right].$$
(21)
Substituting $`x_i^2=\mathrm{\Lambda }_i`$ we may bring the latter integral to the Gaussian form. Expressing it in the spherical coordinates we get the integral (20) and eventually obtain

$$\frac{1}{C_N^{(\alpha ,\beta )}}:=\frac{1}{\mathrm{\Gamma }[\alpha N+\beta N(N1)/2]}\underset{j=1}{\overset{N}{}}\left[\frac{\mathrm{\Gamma }[1+j\beta /2]\mathrm{\Gamma }[\alpha +(j1)\beta /2]}{\mathrm{\Gamma }[1+\beta /2]}\right].$$
(22)
By definition $`C_N^{\mathrm{HS}}=C_N^{(1,2)}`$ and the special case of the above expression reduces to (19).

To obtain the Hilbert-Schmidt volume of the set of mixed states $`_N`$ one has to integrate the volume element (13) over eigenvalues and eigenvectors. By definition the first integral gives $`1/C_N^{\mathrm{HS}}`$, while the second is equal to the volume of the flag manifold. To make the diagonalization transformation (2) unique one has to restrict to a certain order of eigenvalues, say, $`\mathrm{\Lambda }_1<\mathrm{\Lambda }_2<\mathrm{}<\mathrm{\Lambda }_N`$, (a generic density matrix is not degenerate), which corresponds to a choice of a certain Weyl chamber of the eigenvalue simplex $`\mathrm{\Delta }_{N1}`$. In other words, different permutations of the vector of $`N`$ generically different eigenvalues $`\mathrm{\Lambda }_i`$ belong to the same unitary orbit. The number of different permutations (Weyl chambers) equals to $`N!`$, so the volume reads

$$V_N^{(2)}:=\mathrm{Vol}_{\mathrm{HS}}\left(_N\right)=\frac{\sqrt{N}}{N!}\frac{\mathrm{Vol}\left(Fl_{}^{(N)}\right)}{C_N^{\mathrm{HS}}}.$$
(23)
The square root stems from the volume element (13), and the index (2) refers to the general case of complex density matrices. Making use of (19) and (17) we arrive at the final result 111Apart of the first factor $`\sqrt{N}`$, the same formula appeared already in the work of Tucci Tu02

$$V_N^{(2)}=\sqrt{N}(2\pi )^{N(N1)/2}\frac{\mathrm{\Gamma }(1)\mathrm{}\mathrm{\Gamma }(N)}{\mathrm{\Gamma }(N^2)}.$$
(24)
Substituting $`N=2`$ we are pleased to receive $`V_2^{(2)}=\pi \sqrt{2}/3`$ \- exactly the volume of the Bloch ball $`𝐁^3`$ of radius $`R_2=1/\sqrt{2}`$. This result may be also found in the notes by Caves Ca02 , who also derived an explixit integral for the volume of the set of mixed states for arbitrary $`N`$.

The next result $`V_3^{(2)}=\pi ^3/(840\sqrt{3})`$ allows us to characterize the difference between the set $`_3^8`$ and the ball $`𝐁^8`$. The set of mixed states is inscribed into the sphere of radius $`R_3=\sqrt{2/3}0.816`$, while the maximal ball contained inside has the radius $`r_3=R_3/20.408`$. Using Eq. (28) we find the radius of the $`8`$–ball of volume $`V_3`$ is $`\rho _30.519`$. The distance from the center of $`_3`$ to its boundary varies with the direction in $`^8`$ from $`r_3`$ to $`R_3`$, in contrast to the $`N=2`$ case of Bloch ball, for which $`R_2=r_2=\rho _2=1/\sqrt{2}`$. The average HS–distance from the center of $`_3`$ to its boundary is equal to $`\rho _3`$. Similar calculations performed for $`N=4`$ give the maximal radius $`R_4=\sqrt{3/4}0.866`$, the minimal radius $`r_4=R_4/30.289`$ and the ’mean’ radius $`\rho _40.428`$ which generates the ball $`𝐁^{15}`$ of the same volume as $`V_4`$. In general, let $`\rho _N`$ denote the radius of a ball $`𝐁^{N^21}`$ of the same volume as the set $`_N`$.

The volume $`V_N`$ tends to zero if $`N\mathrm{}`$, but there is no reason to worry about it. The same is true for the volume of the $`N`$–ball, see (28). This is just a consequence of the choice of the units. We are comparing the volume of an object in $`^N`$ with the volume of a hypercube $`C^N`$ of side one, and it is easy to understand, that the larger dimension, the smaller is the volume of the ball inscribed into it.

## V Area of the boundary of the set of mixed states

The boundary of the set of mixed states is far from being trivial. Formally it may be written as a solution of the equation det$`\varrho =0`$ which contains all matrices of a lower rank. The boundary $`_N`$ contains orbits of different dimensionality generated by spectra of different rank and degeneracy (see eg. ACH93 ; ZSlo01 ). Fortunately all of them are of measure zero besides the generic orbits created by unitary rotations of diagonal matrices with all eigenvalues different and one of them equal to zero; $`\mathrm{\Lambda }=\{0,\mathrm{\Lambda }_2<\mathrm{\Lambda }_3<\mathrm{}<\mathrm{\Lambda }_N\}`$. Such spectra form the $`N2`$ dimensional simplex $`\mathrm{\Delta }_{N2}`$, which contains $`(N1)!`$ Weyl chambers - this is the number of possible permutations of elements of $`\mathrm{\Lambda }`$ which all belong to the same unitary orbit.

Hence the hyper-area of the boundary may be computed in a way analogous to (23),

$$S_N^{(2)}:=\mathrm{Vol}_{\mathrm{HS}}\left(_N\right)=\frac{\sqrt{N1}}{(N1)!}\frac{\mathrm{Vol}\left(Fl_{}^{(N)}\right)}{C_{N1}^{(3,2)}}.$$
(25)
The change of the parameter $`\alpha `$ in (20) from $`1`$ to $`3`$ is due to the fact that by setting one component of an $`N`$ dimensional vector to zero the corresponding Vandermonde determinant of size $`N`$ leads to the determinant of size $`N1`$ for $`\beta =1`$ and to the square of the determinant for $`\beta =2`$. Applying (19) and (17) we obtain an explicit result

$$S_N^{(2)}=\sqrt{N1}(2\pi )^{N(N1)/2}\frac{\mathrm{\Gamma }(1)\mathrm{}\mathrm{\Gamma }(N+1)}{\mathrm{\Gamma }(N)\mathrm{\Gamma }(N^21)}.$$
(26)
For $`N=2`$ we get $`S_2^{(2)}=2\pi `$ \- just the area of the Bloch sphere $`𝐒^2`$ of radius $`R_2=1/\sqrt{2}`$. The area of the $`7`$-dim boundary of $`_3`$ reads $`S_3^{(2)}=\sqrt{2}\pi ^3/105`$.

In an analogous way we may find the volume of edges, formed by the unitary orbits of the vector of eigenvalues with two zeros. More generally, states of rank $`Nn`$ are unitarily similar to diagonal matrices with $`n`$ eigenvalues vanishing, $`\mathrm{\Lambda }=\{0,\mathrm{},0,\mathrm{\Lambda }_{n+1}<\mathrm{\Lambda }_{n+2}<\mathrm{}<\mathrm{\Lambda }_N\}`$. These edges of order $`n`$ are $`N^2n^21`$ dimensional, since the dimension of the set of such spectra is $`Nn1`$, while the orbits have the structure of $`U(N)/[U(n)\times (U(1))^{Nn}]`$ and dimensionality $`N^2n^2(Nn)`$. Repeating the reasoning used to derive (25) we obtain the volume of the hyperedges

$$S_{N,n}^{(2)}=\frac{\sqrt{Nn}}{(Nn)!}\frac{1}{C_{Nn}^{(1+2n,2)}}\frac{\mathrm{Vol}\left(Fl_{}^{(N)}\right)}{\mathrm{Vol}\left(Fl_{}^{(n)}\right)}.$$
(27)
Note that for $`n=0`$ this expression gives the volume $`V_N^{(2)}`$ of the set $`_N`$, for $`n=1`$ the hyperarea $`S_N^{(2)}`$ of its boundary $`_N`$, for $`n2`$ the area of the edges of rank $`Nn`$. In the extreme case of $`n=N1`$ the above formula gives correctly the volume of the set of pure states, (the states of rank one), Vol$`(𝐏^{N1})=(2\pi )^{N1}/\mathrm{\Gamma }(N)`$, see appendix.

## VI The ratio: area/volume

Certain information about the structure of a convex body may be extracted from the ratio $`\gamma `$ of the (hyper)area of its boundary to its volume. The smaller the coefficient $`\gamma `$ (with the diameter of the body kept fixed), the better the body investigated may be approximated by a ball, for which such a ratio is minimal. And conversely, the larger $`\gamma `$, the less the body resembles a ball, since more (hyper)area is needed to bound a given volume.

To analyze simple examples let us recall the volume of the $`N`$-dimensional unit ball $`𝐁^N^N`$ and the volume $`S_N`$ of the unit $`N`$–sphere $`𝐒^N^{N+1}`$

$$B_N:=\text{vol}(𝐁^N)=\frac{\text{vol}(𝐒^{N1})}{N}=\frac{\pi ^{\frac{N}{2}}}{\mathrm{\Gamma }(\frac{N}{2}+1)}\frac{1}{\sqrt{2\pi }}\left(\frac{2\pi e}{N}\right)^{\frac{N}{2}},$$
(28)
where the Stirling expansion SO87  was used for large $`N`$. For small $`N`$ we obtain the well known expressions, $`S_1=2\pi `$, $`S_2=4\pi `$, $`S_3=2\pi ^2`$, $`S_4=8\pi ^2/3`$ and $`B_2=\pi `$, $`B_3=4\pi /3`$, $`B_4=\pi ^2/2`$. If the spheres and balls have radius $`L`$ then the scale factor $`L^N`$ has to be supplied. In odd dimensions the volume of the sphere simplifies, $`\text{vol}(𝐒^{2k1})=2\pi ^k/(k1)!`$.

Since the boundary of a $`N`$–ball is formed by a $`N1`$ sphere, $`𝐁^N=𝐒^{N1}`$, the ratio $`\gamma `$ for a ball of radius $`L`$ reads

$$\gamma (𝐁^N):=\frac{\text{vol}(𝐁^N)}{\text{vol}(𝐁^N)}=\frac{N}{L}.$$
(29)
Intuitively this ratio will be the smallest possible among all $`N`$-dimensional sets of the same volume. Hence let us compare it with an analogous result for a hypercube $`\mathrm{}_N`$ of side $`L`$ and volume $`L^N`$. The cube has $`2^N`$ corners and $`2N`$ faces, of area $`L^{N1}`$ each. We find the ratio

$$\gamma (\mathrm{}_N):=\frac{\text{vol}(\mathrm{}_N)}{\text{vol}(\mathrm{}_N)}=2\frac{N}{L}$$
(30)
which grows twice as fast as for $`N`$-balls. Another comparison can be made with simplices $`\mathrm{}_N`$, generated by $`(N+1)`$ equally distant points in $`^N`$. The simplex $`\mathrm{}_2`$ is a equilateral triangle, while $`\mathrm{}_3`$ is a regular tetrahedron. The volume of a simplex of side $`L`$ reads $`\text{vol}(\mathrm{}_N)=[L^N\sqrt{(N+1)/2^N}]/N!`$. Since the boundary of $`\mathrm{}_N`$ consists of $`N+1`$ simplices $`\mathrm{}_{N1}`$ we obtain

$$\gamma (\mathrm{}_N):=\frac{\text{vol}(\mathrm{}_N)}{\text{vol}(\mathrm{}_N)}=\sqrt{\frac{2N}{N+1}}\frac{N(N+1)}{L}.$$
(31)
In this case the ratio $`\gamma `$ grows quadratically with $`N`$, which reflects the fact that simplices do have much ’sharper’ corners, in contrast to the cubes, so more (hyper)area of the boundary is required to cover a given volume. Furthermore, if one defines a hyper–diamond as two simplices glued along one face, its volume is twice the volume of $`\mathrm{}_N`$ while its boundary consists of $`2N`$ simplices $`\mathrm{}_{N1}`$, so the coefficient $`\gamma `$ grows exactly as $`N^2`$.

Interestingly, the ratio $`\gamma `$ of the $`N`$–cube is the same as for the $`N`$–ball inscribed in, which has much smaller volume. The same property is characteristic for the $`N`$–simplex. Hence another possibility to characterize the shape of any convex body $`F`$ is to compute the ratio $`\chi _1:=\mathrm{vol}[𝐁_1(F)]/\mathrm{vol}(F)`$, and $`\chi _2:=\mathrm{vol}(F)/\mathrm{vol}[𝐁_2(F)]`$, where $`𝐁_1(F)`$ is the largest ball inscribed in $`F`$ while $`𝐁_2(F)`$ is the smallest ball in which $`F`$ may be inscribed. As stated above for cubes and simplices one has $`\gamma (F)=\gamma [𝐁_1(F)]`$.

Such quotients may be computed for the rather complicated convex body of mixed quantum states analyzed with respect to the Hilbert-Schmidt measure. Using expressions (23) and (25) we find

$$\gamma _N:=\frac{\text{vol}\left(_N\right)}{\text{vol}\left(_N\right)}=\frac{N!\sqrt{N1}}{\sqrt{N}(N1)!}\frac{C_N^{(1,2)}}{C_{N1}^{(3,2)}}=\sqrt{N(N1)}(N^21).$$
(32)
The first coefficients read $`\gamma _2=3\sqrt{2}`$, $`\gamma _3=8\sqrt{6}`$, and $`\gamma _4=15\sqrt{12}`$, so they grow with $`N`$ faster than $`N^2`$. A direct comparison with the results received for balls or cubes would be unfair, since here $`N`$ does not denote the dimension of the set $`_N^D`$. Substituting the right dimension, $`D=N^21`$, we see that the area/volume ratio for the mixed states increases with the dimensionality as $`\gamma D^{3/2}`$. The linear scaling factor $`L`$, equal to the radius $`R_N`$ tends asymptotically to unity and does not influence this behaviour.

Note that the set of mixed states is convex and is inscribed into the sphere of radius $`R_N`$, so for each finite $`N`$ the ratio $`\gamma _N`$ remains finite. On the other hand, the fact that this coefficient increases with the dimension $`D`$ much faster than for balls or cubes, sheds some light into the intricate structure of the set $`_N`$. It touches the hypersphere $`𝐒^{N^22}`$ of radius $`R_N`$ along the $`2N2`$ dimensional manifold of pure states. However, to be characterized by such a value of the coefficient $`\gamma `$ it is a rather ’thin’ set, and a lot of hyper–area of the boundary is used to encompass its volume. In fact, for any mixed state $`\varrho _N`$ its distance to the boundary $`_N`$ does not exceed the radius $`r_N1/N`$. Another comparison can be made with the $`D`$-ball of radius $`L=r_N=[N(N1)]^{1/2}`$, inscribed into $`_N`$. Although its volume is much smaller than this of the larger set of mixed states, its area to volume ratio, $`\gamma =D/L`$ is exactly equal to (32) characterizing $`_N`$. In other words, for any dimensionality $`N`$ the set of mixed quantum states belongs to the class of bodies for which $`\gamma (F)=\gamma (𝐁_1(F))`$ holds.

Using the notion of the effective radius $`\rho _N`$, introduced in section IV, we may express the coefficients $`\chi _i`$ for the set $`_N`$ as a ratio between radii raised to the power equal to the dimensionality, $`D=N^21`$. The exact values of $`\chi _1=(r_N/\rho _N)^D`$ and $`\chi _2=(\rho _N/R_N)^D`$, as well as their product $`\chi =\chi _1\chi _2`$, may be readily obtained from (24). Let us only note the large $`N`$ behaviour, $`\chi (_N)=(N1)^{N^2+1}`$ so it grows with the dimensionality $`D`$ as $`D^{D/2}`$ while $`\chi (𝐁^N)=1`$, $`\chi (\mathrm{}_N)=N^{N/2}`$, and $`\chi (\mathrm{}_N)N^N`$.

## VII Rebits: real density matrices

Even though from physical point of view one should in general consider the entire set $`_N`$ of complex density matrices, we propose now to discuss its proper subset: the set of real density matrices. This set, denoted by $`_N^{}`$, is of smaller dimension $`D_1=N(N+1)/21<D=N^21`$, and any reduction of dimensionality simplifies the investigations. While complex density matrices of size two are known as qubits, the real density matrices are sometimes called rebits CFR02 . In the sense of the HS metric the space of rebits forms the full circle $`𝐁^2`$, which may be obtained as a slice of the Bloch ball $`𝐁^3`$ along a plane containing $`𝕀/2`$.

To find the volume of the set $`_N^{}`$ we will repeat the steps (9)–(24) for real symmetric density matrices which may be diagonalized by an orthogonal rotation, $`\varrho =O\mathrm{\Lambda }O^T`$. The expressions

$$\mathrm{d}\varrho =O[\mathrm{d}\mathrm{\Lambda }+O^1\mathrm{dO}\mathrm{\Lambda }\mathrm{\Lambda }O^1\mathrm{dO}]O^1$$
(33)
and

$$(\mathrm{d}s_{\mathrm{HS}})^2=\underset{i=1}{\overset{N}{}}(\mathrm{d}\mathrm{\Lambda }_i)^2+2\underset{i<j}{\overset{N}{}}(\mathrm{\Lambda }_i\mathrm{\Lambda }_j)^2|(O^1\mathrm{d}O)_{ij}|^2$$
(34)
allow us to obtain the HS volume element, analogous to (13),

$$\mathrm{d}V_{\mathrm{HS}}^{(1)}=\sqrt{N}\underset{j=1}{\overset{N1}{}}\mathrm{d}\mathrm{\Lambda }_j\underset{j<k}{\overset{1\mathrm{}N}{}}|\mathrm{\Lambda }_j\mathrm{\Lambda }_k||\underset{j<k}{\overset{1\mathrm{}N}{}}\sqrt{2}(O^1\mathrm{d}O)_{jk}|$$
(35)
As in the complex case the measure has the product form, and the last factor is the volume element of the orthogonal group (see appendix). Orthogonal orbits of a nondegenerate diagonal matrix form real flag manifolds $`Fl_{}^{(N)}=O(N)/[O(1)]^N`$ of the volume

$$\mathrm{Vol}\left[Fl_{}^{(N)}\right]=\frac{\mathrm{Vol}\left[O(N)\right]}{2^N}=\frac{(2\pi )^{N(N1)/4}\pi ^{N/2}}{\mathrm{\Gamma }[1/2]\mathrm{}\mathrm{\Gamma }[N/2]}.$$
(36)
Here $`O(1)`$ is the reflection group $`_2`$ with volume $`2`$.

The volume element (35) leads to the following probability measure in the simplex of eigenvalues

$$P_{\mathrm{HS}}^{(1)}(\mathrm{\Lambda }_1,\mathrm{},\mathrm{\Lambda }_N)=C_N^{(1,1)}\delta (1\underset{j=1}{\overset{N}{}}\mathrm{\Lambda }_j)\underset{j<k}{\overset{N}{}}|\mathrm{\Lambda }_j\mathrm{\Lambda }_k|,$$
(37)
with the normalization constant given in (22). Note the linear dependence on the differences of eigenvalues, in contrast to the quadratic form present in (18). Taking into account the number $`N!`$ of different permutations of the elements of the spectrum $`\mathrm{\Lambda }`$ we obtain the expression for the volume of the set of $`_N^{}`$,

$$V_N^{(1)}:=\mathrm{Vol}_{\mathrm{HS}}\left(_N^{}\right)=\frac{\sqrt{N}}{N!}\frac{\mathrm{Vol}\left(Fl_{}^{(N)}\right)}{C_N^{(1,1)}},$$
(38)
which gives

$$V_N^{(1)}=\frac{\sqrt{N}}{N!}\frac{2^N(2\pi )^{N(N1)/4}\mathrm{\Gamma }\left[\frac{N+1}{2}\right]}{\mathrm{\Gamma }\left[\frac{N(N+1)}{2}\right]\mathrm{\Gamma }\left[\frac{1}{2}\right]}\underset{k=1}{\overset{N}{}}\mathrm{\Gamma }[1+\frac{k}{2}].$$
(39)
As in the complex case we find the volume of the boundary of $`_N^{}`$, and in general, the volume of edges of order $`n`$ with $`0nN1`$. In the case of real density matrices these edges are $`N(N+1)/21n(n+1)/2`$ dimensional, since the dimension of the set of such spectra is $`Nn1`$, and the orbits have the structure of $`O(N)/[O(n)\times (O(1))^{Nn}]`$ and dimensionality $`N(N1)/2n(n1)/2`$. In analogy to (27) we obtain

$$S_{N,n}^{(1)}=\frac{\sqrt{Nn}}{(Nn)!}\frac{1}{C_{Nn}^{(1+n,1)}}\frac{\mathrm{Vol}\left(Fl_{}^{(N)}\right)}{\mathrm{Vol}\left(Fl_{}^{(n)}\right)},$$
(40)
which for $`n=1`$ gives the volume $`S`$ of the boundary $`_N^{}`$, and allows us to compute the ratio area to volume,

$$\gamma (_N^{}):=\frac{\text{vol}\left(_N^{}\right)}{\text{vol}\left(_N^{}\right)}=\frac{N!\sqrt{N1}}{\sqrt{N}(N1)!}\frac{C_N^{(1,1)}}{C_{N1}^{(2,1)}}=\sqrt{N(N1)}(N1)(1+N/2).$$
(41)
The product of the last two factors is equal to the dimensionality of the set of real density matrices, $`D_1=N(N+1)/21`$. Therefore, just as in the complex case, the ratio area to volume for $`_N^{}`$ coincides with such a ratio $`\gamma =D_1/L`$ for the maximal ball of radius $`L=r_N=[N(N1)]^{1/2}`$ contained in this set. In the simplest case of $`N=2`$ we receive $`V_2^{(1)}=\pi /2`$ \- the volume of the circle $`𝐁^2`$ of radius $`R_2=1/\sqrt{2}`$. The volume of the boundary, $`S=\pi \sqrt{2}`$, equals to the circumference of the circle of radius $`R_2=1/\sqrt{2}`$, and gives $`\gamma =2\sqrt{2}`$ in agreement with (41).

## VIII Concluding remarks

We have found the volume $`V`$ and the surface area $`S`$ of the $`D=N^21`$ dimensional set of mixed states $`_N`$ acting in the $`N`$–dimensional Hilbert space, and its subset $`_N^{}`$ containing real symmetric matrices. Although the volume of the unitary (orthogonal) group depends on the definition used, as discussed in the appendix, the volume of the set of mixed state has a well specified, unambiguous meaning. For instance, for $`N=2`$ the volume $`V_2`$ may be interpreted as the ratio of the volume of the Bloch ball (of radius $`R_2`$ fixed by the Hilbert–Schmidt metric), to the cube spanned by three orthonormal vectors of the HS space: the rescaled Pauli matrices, $`\{\sigma _x,\sigma _y,\sigma _z\}/\sqrt{2}`$.

On one hand, these explicit results may be applied for estimation of the volume of the set of entangled states ZHSL98 ; Zy99 ; Sl00 ; Sl02 ; Sl03 , or yet another subset of $`_N`$. It is also likely to expect that some integrals obtained in this work will be usefull is such investigations.

On the other hand, outcomes of this paper advance our understanding of the properties of the set of mixed quantum states. The ratio of the hyperarea of the boundary of $`D`$–balls to their volume grows linearly with the dimension $`D`$. The same ratio for $`D`$–simplices behaves as $`D^2`$, while for the sets of complex and real density matrices it grows with the dimensionality $`D`$ as $`D^{3/2}`$. Hence these geometrical properties of the convex body of mixed states are somewhere in between the properties of $`D`$–balls and $`D`$–simplices.

Furthermore, we have shown that for any $`N`$ the sets of complex (real) density matrices belong to the family of sets, for which the ratio area to volume is equal to such a ratio computed for the maximal ball inscribed into this set.

It is necessary to emphasize, that a similar problem of estimating the volume of the set of mixed states could be also considered with respect to other probability measures. In particular, analogous results presented by us in SZ03  for the measure Ha98 ; BS01  related to the Bures distance Bu69 ; Uh76  allow us to investigate similarities and differences between the geometry of mixed states induced by different metrics.

It is a pleasure to thank M. Kuś for helpful discussions and to P. Slater for valuable correspondence. Financial support by Komitet Badań Naukowych in Warsaw under the grant 2P03B-072 19 and the Sonderforschungsbereich/Transregio 12 der Deutschen Forschungsgemeinschaft is gratefully acknowledged.

## Appendix A Volumes of the unitary groups and flag manifolds

Although the volume of the unitary (orthogonal) group and the complex (real) flag manifold, we use in our calculations, were computed by Hua many years ago Hu63 , one may find in more recent literature related results, which in some cases seem to be contradicting. However, different authors used different definitions of the volume of unitary group Ma81 ; Tu85 ; Fu01 ; TBS02 ; BST02 ; Ca02 , so we review in this appendix three most common definitions and compare the results.

### A.1 Unitary group $`U(N)`$

We shall recall (15) the metric of the unitary group $`U(N)`$ induced by the Hilbert-Schmidt scalar product and used by Hua Hu63

$$(\mathrm{d}s)^2:=\mathrm{Tr}(U^1\mathrm{d}U)^2=\underset{j=1}{\overset{N}{}}|(U^1\mathrm{d}U)_{jj}|^2+2\underset{j<k=1}{\overset{N}{}}|(U^1\mathrm{d}U)_{jk}|^2,$$
(42)
which is left- and right-invariant under unitary transformations. The volume element is then given by the product of independent differentials times the square root of the determinant of the metric tensor. One has still the freedom of an overall scale factor for (42) which appears then correspondingly in the volume element. To keep invariance the ratio of the prefactors $`c_{\mathrm{diag}}`$ and $`c_{\mathrm{off}}`$ of the diagonal and off–diagonal terms has to be fixed. Nevertheless one may introduce different scalings of the volume elements which we call $`d\nu _A,d\nu _B,d\nu _C`$:

$$\mathrm{d}\nu _A:=\left|\underset{i=1}{\overset{N}{}}(U^1\mathrm{d}U)_{ii}\underset{j<k}{\overset{1\mathrm{}N}{}}\sqrt{2}\mathrm{Re}(U^1\mathrm{d}U)_{jk}\mathrm{Im}(U^1\mathrm{d}U)_{jk}\right|;c_{\mathrm{diag}}=1,c_{\mathrm{off}}=2;$$
(43)
$$\mathrm{d}\nu _B:=2^{N(N1)/2}\mathrm{d}\nu _A;c_{\mathrm{diag}}=1,c_{\mathrm{off}}=1;$$
(44)
$$\mathrm{d}\nu _C:=2^{N/2}\mathrm{d}\nu _B;c_{\mathrm{diag}}=1/2c_{\mathrm{off}}=1.$$
(45)
The product in (43), consistent with (42), has to be understood in the sense of alternating external multiplication of differential forms. Only the first convention (43) labeled by the index A was used in the main part of this work. Note that the normalisation (45) corresponds to the rescaled line element $`(\mathrm{d}s)^2=\frac{1}{2}\mathrm{Tr}(U^1\mathrm{d}U)^2`$. In general we may scale

$$\mathrm{d}\nu __X:=(c_{\mathrm{diag}})^{N/2}(c_{\mathrm{off}})^{N(N1)/2}\mathrm{d}\nu _B$$
(46)
where the label X denotes a certain choice of the prefactors $`c_{\mathrm{diag}}`$ and $`c_{\mathrm{off}}`$ for diagonal or off–diagonal elements in (42). All these volumes correspond to the Haar measure which is unique up to an overall constant scale factor. Thus we deduce that

$$\mathrm{Vol}_A\left[U(N)\right]=2^{N(N1)/2}\mathrm{Vol}_B\left[U(N)\right]\mathrm{and}\mathrm{Vol}_C\left[U(N)\right]=2^{N/2}\mathrm{Vol}_B\left[U(N)\right].$$
(47)
In order to determine the volume of the unitary group let us recall the fiber bundle structure $`U(N1)U(N)𝐒^{2N1}`$, see e.g. Bo91 . This topological fact implies a relation between the volume of the unit sphere $`𝐒^{2N1}`$ and the volume of the unitary group defined by the measure $`d\nu _B`$ (44), for which all components of the vector $`(U^1\mathrm{d}U)_{jk}`$ have unit prefactors,

$$\mathrm{Vol}_B[U(N)]=\mathrm{Vol}_B[U(N1)]\times \mathrm{Vol}[𝐒^{2N1}].$$
(48)
To prove this equality by a direct calculation it is convenient to parametrize a unitary matrix of size $`N`$ as

$$U_N=\left[\begin{array}{cc}e^{i\varphi }& 0\\ 0& U_{N1}\end{array}\right]\left[\begin{array}{cc}\sqrt{1|h|^2}& h^{}\\ h& \sqrt{𝕀hh^{}}\end{array}\right],$$
(49)
where $`\varphi [0,2\pi )`$ is an arbitrary phase and $`h`$ is a complex vector with $`N1`$ components such that $`|h|1`$. This representation shows (we may arrange the two matrices in (49) also in the opposite order) the relation

$$U(N)/[U(1)\times U(N1)]=𝐏^{N1},$$
(50)
since the second factor represents the complex projective space $`𝐏^{N1}`$. In fact, if one calculates the metric (42) we find

$$(\mathrm{d}s_N)^2(\mathrm{d}s_1)^2+(\mathrm{d}s_{N1})^2+2(\mathrm{d}s_h)^2$$
(51)
where $`(\mathrm{d}s_N)^2`$ means the metric for $`U(N)`$ (the sign $``$ shall indicate that we have omitted some shifts in $`(\mathrm{d}s_1)^2`$ and $`(\mathrm{d}s_{N1})^2`$ that are not relevant for the volume) and $`(\mathrm{d}s_h)^2`$ means the metric of the complex projective space $`𝐏^{N1}`$ with radius $`1`$:

$$(\mathrm{d}s_h)^2=\mathrm{d}h^{}\mathrm{d}h+\frac{(h^{}\mathrm{d}h+\mathrm{d}h^{}h)^2}{4(1|h|^2)}+\frac{(h^{}\mathrm{d}h\mathrm{d}h^{}h)^2}{4}.$$
(52)
It is easy to see by diagonalizing this metric (eigenvalues $`1|h|^2`$, $`1/(1|h|^2)`$, and otherwise $`1`$) that the corresponding volume is that of the real ball $`𝐁^{2N2}`$ with radius $`1`$ and dimension $`2N2`$. Thus one obtains

$$\mathrm{Vol}_X[U(N)]=\mathrm{Vol}_X[U(N1)]\times \mathrm{Vol}_X[U(1)]\times c_{\mathrm{off}}^{N1}\mathrm{Vol}[𝐁^{2N2}],$$
(53)
which for the measure (44) with $`c_{\mathrm{diag}}=c_{\mathrm{off}}=1`$ reduces to (48). Applying this relation $`N1`$ times we obtain

$$\mathrm{Vol}_B[U(N)]=\mathrm{Vol}[𝐒^{2N1}]\times \mathrm{}\times \mathrm{Vol}[𝐒^3]\times \mathrm{Vol}[𝐒^1].$$
(54)
Taking into account that Vol$`[𝐒^{2N1}]=2\pi ^N/(N1)!`$ and making use of the relation (47) we may write an explicit result for the volumes calculated with respect to different definitions (4345)

$$\mathrm{Vol}_X[U(N)]=a_X^U\frac{2^N\pi ^{N(N+1)/2}}{0!1!\mathrm{}(N1)!},$$
(55)
where the proportionality constants read $`a_A^U=2^{N(N1)/2}`$, $`a_B^U=1`$ and $`a_C^U=2^{N/2}`$. The result for $`\mathrm{Vol}_A[U(N)]`$ was rigorously derived in Hu63 , $`\mathrm{Vol}_B[U(N)]`$ was given in Fu01 , while $`\mathrm{Vol}_A[U(N)]`$ and $`\mathrm{Vol}_C[U(N)]`$ were compared in Ca02 . In particular, $`\mathrm{Vol}_A[U(1)]=\mathrm{Vol}_B[U(1)]=2\pi `$, while $`\mathrm{Vol}_C[U(1)]=\sqrt{2}\pi `$ and $`\mathrm{Vol}_A[U(2)]=8\pi ^3`$, $`\mathrm{Vol}_B[U(2)]=4\pi ^3`$, $`\mathrm{Vol}_C[U(2)]=2\pi ^3`$.

In general, the volume of a coset space may be expressed as a ratio of the volumes. Consider for instance the manifold of all pure states of dimensionality $`N`$. It forms the complex projective space $`𝐏^{N1}=U(N)/[U(N1)\times U(1)]`$. Therefore

$$\mathrm{Vol}_X[𝐏^{N1}]=\frac{\mathrm{Vol}_X[U(N)]}{\mathrm{Vol}_X[U(1)]\mathrm{Vol}_X[U(N1)]},$$
(56)
which gives the general result

$$\mathrm{Vol}_X[𝐏^k]=a_X^{\mathrm{CP}}\frac{\pi ^k}{k!}=a_X^{\mathrm{CP}}\mathrm{Vol}[𝐁^{2k}]$$
(57)
The scale factors read $`a_A^{\mathrm{CP}}=2^k`$ and $`a_B^{\mathrm{CP}}=a_C^{\mathrm{CP}}=1`$. For instance $`\mathrm{Vol}_A[𝐏^1]=2\pi `$ which corresponds to the circle of radius $`\sqrt{2}`$ , while $`\mathrm{Vol}_B[𝐏^1]=\mathrm{Vol}_C[𝐏^1]=\pi `$, equal to the area of the circle of radius $`1`$. The latter convention is natural if one uses the Fubini–Study metric in the space of pure states, $`D_{FS}(|\phi ,|\psi )=\mathrm{arccos}(\sqrt{\kappa })`$, where the transition probability is given by $`\kappa =|\phi |\psi |^2`$. Then the largest possible distance $`D_{FS}=\pi /2`$, obtained for any orthogonal states, sets the geodesic length of the complex projective space to $`\pi `$ which corresponds to the geodesic distance of two opposite points on the unit circle, being identified. It is worth to add that $`\mathrm{Vol}_C[𝐏^k]=\mathrm{Vol}[𝐒^{2k+1}]/\mathrm{Vol}[𝐒^1]`$ and this relation was used in BST02  to define the volume VolC of complex projective spaces. We see therefore that different conventions adopted in (4345) lead to different sizes (geodesic lengths) of the manifolds analyzed.

Unitary orbits of a generic mixed state with a non–degenerate spectrum have structure of a $`(N^2N)`$–dimensional complex flag manifold $`Fl_{}^{(N)}=U(N)/[U(1)]^N`$. Hence its volume reads

$$\mathrm{Vol}_X[Fl_{}^{(N)}]=\frac{\mathrm{Vol}_X[U(N)]}{\left(\mathrm{Vol}_X[U(1)]\right)^N}=a_X^{\mathrm{Fl}}\frac{\pi ^{N(N1)/2}}{1!2!\mathrm{}(N1)!}$$
(58)
with convention dependent scale constants $`a_A^{\mathrm{Fl}}=2^{N(N1)/2}`$ Hu63  and $`a_B^{\mathrm{Fl}}=a_C^{\mathrm{Fl}}=1`$ BST02 . It is easy to check that the relation

$$\mathrm{Vol}_X[Fl_{}^{(N)}]=\mathrm{Vol}_X[𝐏^1]\times \mathrm{Vol}_X[𝐏^2]\times \mathrm{}\times \mathrm{Vol}_X[𝐏^{N1}]$$
(59)
holds for any definition (4345), since the scale constants do cancel.

For completeness we discuss also the group $`SU(N)`$, the volume of which is not equal to $`\mathrm{Vol}[U(N)]/\mathrm{Vol}[U(1)]`$ Ma81 ; TBS02 ; BST02 . To show this let us parametrize a matrix $`Y_NSU(N)`$ v

$$Y_N=\left[\begin{array}{cc}e^{i\varphi }& 0\\ 0& e^{i[\varphi /(N1)]}Y_{N1}\end{array}\right]\left[\begin{array}{cc}\sqrt{1|h|^2}& h^{}\\ h& \sqrt{𝕀hh^{}}\end{array}\right]=VW,$$
(60)
where $`\varphi [0,2\pi )`$ is an arbitrary phase and $`h`$ is a complex vector with $`N1`$ components such that $`|h|1`$. Condition det$`Y_N=1`$ implies Tr$`Y_N^1\mathrm{d}Y_N=0.`$ For instance, the metric (42) gives, if the volume is concerned

$$(\mathrm{d}s)^2\mathrm{Tr}(V^1\mathrm{d}V)^2\mathrm{Tr}(W^1\mathrm{d}W)^2.$$
(61)
Since the first factor $`V`$ is block diagonal the first term is equal to $`(\mathrm{d}\varphi )^2N/(N1)\mathrm{Tr}(Y_{N1}^1\mathrm{d}Y_{N1})^2`$, while the second one gives the metric on $`𝐏^{N1}`$. Integrating an analogous expression in the general case of an arbitrary metric and using (56) we obtain the following result

$$\mathrm{Vol}_X[SU(N)]=\frac{\mathrm{Vol}_X[U(N)]}{\mathrm{Vol}_X[U(N1)]}\sqrt{\frac{N}{N1}}\mathrm{Vol}_X[SU(N1)],$$
(62)
which iterated $`N1`$ times gives the correct relation

$$\mathrm{Vol}_X[SU(N)]=\sqrt{N}\frac{\mathrm{Vol}_X[U(N)]}{\mathrm{Vol}_X[U(1)]}$$
(63)
with the stretching factor $`\sqrt{N}`$. For instance, working with the measure (45) and making use of (55) we obtain $`\mathrm{Vol}_C[SU(N)]=\sqrt{N}2^{(N1)/2}\pi ^{(N+2)(N1)/2}/[1!\mathrm{}(N1)!]`$, so in particular, $`\mathrm{Vol}_C[SU(2)]=2\pi ^2`$, $`\mathrm{Vol}_C[SU(3)]=\sqrt{3}\pi ^5`$ and $`\mathrm{Vol}_C[SU(4)]=\sqrt{2}\pi ^9/3`$ in consistence with results obtained in Ma81 ; By98 ; BST02 ; Ca02 .

### A.2 Orthogonal group $`O(N)`$

The analysis of the orthogonal group is simpler, since $`(O^1\mathrm{d}O)^T=(O^1\mathrm{d}O)^T`$, so the diagonal elements of $`\mathrm{Tr}(O^1\mathrm{d}O)^2`$ vanish. Thus we shall consider only two metrics (analogous to the measures (4345)) with different scalings,

$$(\mathrm{d}s_A)^2:=\mathrm{Tr}(O^1\mathrm{d}O)^2=2\underset{j<k=1}{\overset{N}{}}|(O^1\mathrm{d}O)_{jk}|^2,$$
(64)
used in section VII of this work, and

$$(\mathrm{d}s_B)^2=(\mathrm{d}s_C)^2:=\frac{1}{2}\mathrm{Tr}(O^1\mathrm{d}O)^2=\underset{j<k=1}{\overset{N}{}}|(O^1\mathrm{d}O)_{jk}|^2,$$
(65)
which both lead to the Haar measure on the orthogonal group.

To obtain the volume of $`O(N)`$ we proceed as in the unitary case and parametrize an orthogonal matrix of size $`N`$ as

$$O_N=\left[\begin{array}{cc}O_1& 0\\ 0& O_{N1}\end{array}\right]\left[\begin{array}{cc}\sqrt{1|h|^2}& h^T\\ h& \sqrt{𝕀hh^T}\end{array}\right],$$
(66)
where $`O_1O(1)=\pm `$, while $`h`$ is here a real vector with $`N1`$ components such that $`|h|1`$. Representing the metric $`(\mathrm{d}s_B)^2`$ by these two matrices we see that the term containing only the vector $`h`$ gives the metric of a real projective space. Integrating the resulting volume element (with scale factor $`1`$) we obtain the volume of $`𝐏^{N1}`$, equal to $`\frac{1}{2}\mathrm{Vol}[𝐒^{N1}]`$. Taking into account a factor of two resulting from $`O(1)`$ we arrive at Vol$`{}_{B}{}^{}[O(N)]=\mathrm{Vol}_B[O(N1)]\mathrm{Vol}[𝐒^{N1}]`$, which applied recursively leads to

$$\mathrm{Vol}_B[O(N)]=\mathrm{Vol}[𝐒^{N1}]\times \mathrm{}\times \mathrm{Vol}[𝐒^1]\times \mathrm{Vol}[𝐒^0]=\underset{k=1}{\overset{N}{}}\frac{2\pi ^{k/2}}{\mathrm{\Gamma }(k/2)},$$
(67)
where $`\mathrm{Vol}[𝐒^0]=\mathrm{Vol}[O(1)]=2`$. To get an equivalent result for the metric (64) we have to take into account the factor $`\sqrt{2}`$ which occures for each of $`N(N1)/2`$ off-diagonal elements. Doing so we obtain

$$\mathrm{Vol}_A[O(N)]=2^{N(N1)/4}\mathrm{Vol}_B[O(N)]=2^{N(N+3)/4}\underset{k=1}{\overset{N}{}}\frac{\pi ^{k/2}}{\mathrm{\Gamma }(k/2)},$$
(68)
in consistency with Hua Hu63 . In particular, $`\mathrm{Vol}_A[O(1)]=\mathrm{Vol}_B[O(1)]=2`$, while $`\mathrm{Vol}_A[O(2)]=4\sqrt{2}\pi `$, $`\mathrm{Vol}_B[O(2)]=4\pi `$, and $`\mathrm{Vol}_A[O(3)]=32\sqrt{2}\pi ^2`$, $`\mathrm{Vol}_B[O(3)]=16\pi ^2`$.

In full analogy to the unitary case we obtain the volume of the real projective manifold

$$\mathrm{Vol}_X[𝐏^{N1}]=\frac{\mathrm{Vol}_X[O(N)]}{\mathrm{Vol}_X[O(1)]\mathrm{Vol}_X[O(N1)]}.$$
(69)
For the metric (65) this expression reduces to $`\mathrm{Vol}_B[𝐏^k]=\frac{1}{2}\mathrm{Vol}[𝐒^k]`$. Hence, this metric may be called ’unit sphere’ metric, while the convention (64) may be called ’unit trace’ metric.

In the similar way we find the volume of the real flag manifolds, used in analysis of real density matrices,

$$\mathrm{Vol}_X[Fl_{}^{(N)}]=\frac{\mathrm{Vol}_X[O(N)]}{\left(\mathrm{Vol}_X[O(1)]\right)^N}=\frac{1}{2^N}\mathrm{Vol}_X[O(N)].$$
(70)
Exactly as in the complex case we observe that the relation

$$\mathrm{Vol}_X[Fl_{}^{(N)}]=\mathrm{Vol}_X[𝐏^1]\times \mathrm{Vol}_X[𝐏^2]\times \mathrm{}\times \mathrm{Vol}_X[𝐏^{N1}]$$
(71)
is satisfied for any definition of the metric.

Computation of the volume of the special orthogonal group $`SO(N)`$ is much easier than in the complex case, since there are no diagonal elements in the metric and hence, no stretching factors. For any normalization one gets

$$\mathrm{Vol}_X[SO(N)]=\frac{\mathrm{Vol}_X[O(N)]}{\mathrm{Vol}_X[O(1)]}=\frac{1}{2}\mathrm{Vol}_X[O(N)]$$
(72)
In particular, we get $`\mathrm{Vol}_B[SO(2)]=2\pi `$ and $`\mathrm{Vol}_B[SO(3)]=\mathrm{Vol}_C[SO(3)]=8\pi ^2`$. The latter results seem to be inconsistent with $`\mathrm{Vol}_C[SU(2)]=2\pi ^2`$, since there exists a one to two relation between both groups. This paradox is resolved by analyzing the scale effects BST02 : the volume of $`SU(2)`$ is two times larger than the volume of the real projective manifold conjugated to $`SO(3)`$ of the appropriate geodesic length, $`\mathrm{Vol}_C[𝐏^3]=\pi ^2`$.


format: text

text:
# IFT-P.007/2003 IFUSP-DFN/03-079 NSF-ITP-03-09 hep-ph/0302039 Probing the LSND mass scale and four neutrino scenarios with a neutrino telescope

## I Introduction

Atmospheric and solar neutrino experiments present today compelling evidence of neutrino flavor oscillations induced, respectively, by the mass squared differences $`|\mathrm{\Delta }m^2atm|3\times 10^3`$ eV2 Gonzalez-Garcia:2002sm  and $`\mathrm{\Delta }m_{}^27\times 10^5`$ eV2 ntz . These results have been independently supported by the K2K experiment k2k , confirming the $`\nu _\mu /\overline{\nu }_\mu `$ conversion observed in atmospheric neutrino data, and by the KamLAND experiment kamland , recently observing $`\overline{\nu }_e`$ disappearance compatible with what has been seen by the solar neutrino experiments.

In the standard three neutrino flavor oscillation framework, the above results imply neutrino oscillation lengths at energies above about 0.3 TeV much greater than the Earth’s diameter, strongly suppressing $`\nu _\mu \nu _e/\nu _\tau `$ and $`\overline{\nu }_\mu \overline{\nu }_e/\overline{\nu }_\tau `$ oscillations. Moreover, the ratio between $`\nu _e/\overline{\nu }_e`$ and $`\nu _\mu /\overline{\nu }_\mu `$ atmospheric fluxes drops as their energy increases so that atmospheric neutrino data above 0.5 TeV are expected to be basically composed of $`\nu _\mu `$ and $`\overline{\nu }_\mu `$ review-atm . The so called prompt atmospheric neutrinos coming from the decay of charmed particles as well as extra-galactic $`\nu _e,\overline{\nu }_e/\nu _\tau ,\overline{\nu }_\tau `$ contributions are very small compared to the conventional atmospheric $`\nu _\mu /\overline{\nu }_\mu `$ fluxes (which mainly come from the decay of charged pions and kaons) in the 0.5-10 TeV energy range Hooper:2002qq .

However, as we will see, atmospheric $`\nu _e,\overline{\nu }_e/\nu _\tau ,\overline{\nu }_\tau `$ fluxes in the TeV energy range can be non-negligible compared to the $`\nu _\mu /\overline{\nu }_\mu `$ if the result of LSND experiment LSND  which indicates $`\overline{\nu }_\mu \overline{\nu }_e`$ transition generated by neutrino oscillations with a large mass scale, $`|\mathrm{\Delta }m^2lsnd|(0.52.0)`$ eV2, turns out to be correct. The LSND result, which will be very soon tested by the MiniBooNE experiment miniboone , if confirmed, can be interpreted in combination with the atmospheric and solar neutrino data as a signature of an additional neutrino, $`\nu _s`$, sterile in nature. The minimum scheme necessary to explain at same time all three indications of neutrino oscillations involves four neutrino mass eigenstates giunti . We are aware that current analyses performed with the entire set of neutrino data from accelerators, reactors, solar and atmospheric neutrinos seem to disfavor four neutrino mass scenarios dis ; strumia . See, however, Refs. critica  which casted doubt on conclusions drawn in Refs. dis ; strumia . More importantly, no experiment until now has confirmed or ruled out the LSND result and one should wait for results from MiniBooNE.

We will show in this letter that in the framework of four neutrino mass schemes, atmospheric neutrino data at the TeV energy scale may contain a non negligible amount of $`\nu _e`$ or $`\overline{\nu }_e`$, due to the interplay of the Earth matter with the LSND mass scale through the so called MSW effect MSW , and of $`\nu _\tau `$ and $`\overline{\nu }_\tau `$, due to almost vacuum oscillations induced by this same mass scale. To our knowledge it has been never mentioned before in the literature that the LSND mass splitting could give rise to resonant $`\nu _\mu \nu _e`$ or $`\overline{\nu }_\mu \overline{\nu }_e`$ conversion for atmospheric neutrinos. These $`\nu _e,\overline{\nu }_e/\nu _\tau ,\overline{\nu }_\tau `$ events can be observed by a large neutrino telescope, such as the proposed IceCube detector icecube , by measuring the angular distribution of upward-going muons and cascade events. This is because the zenithal dependence of upward-going muons as well as of cascade events will be modified by these new contributions with respect to what would be expected for the standard three neutrino oscillation framework. The zenithal distortions can not only discriminate among the different types of four neutrino mass schemes, but also they are sensitive to the sign of the LSND mass splitting, opening a new window to test four neutrino mass scenarios.

## II Framework of four neutrino mixing schemes

We will work in the minimal model structure needed to describe the three evidence for neutrino oscillations (atmospheric, solar and LSND): a four neutrino mass scheme giunti . We will adopt the same parametrization and some useful notations used in Refs dis ; valle-4nu . For what concerns us here, the sign of $`\mathrm{\Delta }m_{}^2`$ and of $`\mathrm{\Delta }m^2atm`$ are irrelevant, only the sign of $`\mathrm{\Delta }m^2lsnd`$ will be significant. We can then classify the possible four neutrino mass schemes into two categories: (a) (3+1), when a single neutrino mass state is separated from the other neutrino mass states by a large mass splitting ($`|\mathrm{\Delta }m^2lsnd|`$); and (b) (2+2), when two pairs of neutrino mass states are separated by a gap given by the $`|\mathrm{\Delta }m^2lsnd|`$ scale. Moreover, when $`\nu _4`$ is a heavy (light) state we will call it normal (inverted) mass hierarchy. We show in Fig. 1, the (2+2) and (3+1) mass schemes for both, normal and inverted, mass hierarchies which will be considered in this paper.

The evolution equations for neutrinos in the flavor basis are

$$ı\frac{d}{dr}\nu _f=\left[U\frac{M^2}{2E_\nu }U^T+A\right]\nu _f,$$
(1)
where $`\nu _f=(\nu _e\nu _\mu \nu _\tau \nu _s)^T`$ and $`E_\nu `$ is the neutrino energy. We define $`\mathrm{\Delta }m^2lsnd=\mathrm{\Delta }m_{42}^2`$ and $`\mathrm{\Delta }m^2atm=\mathrm{\Delta }m_{32}^2`$, with $`\mathrm{\Delta }m_{ij}^2=m_i^2m_j^2`$, for $`i,j=1,2,3,4`$. The diagonalized neutrino mass squared matrix is

$$M^2=\mathrm{diag}(0,\mathrm{\Delta }\mathrm{m}^2\mathrm{lsnd}+\mathrm{\Delta }\mathrm{m}_{}^2,\mathrm{\Delta }\mathrm{m}^2\mathrm{atm}+\mathrm{\Delta }\mathrm{m}_{}^2\mathrm{\Delta }\mathrm{m}^2\mathrm{lsnd},\mathrm{\Delta }\mathrm{m}_{}^2),$$
(2)
with $`\mathrm{\Delta }m_{}^2=\mathrm{\Delta }m_{41}^2`$ for $`(\mathit{2}+\mathit{2})`$ cases and

$$M^2=\mathrm{diag}(0,\mathrm{\Delta }\mathrm{m}_{}^2,\mathrm{\Delta }\mathrm{m}_{}^2+\mathrm{\Delta }\mathrm{m}^2\mathrm{atm},\mathrm{\Delta }\mathrm{m}^2\mathrm{lsnd}+\mathrm{\Delta }\mathrm{m}_{}^2),$$
(3)
with $`\mathrm{\Delta }m_{}^2=\mathrm{\Delta }m_{21}^2`$ for $`(\mathit{3}+\mathit{1})`$ ones. We use for the $`4\times 4`$ mixing matrix $`U`$ the same parametrization as in Ref. valle-4nu ,

$$U=R_{34}R_{24}R_{23}R_{14}R_{13}R_{12},$$
(4)
where $`R_{ij}`$ are rotation matrices in the subspace $`(i,j)`$ with angle $`\theta _{ij}`$. The matter potential matrix is

$$A=\mathrm{diag}(\mathrm{A}_{\mathrm{cc}},0,0,\mathrm{A}_{\mathrm{nc}}),$$
(5)
with $`A_{\mathrm{cc}}=\sqrt{2}G_F\rho Y_e`$ and $`A_{\mathrm{nc}}=A_{\mathrm{cc}}(1Y_e)/(2Y_e)`$, where $`G_F`$ is the Fermi constant, $`Y_e`$ the electron fraction and $`\rho `$ the matter density. For antineutrinos evolution equations similar to Eq. (1) exist but with $`AA`$. We have ignored all CP violation phases, as they do not play any important rôle in our study.

In order to make clear the connection among mixing parameters and experimental observations, we will introduce some useful definitions. Let us first define the quantities $`d_\alpha `$ ($`\alpha =e,\mu ,\tau ,s`$) as

$$d_\alpha =[\begin{array}{cc}|U_{\alpha 4}|^2\hfill & \hfill \text{for }\text{(3+1)}\text{ mass scheme,}\\ |U_{\alpha 1}|^2+|U_{\alpha 4}|^2\hfill & \hfill \text{for }\text{(2+2)}\text{ mass scheme,}\end{array}$$
(6)
and $`\eta _\alpha `$ ($`\alpha =e,\mu ,\tau ,s`$) as

$$\eta _\alpha =[\begin{array}{cc}|U_{\alpha 1}|^2+|U_{\alpha 2}|^2\hfill & \hfill \text{for }\text{(3+1)}\text{ mass scheme,}\\ |U_{\alpha 1}|^2+|U_{\alpha 4}|^2\hfill & \hfill \text{for }\text{(2+2)}\text{ mass scheme,}\end{array}$$
(7)
which can be related to the neutrino survival and oscillation probabilities at short and long-baseline experiments as we will see below.

Note that $`\eta _s`$ measures the sterile admixture in solar neutrino oscillations, this parameter describes the fraction of sterile neutrino participating in solar oscillations: $`\eta _s=0(1)`$ corresponds to pure active (sterile) $`\nu _e\nu _\tau `$ ($`\nu _e\nu _s`$) oscillations. Solar neutrino data prefer a small admixture of sterile neutrino in the $`\mathrm{\Delta }m_{}^2`$ scale, i.e., $`\eta _s1`$. The sterile admixture for atmospheric neutrinos is characterized by the parameter $`d_s`$, in the $`\mathrm{\Delta }m^2atm`$ mass scale, $`d_s=1(0)`$ corresponds to pure active (sterile) $`\nu _\mu \nu _\tau `$ ($`\nu _\mu \nu _s`$) oscillations. Atmospheric neutrino data prefer a small admixture of sterile neutrino in the $`\mathrm{\Delta }m^2atm`$ scale, i.e., $`1d_s1`$.

We can relate $`d_\alpha `$ to the neutrino survival probability in vacuum for short-baseline disappearance experiments, ignoring $`𝒪(\mathrm{\Delta }m^2atm`$) corrections, by

$$P_{\nu _\alpha \nu _\alpha }=1A_{\alpha ;\alpha }\mathrm{sin}^2\left(\frac{\mathrm{\Delta }m^2lsnd}{4E_\nu }L\right)=14d_\alpha (1d_\alpha )\mathrm{sin}^2\left(\frac{\mathrm{\Delta }m^2lsnd}{4E_\nu }L\right),$$
(8)
where $`L`$ is the baseline distance. From the null results of short-baseline experiments $`A_{e;e}`$ and $`A_{\mu ;\mu }`$ are limited to be very small giunti .

The conversion probability, relevant for short-baseline appearance experiments, can be written as

$$P_{\nu _\mu \nu _e}=A_{\mu ;e}\mathrm{sin}^2\left(\frac{\mathrm{\Delta }m^2lsnd}{4E_\nu }L\right),$$
(9)
where

$$A_{\mu ;e}=[\begin{array}{cc}4|U_{e4}|^2|U_{\mu 4}|^2\hfill & \hfill \text{for }\text{(3+1)}\text{ mass scheme,}\\ 4|U_{e1}U_{\mu 1}^{}+U_{e4}U_{\mu 4}^{}|^2\hfill & \hfill \text{for }\text{(2+2)}\text{ mass scheme.}\end{array}$$
(10)
Here the amplitude $`A_{\mu ;e}`$ can be identified as the LSND mixing amplitude $`\mathrm{sin}^22\theta lsnd`$ LSND .

For the long-baseline reactor ($`L<1`$ km) and accelerator experiments ($`L<1000`$ km), we have to include effects of the $`\mathrm{\Delta }m^2atm`$ mass scale, and the conversion probability, $`P_{\nu _\mu \nu _e}ch`$ becomes

$$P_{\nu _\mu \nu _e}ch=12d_e(1d_e)Ach\mathrm{sin}^2\left(\frac{\mathrm{\Delta }m^2atm}{4E_\nu }L\right),$$
(11)
with

$$Ach=[\begin{array}{cc}4\eta _e(1d_e\eta _e)\hfill & \hfill \text{for }\text{(3+1)}\text{ mass scheme,}\\ 4|U_{e2}|^2|U_{e3}|^2\hfill & \hfill \text{for }\text{(2+2)}\text{ mass scheme.}\end{array}$$
(12)
To accommodate the (2+2) mass scheme, $`d_\mu `$, $`Ach`$ and $`A_{\mu ;e}`$ have to be small and $`d_e`$ close to 1 (to keep $`A_{e;e}`$ amplitude small). Also the parameters $`\eta _s`$ and $`d_s`$ are not independent, $`\eta _s=d_s`$ as noted by the authors of Ref. 3+1 . From Ref.valle-4nu  we know that the global fit of atmospheric and solar neutrino data prefers either the limit $`\eta _s=d_s1`$ or the limit $`\eta _s=d_s1`$. For the (3+1) mass scheme, $`d_\mu `$,$`d_e`$, $`Ach`$, $`A_{\mu ;e}`$, $`\eta _s`$ and $`1d_s`$, all, have to be small

In this paper, in order to give an explicit example of the effect of four neutrino mass schemes we will fix the values of the mixing parameters. These were taken within the parameter region allowed by solar and atmospheric neutrino data as well as by the null results of reactors and accelerators (with the exceptions of K2K k2k  and KamLAND kamland giunti ; Gonzalez-Garcia:2002sm . We have scanned a grid of values to find the maximal possible values of the six mixing angles in order to maximize the oscillation effect driven by the LSND mass scale in IceCube, using the constraints on $`d_\alpha `$, $`A_{\mu ;e}`$, $`Ach`$, $`\eta _e`$ and $`\eta _s`$ shown in Ref. valle-4nu . The result of our scan is shown in Table 1. For the neutrino mass splittings, we have used the best fit values $`\mathrm{\Delta }m^2atm=3\times 10^3`$ eV2 and $`\mathrm{\Delta }m_{}^2=7\times 10^5`$ eV2. For definiteness we also have set $`|\mathrm{\Delta }m^2lsnd|=0.5`$ eV2.

We have numerically solved Eq. (1) to compute $`P(\nu _\mu \nu _\alpha )`$ for $`\alpha =e,\mu `$,$`\tau `$ and $`s`$, the oscillation probabilities that are relevant for atmospheric neutrino data. Let us make some observations that can help the reader to understand our numerical results.

(2+2) case:

From the expressions of the evolution equations, one can expect matter enhanced MSW resonant conversion in Earth for the neutrino (antineutrino) channel $`\nu _\mu \nu _e`$ ($`\overline{\nu }_\mu \overline{\nu }_e`$), for the 2+2 inverted (normal) hierarchy if the following condition is satisfied,

$$E_\nu ^\mathrm{R}|\mathrm{\Delta }m^2lsnd|\times \frac{d_ed_\mu }{2A_{\mathrm{cc}}}1.7\text{ TeV}\times (d_ed_\mu )\times \left(\frac{|\mathrm{\Delta }m^2lsnd|}{0.5\text{eV}^2}\right)\times \left(\frac{2.0\mathrm{g}/\mathrm{cc}}{Y_e\rho }\right).$$
(13)
Other resonances can be induced by the $`\mathrm{\Delta }m^2atm`$ mass scale at $`E_\nu <100`$ GeV, but they are irrelevant for our purpose here. In the (2+2) mass scheme, $`d_e1`$ and $`d_\mu 1`$ so that the resonant condition can be achieved in the TeV energy range. For the $`\nu _\mu \nu _\tau `$ channel we expect almost vacuum oscillations.

In Fig. 2 we show, for diametral trajectories ($`\mathrm{cos}\theta _z=1`$), the behavior of the neutrino and antineutrino $`\nu _\mu \nu _\mu `$ and $`\nu _\mu \nu _\tau ,\nu _e,\nu _s`$ transition probabilities for the (2+2) case with normal hierarchy. To provide a better understanding of what is going on, in panels (a)-(d) the matter effects were not taken into account while in panels (e)-(h) they were. We also show for comparison what would be expected for the standard three flavor neutrino oscillation scenario (labeled as 3(g)).

If we compare Fig. 2(c) and Fig. 2(g), we see a sizable $`\overline{\nu }_\mu \overline{\nu }_e`$ conversion probability when matter effects are taken into account. This resonance behavior is due to the LSND mass scale and Earth matter effects MSW . Looking closely to Fig. 2(g), we can see three peaks: one at $`E_\nu ^\mathrm{R}0.57`$ TeV for a typical value of Earth core density ($`\rho _{\mathrm{core}}12\mathrm{g}/\mathrm{cc}`$), and another for $`E_\nu ^\mathrm{R}1.5`$ TeV for a typical value of Earth mantle density, ($`\rho _{\mathrm{mantle}}4\mathrm{g}/\mathrm{cc}`$). A third peak appears for $`E_\nu ^\mathrm{R}0.8`$ TeV and its origin is due to a parametric resonance. Similar behavior was noticed sometime ago in the context of neutrino oscillations induced by $`\mathrm{\Delta }m^2atm`$ for $`\nu _\mu \nu _s`$ smirnov-atm . To determine the behavior of oscillations in $`\nu _\mu \nu _\tau `$ channel, we can use Eq. (9), changing $`e\tau `$, so that $`P_{\nu _\mu \nu _\tau }=A_{\mu ;\tau }\mathrm{sin}^2(\mathrm{\Delta }m^2lsndL/(4E_\nu ))`$. The conversion amplitude for this channel, $`A_{\mu ;\tau }=4\mathrm{cos}^2\theta _{34}\mathrm{cos}^2\theta _{24}\mathrm{sin}^2\theta _{24}`$ depends only on $`\mathrm{sin}^2\theta _{24}`$ and $`\mathrm{sin}^2\theta _{34}`$. We confirmed numerically that this is correct and we have virtually vacuum oscillations induced by $`\mathrm{\Delta }m^2lsnd`$ as you can see comparing Figs. 2(b) and (f). In the case of inverted mass hierarchy, the neutrino and antineutrino probabilities are interchanged.

(3+1) case:

In the (3+1) mass scheme, we cannot expect a strong resonant $`\nu _\mu \nu _e`$ ($`\overline{\nu }_\mu \overline{\nu }_e`$) transition due to the LSND mass scale at TeV energies. This can be explained as follows. First, both $`\nu _\mu `$ and $`\nu _e`$ ($`\overline{\nu }_\mu `$ and $`\overline{\nu }_e`$) are basically distributed among the mass triplet so that we cannot expect a significant direct transition between them driven by $`\mathrm{\Delta }m^2lsnd`$. However, a second order transition can be achieved via $`\nu _4`$ which is mainly composed of $`\nu _s`$ as $`\nu _\mu \nu _s\nu _e`$ ($`\overline{\nu }_\mu \overline{\nu }_s\overline{\nu }_e`$).

For the normal mass hierarchy, the first step, $`\nu _\mu \nu _s`$ ($`\overline{\nu }_\mu \overline{\nu }_s`$), occurs via almost vacuum oscillation (a MSW resonance effect), while the second step, $`\nu _s\nu _e`$ ($`\overline{\nu }_s\overline{\nu }_e`$), occurs via a MSW resonance effect with $`E_\nu ^\mathrm{R}`$ given by Eq. (13) with $`d_ed_\mu `$ replaced by $`d_s`$ (almost vacuum oscillation). The strength of these transitions is regulated by the value of $`\theta _{14}`$ and $`\theta _{24}`$, which are very much constrained by current data (see Table I). This is why we do not expect to observe a large number of $`\nu _e`$ ($`\overline{\nu }_e`$) events in the (3+1) scheme. In the case of inverted mass hierarchy, the neutrino and antineutrino probabilities are interchanged.

On the other hand, $`\nu _\mu \nu _\tau `$ ($`\overline{\nu }_\mu \overline{\nu }_\tau `$) transitions driven by the LSND mass scale can occur directly mainly via vacuum-like oscillation, since there is a significant amount of $`\nu _\tau `$ in the $`\nu _4`$ state. This is due to the large value of $`\theta _{34}`$ (see Table I), which is the maximal value still allowed by data.

These observations are confirmed by our numerical calculations. In Fig. 3 we show, for diametral trajectories ($`\mathrm{cos}\theta _z=1`$), the behavior of the neutrino and antineutrino $`\nu _\mu \nu _\mu `$ and $`\nu _\mu \nu _\tau ,\nu _e,\nu _s`$ transition probabilities for the (3+1) case with normal hierarchy in a similar way as in Fig. 2.

From these observations we should expect at the TeV energy scale $`\nu _\mu \nu _e`$ matter enhanced resonant transitions for neutrinos (antineutrinos) if the LSND mass hierarchy is normal (inverted) in the (2+2) scheme and $`\nu _\mu \nu _\tau `$ as well as $`\overline{\nu }_\mu \overline{\nu }_\tau `$ vacuum-like transitions in both mass schemes.

## III Types of signal events

We have studied the observable consequences of these four neutrino schemes at high energies, in a future neutrino telescope. For definiteness we have assumed a kilometer-scale neutrino telescope as the one proposed by the IceCube collaboration icecube  that, for simplicity, has been simulated with perfect efficiency as well as perfect angular and energy resolution. We have supposed that the IceCube detector can be described by a cylinder with 900 m of height and 500 m radius.

We have calculated two types of measurable signals in this detector: upward-going muons and cascades events. The calculations were done using the atmospheric neutrino fluxes, denoted by $`\mathrm{\Phi }_{\nu _\alpha }`$, $`\alpha =e,\mu `$, of the Bartol group bartol  and the cross sections from Ref. gandhi . We have also included in our calculations the effect of neutrino absorption when crossing the Earth, taking from Ref. gandhi .

### III.1 Upward-going muons

Muon neutrino charged current interactions, occurring inside the Earth close enough to the detector, may produce energetic muon events which can cross the detector volume, giving rise to the so-called upward-going muons. These events are characterized by a very good angular resolution.

We have calculated the number of upward-going muon events as a function of the zenith angle, $`N_\mu (\theta _z)`$, as in Refs. lipari ; nos ,

$$N_\mu (\theta _z)=T_{E_{\mu ,\mathrm{min}}}^{\mathrm{}}A(R_{\mathrm{min}},\theta _z)\frac{d\mathrm{\Phi }_\mu (E_\mu ,\mathrm{cos}\theta _z)}{dE_\mu d\mathrm{cos}\theta _z}𝑑E_\mu +\text{antineutrinos},$$
(14)
with

$`{\displaystyle \frac{d\mathrm{\Phi }_\mu }{dE_\mu d\mathrm{cos}\theta _z}}`$ $`=`$ $`N_A{\displaystyle _{E_\mu }^{\mathrm{}}}𝑑E_{\mu 0}{\displaystyle _{E_{\mu 0}}^{\mathrm{}}}𝑑E_\nu {\displaystyle _0^{\mathrm{}}}𝑑h\kappa _{\nu _\mu }(h,\mathrm{cos}\theta _z,E_\nu )`$ (15)
$`\times {\displaystyle \frac{d\mathrm{\Phi }_{\nu _\mu }(E_\nu ,\theta _z)}{dE_\nu d\mathrm{cos}\theta _z}}P(\nu _\mu \nu _\mu ){\displaystyle \frac{d\sigma (E_\nu ,E_{\mu 0})}{dE_{\mu 0}}}F_{\mathrm{rock}}(E_{\mu 0},E_\mu )S(E_\nu ,\theta _z),`$
where $`T`$ is the livetime that we assumed to be 1 yr, $`N_A`$ is the Avogadro number, $`E_{\mu 0}`$ is the energy of the muon produced in the neutrino interaction and $`E_\mu `$ is the muon energy when entering the detector, $`F_{\mathrm{rock}}(E_{\mu 0},E_\mu )`$ is a function that takes into account the muon energy loss from the point where the muon is created to where it is detected lipari , $`\kappa _\alpha `$ is the slant distance distribution normalized to one pathlength , $`P(\nu _\mu \nu _\mu )`$ is the muon survival probability, $`A(R_{\mathrm{min}},\theta _z)`$ is the effective area for a muon traveling a distance $`R_{\mathrm{min}}`$ inside the detector coming from the direction $`\theta _z`$ lipari , $`\sigma `$ is the $`\nu _\mu N\mu X`$, neutrino cross section, $`\mathrm{cos}\theta _z`$ labels both the neutrino and the muon directions which to a very good approximation at the relevant energies are collinear, $`S(E_\nu ,\theta _z)`$ is a shadow factor that is related to the attenuation of neutrinos traversing the Earth gandhi . For a given zenith angle, the threshold energy cut is obtained by equating $`R(E_{\mu ,\mathrm{min}})=R_{\mathrm{min}}=100`$ m, where $`R(E_{\mu ,\mathrm{min}})`$ is the muon range function.

In Fig. 4 we show the zenithal angle distribution of these events expected in the case of the standard three flavor oscillations (or no oscillations at all) and for each of the four neutrino mass schemes assuming $`|\mathrm{\Delta }m^2lsnd|=0.5`$ eV2 and the oscillation parameters as given in Table 1. It is clear from this plot that the distribution of upward-going muons reflects the $`\nu _\mu /\overline{\nu }_\mu `$ disappearance into other flavors promoted by each of the four neutrino schemes, making them quite distinguishable from the standard three flavor oscillation case. It may be even possible to distinguish among (2+2) and (3+1) schemes, although their normal and inverted cases are virtually the same due to the fact that the mean energy of these events is much lower than the resonant energy ($`E_\nu ^\mathrm{R}0.8`$ TeV for $`\mathrm{\Delta }m^2lsnd=0.5`$ eV2). From the Eq. (14) we note that all muons with an energy sufficient high to travel a distance $`R_{\mathrm{min}}`$ are classified as upward-going muons, the original muon energy is integrated out so that the correlation between the original neutrino energy and the final muon energy is weak.

As a test of capability to discriminate the four neutrino mass schemes from the standard three generation oscillation we performed a simple $`\chi ^2`$ analysis, as follows. If we define $`N_\mu ^i(4\nu )`$ ($`N_\mu ^i`$(3g)) as the number of upward-going muons in the $`i`$-th zenith angle bin in four neutrino mass schemes (standard three neutrino mass scheme) showed in Fig. 4, then we can compute the function

$$\chi ^2=\underset{i=1,20}{}\frac{(\eta N_\mu ^i(4\nu )N_\mu ^i(3g))^2}{(\sigma _{(4\nu )}^i+\sigma _{(3g)}^i)^2},$$
(16)
where $`\sigma _{(4\nu )}^i`$ ($`\sigma _{(3g)}^i`$) is the statistical error on $`N_\mu ^i(4\nu )`$ ($`N_\mu ^i(3g)`$), for simplicity we have ignored systematic uncertainties. The parameter $`\eta `$ is a normalization factor. If we assumed predictions of upward going muons in standard three generation are the correct ones, then the only possible change is a global normalization factor. We scan the parameter $`\eta `$ to look for the best fit and for any of neutrino mass schemes the minimum $`\chi ^2`$ is much higher then the degrees of freedom, 20-1 (20 points minus 1 normalization factor), with a goodness of fit well below $`1\%`$. Then we conclude that for the choice of mixing parameters in this paper, it is very likely that the zenith angular distribution of upward going muons is sufficient to be separated from the usual standard three neutrino oscillations. From this we can say that independent of which four neutrino mass scheme nature chooses, we can always have a distinctive zenith distortion in upward-going muons.

### III.2 Cascades

Electromagnetic and hadronic showers can be produced by ionizing products of the neutrino interactions with the ice. These types of events can be created by electron neutrino charged current interactions ($`\nu _e\mathrm{CC}`$), tau neutrino charged current interactions ($`\nu _\tau \mathrm{CC}`$) and neutral current interactions ($`\nu \mathrm{NC}`$). These events, which are called cascade events, are characterized by a very good energy resolution. In principle, muon neutrinos also produce a hadronic shower and a muon and this case contribute to cascade events. We assume events with a muon track always can be separated from the other cascade events and we will not include them in our computation. We have calculated the number of cascade events for a given zenith angle, $`N_{\mathrm{cascade}}(\theta _z)`$, as in Refs. stanev ; reno ,

$$N_{\mathrm{cascade}}(\theta _z)=TVN_A_{E_{\mathrm{shr},\mathrm{min}}}^{\mathrm{}}\frac{d\mathrm{\Phi }_{\nu _\mu }(E_\nu ,\mathrm{cos}\theta _z)}{dE_\nu d\mathrm{cos}\theta _z}P(\nu _\mu \nu _e)\underset{i}{}f_i(E_\nu )dE_\nu +\text{antineutrinos},$$
(17)
here $`V`$ is the IceCube experiment volume (1 km3), $`f_i(E_\nu ,z)`$, $`i=\nu _e`$CC, $`\nu _\tau `$CC (hadronic plus electromagnetic showers), $`\nu `$NC, are functions which depend on the interaction,

$$\begin{array}{cc}f_{\nu _e\mathrm{CC}}(E_\nu )=𝑑y\frac{d\sigma _{cc}(E_\nu ,y)}{dy}\mathrm{\Theta }(E_\nu E_{\mathrm{shr},\mathrm{min}}),\hfill & \hfill \text{for}\nu _e\text{CC}\\ f_{\nu _\tau \mathrm{CC}}^1(E_\nu )=𝑑y𝑑z\frac{d\sigma _{cc}(E_\nu ,y)}{dy}\frac{dn(E_\tau )}{dz}\mathrm{\Theta }(E_\nu (y+(1y)(1z))E_{\mathrm{shr},\mathrm{min}}),\hfill & \hfill \text{for}\nu _\tau \text{CC hadronic}\\ f_{\nu _\tau \mathrm{CC}}^2(E_\nu )=𝑑y𝑑z^{}\frac{d\sigma _{cc}(E_\nu ,y)}{dy}\frac{dn(E_\tau )}{dz^{}}\mathrm{\Theta }(E_\nu (y+(1y)z^{})E_{\mathrm{shr},\mathrm{min}}),\hfill & \hfill \text{for}\nu _\tau \text{CC electromagnetic}\\ f_{\nu \mathrm{NC}}(E_\nu )=𝑑y\frac{d\sigma _{nc}(E_\nu ,y)}{dy}\mathrm{\Theta }(E_\nu yE_{\mathrm{shr},\mathrm{min}}),\hfill & \hfill \text{for}\nu \text{NC}\end{array}$$

where $`E_\tau `$ is the tau energy, $`y,z,z^{}`$ are the original neutrino energy fractions carried by the relevant lepton/hadron at play, and we have considered only events which have a total shower energy greater than 500 GeV ($`E_{\mathrm{shr},\mathrm{min}}=500`$ GeV).

For $`\nu _\tau `$CC events we can have hadronic and electromagnetic cascades, respectively, from $`\nu _\tau \tau e`$ decay channel and from $`\nu _\tau \tau h`$, where $`h`$ means any hadrons. For $`\nu `$NC ($`\nu _e`$CC) events we have respectively hadronic (electromagnetic) showers. The functions $`dn(E_\tau )/dz`$ and $`dn(E_\tau )/dz^{}`$ are given in Ref. reno . Although we also integrate on the neutrino energy as for the upward-going muons computation, the average shower energy for this sample is more closely related to the average initial neutrino energy.

Since for the energies considered here ($`E_\nu >1`$ TeV), the atmospheric neutrino fluxes are basically composed of $`\nu _\mu `$ and $`\overline{\nu }_\mu `$ review-atm  and according to the usual three neutrinos oscillation scenario electron and tau neutrino appearance are suppressed, therefore one generally expects that only the neutral current contribution can produce cascades at these energies. A possible exception to this would be perhaps charged current contributions induced by $`\nu _e`$/$`\nu _\tau `$ neutrinos emitted in microquasar jets gdlw , but these events are very well collimated with the source and can be easily separated from the atmospheric neutrino data. As we have shown earlier, in four neutrino schemes, electron and tau neutrino components will appear at these energies so that an increase on the number of cascade events should be expected.

We show in Fig. 5 the zenithal distribution of cascade events that can be expected in IceCube after one year of data taking, for the standard three neutrino oscillation case as well as for all the four neutrino schemes, assuming $`|\mathrm{\Delta }m^2lsnd|=0.5`$ eV2 and the oscillation parameters as given in Table 1. All four neutrino cases, for these values of the parameters, can be distinguished from the standard case. We observe that there is an increase on the number of cascade events at $`\mathrm{cos}\theta _z1`$ for the (2+2) inverted scheme due to the resonant appearance of $`\nu _e`$ similar to what is shown in Fig. 2 (g) for $`\overline{\nu }_e`$. There is no such a prominent peak for the (2+2) normal scheme due to the fact that in this case the resonance is in the antineutrino channel (where one has a lower flux and smaller cross section). The increase of the number of cascade events in the (3+1) schemes is almost exclusively due to $`\nu _\tau `$ appearance. In fact, the angular distribution of cascade events reflect the flavor content of the atmospheric neutrino flux as measured at the detector site.

We observe that cascade events are more effective in separating the mass hierarchy of four neutrino schemes than the upward-going muon events. This is not surprising since the 500 GeV energy cut imposed on the cascade showers selects events closer to the resonant energy where the effect of the large $`|\mathrm{\Delta }m^2lsnd|=0.5`$ eV2 is more sizable. For higher values of $`|\mathrm{\Delta }m^2lsnd|`$ the differences among the standard and the four neutrino oscillations distributions could be even larger than what we have shown in Figs. 4 and 5 if not for the fact the mixing angles are much more constrained by data for higher masses valle-4nu . In any case the general behavior of the curves shown in this paper will still hold for $`|\mathrm{\Delta }m^2lsnd|`$ as high as 2.0 eV2.

We note that unlike the case of upward-going muon events, cascade events can be observed also for $`\mathrm{cos}\theta _z>0`$, which correspond to down-going events. For such events, for any four neutrino mass scheme we have considered here, we do not expect any significant deviation of the zenith angle distribution from what is expected in the standard three neutrino oscillation scheme. In the standard scheme, in good approximation, the zenith angle distribution is symmetric with respect to $`\mathrm{cos}\theta _z=0`$. This is because of the fact that for down-going cascade events, there is no matter effect and the distance traveled by neutrinos is much smaller than the typical oscillation length expected for $`|\mathrm{\Delta }m^2lsnd|`$ scale and $`E_\nu 1`$ TeV so that no significant $`\nu _e,\overline{\nu }_e/\nu _\tau ,\overline{\nu }_\tau `$ induced events are expected. Therefore, by comparing the zenith distributions for $`\mathrm{cos}\theta _z>0`$ and $`\mathrm{cos}\theta _z<0`$, it is expected that we can discriminate four neutrino mass schemes from the standard three one with better sensitivity.

Finally, let us note that the uncertainty in the prediction of the atmospheric muon neutrino flux, which is currently about 15%, could potentially spoil the discriminating power among the different signals. This can be, to some extent, overcome by combining cascade and upward-going muon events, and so reducing the uncertainty in the overall flux. Also, the shape of the event distributions, here the key point in disentangling the different signals, is not very much affected by the overall flux normalization.

## IV Conclusions

We have studied the rôle that can be played by a large mass splitting implied by the LSND signal, $`|\mathrm{\Delta }m^2lsnd|(0.52.0)`$ eV2 in a four neutrino mass scheme for atmospheric neutrino data in the energy range $``$ 1 TeV. We have shown that upward-going muon events as well as cascade events, that can be measured by a kilometer scale detector such as the one proposed by the IceCube collaboration, can indicate the presence of the fourth neutrino through the production of $`\nu _e,\overline{\nu }_e`$ and $`\nu _\tau ,\overline{\nu }_\tau `$ which are not expected in the standard three generation oscillation case where we have only the small mass splittings $`\mathrm{\Delta }m_{}^2`$ and $`\mathrm{\Delta }m^2atm`$. Furthermore the zenithal dependence of cascade events is even sensitive to the sign of $`\mathrm{\Delta }m^2lsnd`$ because the matter effect play an important rôle, leading to resonant $`\nu _\mu \nu _e`$ or $`\overline{\nu }_\mu \overline{\nu }_e`$ conversion depending on the sign of $`\mathrm{\Delta }m^2lsnd`$. We note that the standard three generation oscillation scheme cannot induce these signals of $`\nu _e`$/$`\nu _\tau `$ appearance at the 1 TeV scale, and therefore, they constitute an unique signature of a large mass scale present in a four neutrino mass scheme.

A final remark. If MiniBooNE observes a positive signal, the effects shown here should be expected independent of the four neutrino mass scheme. On the other hand, if MiniBooNE results turns out to be negative this may be viewed as a complementary test of the presence of a large neutrino mass splitting in nature.

###### Acknowledgements.

This work was supported by Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP), Conselho Nacional de Ciência e Tecnologia (CNPq), Fundo de Apoio ao Ensino e à Pesquisa (FAEP) and by the National Science Foundation under Grant No. PHY99-07949. We thank T. Stanev for providing the table of the atmospheric neutrino fluxes. We also thank C. Lunardini for bringing to our attention Ref. smirnov-atm  and A. de Gouvea for useful comments. We are grateful for the hospitality of the Kavli Institute for Theoretical Physics, of the University of California in Santa Barbara, where this work was completed.