format: text

text:
# Critical heat kernel estimates via Hardy-Sobolev inequalities

## 1 Introduction

The purpose of this paper is to obtain some new Hardy-Sobolev inequalities and then use them in order to obtain new heat kernel estimates for the Schrödinger operator $`\mathrm{\Delta }V`$ for positive potentials $`V`$ with critical singularities, improving upon analogous estimates of this type.

As a typical example, let us consider the case of a bounded domain $`\mathrm{\Omega }𝐑^N`$, $`N3`$, containing the origin. We obtain upper estimates on the heat kernel of the operator

$$Hu=\mathrm{\Delta }u\frac{\lambda }{|x|^2}u,u|_\mathrm{\Omega }=0,$$
(1.1)
for various values of the real parameter $`\lambda `$. It is well known that the power $`|x|^2`$ is critical and that the associated heat kernel exhibits behaviour which is different from that of the case $`|x|^\gamma `$, $`\gamma <2`$, which is in the Kato class. Recently Milman and Semenov \[MS\] obtained upper heat kernel estimates for the operator (1.1) when $`\lambda <((N2)/2)^2`$. In the case $`0<\lambda <((N2)/2)^2`$ they showed that, modulo the usual Gaussian term, the heat kernel satisfies

$$K(t,x,y)<ct^{\frac{N}{2}}|x|^\alpha |y|^\alpha ,$$

where $`\alpha `$ denotes the smallest solution of $`\alpha (N2\alpha )=\lambda `$. In Theorem 4.11 we extend this result to the critical case $`\lambda =((N2)/2)^2`$, namely we prove that the corresponding kernel satisfies

$$K(t,x,y)<ct^{N/2}|x|^{\frac{N2}{2}}|y|^{\frac{N2}{2}}.$$

This estimate is sharp as can be seen by comparing with the results of Vázquez and Zuazua \[VZ\]. Moreover, for the subcritical case $`\lambda <((N2)/2)^2`$ and for $`\lambda <0`$ we improve the small-time dependence obtained in \[MS\], namely we prove the estimate $`K(t,x,y)<ct^{\frac{N}{2}+\alpha }|x|^\alpha |y|^\alpha `$ instead of $`K(t,x,y)<ct^{\frac{N}{2}+\alpha ϵ}|x|^\alpha |y|^\alpha `$; see Proposition 4.8.

In addition we consider operators that act on the whole of $`𝐑^N`$ with potentials having the critical Hardy singularity near zero, of the form

$$V_ϵ(x)=\{\begin{array}{cc}\left(\frac{N2}{2}\right)^2|x|^2,\hfill & |x|<1,\hfill \\ ϵf(x),\hfill & |x|>1,\hfill \end{array}$$
(1.2)
under appropriate subcritical assumptions on the positive function $`f`$. Thus, in Theorem 4.13 it is shown that if $`ϵ>0`$ is small enough then the heat kernel of $`\mathrm{\Delta }V_ϵ`$ satisfies

$$K(t,x,y)<ct^{N/2}\mathrm{max}\{|x|^{\frac{N2}{2}},1\}\mathrm{max}\{|y|^{\frac{N2}{2}},1\}.$$
(1.3)
We also consider potentials that exhibit the critical behaviour $`((N2)/2)^2|x|^2`$ near infinity, that is

$$\widehat{V}_ϵ(x)=\{\begin{array}{cc}ϵg(x),\hfill & |x|<1,\hfill \\ (\frac{N2}{2})^2|x|^2,\hfill & |x|>1.\hfill \end{array}$$
(1.4)
Under appropriate subcritical assumptions on $`g`$ we obtain Sobolev estimates for $`\mathrm{\Delta }\widehat{V}_ϵ`$ for a sharp range of $`ϵ>0`$. We note here that while the question of Sobolev inequalities for $`\mathrm{\Delta }\widehat{V}_ϵ`$ is rather similar to that for $`\mathrm{\Delta }V_ϵ`$, when it comes to heat kernel estimates essential differences arise. As mentioned earlier, the Sobolev inequality for $`\mathrm{\Delta }V_ϵ`$ yields estimate (1.3) for the corresponding heat kernel. On the other hand, while the short-time behaviour of the heat kernel of $`\mathrm{\Delta }\widehat{V}_ϵ`$ is similar to that of the Laplacian, the long-time behaviour is very different. Zhang \[Z\] used a parabolic Harnack inequality to obtain precise estimates for the heat kernel of $`\mathrm{\Delta }V`$, when $`V`$ is equal to $`\lambda |x|^2`$, $`\lambda <((N2)/2)^2`$, near infinity. This generalizes the estimates given by Davies and Simon \[DS\], where, however, more precise estimates were given. The corresponding problem for the critical case $`\lambda =((N2)/2)^2`$ remains open.

Going back to bounded $`\mathrm{\Omega }𝐑^N`$ and to the operator $`H`$ defined in (1.1), for the critical case $`\lambda =((N2)/2)^2`$ we finally consider additional singularities, that is, we consider potentials of the form $`((N2)/2)^2|x|^2+V_1`$, where $`V_1>0`$ is also critical; $`V_1`$ is defined as a series involving iterated logarithms (see definition (5.17)) and is critical in the sense that the following improved Hardy inequality holds

$$_\mathrm{\Omega }|u|^2\left(\frac{N2}{2}\right)^2_\mathrm{\Omega }\frac{u^2}{|x|^2}_\mathrm{\Omega }V_1u^2,uC_c^{\mathrm{}}(\mathrm{\Omega }),$$
(1.5)
whereas this inequality is no longer true if we replace $`V_1`$ by $`(1+ϵ)V_1`$ for any $`ϵ>0`$. It is remarkable that the extra potential $`V_1`$ does not affect the time dependence of the heat kernel estimates, but only affects the spatial singularity at the origin (cf Theorems 4.11 and 5.3). This is in contrast with Proposition 4.8(ii) where, for $`\lambda <0`$, the potential affects the the time singularity of the heat kernel as well.

Throughout the paper we study a number of concrete potentials. These are chosen precisely because they are critical. By simple monotonicity one can then obtain heat kernel estimates for a whole range of other potentials, including potentials that are not radially symmetric.

To prove the above heat kernel estimates we first use an appropriate change of variables, $`u=\varphi w`$, by means of which, the problem is reduced to obtaining uniform estimates on the heat kernel $`K_\varphi (t,x,y)`$ of an auxiliary operator $`H_\varphi `$ which acts on the function $`w`$; see, e.g., \[MS\]. Those estimates are in turn proved by means of some new Hardy- Sobolev inequalities.

As a typical example of such an inequality we mention the following inequality proved by Brezis and Vázquez \[BV\]:

$$_\mathrm{\Omega }|u|^2\left(\frac{N2}{2}\right)^2_\mathrm{\Omega }\frac{u^2}{|x|^2}K\left(_\mathrm{\Omega }|u|^p𝑑x\right)^{2/p},$$
(1.6)
valid for $`uH_0^1(\mathrm{\Omega })`$ and $`1<p<\frac{2N}{N2}`$; this inequality fails for the critical Sobolev exponent $`p=\frac{2N}{N2}`$. To obtain sharp heat kernel estimates one needs to go up to the critical exponent. In connection with this we mention the following sharp Hardy-Sobolev inequality established in \[FT\]:

$$_\mathrm{\Omega }|u|^2\left(\frac{N2}{2}\right)^2_\mathrm{\Omega }\frac{u^2}{|x|^2}c\left(_\mathrm{\Omega }|u|^{\frac{2N}{N2}}X_1^{1+\frac{N}{N2}}\left(\frac{|x|}{D}\right)𝑑x\right)^{\frac{N2}{N}},$$
(1.7)
valid for $`uH_0^1(\mathrm{\Omega })`$; here $`D=sup_\mathrm{\Omega }|x|`$ and $`X_1(t)=(1\mathrm{log}t)^1`$, $`t(0,1)`$. In the present work we derive new Hardy-Sobolev inequalities that involve potentials such as the ones given in (1.2) or (1.4); see Theorems 3.4, 3.5, 5.1 and 5.2. We should mention that the validity of improved Hardy inequalities is strongly connected to the existence and large time behaviour of solutions of the heat equation with singular potential; see, e.g., Brezis and Vázquez \[BV\], Cabré and Martel \[CM\], Dávila and Dupaigne \[DD\] as well as Vázquez and Zuazua \[VZ\].

As a byproduct of our approach we establish various results concerning improved Hardy inequalities with boundary terms. Such inequalities have recently attracted attention, see Adimurthi and Esteban \[AE\], Wang and Zhu \[WZ\] and references therein.

The structure of the paper is as follows: in Section 2 we present some auxiliary results concerning improved Hardy inequalities with boundary terms. In Section 3 we prove the Hardy-Sobolev inequalities; in Section 4 we apply them to obtain heat kernel estimates. Finally, in Section 5 we prove refined Sobolev inequalities and heat kernel estimates when additional singularities are present.

Acknowledgment We acknowledge partial support by the RTN European network Fronts–Singularities, HPRN-CT-2002-00274.

## 2 Two minimization problems

Throughout this section, $`\mathrm{\Omega }𝐑^N`$, $`N3`$, is a bounded domain containing the origin with $`C^1`$ boundary. Also, we always denote by $`\nu `$ the outward-pointing (with respect to $`\mathrm{\Omega }`$) unit vector on the surface $`\mathrm{\Omega }`$. In Section 2.1 we will work on $`\mathrm{\Omega }`$, while in Section 2.2 we will work on $`\mathrm{\Omega }^c`$. The results of this section will be applied in Section 3.

### 2.1 Bounded domains

For $`\alpha >0`$ we define

$$\lambda _\mathrm{\Omega }(\alpha )=\underset{H^1(\mathrm{\Omega })}{inf}\frac{_\mathrm{\Omega }|u|^2𝑑x+\alpha _\mathrm{\Omega }\frac{x\nu }{|x|^2}u^2𝑑S}{_\mathrm{\Omega }\frac{u^2}{|x|^2}𝑑x}.$$
(2.1)
###### Lemma 2.1

Let $`\mathrm{\Omega }`$ be a bounded domain in $`𝐑^N`$, $`N3`$, containing the origin.
$`(\mathrm{i})`$ If $`0<\alpha \frac{N2}{2}`$, then $`\lambda _\mathrm{\Omega }(\alpha )=\alpha (N2\alpha )`$. Moreover, $`|x|^\alpha H^1(\mathrm{\Omega })`$ is a minimizer for $`0<\alpha <\frac{N2}{2}`$, whereas for $`\alpha =\frac{N2}{2}`$ there is no $`H^1(\mathrm{\Omega })`$ minimizer.
$`(\mathrm{ii})`$ If $`\alpha >\frac{N2}{2}`$ and $`\mathrm{\Omega }`$ is starshaped with respect to zero, then $`\lambda _\mathrm{\Omega }(\alpha )=\left(\frac{N2}{2}\right)^2`$ and there is no $`H^1(\mathrm{\Omega })`$ minimizer.

In case $`\mathrm{\Omega }`$ is not starshaped with respect to zero, concerning the analogue of part (ii) of the above Lemma, we have

###### Lemma 2.2

Let $`\mathrm{\Omega }`$ be a bounded domain in $`𝐑^N`$, $`N3`$, containing the origin, which is not starshaped with respect to zero. Then, there exist finite constants $`\alpha ^{}N2`$ and $`\alpha _{}[\frac{N2}{2},\alpha ^{})`$ depending on $`\mathrm{\Omega }`$ such that:
$`(\mathrm{i})`$ $`\lambda _\mathrm{\Omega }(\alpha ^{})=0`$, whereas $`\lambda _\mathrm{\Omega }(\alpha )>0`$ for all $`\frac{N2}{2}<\alpha <\alpha ^{}`$.
$`(\mathrm{ii})`$ If $`\frac{N2}{2}\alpha \alpha _{}`$, then $`\lambda _\mathrm{\Omega }(\alpha )=\left(\frac{N2}{2}\right)^2`$ and there is no $`H^1(\mathrm{\Omega })`$ minimizer.
$`(\mathrm{iii})`$ If $`\alpha _{}<\alpha <\alpha ^{}`$, then $`\mathrm{max}(0,\alpha (N2\alpha ))\lambda _\mathrm{\Omega }(\alpha )<\left(\frac{N2}{2}\right)^2`$, and there exists an $`H^1(\mathrm{\Omega })`$ minimizer.

Remarks. 1. We note in particular that for any $`\mathrm{\Omega }`$ and any $`\alpha >0`$ there holds

$$\alpha (N2\alpha )\lambda _\mathrm{\Omega }(\alpha )\left(\frac{N2}{2}\right)^2.$$
(2.2)
2. We do not know whether there exists a non-starshaped domain $`\mathrm{\Omega }`$ with smooth boundary so that $`\alpha ^{}=N2`$. Similarly, we do not know whether there there exists such an $`\mathrm{\Omega }`$ for which $`\alpha _{}=(N2)/2`$.

Proof of Lemmas 2.1 and 2.2: Let $`uC^{\mathrm{}}(\overline{\mathrm{\Omega }})`$ be supported outside a neighborhood of zero. For any $`\alpha >0`$ we set $`u(x)=|x|^\alpha v(x)`$. A straightforward calculation shows that

$$_\mathrm{\Omega }|u|^2𝑑x=_\mathrm{\Omega }|x|^{2\alpha }|v|^2𝑑x+\alpha (N2\alpha )_\mathrm{\Omega }\frac{u^2}{|x|^2}𝑑x\alpha _\mathrm{\Omega }\frac{x\nu }{|x|^2}u^2𝑑S,$$
(2.3)
therefore,

$$_\mathrm{\Omega }|u|^2𝑑x+\alpha _\mathrm{\Omega }\frac{x\nu }{|x|^2}u^2𝑑S\alpha (N2\alpha )_\mathrm{\Omega }\frac{u^2}{|x|^2}𝑑x.$$
(2.4)
By a simple density argument this inequality is valid for all $`uH^1(\mathrm{\Omega })`$. This implies in particular the lower bound on $`\lambda _\mathrm{\Omega }(\alpha )`$ in (2.2).

If $`0<\alpha <\frac{N2}{2}`$, then $`|x|^\alpha `$ is in $`H^1(\mathrm{\Omega })`$ and an easy calculation shows that satisfies (2.4) as equality, hence it is a minimizer. If $`\alpha =\frac{N2}{2}`$ then the fact that $`\lambda _\mathrm{\Omega }(\alpha )=\left(\frac{N2}{2}\right)^2`$ follows by considering the minimizing sequence $`u_ϵ(x)=|x|^{\frac{N2}{2}+ϵ}H^1(\mathrm{\Omega })`$, $`ϵ0^+`$.

By the same sequence $`u_ϵ(x)=|x|^{\frac{N2}{2}+ϵ}H^1(\mathrm{\Omega })`$ one can show that $`\left(\frac{N2}{2}\right)^2\lambda _\mathrm{\Omega }(\alpha )`$, for any $`\alpha >0`$ and any $`\mathrm{\Omega }`$, thus proving the upper bound in (2.2).

Suppose now that $`\mathrm{\Omega }`$ is starshaped and $`\alpha >\frac{N2}{2}`$. Then, using first the fact that $`x\nu 0`$ on the boundary of $`\mathrm{\Omega }`$, and then (2.4) (with $`\alpha =\frac{N2}{2}`$)

$`{\displaystyle _\mathrm{\Omega }}|u|^2𝑑x+\alpha {\displaystyle _\mathrm{\Omega }}{\displaystyle \frac{x\nu }{|x|^2}}u^2𝑑S`$ $``$ $`{\displaystyle _\mathrm{\Omega }}|u|^2𝑑x+{\displaystyle \frac{N2}{2}}{\displaystyle _\mathrm{\Omega }}{\displaystyle \frac{x\nu }{|x|^2}}u^2𝑑S`$
$``$ $`\left({\displaystyle \frac{N2}{2}}\right)^2{\displaystyle _\mathrm{\Omega }}{\displaystyle \frac{u^2}{|x|^2}}𝑑x.`$
Hence, in this case $`\lambda _\mathrm{\Omega }(\alpha )=\left(\frac{N2}{2}\right)^2`$.

We next show that when $`\lambda _\mathrm{\Omega }(\alpha )=\left(\frac{N2}{2}\right)^2`$, there is no $`H^1(\mathrm{\Omega })`$ minimizer. Indeed assuming that there is one, then it would be a positive $`H^1(\mathrm{\Omega })`$ solution of the Euler-Lagrange equation

$$\mathrm{\Delta }u+\frac{\left(\frac{N2}{2}\right)^2}{|x|^2}u=0,x\mathrm{\Omega },$$

under suitable Robin boundary conditions. However, this equation has no $`H^1(\mathrm{\Omega })`$ positive solutions; see e.g. \[FT, Theorem C\] for a more general statement. (We note that although in this Theorem Dirichlet condition were imposed, the proof is independent of the boundary conditions.) Thus, Lemma 2.1 has been proved.

Suppose now that $`\mathrm{\Omega }`$ is not starshaped with respect to zero. The existence of $`\alpha ^{}`$ follows from the continuity of $`\lambda _\mathrm{\Omega }(\alpha )`$ with respect to $`\alpha `$ combined with the fact that if $`\mathrm{\Omega }`$ is not starshaped with respect to zero, then one can easily find test functions making the surface integral in (2.1) negative. The fact that $`\alpha ^{}N2`$ follows from the lower bound in (2.2).

From Lemma 2.1(i), we have that $`\lambda _\mathrm{\Omega }(\frac{N2}{2})=\left(\frac{N2}{2}\right)^2`$. We then define $`\alpha _{}`$ as the supremum of all $`\alpha `$ for which $`\lambda _\mathrm{\Omega }(\alpha )=\left(\frac{N2}{2}\right)^2`$. Assuming that $`\alpha _{}>\frac{N2}{2}`$, we will show that for any $`\frac{N2}{2}<\alpha <\alpha _{}`$ there holds $`\lambda _\mathrm{\Omega }(\alpha )=\left(\frac{N2}{2}\right)^2`$. Indeed, if this is not the case then there would exist an $`\alpha `$ in the above interval and $`\varphi H^1(\mathrm{\Omega })`$ such that

$$\frac{_\mathrm{\Omega }|\varphi |^2𝑑x+\alpha _\mathrm{\Omega }\frac{x\nu }{|x|^2}\varphi ^2𝑑S}{_\mathrm{\Omega }\frac{\varphi ^2}{|x|^2}𝑑x}<\left(\frac{N2}{2}\right)^2$$

On the other hand from Lemma 2.1(i), we have that

$$\frac{_\mathrm{\Omega }|\varphi |^2𝑑x+(\frac{N2}{2})_\mathrm{\Omega }\frac{x\nu }{|x|^2}\varphi ^2𝑑S}{_\mathrm{\Omega }\frac{\varphi ^2}{|x|^2}𝑑x}\left(\frac{N2}{2}\right)^2.$$

From the above two inequalities it follows that $`_\mathrm{\Omega }\frac{x\nu }{|x|^2}\varphi ^2𝑑S<0`$. Using $`\varphi `$ as a test function and the fact that $`\alpha _{}>\alpha `$ we conclude that $`\lambda _\mathrm{\Omega }(\alpha _{})<(\frac{N2}{2})^2`$, which is a contradiction. Thus, the estimates of part (ii) and (iii) of Lemma 2.2 have been proved.

The nonexistence of $`H^1(\mathrm{\Omega })`$ minimizer of part (ii) follows exactly as in Lemma 2.1. The existence of $`H^1(\mathrm{\Omega })`$ minimizer of part (iii) will follow later from a more general result; see Proposition 2.5. $`//`$

### 2.2 Complement of bounded domains

Here we consider the complement of a bounded domain and we study the corresponding infimum, that is

$$\mu _\mathrm{\Omega }(\alpha )=\underset{uC_c^{\mathrm{}}(𝐑^N)|_{\mathrm{\Omega }^c}}{inf}\frac{_{\mathrm{\Omega }^c}|u|^2𝑑x\alpha _\mathrm{\Omega }\frac{x\nu }{|x|^2}u^2𝑑S}{_{\mathrm{\Omega }^c}\frac{u^2}{|x|^2}𝑑x}.$$
(2.5)
where $`\mathrm{\Omega }`$, as before, is a bounded domain containing the origin and $`\nu `$ is the outward-pointing (with respect to $`\mathrm{\Omega }`$) unit vector on the surface $`\mathrm{\Omega }`$. Also, $`C_c^{\mathrm{}}(𝐑^N)|_{\mathrm{\Omega }^c}`$ is the set of restrictions on $`\mathrm{\Omega }^c`$ of all functions $`uC_c^{\mathrm{}}(𝐑^N)`$. We also introduce the following norms,

$`u_{𝒟^{1,2}(\mathrm{\Omega }^c)}`$ $`=`$ $`({\displaystyle _{\mathrm{\Omega }^c}}|u|^2𝑑x)^{1/2}+({\displaystyle _{\mathrm{\Omega }^c}}|u|^{\frac{2N}{N2}}𝑑x)^{\frac{N2}{2N}}`$ (2.6)
$`u_{^1(\mathrm{\Omega }^c)}`$ $`=`$ $`({\displaystyle _{\mathrm{\Omega }^c}}|u|^2𝑑x)^{1/2}+({\displaystyle _{\mathrm{\Omega }^c}}{\displaystyle \frac{|u|^2}{|x|^2}}𝑑x)^{1/2}`$ (2.7)
$`u_{𝒲(\mathrm{\Omega }^c)}`$ $`=`$ $`({\displaystyle _{\mathrm{\Omega }^c}}|u|^2𝑑x)^{1/2}+({\displaystyle _\mathrm{\Omega }}|u|^{\frac{2(N1)}{N2}}𝑑S)^{\frac{N2}{2(N1)}}`$ (2.8)
and we denote by $`𝒟^{1,2}(\mathrm{\Omega }^c)`$, $`^1(\mathrm{\Omega }^c)`$ and $`𝒲(\mathrm{\Omega }^c)`$ the completion of $`C_c^{\mathrm{}}(𝐑^N)|_{\mathrm{\Omega }^c}`$ under the corresponding norms. The space $`𝒲(\mathrm{\Omega }^c)`$ is well studied, see e.g. \[M\]. For our purposes however, the natural spaces to use are $`𝒟^{1,2}(\mathrm{\Omega }^c)`$ and $`^1(\mathrm{\Omega }^c)`$. In the next lemma we show that these three spaces coincide (a trivial fact if $`\mathrm{\Omega }^c`$ were replaced by $`\mathrm{\Omega }`$).

###### Lemma 2.3

Let $`\mathrm{\Omega }`$ be a bounded domain in $`𝐑^N`$, $`N3`$, with $`C^1`$ boundary, containing the origin. Then $`𝒟^{1,2}(\mathrm{\Omega }^c)=^1(\mathrm{\Omega }^c)=𝒲(\mathrm{\Omega }^c)`$.

Proof: We will show that all norms are equivalent. Let $`uC_c^{\mathrm{}}(𝐑^N)|_{\mathrm{\Omega }^c}`$. Under our assumptions, it follows easily as in Lemma 2.1 (cf (2.4) with $`\alpha =\frac{N2}{2}`$) that

$$_{\mathrm{\Omega }^c}|u|^2𝑑x\frac{N2}{2}_\mathrm{\Omega }\frac{x\nu }{|x|^2}u^2𝑑S\left(\frac{N2}{2}\right)^2_{\mathrm{\Omega }^c}\frac{u^2}{|x|^2}𝑑x,$$

whence,

$`{\displaystyle _{\mathrm{\Omega }^c}}{\displaystyle \frac{|u|^2}{|x|^2}}𝑑x`$ $``$ $`C\left({\displaystyle _{\mathrm{\Omega }^c}}|u|^2𝑑x+{\displaystyle _\mathrm{\Omega }}|u|^2𝑑S\right)`$
$``$ $`C({\displaystyle _{\mathrm{\Omega }^c}}|u|^2dx+({\displaystyle _\mathrm{\Omega }}|u|^{\frac{2(N1)}{N2}}dS)^{\frac{N2}{(N1)}}).`$
Hence, $`u_{^1(\mathrm{\Omega }^c)}Cu_{𝒲(\mathrm{\Omega }^c)}`$. To obtain the reverse inequality we note that it follows from the standard trace Theorem (e.g \[A, Theorem 5.22\], Chapter V) – applied to $`B\mathrm{\Omega }`$ for some ball $`B\mathrm{\Omega }`$ – that

$`\left({\displaystyle _\mathrm{\Omega }}|u|^{\frac{2(N1)}{N2}}𝑑S\right)^{\frac{N2}{2(N1)}}Cu_{^1(\mathrm{\Omega }^c)}`$
$`\left({\displaystyle _\mathrm{\Omega }}|u|^{\frac{2(N1)}{N2}}𝑑S\right)^{\frac{N2}{2(N1)}}Cu_{𝒟^{1,2}(\mathrm{\Omega }^c)}`$
From the first one it follows that $`u_{𝒲(\mathrm{\Omega }^c)}Cu_{^1(\mathrm{\Omega }^c)}`$, whence $`^1(\mathrm{\Omega }^c)=𝒲(\mathrm{\Omega }^c)`$. From the second one it follows that $`u_{𝒲(\mathrm{\Omega }^c)}Cu_{𝒟^{1,2}(\mathrm{\Omega }^c)}`$. Thus, it remains to prove that $`u_{𝒟^{1,2}(\mathrm{\Omega }^c)}Cu_{𝒲(\mathrm{\Omega }^c)}`$. This inequality follows from Corollary 1 of Section 4.11.1 \[M, p. 258\]. Notice that in the notation of Maz’ja $`𝒲(\mathrm{\Omega }^c)=W_{2,\frac{2(N1)}{N2}}(\mathrm{\Omega }^c,\mathrm{\Omega })`$. $`//`$

An immediate consequence of the above Lemma is that the infimum in (2.5) can be taken over $`𝒟^{1,2}(\mathrm{\Omega }^c)`$, that is

$$\mu _\mathrm{\Omega }(\alpha )=\underset{u𝒟^{1,2}(\mathrm{\Omega }^c)}{inf}\frac{_{\mathrm{\Omega }^c}|u|^2𝑑x\alpha _\mathrm{\Omega }\frac{x\nu }{|x|^2}u^2𝑑S}{_{\mathrm{\Omega }^c}\frac{u^2}{|x|^2}𝑑x}.$$
(2.9)
We now state the analogue of Lemma 2.1 for exterior domains.

###### Lemma 2.4

Let $`\mathrm{\Omega }`$ be a bounded domain in $`𝐑^N`$, $`N3`$, containing the origin.
$`(\mathrm{i})`$ If $`\alpha \frac{N2}{2}`$, then $`\mu _\mathrm{\Omega }(\alpha )=\alpha (N2\alpha )`$. Moreover, $`|x|^\alpha 𝒟^{1,2}(\mathrm{\Omega }^c)`$ is a minimizer for $`\alpha >\frac{N2}{2}`$, whereas for $`\alpha =\frac{N2}{2}`$ there is no $`𝒟^{1,2}(\mathrm{\Omega }^c)`$ minimizer.
$`(\mathrm{ii})`$ If $`0<\alpha <\frac{N2}{2}`$, and $`\mathrm{\Omega }`$ starshaped with respect to zero, then $`\mu _\mathrm{\Omega }(\alpha )=\left(\frac{N2}{2}\right)^2`$ and there is no $`𝒟^{1,2}(\mathrm{\Omega }^c)`$ minimizer.

Proof: The proof is quite similar to the proof of the previous Lemmas 2.1 and 2.2. An alternative proof can be given using the Kelvin transform; see the Remark that follows. $`//`$

Remark There is a duality between the minimization problems (2.1) and (2.5). Indeed, by means of the Kelvin transform, $`u(x)=|y|^{N2}v(y)`$, $`y=x/|x|^2`$, $`x\mathrm{\Omega }^c`$ the domain $`\mathrm{\Omega }^c`$ is transformed to a bounded domain containing the origin that we denote by $`(\mathrm{\Omega }^c)^{}`$. Denoting by $`\nu ^{}`$ the outward pointing normal to $`(\mathrm{\Omega }^c)^{}`$ a straightforward calculation shows that

$$_{\mathrm{\Omega }^c}|_xu|^2𝑑x=_{(\mathrm{\Omega }^c)^{}}|_yv|^2𝑑y+(N2)_{(\mathrm{\Omega }^c)^{}}\frac{y\nu ^{}}{|y|^2}v^2𝑑S_y.$$

Also,

$`{\displaystyle _{\mathrm{\Omega }^c}}{\displaystyle \frac{|u|^2}{|x|^2}}𝑑x`$ $`=`$ $`{\displaystyle _{(\mathrm{\Omega }^c)^{}}}{\displaystyle \frac{|v|^2}{|y|^2}}𝑑y,`$
$`{\displaystyle _{\mathrm{\Omega }^c}}|u|^{\frac{2N}{N2}}𝑑x`$ $`=`$ $`{\displaystyle _{(\mathrm{\Omega }^c)^{}}}|v|^{\frac{2N}{N2}}𝑑y.`$
It can be seen from these relations that $`u𝒟^{1,2}(\mathrm{\Omega }^c)`$ if and only if $`vH^1((\mathrm{\Omega }^c)^{})`$. It then follows easily that $`\mu _\mathrm{\Omega }(\alpha )=\lambda _{(\mathrm{\Omega }^c)^{}}(N2\alpha )`$, and that the existence of a minimizer for $`\mu _\mathrm{\Omega }(\alpha )`$ in $`𝒟^{1,2}(\mathrm{\Omega }^c)`$ is equivalent to the existence of a minimizer in $`H^1(\mathrm{\Omega })`$ for $`\lambda _{(\mathrm{\Omega }^c)^{}}(N2\alpha )`$.

### 2.3 Existence of minimizers

In this section we establish a sufficient condition for the existence of minimizers. We recall from Lemma 2.2 that when $`\mathrm{\Omega }`$ is not starshaped with respect to the origin, $`\alpha ^{}`$ denotes the first zero of $`\lambda _\mathrm{\Omega }(\alpha )`$. We also set $`\alpha ^{}=\mathrm{}`$ in case $`\mathrm{\Omega }`$ is starshaped with respect to zero. Thus, in both cases we have $`\lambda _\mathrm{\Omega }(\alpha )>0`$ for $`0<\alpha <\alpha ^{}`$. Given $`0<\alpha <\alpha ^{}`$, and a nonnegative measurable potential $`V`$ we define

$$\lambda _\mathrm{\Omega }(\alpha ,V):=\underset{\begin{array}{c}uH^1(\mathrm{\Omega })\\ _\mathrm{\Omega }Vu^2𝑑x>0\end{array}}{inf}\frac{_\mathrm{\Omega }|u|^2𝑑x+\alpha _\mathrm{\Omega }\frac{x\nu }{|x|^2}u^2𝑑S}{_\mathrm{\Omega }Vu^2𝑑x}.$$
(2.10)
Note that with this notation $`\lambda _\mathrm{\Omega }(\alpha )=\lambda _\mathrm{\Omega }(\alpha ,|x|^2)`$. Since the numerator in (2.10) is always positive and finite when $`0<\alpha <\alpha ^{}`$, we interpret $`\lambda _\mathrm{\Omega }(\alpha ,V)=0`$ in case there exists $`uH^1(\mathrm{\Omega })`$ such that $`_\mathrm{\Omega }Vu^2𝑑x=+\mathrm{}`$. It is worth mentioning that $`\lambda _\mathrm{\Omega }(\alpha ,V)`$ is not monotone with respect to $`\mathrm{\Omega }`$, unlike the case of Dirichlet boundary conditions.

We denote by $`B_r\mathrm{\Omega }`$ the ball centered at zero with radius $`r`$. We have the following

###### Proposition 2.5

Let $`\mathrm{\Omega }`$ be a bounded domain in $`𝐑^N`$, $`N3`$, containing the origin, and let $`0VL_{loc}^{\frac{N}{2}}(\overline{\mathrm{\Omega }}\{0\})`$. If for some $`r>0`$

$$0<\lambda _\mathrm{\Omega }(\alpha ,V)<\lambda _{B_r}(\alpha ,V)$$
(2.11)
then (2.10) has an $`H^1(\mathrm{\Omega })`$ minimizer.

Note. It is a consequence of (2.11) that $`_\mathrm{\Omega }Vu^2<+\mathrm{}`$ for $`uH^1(\mathrm{\Omega })`$.

Proof. Let $`\{u_j\}H^1(\mathrm{\Omega })`$ be a minimizing sequence of the Rayleigh quotient in (2.10). We may normalize it so that $`_\mathrm{\Omega }Vu_j^2𝑑x=1`$. We claim that $`u_j_{H^1(\mathrm{\Omega })}<C`$. This will follow from two inequalities. The first inequality follows from the fact that $`0<\alpha <\alpha ^{}`$ and $`\lambda _\mathrm{\Omega }(\alpha ^{})=0`$ and reads

$$_\mathrm{\Omega }|u|^2𝑑x+\alpha _\mathrm{\Omega }\frac{x\nu }{|x|^2}u^2𝑑S(1\frac{\alpha }{\alpha ^{}})_\mathrm{\Omega }|u|^2𝑑x.$$
(2.12)
The second one is a consequence of Lemma 2.1 and reads

$`{\displaystyle _\mathrm{\Omega }}|u|^2𝑑x+\alpha {\displaystyle _\mathrm{\Omega }}{\displaystyle \frac{x\nu }{|x|^2}}u^2𝑑S`$ $``$ $`\lambda _\mathrm{\Omega }(\alpha ){\displaystyle _\mathrm{\Omega }}{\displaystyle \frac{u^2}{|x|^2}}𝑑x`$ (2.13)
$``$ $`K\lambda _\mathrm{\Omega }(\alpha ){\displaystyle _\mathrm{\Omega }}u^2𝑑x.`$
Thus, we may extract a subsequence such that $`u_ju_0`$ weakly in $`H^1(\mathrm{\Omega })`$, and $`u_ju_0`$ strongly in $`L^p(\mathrm{\Omega })`$, $`1<p<\frac{2N}{N2}`$. Moreover, since $`VL^{N/2}(\mathrm{\Omega }B_r)`$, standard results give

$$_{\mathrm{\Omega }B_r}Vu_j^2𝑑x_{\mathrm{\Omega }B_r}Vu_0^2𝑑x.$$
(2.14)
Also by the trace theorem

$$_\mathrm{\Omega }\frac{x\nu }{|x|^2}u_j^2𝑑S_\mathrm{\Omega }\frac{x\nu }{|x|^2}u_0^2𝑑S$$
(2.15)
Setting $`u_j=v_j+u_0`$ we easily see that as $`j\mathrm{}`$,

$$_\mathrm{\Omega }|u_j|^2𝑑x=_\mathrm{\Omega }|u_0|^2𝑑x+_\mathrm{\Omega }|v_j|^2𝑑x+o(1),$$
(2.16)
and

$$1=_\mathrm{\Omega }Vu_j^2𝑑x=_\mathrm{\Omega }Vu_0^2𝑑x+_\mathrm{\Omega }Vv_j^2𝑑x+o(1).$$
(2.17)
It then follows from (2.10) that

$`\lambda _\mathrm{\Omega }(\alpha ,V)`$ $`=`$ $`{\displaystyle _\mathrm{\Omega }}|v_j|^2𝑑x+{\displaystyle _\mathrm{\Omega }}|u_0|^2𝑑x+\alpha {\displaystyle _\mathrm{\Omega }}{\displaystyle \frac{x\nu }{|x|^2}}u_0^2𝑑S+o(1)`$ (2.18)
$``$ $`{\displaystyle _\mathrm{\Omega }}|v_j|^2𝑑x+\lambda _\mathrm{\Omega }(\alpha ,V){\displaystyle _\mathrm{\Omega }}Vu_0^2𝑑x+o(1).`$
We then have

$`{\displaystyle _\mathrm{\Omega }}|v_j|^2𝑑x`$ $``$ $`{\displaystyle _{B_r}}|v_j|^2𝑑x`$ (2.19)
$``$ $`\lambda _{B_r}(\alpha ,V){\displaystyle _{B_r}}Vv_j^2𝑑x\alpha {\displaystyle _{B_r}}{\displaystyle \frac{x\nu }{|x|^2}}v_j^2𝑑S`$
$`=`$ $`\lambda _{B_r}(\alpha ,V){\displaystyle _{B_r}}Vv_j^2𝑑x+o(1)`$
$`=`$ $`\lambda _{B_r}(\alpha ,V){\displaystyle _\mathrm{\Omega }}Vv_j^2𝑑x+o(1)`$
$`=`$ $`\lambda _{B_r}(\alpha ,V)\left(1{\displaystyle _\mathrm{\Omega }}Vu_0^2𝑑x\right)+o(1)(j\mathrm{}).`$
Using this and (2.18) we end up with

$$\left(\lambda _\mathrm{\Omega }(\alpha ,V)\lambda _{B_r}(\alpha ,V)\right)\left(1_\mathrm{\Omega }Vu_0^2𝑑x\right)0,$$
(2.20)
whence, since $`\lambda _\mathrm{\Omega }(\alpha ,V)<\lambda _{B_r}(\alpha ,V)`$, it follows that $`_\mathrm{\Omega }Vu_0^2𝑑x1`$. By lower semi continuity we conclude that $`_\mathrm{\Omega }Vu_0^2𝑑x=1`$. It then follows that $`u_0`$ is a minimizer for (2.10). $`//`$

As a consequence we have:
Completion of Proof of Lemma 2.2(iii) (Existence of a minimizer) : Since $`\frac{N2}{2}\alpha _{}<\alpha <\alpha ^{}`$ it follows from Lemma 2.2 that $`0<\lambda _\mathrm{\Omega }(\alpha )<\left(\frac{N2}{2}\right)^2`$. If $`B_r\mathrm{\Omega }`$ is a ball centered at zero it follows by Lemma 2.1 that $`\lambda _{B_r}(\alpha )=\left(\frac{N2}{2}\right)^2`$. By Proposition 2.5, $`\lambda _\mathrm{\Omega }(\alpha )`$ is attained by an $`H^1(\mathrm{\Omega })`$ function. $`//`$

We next state the corresponding result for the exterior of a bounded domain $`\mathrm{\Omega }`$. For $`0<a<N2`$ we define

$$\mu _\mathrm{\Omega }(\alpha ,V):=\underset{\begin{array}{c}u𝒟^{1,2}(\mathrm{\Omega }^c)\\ _{\mathrm{\Omega }^c}Vu^2𝑑x>0\end{array}}{inf}\frac{_{\mathrm{\Omega }^c}|u|^2𝑑x\alpha _\mathrm{\Omega }\frac{x\nu }{|x|^2}u^2𝑑S}{_{\mathrm{\Omega }^c}Vu^2𝑑x}.$$
(2.21)
We then have

###### Proposition 2.6

Let $`\mathrm{\Omega }`$ be a bounded domain in $`𝐑^N`$, $`N3`$, containing the origin, and let $`0VL_{loc}^{\frac{N}{2}}(\overline{\mathrm{\Omega }^c})`$. Also let $`0<\alpha <N2`$. If for some ball $`B_R\mathrm{\Omega }`$ centered at zero

$$0<\mu _\mathrm{\Omega }(\alpha ,V)<\mu _{B_R}(\alpha ,V),$$
(2.22)
then (2.21) has a $`𝒟^{1,2}(\mathrm{\Omega }^c)`$ minimizer.

The proof is similar to that of the previous Proposition.

## 3 Hardy-Sobolev inequalities

### 3.1 Auxilliary inequalities

We begin this section with two known Sobolev inequalities that will be used in the sequel. In Theorems 3.4 and 3.5 we then prove two new Sobolev inequalities.

By the classical inequality of Caffarelli-Kohn-Nirenberg \[CKN\] we have that

$$_{𝐑^N}|w|^2|x|^{2\alpha }𝑑xc\left(_{𝐑^N}|w|^p|x|^{p\beta }𝑑x\right)^{2/p},wC_c^{\mathrm{}}(𝐑^N),$$
(3.1)
with $`p=2N/(N2+2(\beta \alpha ))`$, provided $`\alpha <(N2)/2`$ and $`0\beta \alpha 1`$.

At the critical case $`\alpha =\beta =(N2)/2`$ inequality (3.1) fails. A sharp substitute for bounded $`\mathrm{\Omega }`$ was obtained in \[FT\], where it was shown that, with

$$X_1(t)=(1\mathrm{log}t)^1,t(0,1),$$

and $`D=sup_\mathrm{\Omega }|x|`$ there holds

$$_\mathrm{\Omega }|w|^2|x|^{2N}𝑑xc\left(_\mathrm{\Omega }|w|^{\frac{2N}{N2}}|x|^NX_1^{\frac{2N2}{N2}}(\frac{|x|}{D})𝑑x\right)^{(N2)/N},$$
(3.2)
for all $`wC_c^{\mathrm{}}(\mathrm{\Omega })`$, where the exponent $`\frac{2N2}{N2}`$ of $`X_1(|x|/D)`$ is optimal.

In the sequel we will make essential use of the following one-dimensional result, which is a special case of a more general statement by Maz’ja, cf. \[M, Theorem 3, p. 44\]:

###### Proposition 3.1

Let $`A(r)`$, $`B(r)`$ nonnegative functions such that $`1/A(r)`$ and $`B(r)`$ are integrable in $`(r,\mathrm{})`$ and $`(0,r)`$ respectively, for all positive $`r<\mathrm{}`$. Then, for $`q2`$ the Sobolev inequality

$$_0^{\mathrm{}}(v^{}(r))^2A(r)𝑑rc\left(_0^{\mathrm{}}|v(r)|^qB(r)𝑑r\right)^{2/q},$$

is valid for all $`vC^1(0,\mathrm{})`$ that vanish near infinity, if and only if

$$\underset{r>0}{sup}\left(_0^rB(t)𝑑t\right)\left(_r^{\mathrm{}}\frac{dt}{A(t)}\right)^{q/2}<+\mathrm{}.$$

The above proposition will be applied to higher dimensions by means of the following

###### Lemma 3.2

Let $`N2`$. Suppose that $`VL_{loc}^{\mathrm{}}(𝐑^N\{0\})L_{loc}^1(𝐑^N)`$ is a radially symmetric function. We further assume that inequality

$$_{𝐑^N}|u|^2𝑑x_{𝐑^N}Vu^2𝑑x0,$$
(3.3)
is valid for all radially symmetric functions $`uC_c^{\mathrm{}}(𝐑^N)`$.
$`(\mathrm{i})`$ Then, (3.3) is also valid for nonradial functions, that is, for all $`uC_c^{\mathrm{}}(𝐑^N)`$
$`(\mathrm{ii})`$ If, in addition,

$$0<\text{ess \hspace{0.17em} sup}_{x𝐑^N}|x|^2V(x)=\theta <\mathrm{},$$
(3.4)
then the following improved inequality holds

$`{\displaystyle _{𝐑^N}}|u|^2𝑑x{\displaystyle _{𝐑^N}}Vu^2𝑑x`$ (3.5)
$`{\displaystyle _{𝐑^N}}|u_0|^2𝑑x{\displaystyle _{𝐑^N}}Vu_0^2𝑑x+{\displaystyle \frac{N1}{N1+\theta }}{\displaystyle _{𝐑^N}}|(uu_0)|^2𝑑x,`$
where $`u_0(r)`$ denotes the spherical average of $`uC_c^{\mathrm{}}(𝐑^N)`$, that is

$$u_0(r)=\frac{1}{N\omega _Nr^{N1}}_{B_r(0)}u(x)𝑑S_x.$$
(3.6)
Proof. Let $`uC_c^{\mathrm{}}(𝐑^N)`$ and let

$$u(x)=\underset{m=0}{\overset{\mathrm{}}{}}f_m(\sigma )u_m(r),$$

be its decomposition into spherical harmonics; here $`f_m`$ are orthogonal in $`L^2(S^{N1})`$, normalized by $`\frac{1}{N\omega _N}_{S^{N1}}f_i(\sigma )f_j(\sigma )𝑑S=\delta _{ij}`$. In particular $`f_0(\sigma )=1`$ and the first term in the above decomposition is given by (3.6). The $`f_m`$’s are eigenfunctions of the Laplace-Beltrami operator, with corresponding eigenvalues $`c_m=m(N2+m)`$, $`m0`$. An easy calculation shows that

$$_{𝐑^N}(|u|^2Vu^2)𝑑x=\underset{m=0}{\overset{\mathrm{}}{}}_{𝐑^N}\left\{|u_m|^2+(\frac{c_m}{|x|^2}V)u_m^2\right\}𝑑x=$$

$$=_{𝐑^N}(|u_0|^2Vu_0^2)𝑑x+\underset{m=1}{\overset{\mathrm{}}{}}_{𝐑^N}\left\{|u_m|^2+(\frac{c_m}{|x|^2}V)u_m^2\right\}𝑑x.$$
(3.7)
Part (i) follows immediately since $`c_m>0`$ and $`u_m`$ are radially symmetric.

To prove part (ii) we first observe that

$$_{𝐑^N}|(uu_0)|^2𝑑x=\underset{m=1}{\overset{\mathrm{}}{}}_{𝐑^N}\left\{|u_m|^2+\frac{c_m}{|x|^2}u_m^2\right\}𝑑x.$$

In view of this and (3.7) it is enough to establish that for any $`m1`$, there holds

$$_{𝐑^N}\left\{|u_m|^2+(\frac{c_m}{|x|^2}V)u_m^2\right\}𝑑x\frac{N1}{N1+\theta }_{𝐑^N}\left\{|u_m|^2+\frac{c_m}{|x|^2}u_m^2\right\}𝑑x.$$
(3.8)
or, equivalently,

$$_{𝐑^N}|u_m|^2𝑑x_{𝐑^N}u_m^2\left\{\frac{N1+\theta }{\theta }V\frac{c_m}{|x|^2}\right\}𝑑x.$$

Since $`c_mc_1=N1`$ it is enough to establish this for $`c_m=N1`$. By the definition of $`\theta `$, cf (3.4), it follows easily that

$$\frac{N1+\theta }{\theta }V\frac{N1}{|x|^2}V,$$

and the result follows from (3.3). $`//`$

As a consequence of this we next establish the following result.

###### Lemma 3.3

Let $`N3`$. Suppose that $`VL_{loc}^{\mathrm{}}(𝐑^N\{0\})L_{loc}^1(𝐑^N)`$ is a radially symmetric function, such that

$$0<\text{ess \hspace{0.17em}sup}_{x𝐑^N}|x|^2V(x)=\theta <\mathrm{},$$

and $`WL^{\mathrm{}}(𝐑^N)`$ is a positive radially symmetric function. We further assume that the inequality

$$_{𝐑^N}|u|^2𝑑x_{𝐑^N}Vu^2𝑑xc\left(_{𝐑^N}|u|^{2N/(N2)}W𝑑x\right)^{(N2)/N}$$
(3.9)
is valid for all radially symmetric functions $`uC_c^{\mathrm{}}(𝐑^N)`$. Then inequality (3.9) is true for all $`uC_c^{\mathrm{}}(𝐑^N)`$ (without radial symmetry), provided the constant $`c`$ is replaced by a new constant $`C`$ depending on $`c`$, $`N`$, $`\theta `$ and $`W_L^{\mathrm{}}`$.

Proof: Starting from (3.5) we compute

$`{\displaystyle _{𝐑^N}}\left\{|u|^2Vu^2\right\}𝑑x`$
$``$ $`{\displaystyle _{𝐑^N}}(|u_0|^2Vu_0^2)𝑑x+{\displaystyle \frac{N1}{N1+\theta }}{\displaystyle _{𝐑^N}}|(uu_0)|^2𝑑x`$
$``$ $`c\left({\displaystyle _{𝐑^N}}|u_0|^{2N/(N2)}W𝑑x\right)^{(N2)/N}+`$
$`+c^{}\left({\displaystyle _{𝐑^N}}|uu_0|^{2N/(N2)}𝑑x\right)^{(N2)/N}`$
$``$ $`C\left({\displaystyle _{𝐑^N}}|u|^{2N/(N2)}W𝑑x\right)^{(N2)/N},`$
where, for the last inequalities we used the standard Sobolev inequality, the boundedness of $`W`$ and the triangle inequality. $`//`$

### 3.2 Hardy-Sobolev inequalities

In this section we prove improved Hardy-Sobolev inequalities for potentials that are critical either near zero or near infinity. We first consider a potential which is critical near zero. For $`ϵ>0`$ we define

$$V_ϵ(x)=\{\begin{array}{cc}\left(\frac{N2}{2}\right)^2|x|^2,\hfill & |x|<1\hfill \\ ϵf(x),\hfill & |x|1\hfill \end{array},$$
(3.10)
where $`f`$ is a non-negative, continuous and radially symmetric function on $`\{|x|1\}`$. Moreover we assume $`f`$ to be subcritical, satisfying

$$f(x)K|x|^{2\sigma },|x|1,$$
(3.11)
for some $`\sigma ,K>0`$.

We also define the following auxiliary function (cf. (5.15))

$$\stackrel{~}{X}_1(|x|)=\{\begin{array}{cc}X_1(|x|),\hfill & |x|<1\hfill \\ 1,\hfill & |x|>1\hfill \end{array}.$$
(3.12)
We shall henceforth denote by $`B`$ the unit ball in $`𝐑^N`$ centered at zero, by $`B^c`$ its complement, and, as before, we denote by $`C_c^{\mathrm{}}(𝐑^N)|_{B^c}`$ the set of restrictions on $`B^c`$ of all functions $`uC_c^{\mathrm{}}(𝐑^N)`$. We also denote by $`\nu `$ the outward-pointing unit vector on the surface $`B`$. We have the following

###### Theorem 3.4

Let

$$ϵ_0=\underset{u^1(B^c)}{inf}\frac{_{B^c}|u|^2𝑑x\frac{N2}{2}_Bu^2𝑑S}{_{B^c}fu^2𝑑x}.$$
(3.13)
Then $`ϵ_0>0`$ and for any $`ϵ(0,ϵ_0)`$ there holds

$$_{𝐑^N}|u|^2𝑑x_{𝐑^N}V_ϵu^2𝑑xc\left(_{𝐑^N}|u|^{2N/(N2)}\stackrel{~}{X}_1^{(2N2)/(N2)}(|x|)𝑑x\right)^{(N2)/N},$$
(3.14)
for all $`uC_c^{\mathrm{}}(𝐑^N)`$. Moreover, (3.14) fails for $`ϵ=ϵ_0`$.

Proof.  Since $`f(x)K|x|^2`$ the positivity of $`ϵ_0`$ follows from Lemma 2.4 with $`a=\frac{N2}{2}`$, yielding in fact $`ϵ_0K^1((N2)/2)^2.`$

Let us now fix $`ϵ(0,ϵ_0)`$. By Lemma 3.3, it is enough to prove (3.14) in the case where $`u`$ is radially symmetric, $`u=u(r)`$. Now, there exists a radially symmetric and positive function $`\stackrel{~}{\psi }`$ on $`B^c`$ which solves the Robin problem

$$\begin{array}{cc}\mathrm{\Delta }\stackrel{~}{\psi }+ϵf\stackrel{~}{\psi }=0,\hfill & |x|>1,\hfill \\ \frac{\stackrel{~}{\psi }}{\nu }=\frac{N2}{2}\stackrel{~}{\psi },\hfill & |x|=1.\hfill \end{array}$$

The existence of such a $`\stackrel{~}{\psi }`$ can be easily derived, for example, by a shooting argument from $`\{|x|=1\}`$. We assume that $`\stackrel{~}{\psi }`$ is normalized so that $`\stackrel{~}{\psi }=1`$ on $`\{|x|=1\}`$. The function

$$\psi (x)=\{\begin{array}{cc}|x|^{(N2)/2},\hfill & |x|<1,\hfill \\ \stackrel{~}{\psi }(x),\hfill & |x|>1,\hfill \end{array}$$
(3.15)
then lies in $`C^1(𝐑^N\{0\})`$, is positive, radially symmetric and satisfies $`\mathrm{\Delta }\psi +V_ϵ\psi =0`$ in $`𝐑^N`$. Following \[FT\] we change variables, $`u=\psi v`$, and (3.14) for radially symmetric functions is then written as

$$_0^{\mathrm{}}(v^{})^2\psi ^2r^{N1}𝑑rc\left(_0^{\mathrm{}}|v|^{2N/(N2)}\psi ^{2N/(N2)}\stackrel{~}{X}_1^{2(N1)/(N2)}𝑑r\right)^{(N2)/N}.$$
(3.16)
We claim that $`\psi (r)`$ has a positive limit as $`r+\mathrm{}`$. Indeed, since $`(r^{N1}\psi ^{})^{}=r^{N1}V_ϵ\psi <0`$ and $`\psi ^{}(1)<0`$, $`\psi (r)`$ is decreasing on $`(1,+\mathrm{})`$. If the limit $`lim_{r+\mathrm{}}\psi (r)`$ were zero it would then follow from \[LN, Theorem 2.9\] that $`\psi (r)<cr^{2N}`$ near infinity, which then easily implies $`\psi ^1(B^c)`$. Hence $`\psi `$ can be taken as a test function for the infimum in the right-hand side of (3.13), in which case the value of the Rayleigh quotient is $`ϵ<ϵ_0`$, contradicting the definition of $`ϵ_0`$. Hence $`lim_{r+\mathrm{}}\psi (r)=l>0`$. Using this we deduce (3.16) from Proposition 3.1. Hence (3.14) has been proved.

We finally show that (3.14) fails for $`ϵ=ϵ_0`$. For this we will use Proposition 2.6. Let $`B_RB_1`$. Then $`V_{ϵ_0}(x)ϵ_0KR^\sigma |x|^2`$ on $`(B_R)^c`$, hence

$$\mu _{B_R}(\alpha ,V_{ϵ_0})\frac{R^\sigma }{K}\mu _{B_R}(\alpha ).$$

By Lemma 2.4 (i), for $`\alpha (0,N2)`$ $`\mu _{B_R}(\alpha )`$, is positive and independent of $`R`$; taking $`R`$ large enough we have $`\mu _B(\alpha ,V_{ϵ_0})<\mu _{B_R}(\alpha ,V_{ϵ_0})`$. Hence, by Proposition 2.6 – with $`\alpha =(N2)/2`$ – there exists an $`^1(B^c)`$-minimizer $`\varphi `$ to (3.13). It is standard to show that $`\varphi `$ is simple, radial and of one sign; we normalize it by $`\varphi |_{|x|=1}=1`$ and for $`\theta >0`$ we define the function $`u_\theta H^1(𝐑^N)`$ by

$$u_\theta (x)=\{\begin{array}{cc}|x|^{\frac{N2}{2}+\theta },\hfill & |x|<1;\hfill \\ \varphi (x),\hfill & |x|>1.\hfill \end{array}$$

We then compute the left-hand side of (3.14): in $`B`$ there holds $`\mathrm{\Delta }u_\theta +V_{ϵ_0}u_\theta =\theta ^2u_\theta `$, hence

$`{\displaystyle _{𝐑^N}}(|u_\theta |^2V_{ϵ_0}u_\theta ^2)𝑑x=`$
$`=`$ $`{\displaystyle _B}(u_\theta \mathrm{\Delta }u_\theta +V_{ϵ_0}u_\theta ^2)𝑑x+{\displaystyle _B}u_\theta {\displaystyle \frac{u_\theta }{\nu }}𝑑S+{\displaystyle _{B^c}}(|u_\theta |^2V_{ϵ_0}u_\theta ^2)𝑑x`$
$`=`$ $`\theta ^2{\displaystyle _B}{\displaystyle \frac{u_\theta ^2}{r^2}}𝑑x+N\omega _N({\displaystyle \frac{N2}{2}}+\theta )+{\displaystyle \frac{N2}{2}}{\displaystyle _B}u_\theta ^2𝑑S`$
$`=`$ $`{\displaystyle \frac{N\omega _N\theta }{2}}.`$
On the other hand for the right-hand side of (3.14) we have

$$_{𝐑^N}u_\theta ^{2N/(N2)}\stackrel{~}{X}_1^{\frac{2N2}{N2}}𝑑x_{B^c}u_\theta ^{2N/(N2)}𝑑x=_{B^c}\varphi ^{2N/(N2)},$$

the last term being independent of $`\theta `$. Letting $`\theta 0`$ we conclude that (3.14) fails for $`ϵ=ϵ_0`$. $`//`$

We close this section proving a Sobolev inequality which involves radial potentials with critical behaviour $`((N2)/2)^2|x|^2`$ near infinity. Let $`g`$ be a non-negative, radially symmetric, and continuous in $`B\{0\}`$ function that is subcritical near zero, satisfying

$$g(x)K|x|^{2+\sigma },|x|1,$$

for some $`\sigma ,K>0`$. For $`ϵ>0`$ we define

$$\widehat{V}_ϵ(x)=\{\begin{array}{cc}ϵg(x),\hfill & |x|<1,\hfill \\ (\frac{N2}{2})^2|x|^2,\hfill & |x|>1.\hfill \end{array}$$
(3.17)
We also set

$$\stackrel{~}{Y}_1(|x|)=\{\begin{array}{cc}1,\hfill & |x|<1,\hfill \\ (1+\mathrm{ln}|x|)^1,\hfill & |x|>1.\hfill \end{array}$$

We then have

###### Theorem 3.5

Let

$$\overline{ϵ}_0=\underset{uH^1(B)}{inf}\frac{_B|u|^2𝑑x+\frac{N2}{2}_Bu^2𝑑S}{_Bgu^2𝑑x}.$$

Then $`\overline{ϵ}_0>0`$ and for any $`ϵ(0,\overline{ϵ}_0)`$ there holds

$$_{𝐑^N}|u|^2𝑑x_{𝐑^N}\widehat{V}_ϵu^2𝑑xc\left(_{𝐑^N}|u|^{2N/(N2)}\stackrel{~}{Y}_1^{2(N1)/(N2)}(|x|)𝑑x\right)^{(N2)/N},$$
(3.18)
for all $`uC_c^{\mathrm{}}(𝐑^N)`$. Moreover, (3.18) fails for $`ϵ=\overline{ϵ}_0`$.

Proof. The proof of (3.18) follows closely that of Theorem 3.4, reversing essentially the role of $`B`$ and $`B^c`$ while making the necessary adjustments; in particular, we now use Lemma 2.1 instead of Lemma 2.4. In fact, an alternative and simpler proof consists in simply taking the Kelvin transform of (3.14). The optimality of $`\overline{ϵ}_0`$ is also proven analogously; we omit the details. $`//`$

## 4 Heat kernel estimates

In this section we shall apply the Sobolev inequalities of Section 3 to obtain heat kernel estimates for the Schrödinger operator

$$Hu=\mathrm{\Delta }uVu,u|_\mathrm{\Omega }=0,$$

for various critical potentials $`V`$. We always assume that $`\mathrm{\Omega }`$ is a domain containing the origin in $`𝐑^N`$, $`N3`$. We shall consider the case of bounded $`\mathrm{\Omega }`$, as well as the case $`\mathrm{\Omega }=𝐑^N`$. The operator $`H`$ is defined via quadratic forms, with initial domain $`C_c^1(\mathrm{\Omega }\{0\})`$; it will always be the case that $`H0`$. Note that, equivalently, we could have set $`C_c^1(\mathrm{\Omega })`$ as the initial domain.

We shall use the standard technique of transference to a weighted $`L^2`$ space, which we now describe briefly. Let $`\varphi C^1(\mathrm{\Omega }\{0\})`$ be positive and such that $`\mathrm{\Delta }\varphi L_{loc}^1(\mathrm{\Omega }\{0\})`$. The unitary map

$$L^2(\mathrm{\Omega })uw=\frac{u}{\varphi }L_\varphi ^2:=L^2(\mathrm{\Omega },\varphi ^2dx)$$
(4.1)
satisfies

$$_\mathrm{\Omega }(|u|^2Vu^2)𝑑x=_\mathrm{\Omega }\left(|w|^2Vw^2\frac{\mathrm{\Delta }\varphi }{\varphi }w^2\right)\varphi ^2𝑑x$$
(4.2)
for all $`uC_c^1(\mathrm{\Omega }\{0\})`$. Hence, if in addition $`\varphi `$ satisfies $`\mathrm{\Delta }\varphi +V\varphi =0`$ (weakly) on $`\mathrm{\Omega }\{0\}`$, then

$$_\mathrm{\Omega }(|u|^2Vu^2)𝑑x=_\mathrm{\Omega }|w|^2\varphi ^2𝑑x$$
(4.3)
for all $`uC_c^1(\mathrm{\Omega }\{0\})`$. Hence $`H`$ is unitarily equivalent via (4.1) to the self-adjoint operator $`H_\varphi `$ on $`L_\varphi ^2`$, defined initially on $`C_c^1(\mathrm{\Omega }\{0\})`$ and given formally by

$$H_\varphi w=\frac{1}{\varphi ^2}\mathrm{div}(\varphi ^2w),w|_\mathrm{\Omega }=0.$$

The space $`C_c^1(\mathrm{\Omega }\{0\})`$ is invariant under multiplication (or division) by $`\varphi `$ and hence it is a form core also for $`H_\varphi `$. Moreover, a Sobolev inequality of the form

$$_\mathrm{\Omega }(|u|^2Vu^2)𝑑xc\left(_\mathrm{\Omega }|u|^qW𝑑x\right)^{2/q}$$

is valid for all $`uC_c^1(\mathrm{\Omega }\{0\})`$ if and only if

$$_\mathrm{\Omega }|w|^2\varphi ^2𝑑xc\left(_\mathrm{\Omega }|w|^q\varphi ^qW𝑑x\right)^{2/q}$$

for all $`wC_c^1(\mathrm{\Omega }\{0\})`$. Finally, the heat kernels of $`H`$ and $`H_\varphi `$ are related by

$$K(t,x,y)=\varphi (x)\varphi (y)K_\varphi (t,x,y),t>0,x,y\mathrm{\Omega },$$
(4.4)
and hence one can obtain estimates on $`K(t,x,y)`$ via estimates on $`K_\varphi (t,x,y)`$.

Example. As a typical example let us consider the case of a bounded domain $`\mathrm{\Omega }`$ in $`𝐑^N`$, $`N3`$, and let $`V(x)=\lambda |x|^2`$, $`\lambda ((N2)/2)^2`$. Let $`\varphi (x)=|x|^\alpha `$, $`\alpha `$ being the smallest solution of $`\alpha (N2\alpha )=\lambda `$. Then $`\mathrm{\Delta }\varphi +V\varphi =0`$ on $`\mathrm{\Omega }\{0\}`$ and therefore

$$_\mathrm{\Omega }(|u|^2\lambda \frac{u^2}{|x|^2})𝑑x=_\mathrm{\Omega }|w|^2|x|^{2\alpha }𝑑x.$$
(4.5)
for all $`uC_c^1(\mathrm{\Omega }\{0\})`$ or, equivalently, for all $`wC_c^1(\mathrm{\Omega }\{0\})`$. Moreover, the heat kernel of $`H=\mathrm{\Delta }V`$ is related to the heat kernel of $`H_\varphi `$ by

$$K(t,x,y)=|x|^\alpha |y|^\alpha K_\varphi (t,x,y).$$

We note here that a simple approximation argument shows that for $`\lambda <((N2)/2)^2`$ the form domain of $`H`$ is $`H_0^1(\mathrm{\Omega })`$, but at the critical case $`\lambda =((N2)/2)^2`$ the form domain is strictly larger than $`H_0^1(\mathrm{\Omega })`$; see also \[FT\].

Sobolev inequalities are related to heat kernel estimates by the following standard result \[D, Theorem 2.4.2\]: for any $`q>2`$,

$$\{\begin{array}{c}\text{the upper bound}\hfill \\ K_\varphi (t,x,y)<ct^{q/2},t>0,x,y\mathrm{\Omega }\hfill \\ \text{is equivalent to the Sobolev inequality}\hfill \\ _\mathrm{\Omega }|w|^2\varphi ^2𝑑xc\left(_\mathrm{\Omega }|w|^{\frac{2q}{q2}}\varphi ^2𝑑x\right)^{(q2)/q},wC_c^1(\mathrm{\Omega }\{0\}).\hfill \end{array}$$
(4.6)
In the rest of this section we shall apply the Hardy-Sobolev inequalities of Section 3 in order to obtain upper estimates on the heat kernel $`K(t,x,y)`$ of the operator $`\mathrm{\Delta }V`$ for critical and subcritical potentials $`V`$. For this we shall use (4.4) for appropriate functions $`\varphi `$, together with uniform estimates on $`K_\varphi (t,x,y)`$, obtained by means of (4.6). We present on-diagonal estimates, in the sense that we do not include the usual Gaussian term, which can be added by standard methods.

We assume that $`\mathrm{\Omega }`$ is a domain in $`𝐑^N`$, $`N3`$. We retain the notation introduced in the last example, and, in particular, we have $`H=\mathrm{\Delta }\lambda |x|^2`$, subject to Dirichlet boundary conditions on $`\mathrm{\Omega }`$. Part $`(\mathrm{i})`$ of the next proposition has been proved in \[MS\] but we include a proof for the sake of completeness. Part $`(\mathrm{ii})`$ improves upon the corresponding estimate of \[MS\], who had $`K(t,x,y)<ct^{\frac{N\alpha }{2}ϵ}|x|^\alpha |y|^\alpha `$.

###### Proposition 4.1

(subcritical case) Let $`K(t,x,y)`$ be the heat kernel of $`H=\mathrm{\Delta }\lambda \frac{1}{|x|^2}`$, subject to Dirichlet boundary conditions on $`\mathrm{\Omega }`$. For $`\lambda <((N2)/2)^2`$, let $`\alpha `$ be the smallest solution of $`\alpha (N2\alpha )=\lambda `$.
$`(\mathrm{i})`$ If $`\mathrm{\Omega }`$ is bounded and $`0\lambda <((N2)/2)^2`$ then

$$K(t,x,y)<ct^{N/2}|x|^\alpha |y|^\alpha ,t>0,x,y\mathrm{\Omega }.$$
(4.7)
$`(\mathrm{ii})`$ For any $`\mathrm{\Omega }𝐑^N`$ (bounded or unbounded) and $`\lambda 0`$ then

$$K(t,x,y)<ct^{N/2}\mathrm{min}\{1,(\frac{|x|}{t^{1/2}})^\alpha \}\mathrm{min}\{1,(\frac{|y|}{t^{1/2}})^\alpha \},t>0,x,y\mathrm{\Omega }.$$
(4.8)
Proof. (i) The boundedness of $`\mathrm{\Omega }`$ together with (3.1) imply

$$_\mathrm{\Omega }|w|^2|x|^{2\alpha }𝑑xc\left(_\mathrm{\Omega }|w|^{\frac{2N}{N2}}|x|^{2\alpha }𝑑x\right)^{(N2)/N},wC_c^{\mathrm{}}(\mathrm{\Omega }).$$
(4.9)
By (4.6) this implies $`K_\varphi (t,x,y)<ct^{N/2}`$, from which (4.7) follows using (4.4).

(ii) Comparison with the Laplacian implies that $`K(t,x,y)<ct^{N/2}`$. Moreover, inequality (3.1) for $`\beta p=2\alpha `$ reads

$$_\mathrm{\Omega }|w|^2|x|^{2\alpha }𝑑xc\left(_\mathrm{\Omega }|w|^{\frac{2(N2\alpha )}{N22\alpha }}|x|^{2\alpha }𝑑x\right)^{\frac{N22\alpha }{N2\alpha }}.$$

By means of (4.6) we deduce that $`K_\varphi (t,x,y)<ct^{N/2+\alpha }`$, $`t>0,x,y\mathrm{\Omega }`$. Hence

$$K(t,x,y)<ct^{N/2}\mathrm{min}\{1,(\frac{|x|}{t^{1/2}})^\alpha (\frac{|y|}{t^{1/2}})^\alpha \},t>0,x,y\mathrm{\Omega }.$$
(4.10)
This proves (4.8) when $`x=y`$. The general case follows from the semigroup property since

$`K(t,x,y)`$ $`=`$ $`{\displaystyle _\mathrm{\Omega }}K(t/2,x,z)K(t/2,z,y)𝑑z`$
$``$ $`\left({\displaystyle _\mathrm{\Omega }}K(t/2,x,z)^2𝑑z\right)^{1/2}\left({\displaystyle _\mathrm{\Omega }}K(t/2,z,y)^2𝑑z\right)^{1/2}`$
$`=`$ $`K(t,x,x)^{1/2}K(t,y,y)^{1/2}.`$
$`//`$

###### Theorem 4.2

(critical case) Let $`\mathrm{\Omega }`$ be a bounded domain and $`K(t,x,y)`$ be the heat kernel of $`H=\mathrm{\Delta }(\frac{N2}{2})^2\frac{1}{|x|^2}`$, subject to Dirichlet boundary conditions on $`\mathrm{\Omega }`$. Then

$$K(t,x,y)<ct^{\frac{N}{2}}|x|^{\frac{N2}{2}}|y|^{\frac{N2}{2}},t>0,x,y\mathrm{\Omega }.$$
(4.11)
Proof. Estimate (3.2) implies the weaker inequality

$$_\mathrm{\Omega }|w|^2|x|^{2N}𝑑xc\left(_\mathrm{\Omega }|w|^{\frac{2N}{N2}}|x|^{2N}𝑑x\right)^{(N2)/N}.$$
(4.12)
Hence $`K_\varphi (t,x,y)<ct^{\frac{N}{2}}`$ and (4.11) follows.$`//`$

We next consider the case $`\mathrm{\Omega }=𝐑^N`$, and the potential is critical at zero. More precisely we consider the potential $`V_ϵ`$ defined by (3.10), that is:

$$V_ϵ(x)=\{\begin{array}{cc}\left(\frac{N2}{2}\right)^2|x|^2,\hfill & |x|<1\hfill \\ ϵf(x),\hfill & |x|1\hfill \end{array},$$

where $`f`$ is a non-negative, continuous and radially symmetric function on $`\{|x|1\}`$. Moreover we assume $`f`$ to be subcritical, that is it satisfies (3.11):

$$f(x)K|x|^{2\sigma },|x|1,$$

for some $`\sigma ,K>0`$.

We retain the notation of section 3.1, and in particular we recall the definition (3.13) of $`ϵ_0`$. We have

###### Theorem 4.3

(the operator $`\mathrm{\Delta }V_ϵ`$ on $`𝐑^N`$) For any $`ϵ(0,ϵ_0)`$ the heat kernel of the operator $`\mathrm{\Delta }V_ϵ`$ satisfies

$$K(t,x,y)<ct^{N/2}\mathrm{max}\{|x|^{\frac{N2}{2}},1\}\mathrm{max}\{|y|^{\frac{N2}{2}},1\},t>0,x,y𝐑^N.$$
(4.13)
Proof. Let $`\psi (x)`$ be as in the proof of Theorem 3.4, cf (3.15). It follows from (3.14) that

$$_{𝐑^N}|v|^2\psi ^2𝑑xc\left(_{𝐑^N}|v|^{2N/(N2)}\psi ^{2N/(N2)}\stackrel{~}{X}_1^{(2N2)/(N2)}𝑑x\right)^{(N2)/N},$$
(4.14)
for all $`vC_c^1(𝐑^N\{0\})`$. Since $`\psi ^{2N/(N2)}\stackrel{~}{X}_1^{(2N2)/(N2)}c\psi ^2`$ and $`C_c^1(𝐑^N\{0\})`$ is a form core for $`H_\psi `$ and we conclude that $`K_\psi (t,x,y)<ct^{N/2}`$ whence,

$$K(t,x,y)<ct^{N/2}\psi (x)\psi (y).$$

The required estimate on $`K(t,x,y)`$ follows if we note that

$$c_1\mathrm{max}\{|x|^{(N2)/2},1\}\psi (x)c_2\mathrm{max}\{|x|^{(N2)/2},1\},x𝐑^N.$$

$`//`$

It is well known that the estimates of the above theorems can be improved to yield Gaussian decay of the heat kernel. We have

###### Proposition 4.4

Proposition 4.8 as well as Theorems 4.11 and 4.13 can be improved by adding a factor $`c_\delta \mathrm{exp}\{|xy|^2/((4+\delta )t)\}`$ to the corresponding right-hand sides.

Proof. The proof is standard. One can use Davies’s method of exponential perturbation \[D\] or Theorem 1.1 of \[G\]. Note that the argument is applied to the operator $`H_\varphi `$ – not to $`H`$. $`//`$

## 5 Logarithmic refinements

Our aim in this section is to obtain refined versions of the improved Hardy-Sobolev inequalities of Section 3. As an application, we prove heat kernel estimates for $`H=\mathrm{\Delta }VV_1`$ where $`V`$ is one of the potentials studied in Section 4 (that is, $`V(x)=((N2)/2)^2|x|^2`$ near zero) but $`V_1>0`$ is also critical. The criticality of $`V_1`$ is meant in the sense that the following improved Hardy inequality holds

$$_\mathrm{\Omega }|u|^2_\mathrm{\Omega }Vu^2_\mathrm{\Omega }V_1u^2,uC_c^{\mathrm{}}(\mathrm{\Omega }),$$

whereas this inequality is no longer true if we replace $`V_1`$ by $`(1+ϵ)V_1`$. Of course, $`V_1`$ is of lower order with respect to $`|x|^2`$ (near $`x=0`$) since $`((N2)/2)^2|x|^2`$ is already critical for the validity of the (simple) Hardy inequality. It is remarkable that the addition of the extra potential $`V_1`$ does not affect the time dependence of the heat kernel estimates, but only affects the spatial singularity at the origin, which is increased by a logarithmic factor; see Theorems 5.3 and 5.4.

More precisely, recalling that $`X_1(t)=(1\mathrm{log}t)^1`$, let us introduce the functions

$$X_{k+1}(t)=X_1(X_k(t)),k=1,2,\mathrm{},t(0,1).$$
(5.15)
These are iterated logarithmic functions that vanish at an increasingly slow rate at $`t=0`$ and are equal to one at $`t=1`$. In \[FT\] the following improved Hardy inequality was obtained for a bounded domain $`\mathrm{\Omega }`$ with $`D=sup_\mathrm{\Omega }|x|`$ :

$`{\displaystyle _\mathrm{\Omega }}\left\{|u|^2\left({\displaystyle \frac{N2}{2}}\right)^2{\displaystyle \frac{u^2}{|x|^2}}{\displaystyle \frac{u^2}{4|x|^2}}{\displaystyle \underset{i=1}{\overset{k}{}}}X_1^2({\displaystyle \frac{|x|}{D}})\mathrm{}X_i^2({\displaystyle \frac{|x|}{D}})\right\}𝑑x`$
$`{\displaystyle \frac{1}{4}}{\displaystyle _\mathrm{\Omega }}{\displaystyle \frac{u^2}{|x|^2}}X_1^2({\displaystyle \frac{|x|}{D}})\mathrm{}X_{k+1}^2({\displaystyle \frac{|x|}{D}})𝑑x,uC_c^{\mathrm{}}(\mathrm{\Omega }).`$ (5.16)
The potentials in the left-hand side of (5.16) are critical for each $`k`$, in the sense that (5.16) is sharp: the term $`X_{k+1}^2`$ cannot be replaced by $`c_ϵX_{k+1}^{2ϵ}`$ for any $`ϵ>0`$, and the constant $`1/4`$ in the right-hand side is also optimal. In Theorem 5.3 and for bounded $`\mathrm{\Omega }`$ we obtain upper estimates on the heat kernel of the operator

$$H=\mathrm{\Delta }\left(\frac{N2}{2}\right)^2\frac{1}{|x|^2}\frac{1}{4|x|^2}\underset{i=1}{\overset{k1}{}}X_1^2\mathrm{}X_i^2\frac{\mu }{|x|^2}X_1^2\mathrm{}X_k^2$$
(5.17)
for $`\mu >1/4`$, as well as for the critical case $`\mu =1/4`$; for this we use results obtained in \[FT\]. For the critical case $`\mu =1/4`$ we also consider operators defined on $`𝐑^N`$, in analogy to the operator $`\mathrm{\Delta }V_ϵ`$ of Theorem 4.13; for this we use Theorem 5.1 below, and the corresponding heat kernel estimate is given in Theorem 5.4.

### 5.1 Refined Hardy-Sobolev inequalities

In this subsection we prove two theorems that are refined versions of Theorems 3.4 and 3.5 correspondingly. We recall definition (5.15) and set

$$\stackrel{~}{X}_k(|x|)=\{\begin{array}{cc}X_k(|x|),\hfill & |x|<1,\hfill \\ 1,\hfill & |x|>1,\hfill \end{array}$$

$$\{\begin{array}{cc}Y_k(|x|)=X_k(1/|x|),\hfill & |x|>1,\hfill \\ \stackrel{~}{Y}_k(|x|)=\stackrel{~}{X}_k(1/|x|),\hfill & |x|>0.\hfill \end{array}$$

We point out the differentiation rules for $`X_k(r)`$ and $`Y_k(r)`$:

$$\frac{d}{dr}X_k^a=\frac{a}{r}X_1\mathrm{}X_{k1}X_k^{a+1},\frac{d}{dr}Y_k^a=\frac{a}{r}Y_1Y_2\mathrm{}Y_{k1}Y_k^{a+1},r=|x|,$$
(5.18)
valid for $`0<r<1`$ and $`r>1`$ respectively, which are easily proved by induction.

As in Theorem 3.4, we assume that $`f`$ is a non-negative, continuous and radially symmetric function on $`B^c`$ satisfying (3.11), that is,

$$f(x)K|x|^{2\sigma },|x|1,$$

for some $`\sigma ,K>0`$. For $`ϵ>0`$ we also define

$$V_{k,ϵ}(x)=\{\begin{array}{cc}\left(\frac{N2}{2}\right)^2|x|^2+\frac{1}{4}|x|^2_{i=1}^kX_1^2(|x|)\mathrm{}X_i^2(|x|),\hfill & |x|<1,\hfill \\ ϵf(x),\hfill & |x|>1.\hfill \end{array}$$
(5.19)
We then have

###### Theorem 5.1

Assume that $`k<N2`$ and define

$$ϵ_{k,0}=\underset{u^1(B^c)}{inf}\frac{_{B^c}|u|^2𝑑x\frac{N2+k}{2}_Bu^2𝑑S}{_{B^c}fu^2𝑑x}.$$
(5.20)
Then $`ϵ_{k,0}>0`$ and for $`ϵ(0,ϵ_{k,0})`$ there holds

$$_{𝐑^N}|u|^2𝑑x_{𝐑^N}V_{k,ϵ}u^2𝑑xc\left(_{𝐑^N}|u|^{2N/(N2)}(\stackrel{~}{X}_1\mathrm{}\stackrel{~}{X}_{k+1})^{\frac{2N2}{N2}}𝑑x\right)^{(N2)/N},$$
(5.21)
for all $`uC_c^{\mathrm{}}(𝐑^N)`$.

Remark. The constant $`ϵ_{k,0}`$ is optimal in the sense that inequality (5.21) fails for $`ϵ=ϵ_{k,0}`$. Also the exponent $`(2N2)/(N2)`$ in (5.21) is sharp in the sense that it cannot be replaced by a smaller exponent. The proof of these two facts is rather involved; see \[FT\] for similar arguments. We do not use these facts in the sequel.
Proof. The proof follows closely that of Theorem 3.4, so we only give a sketch of it. The positivity of $`ϵ_{k,0}`$ follows from Lemma 2.4 (i), yielding $`ϵ_{k,0}K^1\mu _B((N2+k)/2)`$. Now let $`ϵ(0,ϵ_{k,0})`$ be fixed and let $`\stackrel{~}{\psi }`$ be the radially symmetric solution to the problem

$$\begin{array}{cc}\mathrm{\Delta }\stackrel{~}{\psi }+V_{k,ϵ}\stackrel{~}{\psi }=0,\hfill & |x|>1,\hfill \\ \frac{\stackrel{~}{\psi }}{\nu }=\frac{N2+k}{2}\stackrel{~}{\psi },\hfill & |x|=1,\hfill \end{array}$$

normalized so that $`\stackrel{~}{\psi }=1`$ on $`\{|x|=1\}`$. The function

$$\psi (x)=\{\begin{array}{cc}|x|^{(N2)/2}X_1^{1/2}\mathrm{}X_k^{1/2},\hfill & |x|<1\hfill \\ \stackrel{~}{\psi }(x),\hfill & |x|>1,\hfill \end{array}$$
(5.22)
is then $`C^1`$, radially symmetric and a direct computation which uses (5.18) shows that $`\mathrm{\Delta }\psi +V_{k,ϵ}\psi =0`$ in $`𝐑^N`$. Exactly as in Theorem 3.4, $`\psi `$ is positive, radially symmetric and has a positive limit as $`r+\mathrm{}`$. We then prove (5.21) in the case where $`u`$ is radially symmetric, using once again Proposition 3.1. The validity of (5.21) for general $`uC_c^{\mathrm{}}(𝐑^N)`$ follows from Lemma 3.3. $`//`$

We finally prove a refined version of Theorem 3.5. Let us fix a a non-negative, continuous and radially symmetric function $`g`$ on $`B=\{|x|<1\}`$, such that

$$g(x)K|x|^{2+\sigma },|x|<1,$$

for some $`\sigma ,K>0`$. Further for $`ϵ>0`$ we define

$$\widehat{V}_{k,ϵ}(x)=\{\begin{array}{cc}ϵg(x),\hfill & |x|<1,\hfill \\ \left(\frac{N2}{2}\right)^2|x|^2+\frac{1}{4|x|^2}_{i=1}^kY_1^2(|x|)\mathrm{}Y_i^2(|x|),\hfill & |x|>1.\hfill \end{array}$$

We then have

###### Theorem 5.2

Assume that $`k<N2`$ and define

$$\overline{ϵ}_{k,0}=\underset{uH^1(B)}{inf}\frac{_B|u|^2𝑑x+\frac{N2k}{2}_Bu^2𝑑S}{_Bgu^2𝑑x}.$$

Then $`\overline{ϵ}_{k,0}>0`$ and for $`ϵ(0,\overline{ϵ}_{k,0})`$ there holds

$$_{𝐑^N}|u|^2𝑑x_{𝐑^N}\widehat{V}_{k,ϵ}u^2𝑑xc\left(_{𝐑^N}|u|^{2N/(N2)}(\stackrel{~}{Y}_1\mathrm{}\stackrel{~}{Y}_{k+1})^{\frac{2N2}{N2}}𝑑x\right)^{(N2)/N},$$
(5.23)
for all $`uC_c^{\mathrm{}}(𝐑^N)`$.

Proof.  We omit the proof, which is similar to that of Theorem 3.5. $`//`$

### 5.2 Refined heat kernel estimates

In Theorems 4.11 and 4.13 we obtained heat kernel estimates for operators $`\mathrm{\Delta }V`$ where $`V(x)=((N2)/2)^2|x|^2`$ near the origin. We shall now prove estimates for $`\mathrm{\Delta }VV_1`$, with $`V_1`$ also critical near the origin. In Theorems 5.3 and 5.4 we consider the cases $`\mathrm{\Omega }`$ bounded and $`\mathrm{\Omega }=𝐑^N`$ respectively.

For $`k1`$ and $`\mu 1/4`$ we define

$$V_k^\mu (x)=\frac{\left(\frac{N2}{2}\right)^2}{|x|^2}+\frac{1}{4|x|^2}\underset{i=1}{\overset{k1}{}}X_1^2\mathrm{}X_i^2+\frac{\mu }{|x|^2}X_1^2\mathrm{}X_k^2,x\mathrm{\Omega },$$
(5.24)
$`(X_i=X_i(|x|/D),D=sup_\mathrm{\Omega }|x|)`$ and consider the operator $`H=\mathrm{\Delta }V_k^\mu `$ subject to Dirichlet boundary conditions on $`\mathrm{\Omega }`$. In \[FT, Proposition 7.2\] the Hardy-Sobolev inequality

$$_\mathrm{\Omega }(|u|^2V_k^{1/4}u^2)𝑑xc\left(_\mathrm{\Omega }|u|^{2N/(N2)}(X_1\mathrm{}X_{k+1})^{\frac{2N2}{N2}}𝑑x\right)^{(N2)/N},uC_c^{\mathrm{}}(\mathrm{\Omega }),$$
(5.25)
was obtained. Let $`\beta `$ be the largest solution of $`\beta (1\beta )=\mu `$ and define

$$\varphi _{k,\beta }(x)=|x|^{\frac{N2}{2}}X_1^{1/2}\mathrm{}X_{k1}^{1/2}X_k^\beta .$$
(5.26)
Using (5.18) we verify that $`\mathrm{\Delta }\varphi _{k,\beta }+V_k^\mu \varphi _{k,\beta }=0`$ and hence the change of variables $`u=\varphi _{k,\beta }w`$ yields

$$_\mathrm{\Omega }(|u|^2V_k^\mu u^2)𝑑x=_\mathrm{\Omega }|w|^2\varphi _{k,\beta }^2𝑑x$$
(5.27)
for all $`wC_c^{\mathrm{}}(\mathrm{\Omega })`$. We have

###### Theorem 5.3

Let $`\mathrm{\Omega }`$ be bounded, $`1k<N2`$, and $`0<\mu 1/4`$. The heat kernel of $`H=\mathrm{\Delta }V_k^\mu `$ satisfies the estimate

$$K(t,x,y)<ct^{N/2}\varphi _{k,\beta }(x)\varphi _{k,\beta }(y),t>0,x,y\mathrm{\Omega };$$
(5.28)
here $`V_k^\mu `$ is given by (5.24) and $`\varphi _{k,\beta }(x)`$ by (5.26).

Proof. For the proof we distinguish two cases.
(1) Case $`\mu <1/4`$. For $`wC_c^{\mathrm{}}(\mathrm{\Omega })`$ we have

$`{\displaystyle _\mathrm{\Omega }}|w|^2\varphi _{k,\beta }^2𝑑x`$ $`=`$ $`{\displaystyle _\mathrm{\Omega }}|u|^2𝑑x{\displaystyle _\mathrm{\Omega }}V_k^\mu u^2𝑑x`$
(by (5.16)) $``$ $`c\left({\displaystyle _\mathrm{\Omega }}(|u|^2V_{k1}^{1/4}u^2)𝑑x\right)`$
(by (5.25)) $``$ $`c\left({\displaystyle _\mathrm{\Omega }}|u|^{2N/(N2)}(X_1\mathrm{}X_k)^{\frac{2N2}{N2}}𝑑x\right)^{(N2)/N}`$
$`=`$ $`c\left({\displaystyle _\mathrm{\Omega }}|w|^{2N/(N2)}|x|^N(X_1\mathrm{}X_{k1})X_k^{\frac{2N22N\beta }{N2}}𝑑x\right)^{(N2)/N}`$
$``$ $`c\left({\displaystyle _\mathrm{\Omega }}|w|^{2N/(N2)}\varphi _{k,\beta }^2𝑑x\right)^{(N2)/N}.`$
This implies that $`K_{\varphi _{k,\beta }}(t,x,y)<ct^{N/2}`$ and (5.28) follows.
(2) Case $`\mu =1/4`$. By \[FT, Lemma 7.5\] the following Sobolev inequality holds

$`{\displaystyle _\mathrm{\Omega }}|w|^2|x|^{2N}X_1^1\mathrm{}X_k^1𝑑x`$
$``$ $`c\left({\displaystyle _\mathrm{\Omega }}|w|^{\frac{2N}{N2}}|x|^NX_1\mathrm{}X_kX_{k+1}^{\frac{2N2}{N2}}𝑑x\right)^{(N2)/N},wC_c^{\mathrm{}}(\mathrm{\Omega }).`$
This implies in particular

$$_\mathrm{\Omega }|w|^2\varphi _{k,\beta }^2𝑑xc\left(_\mathrm{\Omega }|w|^{\frac{2N}{N2}}\varphi _{k,\beta }^2𝑑x\right)^{(N2)/N},wC_c^{\mathrm{}}(\mathrm{\Omega })$$

and hence we have the uniform estimate $`K_{\varphi _{k,\beta }}(t,x,y)<ct^{N/2}`$ as required. //

We finally have the following consequence of Theorem 5.1, where we retain the notation of that theorem:

###### Theorem 5.4

Let $`1k<N2`$, and $`ϵ(0,ϵ_{k,0})`$ with $`ϵ_{k,0}`$ given by (5.20). the heat kernel of the operator $`\mathrm{\Delta }V_{k,ϵ}`$ satisfies

$$K(t,x,y)<ct^{N/2}\psi (x)\psi (y),t>0,x,y𝐑^N;$$

here $`V_{k,ϵ}`$ is given by (5.19) and $`\psi (x)`$ by (5.22).

Proof. The result follows directly from Theorem 5.1 by means of (4.6).$`//`$


text:
# References

EFFECTIVE LAGRANGIAN AND DYNAMICAL SYMMETRY BREAKING IN $`SU(2)U(1)`$ NJL MODEL
S.I. Kruglov

International Educational Centre, 2727 Steeles Ave. W, # 104,
Toronto, Ontario, Canada M3J 3G9

## Abstract

Dynamical symmetry breaking and the formation of scalar condensates in the $`SU(2)U(1)`$ Nambu-Jona-Lasinio model with two coupling constants has been studied in the framework of the mean-field approximation. The bosonization procedures of the model are performed using the functional integration method. The possibility of the spontaneous CP symmetry breaking in the model under consideration has been shown. The mass spectrum of the bound states of fermions, as well as the effective Lagrangian of interacting scalar and pseudoscalar mesons are obtained.

PACS numbers: 11.30.Qc; 12.39.Fe; 24.85.+p.

One of the modern problems is the investigation of nonperturbative effects in quantum chromodynamics (QCD) and obtaining the effective hadron Lagrangians. This area of low energy is difficult to study in the framework of QCD because coupling constant $`\alpha _s`$ is not a small parameter. Therefore, in this domain, some phenomenological models are used. Among them are a model based on local effective chiral Lagrangians (ECL) , , , a model of instanton vacuum ,  (see also ), the Nambu-Jona-Lasinio (NJL) models  which use a contact four-quark interactions, etc.. In this letter the possibility of the spontaneous CP symmetry breaking in the NJL model is shown. It should be noted that the electric dipole moment of particles violates CP invariance and can be explained by the $`\theta `$-term of QCD. Although the effect of CP violation in strong interactions is small, the investigation of mechanisms of CP violation is important.

Let us consider a four-fermion model with scalar-scalar and pseudoscalar-pseudoscalar interactions and possessing the internal symmetry group $`SU(2)U(1)`$ and two coupling constants:

$$(x)=\overline{\psi }^n(x)(\gamma _\mu _\mu +m_0)\psi ^n(x)+\frac{F}{2}\left[\overline{\psi }^n(x)\psi ^n(x)\right]^2\frac{G}{2}\left[\overline{\psi }^n(x)\gamma _5\tau ^a\psi ^n(x)\right]^2,$$
(1)
where $`\tau ^a`$ ($`a=1,2,3`$) are the Pauli matrices, $`_\mu =(/x_i,i/x_0)`$ ($`x_0`$ is the time), $`m_0=\text{diag}(m_{01},m_{02})`$; $`m_{01}`$, $`m_{02}`$ are the current masses of fermions (quarks), $`\gamma _\mu `$ are the Dirac matrices, $`\gamma _5=\gamma _1\gamma _2\gamma _3\gamma _4`$, $`\psi `$ is the doublet of fermions ($`u`$, $`d`$ quarks). Here summation over colour fermion degrees of freedom $`n=1,2,\mathrm{},N_C`$ has been performed. We also assume equal current masses of fermions, $`m_{01}=m_{02}m_0`$. Lagrangian (1) is not invariant under $`\gamma _5`$-chiral transformations, i.e. $`U_A(1)`$-symmetry is broken. Note that the violation of $`U_A(1)`$-symmetry allows one to solve the problem of the differences of masses of $`\eta `$ and $`\eta ^{}`$ mesons in the framework of $`SU(3)_fSU(3)_f`$-model. We shell investigate dynamical symmetry breaking (DSB) and the mass formation using functional integration method .

Considering the generating functional for Green’s functions

$$Z[\overline{\eta },\eta ]=N_0D\overline{\psi }D\psi \mathrm{exp}\left\{id^4x\left[(x)+\overline{\psi }^n(x)\eta ^n(x)+\overline{\eta }^n(x)\psi ^n(x)\right]\right\},$$
(2)
where $`\overline{\eta }^n`$, $`\eta ^n`$ are external sources, and redefining the constant $`N_0`$, one can represent Eq. (2) as follows:

$$Z[\overline{\eta },\eta ]=ND\overline{\psi }D\psi D\mathrm{\Phi }_0D\stackrel{~}{\mathrm{\Phi }}_a\mathrm{exp}\{id^4x[\overline{\psi }^n(x)[\gamma _\mu _\mu +m_0f_0\mathrm{\Phi }_0(x)$$

(3)
$$ig_0\stackrel{~}{\mathrm{\Phi }}_a(x)\gamma _5\tau ^a]\psi ^n(x)\frac{\mu ^2}{2}\stackrel{~}{\mathrm{\Phi }}_a^2(x)\frac{M^2}{2}\mathrm{\Phi }_0^2(x)+\overline{\psi }^n(x)\eta ^n(x)+\overline{\eta }^n(x)\psi ^n(x)]\}.$$

We define constants $`F=f_0^2/M^2`$, $`G=g_0^2/\mu ^2`$, where $`f_0`$, $`g_0`$ are dimensionless coupling constants and constants $`M`$, $`\mu `$ are mass-dimensional. One can integrate the functional integral in Eq. (3) over the fermion fields $`\overline{\psi }`$, $`\psi `$ and obtain

$$Z[\overline{\eta },\eta ]=ND\mathrm{\Phi }_0D\stackrel{~}{\mathrm{\Phi }}_a\mathrm{exp}\left\{iS[\mathrm{\Phi }]+id^4xd^4y\overline{\eta }^n(x)S_f(x,y)\eta ^n(y)\right\},$$
(4)
$$S[\mathrm{\Phi }]=\frac{1}{2}d^4x\left[M^2\mathrm{\Phi }_0^2(x)+\mu ^2\stackrel{~}{\mathrm{\Phi }}_a^2(x)\right]$$

(5)
$$i\text{tr}\mathrm{ln}\left[1+\widehat{G}\left(f_0\mathrm{\Phi }_0(x)+ig_0\stackrel{~}{\mathrm{\Phi }}_a(x)\gamma _5\tau ^a\right)\right],$$

where we introduce the effective action for bosonic collective fields $`S[\mathrm{\Phi }]`$. The fields $`\mathrm{\Phi }_0(x)`$, $`\stackrel{~}{\mathrm{\Phi }}_a(x)`$ can be identified with the $`\sigma `$-meson and triplet of pseudoscalar $`\pi _a`$-mesons . The Green functions for free fermions $`\widehat{G}`$, and for fermions in the external collective fields $`S_f(x,y)`$ are solutions of the equations

$$(\gamma _\mu _\mu +m_0)\widehat{G}(x,y)=\delta (xy),$$
(6)
$$\left[\gamma _\mu _\mu +m_0f_0\mathrm{\Phi }_0(x)ig_0\stackrel{~}{\mathrm{\Phi }}_a(x)\gamma _5\tau ^a\right]S_f(x,y)=\delta (xy).$$
(7)
In four-fermion model under consideration, the symmetric vacuum is not stable. As a result, the physical vacuum is reconstructed, and the appearance of the condensates leads to DSB of the initial $`SU(2)U(1)`$-symmetry. These condensates are coordinate-independent fields $`\mathrm{\Phi }_0`$, $`\stackrel{~}{\mathrm{\Phi }}_a`$ (the mean field approximation) which obey Eq. (7). Eq. (7) in the momentum space for condensates $`\mathrm{\Phi }_0`$, $`\stackrel{~}{\mathrm{\Phi }}_a`$ becomes

$$(i\widehat{p}A)S_f(p)=1,$$
(8)
where $`\widehat{p}=p_\mu \gamma _\mu `$, $`p_\mu =(\text{p},ip_0)`$, $`A=m_0+f_0\mathrm{\Phi }_0+ig_0\stackrel{~}{\mathrm{\Phi }}_a\gamma _5\tau ^a`$. Using the gauge $`\mathrm{\Phi }_00`$, $`\stackrel{~}{\mathrm{\Phi }}_30`$, $`\stackrel{~}{\mathrm{\Phi }}_1=\stackrel{~}{\mathrm{\Phi }}_2=0`$ in which the matrix $`A`$ is diagonal, we obtain from Eq. (8) the Green function

$$S_f(p)=\text{diag}(\frac{i\widehat{p}+m_1+ig_0\stackrel{~}{\mathrm{\Phi }}_3\gamma _5}{p^2+m^2},\frac{i\widehat{p}+m_1ig_0\stackrel{~}{\mathrm{\Phi }}_3\gamma _5}{p^2+m^2}),$$
(9)
where the masses of fermions are defined as follows

$$m_1=m_0f_0\mathrm{\Phi }_0,m^2=m_1^2+g_0^2\stackrel{~}{\mathrm{\Phi }}_3^2.$$
(10)
If the current fermion masses $`m_0=0`$, the dynamical fermion masses $`m0`$. It should be noted that the components containing the term $`ig_0\stackrel{~}{\mathrm{\Phi }}_3\gamma _5`$ in Eq. (9) violate CP parity.

To obtain the vacuum condensates $`\mathrm{\Phi }_0`$, $`\stackrel{~}{\mathrm{\Phi }}_3`$ from Eq. (5), one finds the equations for the fields $`\mathrm{\Phi }_0(x)`$, $`\stackrel{~}{\mathrm{\Phi }}_3(x)`$:

$$\frac{\delta S[\mathrm{\Phi }]}{\delta \mathrm{\Phi }_0(x)}=M^2\mathrm{\Phi }_0(x)+if_0\text{tr}S_f(x,x)=0,$$

(11)
$$\frac{\delta S[\mathrm{\Phi }]}{\delta \stackrel{~}{\mathrm{\Phi }}_a(x)}=\mu ^2\stackrel{~}{\mathrm{\Phi }}_a(x)g_0\text{tr}\left[S_f(x,x)\gamma _5\tau ^a\right]=0.$$

With the help of Eqs. (9), (11), we obtain gap equations for the vacuum condensates:

$$M^2\mathrm{\Phi }_0=2f_0Im_1,$$

(12)
$$\mu ^2\stackrel{~}{\mathrm{\Phi }}_3=2g_0^2I\stackrel{~}{\mathrm{\Phi }}_3,$$

where

$$I=\frac{iN_C}{4\pi ^4}\frac{d^4p}{p^2+m^2}(d^4p=id^3pdp_0).$$
(13)
The quadratically divergent integral in Eq. (13) can be evaluated with help of the momentum cutoff $`\mathrm{\Lambda }`$, and gap equations (12) possess non-trivial ($`\mathrm{\Phi }_00`$, $`\stackrel{~}{\mathrm{\Phi }}_30`$) non-analytic in constants $`F`$, $`G`$ solutions  at $`N_CF\mathrm{\Lambda }>2\pi ^2`$, $`N_CG\mathrm{\Lambda }>2\pi ^2`$. Thus, even at zero current masses of fermions ($`m_0=0`$), as a result of the phase transition, they acquire non-zero masses, and the massive states of fermions correspond to the minimum of effective potential , . The appearance of the non-zero vacuum condensate $`\stackrel{~}{\mathrm{\Phi }}_30`$ indicates the spontaneous CP symmetry breaking.

From Eqs. (12) one arrives at the relationship

$$Fm_0=(FG)f_0\mathrm{\Phi }_0.$$
(14)
It follows from Eq. (14) that if the bare masses of fermions $`m_0=0`$ and $`\mathrm{\Phi }_00`$, the equality $`F=G`$ is valid, and we come to the model with only one coupling constant.

The parameter cutoff ($`\mathrm{\Lambda }`$) specifies the region of non-local fermion-antifermion (quark-antiquark) interactions and this region is determined by the size $`1/\mathrm{\Lambda }`$.

Expending the logarithm in Eq. (5) in small fluctuations of fields $`\mathrm{\Phi }_0^{}(x)`$, $`\stackrel{~}{\mathrm{\Phi }}_a^{}(x)`$ one obtains:

$$\mathrm{\Phi }_0(x)=\mathrm{\Phi }_0+\mathrm{\Phi }_0^{}(x),\stackrel{~}{\mathrm{\Phi }}_3(x)=\stackrel{~}{\mathrm{\Phi }}_3+\stackrel{~}{\mathrm{\Phi }}_3^{}(x),$$

where condensates $`\mathrm{\Phi }_0`$, $`\stackrel{~}{\mathrm{\Phi }}_3`$ are the solutions of gap equations (12). Then the effective action (5) becomes

$$S[\mathrm{\Phi }^{}]=\frac{1}{2}d^4xd^4y\mathrm{\Phi }_A^{}(x)\mathrm{\Delta }_{AB}^1(x,y)\mathrm{\Phi }_B^{}(y)+\underset{n=3}{\overset{\mathrm{}}{}}\frac{i}{n}\text{tr}\left[S_f\left(f_0\mathrm{\Phi }_0^{}+ig_0\stackrel{~}{\mathrm{\Phi }}_a^{}\gamma _5\tau ^a\right)\right]^n,$$

(15)
$$\mathrm{\Delta }_{AB}^1(p)=ig_Ag_B\text{tr}\left[\frac{d^4k}{(2\pi ^4)}S_f(k)T_AS_f(kp)T_B\right]+\delta _{AB}M_A^2,$$

where $`g_A=(f_0,g_0)`$, $`M_A=(M,\mu )`$, $`T_A=(1,i\gamma _5\tau ^a)`$, $`\mathrm{\Phi }_A=(\mathrm{\Phi }_0,\stackrel{~}{\mathrm{\Phi }}_a)`$. Using the quark propagator (9), gap equations (12), and calculating the nonzero elements $`\mathrm{\Delta }_{AB}^1(p)`$ in Eq. (15), we find

$$\mathrm{\Delta }_{11}^1(p)=\mathrm{\Delta }_{22}^1(p)=p^2Z_3^1+𝒪(g_0^2),$$

$$\mathrm{\Delta }_{33}^1(p)=\left(p^2+4g_0^2\stackrel{~}{\mathrm{\Phi }}_3^2\right)Z_3^1+𝒪(g_0^2),$$

(16)
$$\mathrm{\Delta }_{00}^1(p)=\frac{2f_0^2m_0I}{m_0m_1}+\left(p^2+4m_1^2\right)\frac{f_0^2}{g_0^2}Z_3^1+𝒪(f_0^2),$$

$$\mathrm{\Delta }_{03}^1(p)=4m_1f_0\stackrel{~}{\mathrm{\Phi }}_3Z_3^1+𝒪(f_0g_0),$$

where the constant of renormalization is given by

$$Z_3^1=\frac{ig_0^2N_C}{4\pi ^4}\frac{d^4q}{\left(q^2+m^2\right)^2}.$$
(17)
Now we introduce the renormalized fields $`\stackrel{~}{\mathrm{\Phi }}_a(x)=Z_3^{1/2}\stackrel{~}{\mathrm{\Phi }}_a^{}(x)`$, $`\mathrm{\Phi }_0(x)=(f_0/g_0)Z_3^{1/2}\mathrm{\Phi }_0^{}(x)`$ and coupling constants $`g^2=Z_3g_0^2`$, $`f^2=Z_3f_0^2`$. Logarithmic and quadratic integrals are connected by the relation

$$Z_3^1=\frac{g_0^2}{m^2}I\frac{g_0^2}{4\pi ^2}.$$
(18)
It follows from Eq. (17) that the expansion in $`g^2/4\pi ^2`$, $`f^2/4\pi ^2`$ corresponds to the $`1/N_C`$ expansion. Neglecting the terms (radiation corrections) $`𝒪(g^2)`$, $`𝒪(f^2)`$, $`𝒪(fg)`$, one finds the renormalized, quadratic in the boson fields, effective Lagrangian

$$_{free}=\frac{1}{2}\left[\left(_\mu \mathrm{\Phi }_A(x)\right)^2+m_{AB}^2\mathrm{\Phi }_A(x)\mathrm{\Phi }_B(x)\right],$$
(19)
where $`\mathrm{\Phi }_A=(\mathrm{\Phi }_0,\stackrel{~}{\mathrm{\Phi }}_a)`$ and the elements of the mass matrix read

$$m_{00}^2=4m_1^2+\frac{2m_0m^2}{m_0m_1},m_{11}^2=m_{22}^2=0,$$

(20)
$$m_{03}^2=4m_1g\stackrel{~}{\mathrm{\Phi }}_3,m_{33}^2=4g^2\stackrel{~}{\mathrm{\Phi }}_3^2.$$

The mass spectrum of the bosonic fields $`\mathrm{\Phi }_A(x)`$ can be obtained by diagonalizing the mass matrix $`m_{AB}`$ in Eq. (19). In accordance with Eq. (20), the masses of the fields $`\stackrel{~}{\mathrm{\Phi }}_1(x)`$, $`\stackrel{~}{\mathrm{\Phi }}_2(x)`$ are zero, in agreement with the Goldstone theorem . The fields $`\mathrm{\Phi }_0(x)`$, $`\stackrel{~}{\mathrm{\Phi }}_3(x)`$ possess nonzero masses. One may make the $`SO(2)`$-transformations

$$\mathrm{\Phi }_0^{}(x)=\mathrm{\Phi }_0(x)\mathrm{cos}\alpha \stackrel{~}{\mathrm{\Phi }}_3(x)\mathrm{sin}\alpha ,\stackrel{~}{\mathrm{\Phi }}_3^{}(x)=\mathrm{\Phi }_0(x)\mathrm{sin}\alpha +\stackrel{~}{\mathrm{\Phi }}_3(x)\mathrm{cos}\alpha ,$$
(21)
where $`\mathrm{tan}2\alpha =2m_{00}^2/(m_{33}^2m_{00}^2)`$ to diagonalize the mass matrix $`m_{AB}`$. As a result, we obtain the masses of bosonic fields $`\mathrm{\Phi }_0^{}(x)`$, $`\stackrel{~}{\mathrm{\Phi }}_3^{}(x)`$:

$$m_{00}^2=m_{00}^2\mathrm{cos}^2\alpha +m_{33}^2\mathrm{sin}^2\alpha m_{03}^2\mathrm{sin}2\alpha ,$$

(22)
$$m_{33}^2=m_{00}^2\mathrm{sin}^2\alpha +m_{33}^2\mathrm{cos}^2\alpha +m_{03}^2\mathrm{sin}2\alpha ,$$

but the fields $`\stackrel{~}{\mathrm{\Phi }}_1(x)`$, $`\stackrel{~}{\mathrm{\Phi }}_2(x)`$ remain massless. According to Eq. (22) the field $`\stackrel{~}{\mathrm{\Phi }}_3(x)`$ acquires small mass due to the CP-violating condensate $`\stackrel{~}{\mathrm{\Phi }}_3`$. The transformations of the collective fields (21) are induced by the corresponding transformations of fermionic fields $`\psi (x)`$.

Now we evaluate the effective Lagrangian of interacting bosonic fields by counting the components in the sum (15) with $`n=3`$ and $`n=4`$ in the case of small parameters of expansion $`g^2/4\pi ^2<1`$, $`f^2/4\pi ^2<1`$. Note that the fermion loops at $`n>4`$ give small convergent corrections. We make calculations of three- and four-point functions at the particular case of equal coupling constants $`F=G`$ ($`f=g`$) and skipping the small CP-violating condensate $`\stackrel{~}{\mathrm{\Phi }}_3=0`$. After renormalization, we obtain the effective Lagrangian of interacting bosonic fields to an accuracy $`𝒪(g^2)`$, as follows (see also ):

$$_{int}(x)=2gm\left(\mathrm{\Phi }_0^3(x)+\mathrm{\Phi }_0(x)\stackrel{~}{\mathrm{\Phi }}_a^2(x)\right)$$

(23)
$$\frac{g^2}{2}\left[\mathrm{\Phi }_0^4(x)+\left(\stackrel{~}{\mathrm{\Phi }}_a^2(x)\right)^2+6\mathrm{\Phi }_0^2(x)\stackrel{~}{\mathrm{\Phi }}_a^2(x)\right].$$

We have just considered the NJL model with $`SU(2)U(1)`$ internal symmetry group and two coupling constants which can provide the dynamical CP symmetry violation. The scalar collective field $`\mathrm{\Phi }_0`$ is associated with $`\sigma `$-meson, and the pseudoscalar bosons $`\stackrel{~}{\mathrm{\Phi }}_a`$ are associated with the triplet of pions. As usual , pions play the role of the Goldstone bosons. The four-fermion interaction is some approximation to the real quark interactions. However, to take into consideration confinement of quarks, one has to modify the model.


text:
# Statistics of spinons in the spin-liquid phase of Cs2CuCl4

## Abstract

Motivated by a recent experiment on Cs2CuCl4,coldea  we study the spin dynamics of the spin-liquid phase of the spin-1/2 frustrated Heisenberg antiferromagnet on the anisotropic triangular lattice. There have been two different proposalschung ; zhou  for the spin-liquid phase of Cs2CuCl4. These spin-liquid states support different statistics of spinons; the bosonic Sp($`N`$) large-$`N`$ mean field theorychung ; subir  predicts bosonic spinons, while the SU(2) slave-boson mean field theoryzhou ; wen02  leads to fermionic spinons. We compute the dynamical spin structure factor for both types of spin-liquid state at zero and finite temperatures. While at zero temperature both theories agree with experiment on a qualitative level, they show substantial differences in the temperature dependence of the dynamical spin structure factor.

The existence of a spin-liquid state in frustrated quantum magnets has been one of the central issues in the field of strongly correlated systems.anderson73 ; sondhi01 ; balents01 ; nayak01 ; capriotti ; z2  Recent experimental studies of various frustrated magnetic compounds provided excellent opportunities to investigate the competition between geometric frustration and quantum fluctuations, and its role in the occurrence of quantum spin-liquid states.ramirez  Much interest in two-dimensional quantum spin-liquid states has evolved since Anderson proposed that the doped spin-liquid state may hold the key to the puzzles of high temperature superconductivity in Cuprates.anderson87  More recently the search for spin-liquid states has been extended to geometrically frustrated quantum magnets.sondhi01 ; balents01 ; nayak01 ; capriotti

The hallmark of the spin-liquid state is the existence of fractionalized excitations. In this regard, the two-dimensional frustrated Heisenberg antiferromagnet Cs2CuCl4 provides a useful realization of a two-dimensional spin-liquid state: A recent neutron scattering experiment coldea  showed the remarkable result that the dynamical spin structure factor $`𝒮(𝐪,\omega )`$ does not exhibit well-defined peaks corresponding to spin-1 magnons. Instead, there exists a continuum of excitations which has been interpreted as the indication of pairs of deconfined spin-1/2 spinons in the underlying spin-liquid state.

The minimal Hamiltonian describing Cs2CuCl4 is argued to be the spin-1/2 Heisenberg antiferromagnet on an anisotropic triangular lattice:coldea

$`H`$ $`=`$ $`J_1{\displaystyle \underset{<ij>}{}}𝐒_i𝐒_j+J_2{\displaystyle \underset{<<ij>>}{}}𝐒_i𝐒_j,`$ (1)
where $`𝐒_i`$ is the $`S=1/2`$ spin operator on site $`i`$, and $`J_1`$, $`J_2`$ are the exchange couplings along two different types of bonds, as indicated in Fig. 1. There have been at least two different proposals for the spin-liquid state of this model in connection to the experiment on Cs2CuCl4. These proposals are based on two different mean field approaches: the bosonic Sp($`N`$) large-$`N`$ mean field theorychung ; subir  and the SU(2) slave-boson mean field theory.zhou ; wen02 ; su2  One of the distinguishing properties of the two resulting spin-liquid states is the statistics of the fractionalized excitations, i.e., spinons; the bosonic Sp($`N`$) large-$`N`$ mean field theory supports bosonic spinons while the SU(2) slave-boson mean field theory leads to fermionic spinons.

The statistics of the spinons may be a useful characteristic of the underlying spin-liquid states. In particular, an important question is whether spin-liquid ground states with the same symmetry, but different spinon statistics, are necessarily distinct; this question has been only partially answered.demler  In this paper, we would like to achieve a more moderate goal; while the above question about the ground state cannot be answered by investigating mean field theories, one may still be able to identify which mean-field state provides a more faithful approximation of the true spin-liquid state of the material. If the mean-field state is a good representation, residual quantum fluctuations will be small, and are not expected to influence the system’s responses in a qualitative way. We aim to achieve this goal by comparing the spin excitation spectra of the different proposals with the experimentally measured spin structure factor. We compute the dynamical spin structure factor for the spin-liquid states in both the bosonic Sp($`N`$) large-$`N`$ and the SU(2) slave-boson mean field theories. We find that, at temperatures well below the temperature scale set by the exchange couplings, the temperature evolution of the dynamical spin structure factor depends significantly on the spinon statistics: While the fermionic spinons are fairly insensitive to temperature, the bosonic spinon spectrum shows significant changes in this temperature range, as described below.

A useful bosonic representation of the Hamiltonian in Eq. 1 can be obtained by generalizing the physical spin SU(2) $``$ Sp(1) symmetry to Sp($`N`$). The generalized spin operators can be expressed in terms of boson operators $`b_{i\alpha }^{},b_{i\alpha }`$ on every site $`i`$, where $`\alpha =1\mathrm{}2N`$ is a Sp($`N`$) index.subir  The constraint $`b^\alpha b_\alpha =n_b`$ is imposed on each site to fix the number of bosons ($`n_b=2S`$ for $`N=1`$). In the mean-field theory, the Sp($`N`$) Hamiltonian is solved in the $`N\mathrm{}`$ limit with $`\kappa =n_b/N`$ fixed.subir  The mean-field phase diagram as a function of $`J_2/(J_1+J_2)`$ and $`1/\kappa `$ contains both magnetic long-range-order (LRO) and short-range-order (SRO) phases (see Fig. 4 of Ref. chung, ) at zero temperature. For bare parameter values relevant to the material ($`\kappa =1`$ and $`J_2/J_1=3`$), the large-$`N`$ phase diagram predicts a spin-ordered ground state with an incommensurate wavevector in the $`J_2`$-direction.chung  However, finite-$`N`$ corrections will move the phase boundaries, so that the physical spin-1/2 limit could in fact be described by the incommensurate SRO (spin-liquid) phase with deconfined spin-1/2 spinons.chung  The material Cs2CuCl4 actually exhibits long-range order at temperatures below 0.62K, which can be suppressed by an applied magnetic field. Since this ordering is due to the small interlayer coupling ($`J_z<10^2J_2`$),coldea  we expect the excitation spectrum to be indistinguishable from the disordered state at energies above this (small) scale. In our strictly two-dimensional model, no ordered state can occur at any finite temperature. Here, we consider the point $`\kappa =0.64`$ and $`J_2/J_1=3`$ in the large-$`N`$ phase diagram (see Fig. 4 of Ref. chung, ), which lies in the SRO phase close to the LRO phase boundary, as the possible spin liquid state relevant to Cs2CuCl4, which is experimentally known to be close to a quantum phase transition. The spinon dispersion in this case has a small gap at $`𝐪=(0.26\pi ,0.26\pi )`$ of approximately $`0.03J_2`$, which is much smaller than the experimental resolution of about $`0.5J_2`$.coldea

On the other hand, spin-liquid phases can also be obtained from a fermionic representation via the SU(2) slave-boson approach.zhou ; wen02 ; su2  This approach utilizes a hidden SU(2) gauge symmetry in the fermionic representation of the Heisenberg model.su2  Introducing two SU(2) doublets, $`\psi _{i1}^T=(f_i,f_i^{})`$ and $`\psi _{i2}^T=(f_i,f_i^{})`$, to rewrite the destruction operators of spin-up and down states,su2  the mean-field Hamiltonian can be expressed in terms of the $`2\times 2`$ Hermitian matrices $`u_{ij}`$ as

$$H_{\mathrm{mf}}=\underset{ij}{}(\psi _{i\alpha }^{}u_{ij}\psi _{j\alpha }+h.c.)+\underset{i}{}a_0^l\psi _{i\alpha }^{}\tau ^l\psi _{i\alpha }.$$

Here, $`\tau ^l`$ ($`l=1,2,3`$) are the Pauli matrices, and the Lagrange multiplier $`a_0^l`$ is used to enforce the constraint $`\psi _{i\alpha }^{}\tau ^l\psi _{i\alpha }=0`$.zhou ; wen02  It has been proposed that the spin-liquid phase with the following mean-field ansatz is the most likely candidate for the spin-liquid phase of Cs2CuCl4:zhou

$`u_{i,i+\widehat{x}}`$ $`=`$ $`\chi \tau ^2,u_{i,i+\widehat{y}}=\chi \tau ^2,u_{i,i+\widehat{x}+\widehat{y}}=\lambda \tau ^3,`$
$`a_0^{1,2}`$ $`=`$ $`0,a_0^3=a_3.`$ (2)
Here the parameters $`\chi ,\lambda `$ and $`a_3`$ have to be determined self-consistently. According to the classification scheme of Ref. zhou, , this is one of the possible U(1) spin-liquid phases on the anisotropic triangular lattice. The excitation spectrum has gapless pointszhou  at $`(0,0)`$, $`(\pi ,\pi )`$, and $`(q,q)`$, where $`q=\pm \pi /2\pm ϵ`$, with $`ϵ=0.04\pi `$. Although the mean-field solution is expected to be modified by gauge field fluctuations, it may still provide a qualitatively correct description of the spin excitation spectrum.

The spin-liquid states obtained from the two approaches differ in important aspects: one of them is a $`Z_2`$ spin-liquid phase with gapped bosonic spinons (although the gap is small), and the other is a U(1) spin-liquid state with gapless fermionic spinons. It is therefore interesting to ask which spin liquid state is more likely to describe Cs2CuCl4. We therefore calculate in both cases the dynamical spin structure factor defined as $`𝒮(𝐪,\omega )=\frac{1}{\pi }(1+n(\omega ))\chi ^{\prime \prime }(𝐪,\omega ),`$ where $`n(\omega )=1/(e^{\omega /T}1)`$ is the Bose thermal factor, and $`\chi ^{\prime \prime }(𝐪,\omega )`$ is the imaginary part of the dynamical spin susceptibility:

$$\chi ^{\prime \prime }(𝐪,\omega )=3\mathrm{I}\mathrm{m}\left(_{\mathrm{}}^{\mathrm{}}𝑑te^{i\omega t}\theta (t)[S^z(𝐪,t),S^z(𝐪,0)]\right),$$
(3)
where spin rotation invariance has been used to write $`\chi =3\chi _{zz}`$.

In the Sp($`N`$) approach $`\chi ^{\prime \prime }(𝐪,\omega )`$ is obtained as

$`\chi _{\mathrm{Sp}(N)}^{\prime \prime }=3{\displaystyle \underset{k}{}}[({\displaystyle \frac{1}{4}}+A(𝐪))(n(E_𝐤)n(E_𝐤^{}))\delta (\omega +E_𝐤E_𝐤^{})`$
$`+({\displaystyle \frac{1}{4}}+A(𝐪))(n(E_𝐤^{})n(E_𝐤))\delta (\omega +E_𝐤^{}E_𝐤)`$
$`+({\displaystyle \frac{1}{4}}A(𝐪))(1+n(E_𝐤)+n(E_𝐤^{}))\delta (\omega +E_𝐤+E_𝐤^{})`$
$`({\displaystyle \frac{1}{4}}A(𝐪))(1+n(E_𝐤)+n(E_𝐤^{}))\delta (\omega E_𝐤E_𝐤^{})]],`$
where $`A(𝐪)=\frac{\overline{\lambda }^2\mathrm{\Delta }_𝐤\mathrm{\Delta }_𝐤^{}}{4E_𝐤E_𝐤^{}}`$, $`E_𝐤^{}=E_{𝐤+𝐪}`$, and $`\mathrm{\Delta }_𝐤^{}=\mathrm{\Delta }_{𝐤+𝐪}`$. Here $`n(E_𝐤)`$ is the Bose thermal factor, $`E_𝐤=\sqrt{\overline{\lambda }^2\mathrm{\Delta }_𝐤^2}`$ is the bosonic spinon dispersionchung , and $`\mathrm{\Delta }_𝐤=J_1Q_1(\mathrm{sin}k_x+\mathrm{sin}k_y)+J_2Q_2\mathrm{sin}(k_x+k_y)`$. $`Q_1`$ and $`Q_2`$ are mean-field bond variables for nearest-neighbor and next-nearest-neighbor bonds, respectively, and $`\overline{\lambda }`$ is a Lagrange multiplier enforcing the constraint on the number of bosons. The temperature dependent values of $`Q_1`$, $`Q_2`$, and $`\overline{\lambda }`$ are determined self-consistently.

In the SU(2) slave-boson approach, $`\chi ^{\prime \prime }`$ is given by

$`\chi _{SU(2)}^{\prime \prime }=3{\displaystyle \underset{k}{}}[({\displaystyle \frac{1}{4}}+B(𝐪))(f(_𝐤)f(_𝐤^{}))\delta (\omega +_𝐤_𝐤^{})`$
$`+({\displaystyle \frac{1}{4}}+B(𝐪))(f(_𝐤^{})f(_𝐤))\delta (\omega +_𝐤^{}_𝐤)`$
$`({\displaystyle \frac{1}{4}}B(𝐪))(1f(_𝐤)f(_𝐤^{}))\delta (\omega +_𝐤+_𝐤^{})`$
$`+({\displaystyle \frac{1}{4}}B(𝐪))(1f(_𝐤)f(_𝐤^{}))\delta (\omega _𝐤_𝐤^{})],`$
where $`B(𝐪)=\frac{ϵ_𝐤ϵ_𝐤^{}+𝒟_𝐤𝒟_𝐤^{}}{_𝐤_𝐤^{}}`$, $`_𝐤^{}=_{𝐤+𝐪}`$, and $`𝒟_𝐤^{}=𝒟_{𝐤+𝐪}`$. Here, $`_𝐤=2\sqrt{ϵ_𝐤^2+𝒟_𝐤^2}`$ is the dispersion of the fermionic spinons,zhou  $`ϵ_𝐤=\lambda \mathrm{cos}(k_x+k_y)a_3`$, $`𝒟_𝐤=\chi (\mathrm{cos}k_x\mathrm{cos}k_y)`$, and $`f(_𝐤)`$ is the Fermi distribution function. We assume that the values of the order parameter fields $`\chi `$ and $`\lambda `$, which were obtained in Ref. zhou, , do not vary significantly in the temperature range considered here, and we explicitly verified that our results are insensitive to any changes in $`a_3`$ with temperature.

Our results for $`\chi ^{\prime \prime }(𝐪,\omega )`$ for the two spin-liquid states are shown in Figs. 2-4. The spin structure factor $`𝒮(𝐪,\omega )`$ differs from $`\chi ^{\prime \prime }(𝐪,\omega )`$ by an overall thermal factor $`(1+n(\omega ))`$. As a result the low energy spectral weight in $`\chi ^{\prime \prime }(𝐪,\omega )`$ is somewhat suppressed compared to $`𝒮(𝐪,\omega )`$, but the overall intensity distribution looks very similar. To make direct contact with experiment, we show $`\chi ^{\prime \prime }(𝐪,\omega )`$ for both spin liquid states in Fig. 2, with $`𝐪`$ oriented along the direction of $`J_2`$ (see Fig. 1). Figures 3 and 4 show the same quantity along four scan trajectories that were used in the experiment (see Figs. 2 and 3 of Ref. coldea, ). Note that the point $`k=2\pi /b`$ in Fig. 2a of Ref. coldea,  corresponds to $`q=\pi `$ in Fig. 2 here.

Both spin-liquid states show a continuum of spin excitations, which is a hallmark of spin-liquids with spin-1/2 spinons. Moreover, both theories indicate strong scattering around $`(\pi /2,\pi /2)`$ in agreement with the experiment. At zero temperature, the lower edge of the excitation spectrum in both theories has minima at $`(0,0)`$, $`(\pi ,\pi )`$ and close to $`(\pi /2,\pi /2)`$. The spinon spectrum in the Sp($`N`$) approach has a small gap ($`0.03J_2`$) at the single incommensurate minimum while it has two gapless incommensurate points in the SU(2) theory. These slight differences, however, are below the experimental resolution (about $`0.5J_2`$).coldea  The upper boundary of the scattering spectrum in the SU(2) approach is closer to the experimental results. On the other hand, the Sp($`N`$) approach describes some aspects of the experimentally measured spectra at lower energy (see Fig. 2a of Ref. coldea, ) better: there is much less scattering intensity around $`(\pi ,\pi )`$ in the Sp($`N`$) approach compared to the results of the SU(2) theory, and the peak intensity is close to the lower edge as opposed to the upper edge in the case of the SU(2) approach. The details of the spectra described above are expected to change, however, once fluctuations about the mean-field solutions are included. In addition, it is difficult to distinguish the two spin-liquid states from their low energy excitation spectra under current experimental resolution. Therefore, it is fair to say that, at zero temperature, both theories agree reasonably well with experiment on a qualitative level.

At finite temperatures, however, one expects that the different spinon statistics will give rise to substantial differences in the spin excitation spectrum. Indeed, as shown in Figs. 2-4, our results at finite temperatures indicate that the scattering intensity extends to a broader range in the phase space of $`𝐪,\omega `$, and $`\chi ^{\prime \prime }(𝐪,\omega )`$ shows very different behavior with increasing temperature in the two approaches. This difference can be most clearly seen by comparing the temperature evolution of the intensities along the four scan directions used in the experiment (see Fig. 2 of Ref. coldea, ). As shown in Fig. 3, the maximum intensity increases in the Sp(N) approach, and the excitation spectrum becomes more narrowly peaked around the maxima as temperature increases, while the overall intensity remains roughly constant. Additionally, a two-peak structure develops, which resembles the lineshape predicted by a spin-wave calculation in the LRO phase (see Fig. 3 of Ref. coldea, ), and is presumably due to the vicinity of an ordered quantum ground state. On the other hand, as shown in Fig. 4, the evolution of the spin structure factor with temperature is much less significant in the SU(2) theory, where only a very slight shift of spectral weight to lower energies is observed. Since the different temperature dependence arises from distinct statistics of the spinons, these results will not be affected fundamentally by fluctuations about the mean-field ground states.

In summary, we studied the dynamic spin structure factor of different spin-liquid states obtained from the bosonic large-$`N`$ Sp($`N`$) and the fermionic SU(2) mean-field theories of a two-dimensional frustrated spin-1/2 Heisenberg antiferromagnet, as a model for Cs2CuCl4. We found that at zero temperature the dynamic spin structure factor of both spin-liquid states compares favorably with experiment. At finite temperatures, fundamentally different behavior arises due to the different spinon statistics. The signatures we found in the temperature dependence of the spin structure factor can be used to compare both theories with future experiments, in order to determine which theory is best suited to describe the spin liquid state of Cs2CuCl4. More details will be presented in a future publication.

We would like to thank J. B. Marston and R. Coldea for stimulating discussions. This research was supported by the NSERC of Canada, the Sloan Foundation, and the Canadian Institute for Advanced Research.