\section{Introduction}
\label{002.intro}

\subsection{Outline}
\label{002.outline}

Motivated by Penrose's Weyl curvature conjecture (see section \ref{002.WCC}), various candidates for an entropy density for gravity constructed from the Weyl tensor were proposed \cite{Weyl2008,PhysRevD.62.044009,Barrow_2002,AGEP}. The simplest of these proposals was investigated in 5-dimensions by integrating the entropy density over a spacelike hypersurface and comparing it with the standard Bekenstein-Hawking entropy of black holes \cite{Li2016, Zhao:2018aa}. The following integral was computed

\begin{equation}
\int_\Sigma C_{\mu\nu\lambda \rho}C^{\mu\nu\lambda \rho} dV_4 \approx 48\pi^2\left(\frac{R_5}{l_p}\right)^3,
\end{equation}
where $C_{\mu\nu\lambda \rho}$ is the Weyl tensor, $R_5$ and $l_p$ is the Schwarzschild radius and Planck length respectively. This integral should be interpreted as integrating an entropy density over a spacelike hypersurface $\Sigma$. The integral up to a multiplicative constant agrees with the Bekenstein-Hawking entropy of black holes.

If proposals for entropy densities are valid, then we could in principle use them to evaluate the entropy of spacetime geometries other than just over the Kerr-Newman metric. This could give us an insight into what gravitational entropy is and may lead to a better understanding of various issues like the information paradox associated with black hole evaporation \cite{Hawking1975}.

In this paper, we extend the work done in ref\cite{Li2016} and find that when the analysis is replicated for 5-dimensional Gauss-Bonnet gravity, there are issues with interpreting $C_{\mu\nu\lambda \rho}C^{\mu\nu\lambda \rho}$ as an entropy density. This may indicate that a different candidate for an entropy density is required for Gauss-Bonnet gravity.

\subsection{Entropy and the Generalized Second Law}
\label{002.EGSL}
The second law of thermodynamics states that in an isolated system the entropy, $S$, never decreases:
\begin{equation}\label{002.S>0}
\frac{dS}{dt}\geq 0.
\end{equation}
 Originally, entropy was considered as a new extensive thermodynamic property obeying Eq.(\ref{002.S>0}). But later, Ludwig Boltzmann showed that entropy could be understood as the logarithm of the number of different microstates, $W$, compatible with a given macrostate,
 \begin{equation}
S=ln(W).
\end{equation} \footnote{We have set the Boltzmann constant ($k_B=1$).} Boltzmann's approach informs our modern understanding of entropy and of the second law as fundamentally probabilistic.

 In order for the second law of thermodynamics to be valid in the presence of black holes, Jacob Bekenstein proposed that black holes must possess nonzero entropy \cite{PhysRevD.7.2333}. Stephen Hawking later confirmed Bekenstein's idea and showed that black holes must have a temperature and radiate as black bodies \cite{Hawking1975}. The Bekenstein-Hawking entropy of black holes in general relativity is
\begin{equation}\label{002.EqW:BH}
S_{BH}=\frac{A}{4G\hbar}.
\end{equation}
The generalized entropy \cite{PhysRevD.9.3292}
\begin{equation}
S_{gen}=S_{BH}+S_{out},
\end{equation}
where $S_{out}$ is the entropy outside the black hole can be considered as the total entropy of a system when both black holes and matter are present. There is mounting evidence to suggest that classically and semi-classically, $S_{gen}$ is never decreasing \cite{PhysRevD.85.104049,PhysRevD.82.124019}. This is known as the generalized second law (GSL) and supersedes the usual second law of thermodynamics.

Many expect that fundamentally, the origin of $S_{gen}$ and GSL is statistical in nature, as with the usual notion of entropy. But perhaps its origin lies elsewhere and our notion of entropy needs modification. Any insights we can gain about the entropy of black holes and other spacetime geometries may provide hints towards the nature of quantum gravity.

\subsection{Weyl curvature conjecture}
\label{002.WCC}

Understanding black hole thermodynamics and the realization that they have entropy has been important in understanding the status of the second law of thermodynamics and the need to generalize it (GSL). However, no universally agreed generalization for the entropy has been found for other spacetimes. Addressing this issue could help explain the apparent violation of the second law of thermodynamics on cosmological scales \cite{Page:1983aa,Penrose:aa1}. It seems that the universe today has evolved from a remarkably homogeneous and isotropic initial state to a non-equilibrium state which includes complex structures like stars and galaxies. Small perturbations in the initial state grew as a result of gravity and caused structures to form. For gravitational interactions to be made compatible with the second law of thermodynamics, Penrose suggested that gravitational fields should have an associated entropy \cite{Penrose:aa1}. Such a concept if correct could lead to a further generalization of the GSL and explain the homogeneous and isotropic nature of the universe.

The Weyl tensor $C_{\mu\nu\lambda \rho}$ is the trace-free part of the Riemann tensor and in an $n$-dimensional spacetime, the Riemann tensor can be decomposed into its trace-free and trace part:
\begin{eqnarray}
R_{\mu\nu\lambda \rho}=\frac{1}{n-2}(g_{\mu\lambda}R_{\nu\rho} +g_{\nu\rho}R_{\mu\lambda} -g_{\nu\lambda}R_{\mu\rho} -g_{\mu\rho}R_{\nu\lambda}) \nonumber \\
+\frac{1}{(n-1)(n-2)}(g_{\mu\rho}g_{\nu\lambda} -g_{\mu\lambda}g_{\nu\rho})R+C_{\mu\nu\lambda \rho},
\end{eqnarray}
where $R_{\mu\nu\lambda \rho}$ is the Riemannn tensor, $R_{\mu\nu}$ is the Ricci tensor, $R$ is the Ricci scalar and $g_{\mu\nu}$ is the metric.

The Einstein field equation relates the curvature of spacetime to the matter content of spacetime, or more precisely
\begin{equation}\label{002.EE}
R_{\mu\nu}-\frac{1}{2}Rg_{\mu\nu}=8\pi GT_{\mu\nu},
\end{equation}
where $G$ is the gravitational constant and we have set the speed of light $c=1$. The left hand side of this equation only involves the Ricci tensor and Ricci scalar and hence locally, the Weyl tensor is independent of the energy momentum tensor. It represents gravitational effects such as tidal effects and gravitational waves. Thus it seems plausible that information encoded in the Weyl tensor is related to the thermodynamics of gravity.

 The Weyl tensor also vanishes if and only if the spacetime is conformally flat in spacetimes with dimension $n>3$. Hence initially, in 4 or higher dimensions, when the universe was homogenous and isotropic, the Weyl tensor vanished and grew afterwards as complex structures formed. This motivated Penrose to conjecture (The Weyl curvature conjecture) that the low entropy content of the universe is related to the vanishing of the Weyl tensor \cite{Penrose:aa1, Goode_1985, 1985CQGra...2...99G, 1989NYASA.571..249P}.

To realize Penrose's conjecture, candidates for an entropy density for gravity has been proposed using the Weyl tensor \cite{Weyl2008,PhysRevD.62.044009,Barrow_2002,AGEP}. For example the simplest such entropy density below uses a scalar constructed from the Weyl tensor;
\begin{equation}\label{002.P}
P^2=C^{\mu\nu\lambda\rho}C_{\mu\nu\lambda\rho}.
\end{equation}
In $4$-dimensional spacetime, this can not be interpreted as an entropy density but can be in $5$ dimensions, at least from a dimensional analysis point of view\footnote{We set the Boltzmann constant to 1 so that entropy densities have units of $[volume]^{-1}$}. This proposal in $5$ dimensions was investigated by the authors of ref\cite{Li2016} to see if it reproduced the standard Bekenstein-Hawking entropy of black holes. They showed that on a Schwarzschild metric in $5$-dimensional spacetime,
\begin{eqnarray}\label{002.W}
\int_\Sigma C_{\mu\nu\lambda \rho}C^{\mu\nu\lambda \rho} dV_4 &=  48\pi^2\left[\left(\left(\frac{R_5}{l_p}\right)^3+\frac{2R_5}{l_p}\right)\sqrt{1-\left(\frac{l_p}{R_5}\right)^2} +2\right]\nonumber \\
& \approx48\pi^2\left(\frac{R_5}{l_p}\right)^3
\end{eqnarray}
to highest order\footnote{Lower orders in $\frac{R_5}{l_p}$ were dropped since corrections to Bekenstein-Hawking entropy to these orders may require a full description of quantum gravity in the regime when $R_5 \approx l_p$. When $R_5>>l_p$, we expect the lower order terms to be suppressed. In other words, we restrict our analysis to large black holes.} in $\frac{R_5}{l_p}$, where $R_5>>l_p$ is the Schwarzchild radius and Planck length respectively, and $\Sigma$ is the spacelike hyper-surface orthogonal to the timelike Killing vector. Hence in $5$ dimensions this integral is dimensionless and up to a multiplicative constant agrees with the Bekenstein-Hawking entropy of black holes (Eq.(\ref{002.EqW:BH})).

Any proposal for an entropy density must be able to recover the standard entropy of black holes. In this paper, we also investigate Eq.(\ref{002.P}) using methods similar to those of ref\cite{Li2016}, but in 5-dimensional Gauss-Bonnet gravity.

\section{Gauss-Bonnet gravity}
\label{002.GBG}

Lovelock theories of gravity are a class of generalizations of general relativity in $n$-dimensional spacetime which gives second order equations of motion for the metric \cite{doi:10.1063/1.1665613}. In $4$ dimensions, the Einstein field equations (Eq.(\ref{002.EE})) with a cosmological constant are the unique class of Lovelock theories.

In $5$ dimensions, Gauss-Bonnet gravity is the most general Lovelock theory. Setting $c=1$, the action for Gauss-Bonnet gravity is given by \cite{boulware:1985aa,Clunan:2004aa}
\begin{equation}
\frac{1}{16\pi G} \int d^5x\sqrt{-g}(R+\alpha (R^2-4R^{\mu\nu}R_{\mu\nu}+R^{\mu\nu \lambda \rho}R_{\mu\nu \lambda \rho})),
\end{equation}
where $\alpha$ is an arbitrary constant with dimensions of $[length]^2$. The $(R^2-4R^{\mu\nu}R_{\mu\nu}+R^{\mu\nu \lambda \rho}R_{\mu\nu \lambda \rho})$ term in $4$ dimensions is a topological term which can be removed from the action without affecting the equation of motion. But in $5$ dimensions, this term in general can not be removed. These higher-curvature terms may arise from quantum gravity corrections so we will  assume $\frac{\alpha}{l_p^2}\approx 1$.

We work with the following family of static vacuum (black hole) solutions for $5$-dimensional Gauss-Bonnet gravity \cite{boulware:1985aa,Clunan:2004aa}

\begin{equation}\label{002.Eq:g}
ds^2=-F(r)^2dt^2+\frac{1}{F(r)^2}dr^2+r^2d\Omega^2,
\end{equation}
where $d\Omega^2$ is the metric on the $S^3$ sphere and
	\begin{equation}\label{002.Eq:g2}
F^2(r)=k+\frac{r^2}{4\alpha}\left( 1-\sqrt{1+\frac{128\pi G\alpha M}{3 \Sigma_k r^{4}}+\frac{4\alpha\Lambda}{3}}  \right)
	\end{equation} with $\Sigma_k$ being the volume of the 3-sphere and $\Lambda$ the cosmological constant. $k$ can be either $-1$, $0$ or $1$ although $k=-1 $ and $k=0$ are only possible when $\Lambda<0$. $M$ parametrizes the family of solutions and is also the mass of the black hole. For simplicity, we now set $k=1$ and $\Lambda=0$ unless explicitly stated. Our calculations are independent of this choice. See \ref{002.AB} for more details.

To first order in $\alpha$, $F^2(r)=f^2(r)+\alpha\frac{M_1^2}{8r^6}$ with $f^2(r)=\left(1-\frac{M_1}{4r^2}\right)$ and the Weyl invariant to first order in $\alpha$ is
	\begin{equation}\label{002.P2}
C_{\mu\nu\lambda \rho}C^{\mu\nu\lambda \rho}=\frac{9M_1^2}{2r^8}-\alpha \frac{21M_1^3}{r^{12}} +O(\alpha^2),
\end{equation}
where $M_1=\frac{64\pi GM}{3\Sigma_k}$.

	\section{Black hole entropy in Gauss-Bonnet Gravity}
\label{002.BHEGBG}

When the Einstein Hilbert action is modified, the expression for the black hole entropy also needs to be modified. For a stationary black hole's entropy in 5-dimensional Gauss-Bonnet gravity, this is given by \cite{Clunan:2004aa,Hawking1983}
\begin{equation}\label{002.GB}
S_{BH}=\frac{R_5^3\Sigma_k}{4l_p^3}\left(1+\frac{12\alpha k}{R_5^2}\right)=\frac{A}{4l_p^3}\left(1+\frac{12\alpha k}{R_5^2}\right),
\end{equation}
\footnote{We have explicitly reintroduced the parameter $k$ and $\Lambda$ for this expression. Also note that the black hole entropy is independant of $\Lambda$.}  where $A=R_5^3\Sigma_k$ is the surface area of the horizon at $r=R_5$ with $R_5$ being the largest root of $F^2(R_5)=0$. The expression for black hole entropy changes because the behavior of the black hole horizon changes when the Einstein field equation is modified. This can be seen by considering the first law of black hole mechanics \cite{Clunan:2004aa} or via Wald's Noether charge method \cite{Wald:1993aa}.

In this paper, we will work in 5 dimensions and integrate the expression $\int C_{\mu\nu\lambda \rho}C^{\mu\nu\lambda \rho} dV_4$ on a spacelike hypersurface in analogy to Eq.(\ref{002.W}) to see if we recover the entropy of a black hole in Gauss-Bonnet gravity (Eq.(\ref{002.GB})). This test in Gauss-Bonnet gravity is more stringent than in general relativity as more terms need to match. By dimensional analysis, the integral $\int C_{\mu\nu\lambda \rho}C^{\mu\nu\lambda \rho} dV_4$ to zeroth order in $\alpha$ can be written as

\begin{equation}\label{002.n1}
	a\left(\frac{R_5}{l_p}\right)^{n_1}+\dots
\end{equation}

where $a$ and $n_1$ are constants to be determined. We only focus on the highest order term in $\frac{R_5}{l_p}$ with lower order terms contained in `\dots' which we regard as small for large black holes. We can use Eq.(\ref{002.W}) to see that $n_1=3$ and thus this reproduces the expression for black hole entropy (Eq.(\ref{002.GB})) up to a multiplicative constant (at zeroth order in $\alpha$). This result will be reproduced in the next section.

Similarly, to first order in $\alpha$, the integral $\int C_{\mu\nu\lambda \rho}C^{\mu\nu\lambda \rho} dV_4$ can be written as

\begin{equation}\label{002.n2}
	b\frac{\alpha}{R_5^2}\left(\frac{R_5}{l_p}\right)^{n_2}+\dots
\end{equation}
where $b$ and $n_2$ are constants to be determined with lower order terms in $\frac{R_5}{l_p}$ ignored. For Eq.(\ref{002.P}) to pass as an entropy density in Gauss-Bonnet gravity, we must have $n_1=n_2=3$\footnote{Also, $b=12a$}. However, we show below that $n_2=7$.

\section{Dimensional Analysis}\label{002.DA}

Fortunately, we can determine $n_1$ and $n_2$ directly through dimensional analysis. However, we provide a more detailed calculation in \ref{002.AA}.

We can write $F^2(r)$ from Eq.(\ref{002.Eq:g2}) as

\begin{equation}
F^2(r)=1-\frac{A}{r^2}+\frac{\alpha B}{r^6} + O(\alpha^2),
\end{equation}
 and Eq.(\ref{002.P}) as
\begin{equation}
P^2=C^{\mu\nu\lambda\rho}C_{\mu\nu\lambda\rho}=\frac{D}{r^8}-\frac{\alpha E}{r^{12}}+O(\alpha^2),
\end{equation}
where $A$, $B$, $D$ and $E$ are positive constants. Therefore, the integral $\int C_{\mu\nu\lambda \rho}C^{\mu\nu\lambda \rho} dV_4$ up to first order in $\alpha$ becomes

\begin{eqnarray}
	 \int C_{\mu\nu\lambda \rho}C^{\mu\nu\lambda \rho} dV_4 &=2\pi^2\int\left(\frac{D}{r^8}-\frac{\alpha E}{r^{12}}\right)r^3g_{rr}dr \nonumber \\
	& =2\pi^2\int\left(\frac{D}{r^8}-\frac{\alpha E}{r^{12}}\right)\frac{r^3}{|F(r)|}dr.
\end{eqnarray}

We find that the essential contribution to the above integral comes from the small radius (i.e $r \rightarrow 0$) (see \ref{002.AA}) . In this case, we have that

\begin{equation}
	\frac{r^3}{|F(r)|}=\frac{r^3}{|1-\frac{A}{r^2}+\frac{\alpha B}{r^6}|^{\frac{1}{2}}}\approx \frac{r^3}{(\frac{A}{r^2}-\frac{\alpha B}{r^6})^{\frac{1}{2}}}\approx Fr^4+\alpha G
\end{equation}
where $F$ and $G$ are positive constants. Hence

\begin{eqnarray}\label{002.Int}
	 \int C_{\mu\nu\lambda \rho}C^{\mu\nu\lambda \rho} dV_4 &\propto \int_{r\approx 0}\left(\frac{D}{r^8}-\frac{\alpha E}{r^{12}}\right)(Fr^4+\alpha G)dr \nonumber \\
	 & \approx \int_{r\approx 0} \left(\frac{H}{r^4}+\frac{\alpha I}{r^8} \right)dr
\end{eqnarray}
where $H$ and $I$ are constants, with $H$ positive.

The classical theory of general relativity and generalizations like Gauss-Bonnet gravity are expected to break down at extremely small radii where we expect a quantum theory of gravity to have large effects. In particular, these theories are invalid at the spacetime singularity ($r=0$) so we bound the integral below by Planck's constant $l_p$ in Eq.(\ref{002.Int}). In this case, we find that the essential contribution\footnote{The essential contribution are the highest order non-zero terms in $\frac{R_5}{l_p}$} to the integral comes from the $r\approx l_p$ piece which gives
\begin{eqnarray}\label{002.EDBH}
	 \int C_{\mu\nu\lambda \rho}C^{\mu\nu\lambda \rho} dV_4 \propto J\left(\frac{R_5}{l_p}\right)^3+K \frac{\alpha}{R_5^2} \left(\frac{R_5}{l_p}\right)^7
\end{eqnarray}
where $J$ and $K$ are constants. We plugged back the black hole radius $R_5$ in this expression to emphasize that the integral is dimensionless. This shows that $n_1=3$ and $n_2=7$. Since $n_2\neq3$, we find that the integral above does not reduce to the expected expression for black hole entropy in Gauss-Bonnet gravity (Eq.(\ref{002.GB})) at order $\alpha$. Thus Eq.(\ref{002.P}) is not a valid entropy density for Gauss-Bonnet gravity.

	\section{Discussion}
\label{002.DIS}

The second law of thermodynamics is one of the most important in physics because of its wide range of applicability. For this to be valid when a system includes a black hole, one has to assign an entropy to the black hole, $S_{BH}$. If the notion of black hole entropy can be extended to other spacetimes and cosmological settings, then the apparent violation of the second law of thermodynamics on cosmological scales maybe resolved \cite{Page:1983aa,Penrose:aa1}. Penrose conjectured that the initially  low  entropy of the universe is related to the vanishing of the Weyl tensor. Motivated by this, various proposals for an entropy density constructed from the Weyl tensor have been suggested to realize this conjecture \cite{Weyl2008,PhysRevD.62.044009,Barrow_2002,AGEP}. In $5$ dimensions, the simplest scalar constructed from the Weyl tensor was proposed as an entropy density (see Eq.(\ref{002.P})) and investigated in ref\cite{Li2016}. They found that this entropy density recovers the Bekenstein-Hawking entropy of black holes.

When we replicated this investigation in Gauss-Bonnet gravity, a generalization of general relativity, the proposed entropy density failed to recover the standard black hole entropy at order $O(\alpha)$. By dimensional analysis, we see that this is due to the term $(-21\alpha\frac{M1^3}{r^{12}})$ in Eq.(\ref{002.P2}) which ultimately leads to $n_2=7\neq3$ in Eq.(\ref{002.n2}). For the entropy density to reproduce the black hole entropy, we require $n_1=n_2=3$. In other words, we require the entropy density $s$ to take the following form;

\begin{equation}\label{002.sd}
	s=\frac{L}{r^8}+\frac{N\alpha}{R_5^2r^8}+O(\alpha^2).
\end{equation}
where $L$ and $N$ are some constants. Other candidates for an entropy density multiply Eq.(\ref{002.P}) by scalars constructed from other curvature terms \cite{Weyl2008, PhysRevD.62.044009}. An example of such a candidate is
\begin{equation}
	P'^2=C^{\mu\nu\lambda\rho}C_{\mu\nu\lambda\rho}/R_{\delta}^{\gamma}R_{\gamma}^{\delta}.
\end{equation}
By dimensional analysis, none of these candidates can change the form of the entropy density from Eq.(\ref{002.P2}) to Eq.(\ref{002.sd}).

In Gauss-Bonnet gravity, the Einstein field equations are modified so that the Weyl tensor couples to the energy momentum tensor and is not necessarily locally independent of the matter content in spacetime. This and our investigation above suggests that the Weyl tensor may need to be replaced in alternative theories of gravity with some other tensor constructed from the Riemann tensor which is locally independent of the matter content.

In vacuum with the cosmological constant set to zero ($\Lambda=0$), only the gravitational degrees of freedom are involved, so we expect the Riemann tensor to represent only these degrees of freedom. Additionally, in general relativity, vacuum solutions with $\Lambda=0$ lead to $R_{\mu\nu}=0$ and so $C_{\mu\nu\lambda\rho}=R_{\mu\nu\lambda\rho}$. Thus, in vacuum with $\Lambda=0$,
\begin{equation}
	P^2=C^{\mu\nu\lambda\rho}C_{\mu\nu\lambda\rho}=R^{\mu\nu\lambda\rho}R_{\mu\nu\lambda\rho}.
\end{equation}

In Gauss-Bonnet gravity, even if the Weyl tensor needs to be replaced by curvature terms locally independent of the matter content, we still expect the Riemann tensor to represent only the gravitational degrees of freedom in a vacuum with $\Lambda=0$. Thus one may expect that in vacuum with $\Lambda=0$,
\begin{equation}
P^2=\tilde{C}^{\mu\nu\lambda\rho}\tilde{C}_{\mu\nu\lambda\rho}=R^{\mu\nu\lambda\rho}R_{\mu\nu\lambda\rho}
\end{equation}
where $\tilde{C}^{\mu\nu\lambda\rho}$ is the conjectured curvature term locally independent of the matter content replacing the Weyl tensor. However, for the black hole solutions in Gauss-Bonnet gravity with $\Lambda=0$, we find that $\int R_{\mu\nu\lambda \rho}R^{\mu\nu\lambda \rho} dV_4=\int C_{\mu\nu\lambda \rho}C^{\mu\nu\lambda \rho} dV_4$ up to first order in $\alpha$ so our results don't change. Thus, the issues resulting from dimensional analysis would still need to be tackled head on.

Another issue with Eq.(\ref{002.P2}) is that it leads to an expression for the black hole entropy, which is independent of the $k$ parameter (see \ref{002.AB}). This is incompatible with the standard black hole entropy, Eq.(\ref{002.GB}), as it depends on $k$.

We expect that some of these issues could be resolved if the entropy density is a sums of scalars built from curvature terms locally independent of the matter content. For example, using a Bel-Robinson-like tensor for Gauss-Bonnet to find a ``super-energy-momentum tensor" \cite{AGEP} may help in building the curvature terms that would replace its Weyl tensor counterpart.

Additionally any curvature term locally independent of the matter content that is to replace the Weyl tensor in Gauss-Bonnet gravity would need to be checked that it satisfies similar properties to those discussed in section (\ref{002.WCC}), which motivated the Penrose Weyl curvature conjecture in the first place. We leave this to future work.

Finally, we note that the expression for the entropy density in Eq.(\ref{002.P2}) includes a negative contribution proportional to $\alpha$ which dominates for $l_p<r\lessapprox(\alpha^{\frac{1}{2}}R_5)^{\frac{1}{2}}\approx (l_pR_5)^{\frac{1}{2}}$ where the upper bound can be large for large black holes. This further shows that the entropy density needs modification in Gauss-Bonnet gravity since a negative entropy density is not physically meaningful. We hope that future work on finding a suitable replacement to Eq.(\ref{002.P2}) will resolve this issue.

One may worry that this analysis was artificially restricted to 5-dimensions. However, using similar methods, one could test other proposals in other dimensions, provided the constructed entropy density via the Weyl tensor has the correct physical dimension. For instant, one could investigate $\tilde{P}^{(N)}:=(C_{abcd}C^{abcd})^{\frac{N-1}{4}}=P^{\frac{N-1}{2}}$ instead of Eq.(\ref{002.P}) as an entropy density in $N$-dimensions. To test Gauss-Bonnet gravity, a 5-or higher-dimensional spacetime is required for a non-trivial test because in lower dimensions Gauss-Bonnet gravity reduces to general relativity. In higher dimensions, non-trivial tests could also be conducted in other Lovelock theories of gravity, which are further generalizations of general relativity and Gauss-Bonnet gravity. We expect similar issues to arise in these cases as well.

\section*{ACKNOWLEDGMENTS}

We would like to thank our referees for constructive feedback and helpful comments. In particular, one of our referee suggested to highlight some calculations in section \ref{002.DA} that can be done simply by dimensional analysis.


\section{Introduction}
\label{sec:intro}

\begin{figure}
    \centering
    \begin{tabular}{c c c}
    \begin{minipage}[c]{0.3\linewidth}
    \includegraphics[width=\linewidth, height=1.4\linewidth]{res/img/example/2.jpg}
    \end{minipage} &
    \begin{minipage}[c]{0.3\linewidth}
    \includegraphics[width=\linewidth, height=1.4\linewidth]{res/img/example/3.jpg}
    \end{minipage} &
    \begin{minipage}[c]{0.3\linewidth}
    \includegraphics[width=\linewidth, height=1.4\linewidth]{res/img/example/5.jpg}
    \end{minipage} \\
    (a) & (b) & (c) \\
    \multicolumn{3}{c}{\includegraphics[width=\linewidth]{res/img/imbalance.pdf}} \\
    \multicolumn{3}{c}{(d)}\\
    \end{tabular}
    \caption{(a): $\textless$vase-sitting on-table$\textgreater$; (b): $\textless$man-sitting on-chair$\textgreater$; (c): $\textless$dog-sitting on-chair$\textgreater$. (a)(b)(c) have completely different visual appearances but are considered as the same relation class. (d): The long-tailed distribution of independent relation classes}
    \label{fig:motivation}
\end{figure}

\begin{figure*}
    \centering
    \begin{tabular}{c c c}
    \begin{minipage}[c]{0.36\linewidth}
    \includegraphics[width=\linewidth]{res/img/intro_2.pdf}
    \end{minipage} &
    \begin{minipage}[c]{0.23\linewidth}
    \includegraphics[width=\linewidth]{res/img/intro_0.pdf}
    \end{minipage} &
    \begin{minipage}[c]{0.36\linewidth}
    \includegraphics[width=\linewidth]{res/img/intro_1.pdf}
    \end{minipage} \\
    (a) & (b) & (c) \\
    \end{tabular}
    \caption{(a): The global knowledge graph of VG; (b): Unstructured output space in which the relation classifier is shared among all subject-object pairs; (c): Structured output space of HOSE-Net in which the relation classifier is context-specific.}
    \label{fig:intro}
\end{figure*}
In recent years, visual recognition tasks for scene understanding has gained remarkable progress, particularly in object detection and instance segmentation. While accurate identification of objects is a critical part of visual recognition, higher-level scene understanding requires higher-level information of objects.
Scene graph generation aims to provide more comprehensive visual clues than individual object detectors by understanding object interactions. Such scene graphs serve as structural representations of images by describing objects as nodes (``subjects/objects") and their interactions as edges (``relation"), which benefit many high-level vision tasks such as image caption\cite{li2019know,yang2019auto,guo2019aligning}, visual question answering\cite{teney2017graph,peng2019cra} and image generation\cite{johnson2018image}.

In scene graph generation, we actually obtain a set of visual phases$\textless$ subject-relation-object $\textgreater$ and the locations of objects in the image.
The triples of each scene graph form a local knowledge graph of the image and the triples of the whole training set form a global knowledge graph of relationships as shown in Figure~\ref{fig:intro} (a).
It remains a challenging task because deep neural networks cannot directly predict structured data due to its continuous nature.
It's a common practice to divide the scene graphs into classifiable graph elements.
\cite{sadeghi2011recognition} divides them into visual phase classes.
However, it's infeasible due to the hyper-linear growth concerning the number of objects and relations.
A widely-adopted strategy is to divide them into independent object classes and relation classes\cite{lu2016visual}.
Most methods classify the objects separately and then apply local graph structures to learn contextual object representations for relation classification\cite{xu2017scene,qi2019attentive,chen2019knowledge}.
However, they ignore the fact that the output space should also be contextual and structure-aware and adopt an unstructured one as shown in Figure~\ref{fig:intro} (b).
Hence these methods suffer from drastic intra-class variations.
For example, given the relation ``sit on'', the visual contents vary from ``vase-sit on-table" to ``dog-sit on-chair" as shown in Figure~\ref{fig:motivation} (a)(b)(c).
On the other hand, the distribution of these independent relation classes is seriously unbalanced as shown in Figure~\ref{fig:motivation} (d).

To mitigate the issues mentioned above, we propose a novel higher order structure embedded network (HOSE-Net), which consists of a visual module, a structure-aware embedding-to-classifier (SEC) module and a hierarchical semantic aggregation(HSA) module.
The SEC module is designed to construct a contextual and structured output space.
First, since objects serve as contexts in relationships, SEC learns context embeddings which embeds the objects' behavior patterns as subjects or objects and transfers this knowledge among the classifiers it connects to based on the overall class structure.
It adopts a graph convolution network\cite{kipf2016semi} to propagate messages on the local graphs with the guidance of object co-occurrence\cite{mensink2014costa} statistics.
Second, SEC learns a mapping function to project the context embeddings to related relation classifiers.
This mapping function is shared among all contexts which implicitly encodes the statistical correlations among objects and relations and organize a global knowledge graph based output space shown in Figure~\ref{fig:intro} (c).
Since the unbalanced relation data are distributed into different subspaces, SEC can alleviate the long-tailed distribution and the intra-class variations.

However, even if the context-specific classifiers can share statistical strengths via the context embeddings, distributing the training samples into a large set of subspaces can still suffer from sparsity issues.
To address this problem, we are inspired by the thought that object-based contexts can be redundant or noisy since relations are often defined in more abstract contexts.
For example, ride in ``man-ride-horse'' and ``woman-ride-elephant'' can be summarized as ``people-ride-animal''.
Accordingly, we propose a hierarchical semantic aggregation (HSA) module to mine the latent higher order structures in the global knowledge graph.
HSA hierarchically clusters the graph nodes following the principle that, if two objects have similar behavior patterns in the relationships they involved, the contexts they create can be embedded together, which is designed to find a good strategy to redistribute the samples into a smaller set of subspaces.
An object semantic hierarchy is generated in the process even if HSA just uses the graph structures.
It's not hard to understand because a semantic hierarchy is based on the properties of objects which also very relevant to their behavior patterns in relationships.

In summary, the proposed Higher Order Structure Embedded Network(HOSE-Net) uses embedding methods to construct a structured output space.
By modeling the inter-dependencies among object labels and relation labels, the serious intra-class variations and the long-tailed distribution can be alleviated. Moreover, clustering methods are used to make the structured output space more scalable and generalized.

\begin{figure*}
    \centering
    \includegraphics[width=\linewidth]{res/img/overview.pdf}
    \caption{The framework of our HOSE-Net. It consists of three modules: (1) a visual module which outputs detection results and prepare subject-object pairs for relation representation learning;
    (2) a SEC module which embeds the object labels into context embeddings by message passing and maps them to classifiers;
    (3) a HSA module which distill higher order structural information for context embedding learning.}
    \label{fig:overview}
\end{figure*}

Our contributions are as follows:
\begin{enumerate}[(1)]
    \item We propose to map the object-based contexts in relationships into a high-dimensional space to learn contextual and structure-aware relation classifiers via a novel structure-aware embedding-to-classifier module. This module can be integrated with other works focusing on visual feature learning.
    \item We design a hierarchical semantic aggregation module to distill higher order structural information for learning a higher order structured output space.
    \item We extensively evaluate our proposed HOSE-Net, which achieves new state-of-the-art performances on challenging Visual Genome \cite{krishna2017visual} and VRD~\cite{lu2016visual} benchmarks.

\end{enumerate}

\section{Related Work}
\textbf{Scene Graph Generation.}
Recently, the task of scene graph generation is proposed to understand the interactions between objects.
\cite{sadeghi2011recognition} decomposes the scene graphs into a set of visual phase classes and designs a detection model to directly detect them from the image.
Considering each visual phase as a distinct class would fail since the number of visual phase triples can be very large even with a moderate number of objects and relations.
An alternative strategy is to decompose the scene graphs into object classes and relation classes in which way the graph structures of the output data is completely collapsed.
Most of these methods focus on modeling the inter-dependencies of objects and relations in the visual representation learning.
\cite{dai2017detecting} embeds the statistical inference procedure into the deep neural network via structural learning.
\cite{xu2017scene} constructs bipartite sub-graphs of scene graphs and use RNNs to refine the visual features by iterative message passing.
\cite{chen2019knowledge,qi2019attentive,cui2018context} uses graph neural networks to learn contextual visual representation.
However, these methods still suffer from highly diverse appearances within each relation class because they all adopt a flat and independent relation classifiers.
In this paper, we argue that the structural information including the local and the global graph structures of the output data is vital for regularizing a semantic space.
\\
\textbf{Learning Correlated Classifiers With Knowledge Graph}.
Zero-shot learning(ZSL) models need to transfer the knowledge learned from the training classes to the unknown test classes.
A main research direction is to represent each class with learned vector representations.
In this way, the correlations between known classes and unknown classes can help to transfer the knowledge learned from the training classes to the unknown test classes by mapping the embeddings to visual classifiers.
Knowledge Graphs (KGs) effectively capture explicit relational knowledge about individual entities hence many methods\cite{wang2018zero,kampffmeyer2019rethinking,hascoet2019semantic,gao2019know,zhang2019tgg} use KGs to learn the class correlations.
In scene graph generation, the relation classes are correlated by object classes as in the knowledge graph and the structural information is vital for a well-defined output space.
We indirectly learn vector representations of the objects' role in relationships which are mapped to the visual relation classifiers via the knowledge graph structure.

\section{Approach}

\subsection{Overview}
We formally define a scene graph as $G = \left \{ B, O, R \right \}$. $O = \left \{ o_1, o_2, \dots, o_n \right \}$ is the object set and $o_i$ denotes the i-th object in image. $B = \left \{ b_1, b_2, \dots, b_n \right \}$ is the bounding box set and $b_i$ denotes the bounding box of $o_i$. $R = \left \{ r_{o_1 \rightarrow o_2}, r_{o_2 \rightarrow o_3}, \dots, r_{o_{(n-1)} \rightarrow o_n} \right \}$ is the edge set and $r_{o_i \rightarrow o_j}$ denotes the relation between subject $o_i$ and object $o_j$.
The probability distribution of the scene graph $\Pr(G | I)$ is formularized as:
\begin{equation}
    \Pr(G | I) = \Pr(B|I)\Pr(O|B,I)\Pr(R|B,O,I)
\end{equation}
We follow the widely-adopted two-stage pipeline\cite{zellers2018neural} to generate scene graphs. The first stage is object detection including object localization ($\Pr(B|I)$) and object recognition ($Pr(O|B, I)$). The second stage is relation classification ($\Pr(R|B,O,I)$).
Our proposed HOSE-Net consists of a visual module, a SEC module and a HSA module.

Section ~\ref{sec:visrep} introduces the visual module.
The major component is an object detector,
which outputs $B$, $O$ and the region features $F=\left \{ f_1, f_2, \dots, f_n \right \}$.
Then a set of object pairs $\left \{ (f_s, f_t), (o_s, o_t), (b_s,b_t) \right \}$ are produced, where $ s \neq t;s, t = 1 ... n $.
The union box feature $f_u$ for each pair is extracted by a relation branch.
The spatial feature $f_{spt}$ for each pair is learned from $(b_s,b_t)$ by a spatial module.
Section ~\ref{sec:secm} introduces the structure-aware embedding-to-classifier(SEC) module.
First, we construct local graphs to transfer statistical information between context embeddings of $O$.
Then the context embeddings are mapped to a set of primitive classifiers.
The classifier for each relation representation is adaptively generated by concatenating the primitive classifiers according to the pair label $(o_s, o_t)$.
Section ~\ref{sec:hsa} introduces the hierarchical semantic aggregation(HSA) module.
Based on the resulting semantic hierarchy, HSA creates a context dictionary $\mathcal{D}$ to map $o_i$ to one-hot encoding $c_i \in \mathbb{R}^{K}$ where $K  \in \left [ 1,N \right ] $ is the number of context embeddings and $N$ is the number of object classes.
The overall pipeline in shown in Figure~\ref{fig:overview}.

\subsection{Visual Representation}\label{sec:visrep}
\textbf{Object Detection}. In the first stage, the object detection is implemented by a Faster RCNN\cite{ren2015faster}. With the detection results, a set of subject-object region feature pairs $(f_s, f_t)$ with label information $(o_s, o_t)$ and coordinates of subject box $(x_s,y_s,w_s,h_s)$, object box $(x_t,y_t,w_t,h_t)$, union box$(x_u,y_u,w_u,h_u)$ is produced.
Then a separate relation branch uses three bottlenecks to refine the image feature and extract the union box feature $f_u$ of each subject-object pair by roi pooling.
While the Faster RCNN branch focuses on learning discriminative features for objects, the relation branch focuses on learning interactive parts of two objects.\\
\\
\textbf{Relation Representation}. Most existing methods explore the the visual representation learning for relations. To establish the connections between objects, they usually build graphs to associate the detected regions and use message passing frameworks to learn contextualized visual representations. Then the fusion features of the subjects and objects are projected to a set of independent relation labels by a softmax function.
Whether the relation classifiers are structured and contextualized has been little explored.
To verify the effectiveness of adopting a structured output space, we don't use a graph structure for learning the visual representations.
Given the triple region features from the detection module $(f_s, f_t, f_u)$, the visual representation of the relation is:
\begin{equation}

    r_{st} = \Psi_{st}([f_u; f_s; f_t])
\end{equation}
where $[;]$ is the concatenation operation and $\Psi_{st}$ is a linear transformation. $[f_u, f_s, f_t] \in \mathbb{R}^{3d_f}$. \\
\\
\textbf{Spatial Representation}.
The relative positions of the subject boxes and the object boxes are also valuable spatical clues for recognizing the relations.
The normalized box coordinates $\widehat{b_i}$ are computed as $[\frac{x}{w_{img}},\frac{y}{h_{img}},\frac{x+w}{w_{img}},\frac{y+h}{h_{img}},\frac{wh}{w_{img} h_{img}}]$ where $w_{img}$ and $h_{img}$ are the width and height of the image.
The relative spatial feature $b_{st}$ is encoded as $[\frac{x_s-x_t}{w_t},\frac{y_s-y_t}{h_t},log\frac{w_s}{w_t},log\frac{h_s}{h_t}]$. The final spatial representation is the concatenation of the normalized features and the relative features of the subject and object boxes:
\begin{equation}
    f_{spt} = \Psi_{spt}([\widehat{b_s},\widehat{b_t},b_{st}])
\end{equation}
where $\Psi_{spt}$ is a linear transformation, $[\widehat{b_s},\widehat{b_t},b_{st}] \in \mathbb{R}^{14}$.

\subsection{Structure-Aware Embedding-to-Classifier}\label{sec:secm}
Given the object label information $O = \left \{ o_1, o_2, \dots, o_n \right \}$, our proposed SEC module generates dynamic classifiers for relation representations according to the pair label $(o_s,o_t)$.
First, we embed the object labels into higher level context embeddings.
The one-hot context encodings of objects $\mathcal{C} = \left \{ c_1, c_2, \dots, c_n \right \},c_i \in \mathbb{R}^{K}$ are obtained through the context dictionary $\mathcal{D}$ which will be discussed in \ref{sec:hsa}.
The context embeddings $\mathcal{E} = \left \{ e_1, e_2, \dots, e_n \right \}, e_i \in \mathbb{R}^{d_e}$ are genereted as follows:
\begin{equation}
    e_i = W_{e}c_i
\end{equation}
where $W_{e} \in \mathbb{R}^{d_e \times K}$ is a context embedding matrix to be learned.

Then the context embeddings $\mathcal{E}$ are fed into a graph convolution network to learn local contextual information based on object co-occurrences.
We model the co-occurrence pattern in the form of conditional probability, i.e., $\mathcal{P}_{ij}$, which denotes the probability of occurrence of the j-th object class when the i-th object class appears. We compute the co-occurrences of object pairs in the training set and get the matrix $\mathcal{T}\in \mathbb{R}^{N \times N}$, $N$ is the number of object classes. $\mathcal{T}_{ij}$ denotes the co-occurring times of label pairs. The conditional probability matrix $\mathcal{P}$ is computed by:
\begin{equation}
    \mathcal{P}_{ij} = \frac{\mathcal{T}_{ij}}{\sum_{j}^{N}\mathcal{T}_{ij}}
\end{equation}
where $\sum_{j}^{N}\mathcal{T}_{ij}$ denotes the total number of i-th object class occurrences in the training set.
Then the adjacency matrix $A \in \mathbb{R}^{n \times n}$ of the local contextual graph is produced by:
\begin{equation}
    A_{ij} = \mathcal{P}_{o_i, o_j}
\end{equation}
The update rule of each GCN layer is:
\begin{equation}
    \mathcal{E} ^{l + 1} = f(\mathcal{E} ^{l}, A)
\end{equation}where $f$ is the graph convolution operation of \cite{kipf2016semi}.
The node output of the final GCN layer is the primitive classifiers $W_{prim} = \left \{ w_1, w_2, \dots, w_n \right \}, w_i \in \mathbb{R}^{\frac{d_{cls}}{2}}$ formulated as:
\begin{equation}
    w_i = {e_i}^{l + 1} = \sigma(\sum_{j\in \mathbb{N}_{i}}A_{ij}U{e_j}^{l})
\end{equation}
where $U \in \mathbb{R}^{d_e \times \frac{d_{cls}}{2}}$ is the transformation matrix to be learned. $\mathbb{N}_{i}$ is the neighbor node set of $e_i$. $\sigma$ is the nonlinear function.
For each subject-object label pair $(o_s,o_t)$, the visual classifier $W_{st}$ is a composition of two primitive classifiers according to its context:
\begin{equation}
    W_{st} = [w_{o_s};w_{o_t}] \in \mathbb{R}^{d_{cls}}
\end{equation}
where $[;]$ is the concatenation operation.
Apply the learned classifier to the relation representations to get the predicted scores:
\begin{equation}
    \hat{y} = W_{st}[r_{st};f_{spt}]
\end{equation}

\subsection{Hierarchical Semantic Aggregation}\label{sec:hsa}

\begin{figure}
    \centering
    \begin{minipage}[c]{\linewidth}
    \includegraphics[width=\linewidth]{res/img/graph_1.pdf}
    \end{minipage}
    (a)
    \begin{minipage}[c]{\linewidth}
    \includegraphics[width=\linewidth]{res/img/graph_2.pdf}
    \end{minipage}
    (b)
    \caption{The connectivity subgraph of (street, sidewalk) is illustrated in two parts: (street, sidewalk) are objects (a) or subjects (b) in $\textless$subject-relation-object$\textgreater$. The blue edges are the connected path and
    the yellow edges are the unconnected path.}
    \label{fig:scene_graph_example}
\end{figure}
Even if the global knowledge graph exhibits rich, lower-order connectivity patterns captured at the level of objects and relations, new problems emerge if we create context embeddings for all object classes. When the number of classes $N$ increases, a lot of context-specific classifiers can't get sufficient training samples due to data sparsity and can not be scalable.
The motivation of HSA is, although relations exist among concrete objects, the objects actually have many similar higher level behavior patterns in the overall contexts.
And there exists higher order connectivity patterns on the class structure which are essential for understanding the object behaviors in relationships.

We design an clustering algorithm for mining the higher order structural information based on behavior patterns of objects.
The connectivity pattern with respect to two nodes $q_s, q_t$ of knowledge graph $KG$ is represented in a subgraph $SG$.
$SG$ includes two sets of connection nodes $L_s,L_o$ between $q_s, q_t$:
\begin{eqnarray}
q_i\in L_s \Leftrightarrow r_{q_i \rightarrow q_s} = r_{q_i \rightarrow q_t}\\
q_i\in L_o \Leftrightarrow r_{q_i \leftarrow q_s} = r_{q_i \leftarrow q_t}
\end{eqnarray}
where $r_{q_i \rightarrow q_j}$ denotes the relation between $q_i$ and $q_j$.
Figure~\ref{fig:scene_graph_example} illustrates an example, which visualize the common patterns between street and sidewalk: both of them are made of tile, can be covered in snow, a person can walk on them and have buildings nearby.

If the behavior patterns of $q_s$ and $q_t$ have a large overlap, they can be clustered into a higher-level node.
The similarity score between $q_s, q_t$ is defined as:
\begin{equation}
\begin{split}
    f_{sim}(q_s, q_t) & = \frac{\left | L_s \right |}{d_{in}(q_s) + d_{in}(q_t) - \left | L_s \right |} \\
     & +  \frac{\left | L_o \right |}{d_{out}(q_s) + d_{out}(q_t) - \left | L_o \right |},
\end{split}
\end{equation}
where $d_{in}(q_i)/d_{out}(q_i)$ denotes the number of incoming/outgoing edges of node $q_i$, which represents the occurrence times of $q_i$ in all relationships as object/subject respectively. $\left | L_s \right|/\left | L_o \right |$ denotes the number of nodes in $L_s/L_o$, which represents the number of common behavior patterns of $q_s, q_t$ as object/subject.
This measure is fully based on the graph structure, not on the distributed representations of nodes from external knowledge graphs.
\\
\begin{algorithm}[tb]
\caption{}
\label{alg:optim}
\begin{algorithmic}[1]
\Require{$KG = \left \{ (q_1^0,q_2^0,...,q_N^0),(r_{q_1^0 \rightarrow q_2^0},...,r_{q_i^0 \rightarrow q_j^0},...) \right \}$,similarity measure function $f_{sim}$, cluster number $K$}
    \For{\texttt{$i = 1,2,...,N$}}
        \State{$\lambda_{i} = 1 $}
        \For{\texttt{$j = 1,2,...,N$}}
            \State{$Sim(i,j) = f_{sim}(q_i^0,q_j^0)$}
            \State{$Sim(j,i) = Sim(i,j)$}
        \EndFor
    \EndFor
    \State {Set current cluster number $num = N$}
    \While{$num > K$}\Comment{Find the two most similar node cluster}
        \State{ $q_{i}^{l_i}, q_{j}^{l_j} \gets SELECTMAX(Sim(i,j) / (\lambda_{i} + \lambda_{j}))$}
        \State $q_{i}^{l_{ij}} \gets MERGE(q_{i}^{l_i},q_{j}^{l_j})$
        \State $KG,Sim \gets UPDATE(KG,Sim)$
        \State $\lambda_{i} \gets \lambda{i} + \lambda{j} + 1$
        \State $REINDEX(\lambda)$
        \State $num \gets num - 1$
    \EndWhile
    \State{$\mathcal{D} \gets GETDICT((q_1^0,q_2^0,...,q_N^0),(q_1^l,q_2^l,...,q_K^l))$}
\Ensure{$\mathcal{D}$}
\end{algorithmic}
\end{algorithm}

\noindent\textbf{Algorithm.} We use hierarchical agglomerative clustering to find the node clusters shown in Algorithm~\ref{alg:optim}.
At each iteration, we merge the two clusters which have the most similar behavior patterns and update the knowledge graph by replacing the two clustered nodes with a higher level node.
Since the given triples are incomplete and unbalanced, we introduce a penalty term $\lambda$ to avoid the objects which have frequent occurrences in annotated relationships dominating the clustering.
When the number of clusters reaches the given K, the algorithm stops iterating.
We encode the clustering results as a dictionary $\mathcal{D}$ to map the N objects to one-hot encodings of dimension $K$ hence the objects within the same cluster will have the same context embedding.
In this way, the output space is reorganized into a smaller one.
Even if the clusters are not reasonable for all relations, experiments show that the context embeddings can still learn the upside in a high-dimensional space.
\\

\begin{figure*}
    \centering
    \begin{tabular}{c c}
    \begin{minipage}[c]{0.48\linewidth}
    \includegraphics[height=\linewidth]{res/img/tree.pdf}
    \end{minipage} &
    \begin{minipage}[c]{0.52\linewidth}
    \includegraphics[height=\linewidth]{res/img/tree1.pdf}
    \end{minipage} \\
    (a) & (b) \\
    \end{tabular}
    \caption{(a) The semantic hierarchy of VG. (b) The semantic hierarchy of VRD.}
    \label{fig:optim_result}
\end{figure*}
\section{Experiments}

\begin{table*}
    \centering
    \begin{tabular}{l l | l | l l | l l | l l}
    \hline
SEC & HSA & & \multicolumn{2}{c}{SGDET} \vline & \multicolumn{2}{c}{SGCLS} \vline & \multicolumn{2}{c}{PRDCLS}\\
 & & Recall at & 50 & 100 & 50 & 100 & 50 & 100 \\
\hline
\xmark & \xmark & Baseline $(K=1)$ & 28.1 & 32.5 & 34.8 & 36.4 & 64.6 & 67.3\\
\cmark & \xmark & HOSE-Net $(K=150)$ & 28.6 & 33.1 & 36.2 & 37.3 & 66.5 & 69.0\\
\cmark & \cmark & HOSE-Net $(K=40)$ & \textbf{28.9} & \textbf{33.3} & \textbf{36.3} & \textbf{37.4} & \textbf{66.8} & \textbf{69.2}\\
\hline
    \end{tabular}
    \caption{Ablation study on the SEC module and HSA module.}
    \label{tab:compair_m}
\end{table*}

\begin{figure}
    \centering
    \includegraphics[width=\linewidth, height=0.45\linewidth]{res/img/ablation.pdf}
    \caption{Ablation study on the clustering number. We draw the performance curves of SGCLS and PRDCLS on K = 1, 10, 20, 30, 40, 50, 70, 100, 130, 150.}
    \label{fig:compair_m}
\end{figure}

\begin{table*}
    \centering
    \begin{tabular}{l | l l | l l | l l}
    \hline
 & \multicolumn{2}{c}{SGDET} \vline & \multicolumn{2}{c}{SGCLS} \vline & \multicolumn{2}{c}{PRDCLS}\\
 Recall at & 50 & 100 & 50 & 100 & 50 & 100 \\
\hline
SEC ($K=150$) & 28.6 & 33.1 & 36.2 & 37.3 & 66.5 & 69.0\\
SEC + kmeans with word2vec embedding($K=40$) & 28.7 & 33.1 & 36.2 & 37.3 & 66.4 & 68.9\\
SEC + HSA ($K=40$) & \textbf{28.9} & \textbf{33.3} & \textbf{36.3} & \textbf{37.4} & \textbf{66.8} & \textbf{69.2}\\
\hline
    \end{tabular}
    \caption{ Comparison between the clustering in HSA module and kmeans with word2vec embedding clustering.}
    \label{tab:kmeans}
\end{table*}

\begin{table*}
\centering
\begin{tabular}{l | l l l l l l | l l  l l  l l}
\hline
 & \multicolumn{6}{c}{Graph Constraint}  \vline & \multicolumn{6}{c}{No Graph Constraint}\\
 & \multicolumn{2}{c}{SGDET}  & \multicolumn{2}{c}{SGCLS}  & \multicolumn{2}{c}{PRDCLS} \vline &  \multicolumn{2}{c}{SGDET}  & \multicolumn{2}{c}{SGCLS}  & \multicolumn{2}{c}{PRDCLS}\\
Recall at & 50 & 100 & 50 & 100 & 50 & 100 & 50 & 100 & 50 & 100 & 50 & 100 \\
\hline
VRD~\cite{lu2016visual} & 0.3 & 0.5 & 11.8 & 14.1 & 27.9 & 35.0 & - & - & - & - & - & -\\
ISGG~\cite{xu2017scene} & 3.4 & 4.2 & 21.7 & 24.4 & 44.8 & 53.1 & - & - & - & - & - & -\\
MSDN~\cite{xu2017scene} & 7.0 & 9.1 & 27.6 & 29.9 & 53.2 & 57.9 & - & - & - & - & - & - \\
AsscEmbed~\cite{newell2017pixels} & 8.1 & 8.2 & 21.8 & 22.6 & 54.1 & 55.4 & 9.7 & 11.3 & 26.5 & 30.0 & 68.0 & 75.2\\
Message Passing+~\cite{zellers2018neural} & 20.7 & 24.5 & 34.6 & 35.4 & 59.3 & 61.3 & 22.0 & 27.4 & 43.4 & 47.2 & 75.2 & 83.6\\
Frequency~\cite{zellers2018neural} & 23.5 & 27.6 & 32.4 & 34.0 & 59.9 & 64.1 & 25.3 & 30.9 & 40.5 & 43.7 & 71.3 & 81.2\\
Frequency+Overlap~\cite{zellers2018neural} & 26.2 & 30.1 & 32.3 & 32.9 & 60.6 & 62.2 & 28.6 & 34.4 & 39.0 & 43.4 & 75.7 & 82.9\\
MotifNet-LeftRight~\cite{zellers2018neural} & 27.2 & 30.3 & 35.8 & 36.5 & 65.2 & 67.1 & 30.5 & 35.8 & \textbf{44.5} & 47.7 & 81.1 & 88.3\\
GraphRCNN~\cite{yang2018graph} & 11.4 & 13.7 & 29.6 & 31.6 & 54.2 & 59.1 & - & - & - & - & - & - \\
KERN~\cite{chen2019knowledge} & 27.1 & 29.8 & \textbf{36.7} & 37.4 & 65.8 & 67.6 & - & - & - & - & - & -\\
VCTREE~\cite{tang2019learning} & 27.7 & 31.1 & 37.9 & \textbf{38.6} & 66.2 & 67.9 & - & - & - & - & - & -\\
\hline
HOSE-Net ($K=40$) & \textbf{28.9} & \textbf{33.3} & 36.3 & 37.4 & \textbf{66.7} & \textbf{69.2} & \textbf{30.5} & \textbf{36.3} & 44.2 & \textbf{48.1} & \textbf{81.1} & \textbf{89.2}\\
\hline
\hline
RelDN$^\ast$~\cite{zhang2019graphical} & 28.3 & 32.7 & 36.8 & 36.8 & 68.4 & 68.4 & 30.4 & \textbf{36.7} & 48.9 & 50.8 & 93.8 & 97.8\\
HOSE-Net$^\ast$ ($K=40$) & \textbf{28.9} & \textbf{33.3} & \textbf{37.3} & \textbf{37.3} & \textbf{70.1} & \textbf{70.1} & \textbf{30.5} & 36.3 & \textbf{49.7} & \textbf{51.2} & \textbf{94.6} & \textbf{98.2}\\
\hline
\end{tabular}
\caption{Comparison with state-of-the-art methods on Visual Genome. HOSE-Net$^\ast$ uses the evaluation metric in ~\cite{zhang2019graphical}}
\label{tab:sota_vg}
\end{table*}

\subsection{Datasets}

\textbf{Visual Genome\cite{krishna2017visual}.} It is a large scale dataset with 75729 object classes and 40480 relation classes. There are several modified versions for scene graph generation. In this paper, we follow the same train/val splits in which the most frequent 150 objects and 50 relations are chosen.

We measure our method on VG in three tasks:
\begin{enumerate}
    \item predicate classification (PRDCLS): Given the ground truth annotations of the object classes and bounding boxes, predict the relation type of each object pair.
    \item Scene graph classification (SGCLS):  Given the ground truth annotations of object bounding boxes, predict the object classes and the relation type of each object pair.
    \item Scene graph detection (SGDET): Predict the bounding boxes, the object classes and the relation type of each object pair.
\end{enumerate}

We use Recall@50, Recall@100 as our evaluation metrics. Recall@x computes the fraction of relationship hits in the top x confident relationship predictions. The reason why precision and average precision (AP) are not proper metrics for this task is, only a fraction of relationships are annotated and they will penalize the right detection if it is not in the ground truth.
We report the Graph Constraint Recall@x following \cite{lu2016visual} which only involves the highest score relation prediction of each subject-object pair in the recall ranking. We also report the No Graph Constraint Recall@x following \cite{newell2017pixels} which involves all the 50 relation scores of each subject-object pair in the recall ranking. It allows multiple relations exist between objects.

\noindent\textbf{VRD\cite{lu2016visual}} contains 4000 training and 1000 test images including 100 object classes and 70 relations.

We follow \cite{yu2017visual} to measure our method on VRD in two tasks:
\begin{enumerate}
    \item Phase detection: Predict the visual phase triplets $\textless$subject-relation-object$\textgreater$ and localize the union bounding boxes of each object pair.
    \item Relationship detection: Predict the visual phase triplets $\textless$subject-relation-object$\textgreater$ and localize the bounding boxes of subjects and objects.
\end{enumerate}

We report Recall@50 and Recall@100 at involving 1 ,10 and 70 relation predictions per object pair in recall ranking as the evaluation metrics.

\begin{table*}
    \centering
    \resizebox{\linewidth}{!}{
    \begin{tabular}{l | l l l l l l | l l l l l l}
    \hline
     & \multicolumn{6}{c}{Relationship Detection} \vline & \multicolumn{6}{c}{Phrase Detection}\\
     & \multicolumn{2}{c}{rel=1} & \multicolumn{2}{c}{rel=10} & \multicolumn{2}{c}{rel=70} \vline& \multicolumn{2}{c}{rel=1} & \multicolumn{2}{c}{rel=10} & \multicolumn{2}{c}{rel=70}\\
     Recall at & 50 & 100 & 50 & 100 & 50 & 100 & 50 & 100 & 50 & 100 & 50 & 100\\
     \hline
     VTransE~\cite{zhang2017visual} & 19.4 & 22.4 & - & - & - & - & 14.1 & 15.2 & - & - & - & -\\
     ViP-CNN~\cite{li2017vip} & 17.32 & 21.01 & - & - & - & - & 22.78 & 27.91 & - & - & - & -\\
     VRL~\cite{liang2017deep} & 18.19 & 20.79 & - & - & - & - & 21.37 & 22.60 & - & - & - & -\\
     KL distilation~\cite{yu2017visual} & 19.17 & 21.34 & \textbf{22.56} & \textbf{29.89} & \textbf{22.68} & \textbf{31.89} & 23.14 & 24.03 & 26.47 & 29.76 & 26.32 & 29.43\\
     MF-URLN~\cite{zhan2019exploring} & 23.9 & 26.8 & - & - & - & - &  31.5 & 36.1 & - & - & - & -\\
     Zoom-Net~\cite{yin2018zoom} & 18.92 & 21.41 & - & - & 21.37 & 27.30 & 24.82 & 28.09 & - & - & 29.05 & 37.34\\
     CAI + SCA-M~\cite{yin2018zoom}      & 19.54 & 22.39 & - & - & 22.34 & 28.52 & 25.21 & 28.89 & - & - & \textbf{29.64} & \textbf{38.39}\\
     RelDN (ImageNet)~\cite{zhang2019graphical} & 19.82 & 22.96 & 21.52 & 26.38 & 21.52 & 26.38 & 26.37 & 31.42 & 28.24 & 35.44 & 28.24 & 35.44\\
     \hline
     HOSE-Net ($K=18$) & \textbf{20.46} & \textbf{23.57} & 22.13 & 27.36 & 22.13 & 27.36 & \textbf{27.04} & \textbf{31.71} & \textbf{28.89} & \textbf{36.16} & 28.89 & 36.16\\
     \hline
    \end{tabular}
    }
    \caption{Comparison with state-of-the-art methods on VRD.}
    \label{tab:sota_vrd}
\end{table*}

\subsection{Implementation Details}

HOSE-Net adopts a two-stage pipeline. The object detector is Faster RCNN with a VGG backbone initialized by COCO pre-trained weights for Visual Genome and ImageNet pre-trained weights for VRD and then finetuned on the visual relationship datasets. The backbone weights are fixed.
For stable training, we add an unstructured relation classifier as a separate branch for joint training.
Considering the dataset scale and dataset quality, we adopt different training mechanisms for Visual Genome and VRD. In Visual Genome experiments, we set $lr = 0.001$ for the structured classifier and $lr = 0.01$ for the unstructured one. During testing, we evaluate the structured classifier. In VRD, the loss weight of the unstructured classifier is 0.7, and the structured one is 0.3.
During testing, the result is the weight sum of the two classifiers.
Since the statistical bias is a widely-adopted strategy in the two-stage pipeline, we train a bias vector and fuse the bias results with the visual module results during testing following \cite{zellers2018neural}.

The proposed framework is implemented by PyTorch. All experiments are conducted on servers with 8 NVIDIA Titan X GPUs with 12 GB memory.
The batch size at the training phase is set to 8. $d_f$ is set to 4096 and $d_e$ is set to 512.

\subsection{Ablation Study} Now we perform ablation studies to better examine the effectiveness of our framework.
\\
\textbf{Structured Output Space with Cluster Number K.} We perform an ablation study to validate the effectiveness of the SEC module which learns a structured output space and the HSA module which incorporates higher order structure into the output space with respect to the cluster number K.
$K = 1$ is our baseline model which uses the conventional unstructured relation classifiers.
$K = 150$ only employs SEC module to learn a low order structured output space.
In HSA module, K is a hyper parameter which can be a trade-off between the performance and the model complexity.
We know that all clustering algorithms suffer from the lack of automatic decisions for an optimal number of clusters. While trying all possible combinations is prohibitively expensive, we have got a comprehensive set of results for comparison. The performance curve on $K=1,10,20,30,40,50,60,70,100,130,150$ are shown in the Figure~\ref{fig:compair_m}.
We find $K=40$ works the best.
Table~\ref{tab:compair_m} presents results when $K=1,150,40$. \\
The comparison shows that:
\begin{enumerate}[1)]
    \item Adopting a structured output space ($K=40,150$) is superior to an unstructured one($K=1$) which verifies the effectiveness of the SEC module.
    \item Adopting a higher order structured output space ($K=40$) outperforms lower order one ($K=150$) which verifies the effectiveness of the HSA module.
\end{enumerate}
We also show the resulting semantic hierarchy of objects from the HSA module on VG and VRD in Figure~\ref{fig:optim_result}. Although the HSA module is not designed to sort out the objects, the unsupervised process of searching higher order connectivity patterns in the knowledge graph can contribute to an object taxonomy. At the lower levels, the object classes are classified according to more specific properties, eg. roof with railing, street with sidewalk, train with car. At the higher levels, the clusters have more abstract semantics and are classified according to more general properties, eg. glass-bottle-cup with basket-box-bag, toilet-sink-drawer with shelf-cabinet-counter.

\noindent\textbf{Clustering in HSA}.
Our behavior pattern based hierarchical clustering purely relies on the knowledge graph structure of the ground truth.
To verify the effectiveness of our clustering, we also conduct K-means clustering on word2vec embeddings of objects to obtain an external knowledge based clusters.
Table~\ref{tab:kmeans} presents the results of adopting HSA clustering results ($K=40$), adopting K-means with word2vec embedding clustering results and not adopting context clustering $K=150$).\\
The comparison shows that:
\begin{enumerate}[1)]
    \item HOSE-Net with SEC($K=150$) shows comparable results to HOSE-Net with SEC and K-means with word2vec embedding($K=40$), which means, the clustering results can't improve the performance.\\
    \item HOSE-Net with SEC and HSA($K=40$) outperforms HOSE-Net with SEC and K-means with word2vec embedding($K=40$), which proves that our structure-based clustering with internal relation knowledge can truly produces helpful clustering results to boost this task.
\end{enumerate}

\subsection{Comparison to State of the Art}
\textbf{Visual Genome:} Table~\ref{tab:sota_vg} shows the performance of our model outperforms the state-of-the-art methods. Our object detector is adopted from \cite{zhang2019graphical} with $mAP = 25.5$ , $IoU = 0.5$. The number of clusters for comparison is 40. These methods all adopt flat relation classifiers. VRD\cite{lu2016visual}, AsscEmbed\cite{newell2017pixels}, Frequency\cite{zellers2018neural} predict the objects and the relations without joint inference. The other works are engaged in modeling the inter-dependencies among objects and relations. MotifNet-LeftRight\cite{zellers2018neural} encodes the dependencies through bidirectional LSTMs. MSDN\cite{li2017scene},ISGG\cite{xu2017scene},KERN\cite{chen2019knowledge} rely on message passing mechanism. SGP\cite{herzig2018mapping} employs structured learning.  In comparison, our framework doesn't refine the visual representations but still achieves new state-of-the-art results on SGDET, SGCLS, PRDCLS with and without graph constraint.
RelDN\cite{zhang2019graphical} proposes contrastive losses and reports Top@K Accuracy (A@K) on PredCls and SGCls in which the ground-truth subject-object pair information is also given.
We also compare with RelDN at A@K as shown in Table~\ref{tab:sota_vg}.

\noindent\textbf{VRD:} Table~\ref{tab:sota_vrd} presents results on VRD compared with state-of-the-art methods. The number of clusters for comparison is 18. The implementation details of most methods on VRD are not very clear. As shown in \cite{zhang2019graphical}, pre-training on COCO can provide stronger localization features than pre-training on ImageNet. For a fair comparison, we use the ImageNet pre-trained model. We achieve new state-of-the-art results on Relationship Detection and Phrase Detection.

\section{Conclusions}
In this work, we propose Higher Order Structure Embedded Network to address the problems caused by ignoring the structure nature of scene graphs in existing methods.
First we propose a Structure-Aware Embedding-to-Classifier module to redistribute the training samples into different classification subspaces according to the object labels and connect the subspaces with a set of context embeddings following the global knowledge graph structure.
Then we propose a Hierarchical Semantic Aggregation module to mine higher order structures of the global knowledge graph which makes the model more scalable and trainable.

\section{ACKNOWLEDGMENTS}
This work was supported by NSFC project Grant No. U1833101, SZSTI under Grant No. JCYJ20190809172201639 and the Joint Research Center of Tencent and Tsinghua.

\bibliographystyle{ACM-Reference-Format}