\section{Introduction}
\label{sec:Introduction}

The influence of Coulomb interactions on electron properties of graphene is a subject of much interest~\cite{CastroNeto2009tep, Kotov2012eei}.
Prominent examples of interaction effects are the enhancement of the quasiparticle velocity $v$
above the typically quoted value $v = 1.0 \times 10^8\,\text{cm}/\text{s}$ at low momenta~\cite{Elias2011dcr, Li2008dcd, Siegel2011mbi} (as predicted
by the early theoretical work\cite{Gonzalez1994nfl, Gonzalez1999mfl}),
and the emergence of additional dispersion branches\cite{Bostwick2010oop, Walter2011esa} believed to be plasmarons --- bound states of electrons and plasmons.
The strength of Coulomb interactions in graphene is determined by the dimensionless parameter ($\hbar \equiv 1$ in this paper)
\begin{equation}
\alpha = \frac{e^2}{\kappa v} = \frac{2.2}{\kappa}\,,
\label{eqn:alpha}
\end{equation}
which can be controlled by varying the effective dielectric constant $\kappa$ of the graphene environment.

Interactions modify not only the quasiparticle properties but also the response functions of graphene, e.g.,
the irreducible (or proper) polarization $P(q,\omega)$.
This quantitiy is defined to be the coefficient of proportionality between the change in electron density and the self-consistently determined screened scalar potential.
The observables directly related to $P$ are the dielectric function $\epsilon$ and the longitudinal conductivity $\sigma$:
\begin{align}
\epsilon(q, \omega) &= 1 - \frac{2\pi e^2}{q} \,  P(q, \omega)\,,
\label{eqn:epsilon}\\
\sigma(q, \omega) &= e^2\, \frac{i \omega}{q^2}\, P(q, \omega)\,.
\label{eqn:sigma}
\end{align}
Our work is motivated in part by two recent experiments\cite{Reed2010tef, Wang2012mdq}
that suggested that the standard random phase approximation\cite{Mahan1990mpp} (RPA)
significantly underestimates the static dielectric function $\epsilon(q, 0)$ of graphene.
The RPA amounts to replacing the exact polarization function $P$ by the noninteracting value $P_0$.
For neutral graphene at zero temperature,
which is the case studied here, function $P_0$ is given by\cite{Gonzalez1994nfl}
\begin{equation}
P_0(q,\omega) = -\frac{q^2}{4 \sqrt{v^2 q^2 - (\omega + i0)^2}}
\label{eqn:P_0}
\end{equation}
at small enough $q$ and $\omega$ where the Dirac approximation is valid.
Equations~\eqref{eqn:epsilon} and \eqref{eqn:P_0} entail~\cite{Ando2002dca}
\begin{equation}
\epsilon_{\text{RPA}}(q, 0) = 1 + \frac{\pi}{2}\, \alpha\,.
\label{eqn:epsilon_RPA}
\end{equation}
For $\kappa = 1$ this formula gives $\epsilon_{\text{RPA}} \approx 4.6$.
On the other hand,
a much larger value $\epsilon = 15.4^{+39.6}_{-6.4}$
was inferred from inelastic x-ray scattering on bulk graphite\cite{Reed2010tef}.
Similarly, for graphene on a boron nitride substrate ($\kappa \approx 2.5$,
$\alpha \approx 0.9$) $\epsilon_{\text{RPA}} \approx 2.4$,
whereas the study of charge profile near Coulomb impurities by means of scanning tunneling microscopy suggests\cite{Wang2012mdq} $\epsilon = 3.0 \pm 0.1$.

\begin{figure}[t]
	\begin{center}
		\includegraphics[width=2.6in]{pol1cor_Fig1}
	\end{center}
	\caption{First-order diagrams for the polarization function include the self-energy part (a) and the vertex part (b). Solid lines represent electron Green's function, dashed lines the Coulomb interaction, and dotted lines the external momentum and frequency.}
	\label{fig:Diagrams}
\end{figure}

The first-order interaction correction $P_1 = \mathcal{O}(\alpha)$ beyond the RPA is represented by the diagrams depicted in Fig.~\ref{fig:Diagrams}.
We show that by including this correction,
\begin{equation}
\label{eqn:pol}
P(q,\omega) \to P_0(q,\omega) + P_1(q,\omega)\,,
\end{equation}
one can significantly reduce the discrepancy between the theory and experiment.
Our result for $\epsilon(q, 0)$ is\footnote{Similar results have been obtained by F.~Guinea and also by A.~Principi and M.~Polini (private communications).}
\begin{equation}
\epsilon(q, 0) = 1 + \frac{\pi}{2}\, \alpha_q + 0.778 \alpha_q^2\,,
\quad \alpha_q \ll 1\,.
\label{eqn:epsilon_dc}
\end{equation}
(The difference between $\alpha_q$ and $\alpha$ will be clarified in Sec.~\ref{sec:results}.)
This formula yields $\epsilon \approx 3.0$ at $\alpha_q = 0.9$, in excellent agreement with Ref.~\onlinecite{Wang2012mdq}.
In turn, if we use Eq.~\eqref{eqn:epsilon_dc} for $\alpha_q = 2.2$, we get $\epsilon \approx 8.2$,
which is close to the lower estimate of the static dielectric constant in Ref.~\onlinecite{Reed2010tef}, although, the use of a
perturbative formula is questionable at such large interaction
strengths.

Another topic of interest is the dynamic response of graphene.
For example, in the ``optical'' limit $\omega\gg v q$,
Eqs.~\eqref{eqn:sigma} and \eqref{eqn:P_0} imply
that noninteracting Dirac fermions have a frequency-independent conductivity\cite{Ludwig1994iqh}
\begin{equation}
\sigma_0 (0,\omega)=  \frac{e^2}{4}\,.
\label{eqn:sigma_0}
\end{equation}
Accordingly, deviations of the infrared optical conductivity of graphene from the universal value $\sigma_0$ may signal interaction effects.
Experimentally, no such deviations have been observed\cite{Mak2008mot, Nair2008fsc, Li2008dcd}
while the theoretical calculation of the interaction corrections has been a subject of a debate.
Our analysis favors the result
\begin{equation}
\frac{\sigma(0, \omega)}{\sigma_0} - 1 \simeq \frac{19 - 6\pi}{12}\, \alpha\,,
\quad \alpha \ll 1\,,
\label{eqn:sigma_I}
\end{equation}
derived in~Refs.~\onlinecite{Mishchenko2008mci, Sheehy2009oto}.
We explain why a different coefficient was obtained in Ref.~\onlinecite{Juricic2010coi}.
The numerical smallness of $(19 - 6\pi) / 12 \approx 0.01$ may be the reason why
the interaction corrections have not been observed so far.

To get a more complete understanding of the interaction corrections
we also compute $P_1(q, \omega)$ at arbitrary $\omega / v q$ ratios.
This enables us to see the evolution of $P_1(q, \omega)$ in the full momentum-frequency parameter space, from the small value in the optical limit to the divergence at the spectral boundary $\omega = v q$ to a sizable effect in the static limit.
Besides theoretical interest,
motivation for this calculation comes from a rapidly burgeoning effort in probing
regimes of intermediate $\omega/v q$ by state-of-the-art experimental techniques, such as the near-field optics\cite{Fei2011ino, Fei2012gto, Chen2012oni} and electron energy loss spectroscopies\cite{Liu2010pps, Koch2010spp, Tegenkamp2011peh, Shin2011ooi}.
Such regimes are also pertinent for the electromagnetic response of graphene nanoribbons\cite{Ju2011gpf}.

The remainder of the paper is organized as follows. In Sec.~\ref{sec:results} we summarize our main results. Derivation of these results is outlined in Sec.~\ref{sec:derivation}. In Sec.~\ref{sec:discussion} we compare our findings with previous work. The Appendix~\ref{sec:optical} is devoted to the critical analysis of the controversy regarding the optical limit.

\section{Main results}
\label{sec:results}

In order to present our results we first need to explain the notation $\alpha_q$ in Eq.~\eqref{eqn:epsilon_dc} above.
Crudely, $v$ and $\alpha$ represent the bare velocity and the bare coupling constant of the theory,
whereas $v_q$ and $\alpha_q$ denote their renormalized values.
More precisely, $v_q$ is defined to be the phase velocity of quasiparticles with momentum $q$:
\begin{equation}
\label{eqn:vel_I}
v_q \equiv v + \frac{\Sigma_q}{q}\,,
\end{equation}
where $\Sigma_q$ is the on-shell self-energy.
At the level of the first-order perturbation theory one finds\cite{Gonzalez1994nfl, Gonzalez1999mfl}
\begin{equation}
\Sigma_q = \frac{e^2 q}{4\kappa}\, \ln \frac{\Lambda}{q}\,,
\label{eqn:Sigma_q}
\end{equation}
where $\Lambda$ is the high-momentum cutoff.
Therefore, $v_q$ and $\alpha_q$ are given by
\begin{align}
v_q &= v + \frac{e^2}{4\kappa}\,
\ln \frac{\Lambda}{q}\,,
\label{eqn:vel_II}\\
\alpha_q &= \frac{e^2}{\kappa v_q}
          = \left(\frac{1}{\alpha} + \frac{1}{4}\, \ln \frac{\Lambda}{q}\right)^{-1}\,.
\label{eqn:alpha_q}
\end{align}
According to the renormalization group approach,
such expressions are not just first-order approximations.
They are asymptotically exact at low enough $q$ where $\alpha_q \ll 1$.
However, $\alpha$ should be understood as the running coupling constant evaluated at some other cutoff $\Lambda$.
The choice of $\alpha$ is largely arbitrary because a change in $\alpha$ in Eq.~\eqref{eqn:alpha_q} can be absorbed into $\Lambda$.
Nevertheless, $\alpha_q$, which is determined by the observable quantity,
the phase velocity $v_q$, is unambiguous.
Relations between various observable quantities are expressible in terms of the renormalized parameters only. For example, phase velocities at two different momenta $q$ and $k$ are linked by the relation
\begin{equation}
\frac{v_q}{v_k} = 1 + \frac{\alpha_k}{4}\, \ln\frac{k}{q}\,,
\end{equation}
which is free from the arbitrary parameters $\alpha$ and $\Lambda$.
Similarly, the polarization function can and (if one desires higher accuracy) should
be expressed in terms of $\alpha_q$ and $v_q$. This point will be stressed again in Sec.~\ref{sec:discussion}.

Let us now present our findings.
The first-order correction to the polarization function is written as
\begin{equation}
P_1(q,\omega) = \frac{\alpha_q q}{v_q}\, p_1(x)\,,
\label{eqn:p_1}
\end{equation}
where $x$ is the dimensionless ratio
\begin{equation}
x = \frac{\omega}{v_q q}
\label{eqn:x}
\end{equation}
and $p_1(x)$ is the complex dimensionless function, whose real and imaginary parts are displayed in Figs.~\ref{fig:p1}(a)--\ref{fig:p1}(d).
In general, $p_1(x)$ has to be evaluated numerically.
However,
analytical results are available in several limits,
as discussed later in this section.

\begin{figure*}
\begin{center}
\includegraphics[width=7.0in]{pol1cor_Fig2.pdf}
\end{center}
\caption{(Color online) Panels (a) and (b) depict function $p_1(x)$ where $x = \omega / v_q q$.
The red dashed line, the blue dotted line, and the black solid line are, respectively, the self-energy term, the vertex term,
and their sum, which is $p_1(x)$.
Although not shown in the figure,
very close to $x = 1$ the divergence of the self-energy term overwhelms that of the vertex term,
which causes the sign change of $\mathrm{Re}\, p_1(x)$ at
$x = 0.998$ and $\mathrm{Im}\, p_1(x)$ at $x = 1.005$.
Panels (c) and (d) illustrate $x p_1(x)$,
the quantity which (if multiplied by $4i$) gives the interaction correction to the universal conductivity,
$\sigma / \sigma_0 - 1$.
The meaning of the three curves is the same as in panels (a) and (b).
Panels (e) and (f) depict the real and imaginary
parts of the dielectric function for $\alpha_q = 0.3$.
The green solid line is the RPA result
and the black curves (solid and dashed) include the interaction correction.
The dashed part of the curve is within $\alpha_q$ from the absorption threshold.
The first-order perturbation theory is expected to be unreliable in that region.
Unphysical behavior near the threshold is exemplified by the negative sign of $\text{Im}\,\epsilon$ in the inset of panel (f). In panel (f) the second order interaction correction (black curve) becomes larger than the RPA value (green curve) at $\omega/v_q q \approx 9.1$ (not shown).
\label{fig:p1}
}
\end{figure*}

The full polarization function to order $\mathcal{O}(\alpha_q)$ is
\begin{equation}
P(q,\omega)=\frac{q}{v_q}
\left[-\frac{1}{4}\frac{1}{\sqrt{1 - (x+i0)^2}} + \alpha_q p_1(x)\right]\,.
\label{eqn:P_from_p_1}
\end{equation}
The finite imaginary part of $P$ appears when $x$ exceeds unity, which we refer to as the absorption threshold.
Once $p_1(x)$ is known, one can use Eqs.~\eqref{eqn:epsilon} and \eqref{eqn:P_from_p_1} to get the dielectric function $\epsilon(q, \omega)$.
For example, in the static limit we obtain Eq.~\eqref{eqn:epsilon_dc}.
The real and imaginary parts of $\epsilon(q, \omega)$ as a function of $x$ are
illustrated by Figs.~\ref{fig:p1}(e) and~\ref{fig:p1}(f) for the case
of a suitably low $\alpha_q = 0.3$.
The RPA predictions are also plotted in Figs.~\ref{fig:p1}(e) and~\ref{fig:p1}(f) for comparison.
As one can see, the RPA underestimates $\mathrm{Re}\, \epsilon(q, \omega)$ at $x < 1$ and overestimates it at $x > 1$.
Near the absorption threshold, at $|1 - x| < \alpha_q$,
the first-order results are deemed unreliable and are shown by the dashed line.
In that region higher order corrections become as important as the first-order ones.
An example of inapplicability of the first-order perturbation theory near $x = 1$
is the negative sign of $\mathrm{Im}\, \epsilon(q, \omega)$ at sufficiently large $\alpha_q$,
see the inset of Fig.~\ref{fig:p1}(f).

Let us next discuss the analytical results for the function $P_1(q, \omega)$.
This function can be written as
\begin{equation}
P_1(q,\omega) = 2 P_a(q,\omega) + P_b(q,\omega)\,,
\end{equation}
where $P_a$ and $P_b$ are the self-energy and vertex terms represented by
the corresponding diagrams in Fig.~\ref{fig:Diagrams}.
(The factor of $2$ comes from the symmetry of the self-energy diagram.)
Below we describe these terms separately.

For the self-energy contribution we have the result
\begin{equation}
2 P_a(q,\omega)=\frac{\alpha q}{8 \pi v}\,
\left[\frac{\pi}{2}\frac{1}{(1-y^2)^{3/2}}\ln \frac{\Lambda}{q}+I_a(y)\right]\,,
\label{eqn:Pa2}
\end{equation}
where $y = \omega / (v q) + i 0$.
The expression for function $I_a(y)$, which is rather cumbersome, is given by Eq.~\eqref{eqn:I_a} (Sec.~\ref{sec:derivation}).

Equation~\eqref{eqn:Pa2} is written in terms of the ``bare'' parameters.
As explained above, we should rewrite it in terms of the renormalized ones.
To do so we combine $2 P_a(q, \omega)$ with the zeroth order polarization function $P_0(q, \omega)$ [Eq.~\eqref{eqn:P_0}] and get,
to the order $\mathcal{O}(\alpha_q)$,
\begin{equation}
\begin{split}
P_0(q,\omega) + 2 P_a(q,\omega)
&= -\frac{1}{4}\, \frac{q}{v_q}\, \frac{1}{\sqrt{1 - (x + i0)^2}}\\
&+ \frac{1}{8 \pi}\, \frac{\alpha_q q}{v_q}\, I_a(x + i0)\,.
\label{eqn:P02Pa}
\end{split}
\end{equation}
The desired renormalized $P_0(q,\omega)$ and $2 P_a(q,\omega)$ are, respectively,
the first and the second line of Eq.~\eqref{eqn:P02Pa}.
The limiting forms of such renormalized $2 P_a$ are as follows.
In the static limit we find
\begin{gather}
2 P_a(q,0) = -\frac{C_a}{4}\, \frac{\alpha_q q}{v_q}\,,
\label{eqn:Pa_dc}\\
C_a = \frac{1}{8} - \frac{\ln 2}{2} + \frac{1}{\pi}\left(G - \frac{1}{6}\right) \approx 0.017\,,
\label{eqn:C_a}
\end{gather}
where $G = 0.916\ldots$ is the Catalan constant.
In the optical limit, we obtain
\begin{equation}
2 P_a(q,\omega) \simeq \frac{1}{16}\,
\frac{\alpha_q q^2}{i \omega}\,,
\quad \omega \gg v_q q\,.
\label{eqn:P02Pa_optical}
\end{equation}
We expect that the accuracy of this expression is improved if in place of $\alpha_q$ one uses $\alpha_k$, where $k$ is such that $\omega = v_k k$.

Near the absorption threshold, $|x - 1| \ll 1$, we get
\begin{equation}
2 P_a \simeq
\frac{1}{16}\, \frac{\alpha_q q}{v_q}\, \left[
\frac{2\ln 2 - \frac{5}{6}}{(1 - x^2)^{3/2}}
-\frac{1}{3}\, \frac{1}{(1 - x^2)^{1/2}}
\right]\,.
\label{eqn:Pa_theshold}
\end{equation}
Note that the coefficient in front of the dominant $(1 - x^2)^{-3/2}$ singularity depends on the renormalization procedure.

Our result for the vertex term is
\begin{equation}
P_b(q, \omega) = -\frac{1}{4}\, \frac{\alpha_q q}{v_q}\, I_b(x + i 0)\,,
\label{eqn:P_b_II}
\end{equation}
where function $I_b(y)$ is given by Eqs.~\eqref{eqn:Ib}--\eqref{eqn:ImIb} of Sec.~\ref{sec:derivation}
and is computed by numerical quadrature.
The limiting forms of $P_b$ are as follows.
In the static limit we find
\begin{equation}
P_b(q,0)=-\frac{q}{4 v_q} C_b \alpha_q\,,
\end{equation}
where $C_b\approx0.48$.
In the optical limit, $\omega\gg v_q q$, we have
\begin{equation}
P_b(q,\omega) \simeq \frac{C'_b}{4}\, \frac{\alpha_q q^2}{i\omega}\,,
\quad
C'_b \approx -0.237\,,
\label{eqn:P_b_optical}
\end{equation}
which is consistent with\cite{Mishchenko2008mci} $C'_b = (8 - 3 \pi) / 6$.
Near the absorption threshold $|x - 1|\ll 1$,
we find the analytical expression
\begin{equation}
P_b(q,\omega) \simeq \frac{1}{6 \pi}\, \frac{\alpha_q q}{v_q}\,
\frac{1}{x - 1}\,
\left[\ln \left(\frac{8}{1 - x}\right) - 3\right],
\label{eqn:P_b_threshold}
\end{equation}
which agrees with the result of Ref.~\onlinecite{Gangadharaiah2008crf},
further providing the subleading divergent term (determined by the numerical factor inside the logarithm).

\section{Derivation}
\label{sec:derivation}

In this section we discuss how our results for the self-energy and vertex corrections have been derived. Within the Matsubara formalism~\cite{Mahan1990mpp}, the diagrams we compute are expressed by the integrals
\begin{widetext}
\begin{align}
P_a(q, i\Omega) &= -\frac{N}{\beta} \sum_{\nu} \int\frac{d^2{\bf k}}{(2\pi)^2}
{\rm tr}
[{\hat G}({\bf k},i\nu) {\hat G}({\bf k}+{\bf q},i\nu + i\Omega)
{\hat \Sigma}({\bf k}+{\bf q}) {\hat G}({\bf k}+{\bf q},i\nu + i\Omega)]\,,
\label{eqn:Pa}\\
P_b(q, i\Omega) &= -\frac{N}{\beta^2} \sum_{\nu,\nu'} \int\frac{d^2 {\bf k}}{(2\pi)^2} \frac{d^2 {\bf k}'}{(2\pi)^2}
V({\bf k}-{\bf k}')
{\rm tr}
[\hat{G}({\bf k},i\nu) \hat{G}({\bf k}+{\bf q},i\nu + i\Omega)
\hat{G}({\bf k}' + {\bf q}, i\nu' + i\Omega) \hat{G}({\bf k}',i\nu')]\,,
\label{eqn:Pb}
\end{align}
where $N = 4$ is the spectral degeneracy and the sums are performed over fermionic Matsubara frequencies $\nu = \pi (2 n + 1) \beta$.
Green's function $\hat{G}$, the self-energy matrix ${\hat \Sigma}$, and the Coulomb interaction kernel $V$ are given by
\begin{equation}
\hat{G}({\bf k}, i\nu) = (i\nu - v\,\mathbf{k} \cdot \bm{\sigma})^{-1}\,,
\quad
{\hat \Sigma}({\bf k}) = \frac{e^2}{4 \kappa} \mathbf{k}\cdot \bm{\sigma}
 \ln\frac{\Lambda}{|\mathbf{k}|}\,,
\quad
V(\mathbf{k})= \frac{2 \pi e^2}{\kappa |\mathbf{k}|}\,,
\label{eqn:V}
\end{equation}
where $\bm{\sigma} = \{\sigma_x, \sigma_y\}$ is the vector of the Pauli matrices.
To obtain results at real frequencies $\omega$
the analytical continuation $i\Omega \to \omega + i0$ has to be done at the end of the calculation, as usual.
In the zero temperature limit $\beta \to \infty$,
the integration over $\nu$ and the analytic continuation lead to
\begin{align}
2 P_a(q,\omega) &= 2 N\int\frac{d^2 \mathbf{k}}{(2\pi)^2}\,
\Sigma_k(1-\hat{n}_{\bf k} \cdot\hat{n}_{{\bf k}+{\bf q}})
\frac{E_{\mathbf{k},\mathbf{q}}^2 + \omega^2}
     {[E_{\mathbf{k},\mathbf{q}}^2 - (\omega + i0)^2]^2}\,,
\quad
E_{\mathbf{k},\mathbf{q}} \equiv v(|\mathbf{k}|+|\mathbf{k} + \mathbf{q}|)\,,
\quad
\hat{n}_{\mathbf{k}} \equiv \frac{\mathbf{k}}{|\mathbf{k}|}\,,
\label{eqn:Pa0}\\
P_b(q,\omega) &= -\frac{N}{2}\int\frac{d^2 \mathbf{k}}{(2\pi)^2}
\frac{d^2 \mathbf{k}'}{(2\pi)^2}
\frac{V(\mathbf{k} - \mathbf{k}')}{[E_{\mathbf{k},\mathbf{q}}^2 - (\omega + i0)^2]
                           [E_{\mathbf{k}',\mathbf{q}}^2 - (\omega + i0)^2]}
\Bigl\{
\omega^2 (\hat{n}_{\mathbf{k}} - \hat{n}_{\mathbf{k} + \mathbf{q}}) \cdot(\hat{n}_{\mathbf{k}'} - \hat{n}_{\mathbf{k}' + \mathbf{q}})
\notag\\
&+ E_{\mathbf{k},\mathbf{q}}E_{\mathbf{k}',\mathbf{q}}
[(1 - \hat{n}_{\mathbf{k}} \cdot \hat{n}_{\mathbf{k} + \mathbf{q}})
(1 - \hat{n}_{\mathbf{k}'} \cdot \hat{n}_{\mathbf{k}' + \mathbf{q}})
+ (\hat{n}_{\mathbf{k}} \times \hat{n}_{\mathbf{k} + \mathbf{q}}) \cdot
(\hat{n}_{\mathbf{k}'} \times \hat{n}_{\mathbf{k}' + \mathbf{q}})]
\Bigr\}\,.
\label{eqn:Pb1}
\end{align}
These integrals can be simplified by transformation from the Cartesian $(k_x, k_y)$ to the elliptic coordinate system $(\mu, \nu)$, where $0 \leq \mu < \infty$ and $0 \leq \nu < 2\pi$.
In this system the coordinate grid is made of ellipses and hyperbolas
with the foci at ${\bf k}=0$ and ${\bf k}=-{\bf q}$.
The transformation formulas
are $k_x + i k_y = (q / 2) [\cosh(\mu + i\nu) - 1]$ and  $d^2 k = (q / 2)^2 (\cosh^2\mu - \cos\nu)$.
The integral for $2 P_a$ becomes
\begin{equation}\label{eqn:Pa_elliptic}
2 P_a(q,\omega)=\frac{N\alpha q}{8\pi^2 v}\int d\mu d\nu
\left[\ln \left(\frac{2\Lambda}{q}\right) -
      \ln (\cosh \mu - \cos \nu) \right]
(\cosh \mu-\cos \nu)
\frac{4\cosh^2 \mu + (2\omega / v q)^2}{[4\cosh^2 \mu - (2\omega / v q + i0)^2]^2}\,,
\end{equation}
which can be evaluated analytically in terms of the dilogarithm function $\mathrm{Li}_2(z)$.
For $\omega > 0$ we get Eq.~\eqref{eqn:Pa2} with
\begin{equation}
\begin{split}
I_a(x) &= \frac{1}{3}\frac{1+2x^2}{1-x^2} - \frac{x}{6}\frac{5-2 x^2}{1-x^2}
\ln \left(\frac{1-x}{1+x}\right)
-\frac{\pi}{12}\frac{3-12 \ln 2 +6x^2-4x^4}{(1-x^2)^{3/2}}\\
&-\frac{i}{(1-x^2)^{3/2}}
\left[ \frac{\pi^2}{4}-\mathrm{Li}_2\left(x + i\sqrt{1-x^2}\,\right)
 + \mathrm{Li}_2\left(-x-i\sqrt{1-x^2}\,\right)
 + \frac{i\pi}{2}\,\ln\left(x + i\sqrt{1-x^2}\,\right)\right]\,.
\end{split}
\label{eqn:I_a}
\end{equation}
The vertex correction to the polarizability is represented by the integral in Eq.~\eqref{eqn:Pb1}.
Although not amenable to analytic evaluation, this integral can still be simplified by employing the elliptic coordinates:
\begin{equation}
\begin{split}
P_b(q,\omega) &=-\frac{Nq\alpha}{32 \pi^3 v}\, \int\frac{d\mu d\mu' d\nu d\nu'}{\sqrt{\cosh(\mu+\mu')-\cos(\nu+\nu')} \sqrt{\cosh(\mu-\mu')-\cos(\nu-\nu')}}\,
\frac{\cosh\mu\cosh\mu'\sin\nu\sin\nu'}{(\cosh^2 \mu - y^2)(\cosh^2 \mu' - y^2)}\\
&\times
[(\sin\nu\sin\nu'+\sinh\mu\sinh\mu')+x^2(\cosh\mu\cosh\mu'+\tanh\mu\tanh\mu'\cos\nu\cos\nu')]\,,
\label{eqn:P_b_IV}
\end{split}
\end{equation}
where $y = x + i0$.
Integrating over $\nu$ and $\nu'$, one is lead to Eq.~\eqref{eqn:P_b_II} with $I_b$ given by the two-dimensional integral
\begin{gather}
I_b(y) = \frac{N}{\pi^3}\int_0^{\infty}\int_0^{\infty}
\frac{da db F(a, b)[\cosh a + (1 + 2 y^2) \cosh b]}
     {[1 - 2 y^2 + \cosh(a + b)] [1 - 2 y^2 + \cosh(a - b)]}\,,
\label{eqn:Ib}\\
F(a, b) = \frac{\cosh(a / 2)}{\cosh(b / 2)} \left\{ [K_b \cosh b - E_b (1 + \cosh b) ] E_a
- \frac{1}{3}\, [-E_a \cosh a + K_a(\cosh a - 1)] K_b \right\}\,,
\label{eqn:F}\\
K_\tau = K\left(\operatorname{sech}^2 \frac{\tau}{2}\right)\,,
\quad
E_\tau = E\left(\operatorname{sech}^2 \frac{\tau}{2}\right)\,,
\label{eqn:KE}
\end{gather}
\end{widetext}
where $K(z)$ and $E(z)$ denote the complete elliptic integrals of the first and the second kind.

For $x < 1$, the calculation of $P_b$ using these formulas is easily done numerically
and the result is real.
For $x > 1$, the standard numerical quadrature routines fail because the integration path in Eq.~\eqref{eqn:Ib} passes near the zeros of the denominator.
For such $x$, one can derive the following alternative formulas for $P_b$.
Denoting $x = \cosh(\rho/2)$, where $\rho > 0$ is real,
we see that the poles of the integrand are at $a_{1, 2} = \rho \mp b$.
The result for $I_b$ is complex, with the principal value integral (denoted by $\mathcal{P}$) giving the real part. The $a$ integral can be computed as the integral over contour depicted by the dashed line in Fig.~\ref{fig:contour}, which can be deformed into the union of contours $C_1$ and $C_2$. The real part of the integrand vanishes on $C_2$,
and thus only $C_1$ contributes to $\mathrm{Re}\, I_b$.

\begin{figure}
\begin{center}
\includegraphics[width=2.5in]{pol1cor_Fig3}
\end{center}
\caption{Integration contours in the complex $a$ plane to evaluate $\operatorname{Re} I_b(x)$ for $x>1$ in Eq.~\eqref{eqn:Ib}.
The dots indicate the poles of the integrand.
\label{fig:contour}
}
\end{figure}

In turn, the imaginary part of $I_b$ can be obtained using the Sokhotski--Plemelj identity
\begin{equation}
\frac{1}{z \pm i 0} = \mathcal{P}\,\frac{1}{z} \mp i\pi \delta(z)\, ,
\label{eqn:Plemelji}
\end{equation}
followed by simple algebraic manipulations.
In the end, we obtain the alternative formulas for the real and imaginary parts of $I_b$ suitable for $x > 1$:
\begin{widetext}
\begin{align}
\mathrm{Re}\, I_b(x + i0) &= \int_0^\infty db\int_0^\pi \frac{du}{\cosh \rho -\cos u}
\mathrm{Im}\, \left[\frac{F_s(b + iu, b)}{\cosh(2b + iu)-\cosh\rho}\right]\,,
\label{eqn:reIb}\\
\mathrm{Im}\, I_b(x + i0) &= \frac{1}{\pi^2 \sinh \rho \sinh(\rho/2)}
\int_0^\infty \frac{d\mu}{\sinh \mu}
\left[\frac{F_s(\mu + \rho,\rho)}{\cosh(\mu + \rho/2)} - \frac{F_s(\mu - \rho,\rho)}{\cosh(\mu - \rho/2)}\right]\,,
\label{eqn:ImIb}\\
F_s(a, b) &= F(a, b) [\cosh a +(\cosh \rho + 2) \cosh b]+ (a \leftrightarrow b)\,,
\quad
\rho = 2 \ln \left(x + \sqrt{x^2 - 1}\,\right)\,.
\end{align}
\end{widetext}
Straightforward numerical evaluation and asymptotic analysis of these expressions lead
to the results presented in Sec.~\ref{sec:results} above.

Since all the above calculations have been done within the Dirac approximation,
it is instructive to discuss how they can be generalized to a more realistic lattice model.
We will consider in particular the self-energy term (the vertex correction can be analyzed similarly).
We will show that the correction calculated from the Dirac model, Eq.~\eqref{eqn:Pa0},
and that computed from the lattice model is vanishingly small in the limit of interest,
$\{\omega \ll v / |\bm{a}|, q \ll 1/ |\bm{a}|\}$,
where $\bm{a}$ is the vector connecting a pair of nearest lattice sites.

Let the kinetic energy matrix on the lattice be
$\hat{H} = \mathcal{H}_{\mathbf{k}}\cdot \bm{\sigma}$, where $\mathcal{H}_{\mathbf{k}}=(\mathcal{H}^x_{\mathbf{k}},\mathcal{H}^y_{\mathbf{k}})$, and
let the on-site density distributions be described by a form-factor
function $\mathcal{F}(\mathbf{q})$,
which is close to unity at $q \ll 1 / b$ and rapidly decays to zero at $q \gg 1 / b$,
where $b \sim a$ is the characteristic size of the site orbitals.
In terms of these notations
the self-energy on the lattice
$\hat{\Sigma}_{\mathbf{q}}=\mathbf{\Sigma}_{\mathbf{q}} \cdot \bm{\sigma}$, with $\mathbf{\Sigma}_{\mathbf{q}}=(\Sigma^x_{\mathbf{q}},\Sigma^y_{\mathbf{q}})$,
is given by the equation
\begin{widetext}
\begin{equation}
\Sigma^x_\mathbf{q} + i \Sigma^y_\mathbf{q}
= \int\frac{d^2 \mathbf{k}}{(2\pi)^2}
\mathcal{V}({\bf q}-{\bf k}) e^{i(\mathbf{q} - \mathbf{k}) \cdot \bm{a}} (\hat{\mathcal{H}}^x_{\mathbf{k}} + i \hat{\mathcal{H}}^y_{\mathbf{k}}) \,,
\quad \mathcal{V}(\mathbf{q})
=\frac{2 \pi e^2}{\kappa |\mathbf{q}|}
\mathcal{F}(\mathbf{q})\,,\quad
\hat{\mathcal{H}}_{\mathbf{k}} \equiv \frac{\mathcal{H}_\mathbf{k}}{|\mathcal{H}_\mathbf{k}|} \,.
\label{eqn:Sigma_lat}
\end{equation}
Note that the form-factor $\mathcal{F}(\mathbf{q})$ regularizes the \emph{short-range}
behavior of the interaction potential
and serves as a cutoff on the momentum transfer introduced in
Refs.~\onlinecite{Mishchenko2008mci, Sheehy2009oto}.
Note also that the integration in Eq.~\eqref{eqn:Sigma_lat} is over the entire momentum space to account for Umklapp processes.

From Eq.~\eqref{eqn:Sigma_lat} we see that the self-energy in the lattice model and the Dirac model [Eq.~\eqref{eqn:Sigma_q}] have the same functional form if the deviation $\delta \mathbf{k}$ of $\mathbf{k}$ from a corner of the BZ $\mathbf{K}$ is small:
\begin{equation}
\mathbf{\Sigma}_{\mathbf{K}+\delta\mathbf{k}}
= \frac{e^2}{4\kappa}\, \delta\mathbf{k}
\ln \left|\frac{\Lambda_{\text{lat}}}{\delta \mathbf{k}}\right|
\left[1 + \mathcal{O}\left(\left|\frac{\delta \mathbf{k}}{\Lambda_{\text{lat}}} \right|\right)\right]\,,
\quad
\Lambda_{\text{lat}} \sim \frac{1}{a}\,.
\end{equation}
The lattice analog of Eq.~\eqref{eqn:Pa0} is
\begin{equation}
2 P^{\mathrm{lat}}_a(q,\omega)
= N_s \int_{\mathrm{BZ}}\frac{d^2 \mathbf{k}}{(2\pi)^2} \mathbf{\Sigma}_{\mathbf{k}}\cdot(\hat{\mathcal{H}}_{\mathbf{k}}
- \hat{\mathcal{H}}_{\mathbf{k} + \mathbf{q}})
\frac{\mathcal{E}_{\mathbf{k},\mathbf{q}}^2 + \omega^2}
     {[\mathcal{E}_{\mathbf{k},\mathbf{q}}^2 - (\omega + i0)^2]^2}\,,
\quad
\mathcal{E}_{\mathbf{k},\mathbf{q}} \equiv |\mathcal{H}_\mathbf{k}|+|\mathcal{H}_{\mathbf{k} + \mathbf{q}}|\,,
\label{eqn:P_a_lat}
\end{equation}
\end{widetext}
where the integral is now taken over a Brillouin zone (BZ) and $N_s = 2$ accounts for the spin degeneracy only.
It is easy to see that this integral is convergent
and that the difference between Eq.~\eqref{eqn:Pa0} and Eq.~\ref{eqn:P_a_lat} is of the order of $(q / \Lambda_{\text{lat}}) \ln (q / \Lambda_{\text{lat}})$.
This difference is vanishingly small in the limit of interest, as we stated above.

\section{Discussion}
\label{sec:discussion}

In this section we compare our work with previous literature and discuss some effects beyond the first-order perturbation theory.

The interaction correction to the static dielectric function $\epsilon(q, 0)$ has been considered in Ref.~\onlinecite{Kotov2008eei}.
Our Eq.~\eqref{eqn:epsilon_dc} for this quantity differs from the result obtained therein by the numerical coefficient of the quadratic term,
$0.778$ versus $0.53$.
This discrepancy is for two reasons.
First is the apparent error in the numerical evaluation of the vertex diagram in Ref.~\onlinecite{Kotov2008eei}.
Second is the different treatment of the self-energy contribution.
In Ref.~\onlinecite{Kotov2008eei}, this contribution is assumed to be completely absorbed into the velocity renormalization.
In our renormalization scheme, only the first term in Eq.~\eqref{eqn:Pa2} is absorbed while the second leaves a finite remainder, Eq.~\eqref{eqn:Pa_dc}.
This difference is not merely a matter of convention.
It has to do with the principal distinction regarding the relations between observable and nonobservable quantities.
While the former relations do not depend on the renormalization scheme, the latter do.
In our case,
the expansion of the static dielectric function $\epsilon(q, 0)$ in powers of $\alpha_q$ (defined by the quasiparticle phase velocity $v_q$) is unique and given by Eq.~\eqref{eqn:epsilon_dc}.
On the other hand, the expansion of $\epsilon(q, 0)$ in powers of the ``bare'' coupling $\alpha$ has the form
\begin{equation}
\epsilon(q, 0) \simeq 1 + \frac{\pi}{2}\, \alpha + \left(0.778 - \frac{\pi}{8}\, \ln \frac{\Lambda}{q}\right) \alpha^2\,,
\label{eqn:epsilon_dc_II}
\end{equation}
in which the coefficient for $\alpha^2$ depends on the nonuniversal cutoff parameter $\Lambda$.
Clearly, this coefficient can be discussed only after the renormalization scheme is precisely defined,
as we have done here.

Next, the optical limit of the polarization function has been vigorously debated~\cite{Mishchenko2008mci, Sheehy2009oto, Katsnelson2008opo, Juricic2010coi, Giuliani2012erg} in the context of the interaction correction to the universal conductivity of graphene. Our method of calculation and the final result, Eq.~\eqref{eqn:sigma_I},
are the same as in Ref.~\onlinecite{Mishchenko2008mci}.
The interaction correction to the optical conductivity of \emph{doped} graphene
has been studied in Ref.~\onlinecite{Abedinpour2011dwp}.
In the limit $\omega \gg \mu$,
where $\mu$ is the chemical potential measured with respect to the Dirac point,
one expects to recover the behavior characteristic of neutral graphene.
In this limit the results of Ref.~\onlinecite{Abedinpour2011dwp} agree with those of Ref.~\onlinecite{Mishchenko2008mci} and therefore with ours.
The origin of the discrepancy with Ref.~\onlinecite{Juricic2010coi}
is discussed in Appendix~\ref{sec:optical}.

Finally, the behavior of $P(q, \omega)$ near the absorption threshold $|\omega/{v_q q} - 1|\ll 1$
has been studied by the authors of Ref.~\onlinecite{Gangadharaiah2008crf} (henceforth GFM).
In this special region the perturbative expansion of the dielectric function diverges, see Fig.~\ref{fig:threshold}.
The divergence of the self-energy term is stronger, $\sim (1 - x^2)^{-3 / 2}$ [Eq.~\eqref{eqn:Pa_theshold}] than that $\sim (1 - x)^{-1} \ln (1 - x)$ [Eq.~\eqref{eqn:P_b_threshold}] of the vertex term.
However, GFM argued that the divergence of the self-energy term can be trivially absorbed into the velocity renormalization thus leaving only the divergence of the vertex.
Summing the ladder series for the vertex corrections,
they obtained a nonperturbative expression for $\epsilon(q, \omega)$.
This expression is also plotted in Fig.~\ref{fig:threshold} assuming the renormalized velocity of GFM coincides with $v_q$.

\begin{figure}
\begin{center}
\includegraphics[width=2.8in]{pol1cor_Fig4}
\end{center}
\caption{(Color online)
The real part of the dielectric function near the absorption threshold for $\alpha_q = 0.3$.
The black solid line is our first-order theory,
the red dash-dotted line is the ladder sum from Ref.~\onlinecite{Gangadharaiah2008crf}.
\label{fig:threshold}
}
\end{figure}

An important qualitative prediction of GFM theory is vanishing of $\mathrm{Re}\, \epsilon$ at certain $x < 1$,
see Fig.~\ref{fig:threshold}, which signals the presence of a new collective mode --- ``excitonic plasmon.''
While we find this prediction very interesting,
we wish to express some reservations in the validity of GFM approach.
We believe that the nonperturbative treatment should begin with the resummation of the self-energy not the vertex term because the former is more divergent.
Such a resummation would make the quasiparticle velocity $v_q$ momentum-dependent, cf.~Eq.~\eqref{eqn:vel_I}.
In other words, the linear Dirac spectrum would be replaced by a spectrum with a finite curvature.
As in the case of the static response, it is not possible to faithfully represent the effect of such a curvature
by simply replacing $v$ with another constant number.
It is easy to see that the
finite curvature of the spectrum modifies the behavior of the dielectric function over the range of frequencies $\sim \alpha q$,
which is much wider than the interval $\alpha^2 q$ where the higher-order terms considered by GFM are important,
see also Fig.~\ref{fig:threshold}.
Our preliminary analysis suggests that this significantly modifies the analytical structure of the ladder sum compared to what was obtained by GFM.
This intriguing problem warrants further study.

Our work is supported by the Grants NSF PHY11-25915, by UCOP (M.M.F.), by the KITP Graduate Fellows Program (I.S.), the Welch Foundation Grant No. TBF1473 (I.S.) and the NRI SWAN program (I.S.). We are grateful to the KITP at UCSB, where this work has been carried out, for hospitality. We are thankful to I.~Herbut, P.~Koroteev, V.~Kotov, V.~Mastropietro, E.~Mishchenko, M.~Polini, O.~Vafek, and B.~Uchoa for discussions and comments on the manuscript. We thank M.~Vozmediano for bringing Ref.~\onlinecite{Teber} to our attention.


\section{Approach}
\label{sec:approach}
We elaborate our approach to the query-focused video summarization in this section. Denote by ${\cal V} = \{{\cal V}_t\}_{t=1}^T$ a video that is partitioned to $T$ segments, and by $q$ the query about the video. In our experiments, every segment ${\cal V}_t$ consists of $10$ video shots each of which is 5-second long and is used in Section~\ref{subsec:GS} to collect the concept annotations.

\begin{figure*}[t]
	\includegraphics[width=\textwidth]{./Framework}
	\caption{\small{Our query-focused video summarizer: Memory network (right) parameterized sequential determinantal point process (left).}}
	\label{fig:framework}
	\vspace{-10pt}
\end{figure*}

\subsection{Query Conditioned Sequential DPP}
The sequential determinantal point process (DPP)~\cite{gong2014diverse} is among the state-of-the-art models for generic video summarization. We condition it on the query $q$ as our overarching video summarization model,
\begin{align}
&P(Y_1=\bm{y}_1, Y_2 = \bm{y}_2, \cdots, Y_T=\bm{y}_T | \mathcal{V}, q) \\
= &P(Y_1=\bm{y}_1 | \mathcal{V}_1, q)\prod_{t=2}^T P(Y_t = \bm{y}_t | \mathcal{V}_t, \bm{y}_{t-1}, q)
\end{align}
where the $t$-th DPP variable $Y_t$ selects subsets from the $t$-th segment ${\cal V}_t$, i.e., $\bm{y}_t\subseteq{\cal V}_t$, and the distribution $P(Y_t = \bm{y}_t | \mathcal{V}_t, \bm{y}_{t-1}, q)$ is specified by a conditional DPP~\cite{kulesza2012determinantal},
\begin{align}
P(Y_t = \bm{y}_t | \mathcal{V}_t, \bm{y}_{t-1}, q) = \frac{\det [\bm{L}(q)]_{\bm{y}_t\cup\bm{y}_{t-1}}}{\det\big(\bm{L}(q) + \bm{I}_t\big)}.
\end{align}
The nominator on the right-hand side is the principle minor of the (L-ensemble) kernel matrix $\bm{L}(q)$ indexed by the subsets $\bm{y}_t\cup\bm{y}_{t-1}$. The denominator calculates the determinant of the sum of the kernel matrix and a corrupted identity matrix whose elements indexed by $\bm{y}_{t-1}$ are 0's. Readers are referred to the great tutorial~\cite{kulesza2012determinantal} on DPP for more details.

Note that the DPP kernel $\bm{L}(q)$ is parameterized by the query $q$. We have to carefully devise the way of parameterizing it in order to take account of the following properties. In query-focused video summarization,  a user selects a shot to the summary for two possible reasons. One is that the shot is quite related to the query and thus becomes appealing to the user. The other may attribute to the contextual importance of the shot; e.g., the user would probably  choose a shot to represent a prominent event in the video even if the event is not quite relevant to the query. To this end, we use a memory network to model the two types of importance (query-related and contextual) of a video shot simultaneously.

\eat{
In this section, we briefly describe the state-of-the-art in query-focused video summarization, depicting their shortcomings. Subsequently, we identify and explain the building blocks of our pipeline followed by a novel framework that addresses the limitations and outperforms the existing work.

\subsection{Query-Focused Video Summarization}
\label{subsec:qfvs}
Video summarization is highly subjective; given a video, different users may summarize it differently. The reason behind this is twofold: 1) each user finds some events (collection of concepts) more important and appealing than other users, or 2) some concepts are main focus of certain videos, e.g. given a video of an \textit{auto show} has to be summarized in a way that is collective mainly around different cars. Therefore in ~\cite{sharghi2016query}, authors designed the first framework for query-focused video summarization to address different user needs and also content-aware video indexing. They use a hierarchical graphical model which requires both static and dynamic features. The first stage in their approach is to generate \textit{query-dependent} (dynamic) from static features, that are high-level SentiBank ~\cite{borth2013sentibank} concept scores for all dictionary elements. The scheme employed to acquire the dynamic feature set is a re-weighting function that scales down the concept detection scores for the concepts that are not present in the user preferences list while preserving full score for those who user has indicated to be interested in. The dynamic feature set is then fed to the first summarizer unit to acquire \textit{query-relevant} summary of the video. At the second stage, static features are used to \textit{complement} the summary from the first stage by adding parts of the video that may not correspond to any query, but are import in the context of the video; hence, both the \textit{query} and the \textit{story} of the video is maintained in the system summary. Even though this approach is able to generate descent summaries, we argue the existence of the following unappealing constraints:\\
\begin{itemize}
	\item Mandating high-level concept detectors as features
	\item Restricted query representation
	\item High supervision in training the model
\end{itemize}

}

\eat{
Requiring high concept detectors as features is very limiting, mainly because itself is an open problem in computer vision; it is not easy to train high-level detectors that work well in unconstrained settings (such as an untrimmed egocentric video). Additionally, adding any item to the dictionary means one needs to either find or train a detector, which in various cases is extremely hard. As for the query, ideally, it must be free entry text, such that a user can enter words or sentences that describes their desire instead of using limited elements in a predefined dictionary. It is hard to augment the approach in ~\cite{sharghi2016query} to address this issue. Finally, training the model requires high supervision; one not only needs to know the groundtruth summaries, but they must be able to infer which elements in the groundtruth summary is query-relevant/irrelevant. In this work we propose an attention-based query-focused video summarization approach, that addresses the limitations mentioned above, while pushing the performance drastically.

\subsection{Background}
In this section, we provide some background information that facilitates understanding our proposed approach. Our Attention-based Query-Focused video summarization approach consists of two units working jointly to generate summaries. At the first step, a memory network cell receives the shot features as well as the query, and generates a query-dependent feature that is then fed to a determinantal point process ~\cite{kulesza2012determinantal} (DPP) summarizer unit. In what follows, we first explain how memory network generates query-dependent features, and subsequently we review basics of DPP unit.
}

\subsection{Memory Network to Parameterize DPP Kernels}
The memory network~\cite{sukhbaatar2015end} offers a neural network architecture to naturally attend a question to ``facts'' (cf.\ the rightmost panel of Figure~\ref{fig:framework}). In our work, we shall measure the relevance between the query $q$ and a video shot and incorporate such information into the DPP kernel $\bm{L}(q)$. Therefore, it is straightforward to substitute the question in memory network by our query, but the ``facts'' are less obvious.

As discussed in Section~\ref{subsec:Dict}, there could be various scenarios for a query and a shot. All the query concepts may appear in the shot but possibly in different frames; one or two concepts of the query may not be present in the shot; it is also possible that none of the concepts are relevant to any frame in the shot. In other words, the memory network is supposed to screen all the video frames in order to determine  the shot's relevance to the query. Hence, we uniformly sample 8 frames from each shot as the ``facts''.  The video frames are represented using the same feature as~\cite{sharghi2016query} (cf.\ $\bm{f}_1,\cdots,\bm{f}_K$ on the rightmost panel of Figure~\ref{fig:framework}).

The memory network takes as input the video frames $\{\bm{f}_k\}$ of a shot and a query $q$. The frames are transformed to memory vectors $\{\bm{m}_k\}$ through an embedding matrix $A$. Similarly, the query ${q}$, represented by a binary indication vector, is mapped to the internal state $\bm{u}$ using an embedding matrix $C$. The attention scheme is implemented simply by a dot product followed by a softmax function,
\begin{equation}
p_k = \text{Softmax}(\bm{u}^T\bm{m}_k),
\end{equation}
where $p_k$ carries how much attention the query $q$ incurred over the frame $\bm{f}_k$.

Equipped with the attention scores $\{p_k\}$, we assemble another embedding $\{\bm{c}_k\}$ of the frames, obtained by the mapping matrix $B$ in figure~\ref{fig:framework}, into the video shot representation  $\bm{o}$:
\begin{equation}
\bm{o} = \sum_{k}p_i\bm{c}_k,
\end{equation}
which is conditioned on the query $q$ and entails the relevance strength of the shot to the query. As a result, we expect the DPP kernel parameterized by the following
\begin{equation}
[\bm{L}(q)]_{ij} = \bm{o}_i^T {D}^T {D} \bm{o}_j \label{eWeightCombine}
\end{equation}
is also flexible in modeling the importance of the shots to be selected into the video summary. Here $i$ and $j$ index two shots, and ${D}$ is another embedding matrix. Note that the contextual importance of a shot can be inferred from a shot's similarities to the others by the kernel matrix, while the query-related importance is mainly by the attention scheme in the memory network.

\subsection{Learning and Inference}
We learn the overall video summarizer, including the sequential DPP and the memory network, by maximizing the log-likelihood of the user summaries in the training set. We use stochastic gradient descent with mini-batching to optimize the embedding matrices $\{A, B, C, D\}$. The learning rates and numbers of epochs are chosen using the validation set. At the test stage, we sequentially visit the video segments ${\cal V}_1, \cdots, {\cal V}_T$ and select shots from them using the learned summarization model.

It is notable that our approach requires less user annotations than the SH-DPP~\cite{sharghi2016query}. It learns directly from the user summaries and implicitly attend the queries to the video shots. However, SH-DPP requires very costly annotations about the relevances between video shots and queries. Our new dataset does supply such supervisions, so we shall include SH-DPP as the baseline method in our experiments.

\eat{
In \textit{question-answering} frameworks, the response vector is subsequently passed to an answering module (e.g. softmax) to infer the answer to the question.

\subsubsection{Determinantal Point Process}
\label{subsec:dpp}
\textbf{DPP} ~\cite{kulesza2012determinantal} has been introduced and used in order to model negative correlation. DPP fits summarization tasks as it is able to integrate two components closely; individual importance, and collective diversity. Vanilla DPP was used effectively for document summarization, achieving the state-of-the-art.

Define the groundset by $\cal Y$ $= \{1,2,...,N\}$. A DPP defines a probability distribution over a subset selection random variable by:

\begin{align}
P(Y=y) = {\det(\mat{L}_y)}/{\det(\mat{L}+\mat{I})}, \quad \forall y\subseteq \cal Y,  \label{eDef}
\end{align}
}

\eat{
where $\mat{L} \in \mathbb{S}^{N \times N}$ is a positive semidefinite kernel matrix, $\mat{I}$ is identity matrix, and $\mat{L}_y$ is the submatrix indexed by $y$. The individual importance of an item in the set is represented by $P(i \in y) \propto \mat{L}_{ii}$ while the repulsion of any two items can be inferred by $P(i,j \in y) \propto \mat{L}_{ii}\mat{L}_{jj} - \mat{L}^2_{ij}$. In other terms, DPP assigns the highest probability to a subset which best spans the groundset while preserving the diversity in the selected elements.

DPP has been successfully used in document-summarization tasks, achieving state-of-the-art performance ~\cite{chao2015large,kulesza2012determinantal}. ~\cite{gong2014diverse,sharghi2016query} succeeded in employing DPP for video summarization. Inspired by them, we use a conditional version of DPP as the summarizer unit in our pipeline.

\subsection{Framework}
As explained in \ref{subsec:qfvs}, the framework in ~\cite{sharghi2016query} uses a re-weighting scheme in order to generate query-dependent features, however, this introduces unappealing limitations; 1) one has to have access to detectors for every concept in the dictionary, 2) hard restriction over query representation, and 3) hierarchical supervision required to train the model. Our novel \textbf{attention-based} DPP summarizer heals these problems as it is able to: 1) latently learn to distinguish concepts based on low-level cues in the shot, hence no restriction over the input features, 2) query embedding layer in the memory network can be easily swapped with a recurrent neural network (such as LSTM) that is able to transform free text into rich representations, and 3) model is trained end-to-end, and having one layer of DPP summarizer lowers the supervision required to train the model.

Figure \ref{fig:framework} shows our proposed method unrolled in time. The framework consists of two elements: attention-based memory network unit and Determinantal Point Process (DPP) summarizer unit. Memory network unit (that is shown on the right side of Figure ~\ref{fig:framework}) takes as input the feature set of a shot $\mat{f}_1 ...\mat{f}_k$ and the query $\mat{q}$, embeds them into the same space via embedding matrices $\mat{A}$ and $\mat{C}$ respectively. Obviously, these two embedding matrices share the size of their first dimension, so that embedded features have the same length. This allows our approach to receive a feature set and a query that are not initially in the same space, letting it accept features of any nature. In the figure, embedded features are shown by $\mat{m}_i$, and the query $\mat{q}$ after transformation via embedding matrix $\mat{C}$ is shown by $\mat{u}$. In this space, the inner product of $\mat{m}_i$ and $\mat{u}$ is meaningful, and represents how well the $\text{i}^\text{th}$ feature of the shot correlates with $\mat{u}$. Having computed the inner products of all features of the shot with $\mat{u}$, we convert them into probability using a softmax layer, shown by $p_i$. The softmax layer represents the attention in the framework; if one of the features in the set has high correlation with $\mat{u}$, then softmax assigns a value of close to $1$ to it while giving much less \textit{attention} to the rest of the features. Probabilities $p_i$ are then used to combine $\mat{c}_i$ to generate $\mat{o}$, that are transformed input features via another embedding matrix $\mat{B}$. Hence, the memory network is able to convert static feature set of a shot into \textit{one} feature vector based on their correlation with the query. By changing the size of embedding matrix $\mat{B}$, one can control the output feature dimension, that is another advantage to our approach.

Having acquired query-relevant features $\mat{o}_i$, at the next step they are fed into the summarizer, that is a conditional DPP unit. At time step $t$, the groundset is defined as, $\cal Y$$_t = \{\mat{o}_i,...,\mat{o}_{i+N}\} \cup \{Y_{t-1} = y_{t-1}\}$, where $N$ represents the number of shots to select from and $y_{t-1}$ is the subset selection output from time step $t-1$. This is to enforce Markov diversity between two consecutive time step to ensure less redundancy:

\begin{equation}
P(Y_t = y_t) = \frac{\det \mat{L}_{y_t \cup y_{t-1}}}{\det (\mat{L} + \mat{I}_t)}
\end{equation}

where $\mat{L}$ represents the kernel, $Y_t$ being the subset selection random variable, $\mat{L}_{y_t \cup y_{t-1}}$ indicating the square sub-kernel whose columns and rows are indexed by $y_t \cup y_{t-1}$, and $\mat{I}_t$ representing an identity like with the exception of having zero diagonal elements indexed by $y_{t-1}$.

}

\section{Conclusion}
\label{sec:conc}
\vspace{-5pt}

In this work, our central theme is to study the \textit{subjectiveness} in video summarization. We have analyzed the key challenges caused the subjectiveness and proposed some solutions. In particular, we compiled a dataset that is densely annotated with a comprehensive set of  concepts and designed a novel evaluation metric that benefits from the collected annotations. We also devised a new approach to generating personalized summaries by taking user queries into account. We employed memory networks and determinantal point processes in our summarizer, so that our model leverages their attention schemes and diversity modeling capabilities, respectively. Extensive experiments verify the effectiveness of our approach and reveals some nice behaviors of our evaluation metric.
\vspace{-10pt}
\paragraph{Acknowledgements.}
This work is supported by NSF IIS \#1566511, a gift from Adobe Systems, and a GPU from NVIDIA. We thank Fei Sha, the anonymous reviewers and area chairs, especially R2, for their insightful suggestions.
\section{Dataset}
\label{sec:dataset}

In this section, we provide the details on compiling a comprehensive dataset for video summarization. We opt to build upon the currently existing UT Egocentric (UTE) dataset~\cite{lee2012discovering} mainly for two reasons: 1) the videos are consumer grade, captured in uncontrolled everyday scenarios, and 2) each video is 3--5 hours long and contains a diverse set of events, making video summarization a naturally desirable yet challenging task. In what follows, we first explain how we define a dictionary of concepts and determine the best queries over all possibilities for the query-focused video summarization. Then we describe the procedure of gathering user summaries for the queries. We also show informative statistics about the collected dataset.

\begin{figure*}
	\centering
	\includegraphics[width=\linewidth]{./shot_tags}
	\vspace{-18pt}
	\caption{\small{All annotators agree with each other on the prominent concepts in the video shot, while they miss different subtle concepts. }} 	\label{fig:tags}
	\vspace{-2pt}
\end{figure*}

\subsection{Concept Dictionary and Queries}
\label{subsec:Dict}
We plan to have annotators to transform the semantic information in each video shot to a binary semantic vector (cf.\ Figures~\ref{fig:captionvstags} and \ref{fig:tags}), with 1's indicating the presence of the corresponding concepts and 0's the absence. Such annotations serve as the foundation for an efficient and automatic evaluation method for video summarization described in Section~\ref{subsec:tag}. The key is thus to have a dictionary that covers a wide range and multiple levels of concepts, in order to have the right basis to encode the semantic information.

\eat{
\begin{table}[t]
	\centering
	\begin{small}
		\begin{tabular}{ll}
			\hline \toprule
			\multicolumn{1}{c}{\normalsize{Concepts}} \\
			\hline
			Baby, ~Beach, ~Bed, ~Blonde, ~Boat, ~Book, ~Building, ~Car, Chair\\
			Chocolate, ~Computer, Cup/glass, ~Dance, Desk, Drink, Exercise\\
			Face, ~Flower, ~Food, ~Garden, ~Grass, ~Hall, ~Hands, ~~Hat, ~Kids\\
			Lady, ~Lake, ~Market, ~Men, ~Musical/instrument, ~Ocean, ~Office\\
			Park, ~Party, ~Pets/animal, ~Phone, ~Room, ~School, ~Shoes, ~Sign\\
			Sky, ~Sports, ~Street, ~Student, ~Sun, ~Toy, ~Tree, ~Window\\
			\bottomrule
		\end{tabular}
	\end{small}
	\caption{\small{Dictionary of concepts}}
	\label{table:dict}
	\vspace{-10pt}
\end{table}
}

In~\cite{sharghi2016query}, we have constructed a lexicon of concepts by overlapping nouns in the video shot captions~\cite{yeung2014videoset} with those in the SentiBank~\cite{borth2013sentibank}. Those nouns serve as a great starting point for us since they are mostly entry-level~\cite{ordonez2013large} words. We prune out the concepts that are weakly related to visual content (e.g., \textsc{''Area''}, which could be interpreted in various ways and applicable to most situations). Additionally, we merge the redundant concepts such as \textsc{''Children''} and \textsc{''Kids''}. We also add some new concepts in order to construct an expressive and comprehensive dictionary. Two strategies are employed to find the new concept candidates. First, after watching the videos, we manually add the concepts that appear for a significant frequency, e.g., \textsc{''Computer''}. Second, we use the publicly available statistics about YouTube and Vine search terms to add the terms that are frequently searched by users, e.g., \textsc{''Pet/Animal''}. The final lexicon is a concise and diverse set of 48 concepts (cf.\ Figure~\ref{fig:stats}) that are deemed to be comprehensive for the UTE videos of daily lives.

\begin{figure*}[t]
	\centering
	\includegraphics[width=\linewidth]{./Summary_Comparison}
		\vspace{-18pt}
	\caption{\small{Two summaries generated by the same user for the queries $\{\textsc{Hat},\textsc{Phone}\}$ and $\{\textsc{Food},\textsc{Drink}\}$, respectively. The shots in the two summaries beside the green bars exactly match each others, while the orange bars show the query-specific shots.}}
	\label{fig:sumcompare}
\vspace{-10pt}
\end{figure*}

We also construct queries, to acquire query-focused user summaries, using two or three concepts as opposed to singletons. Imagine a use case of video search engines. The queries entered by users are often more than one word. For each video, we formalize 46 queries. They cover the following four distinct scenarios: i) all the concepts in the query appear in the same video shots together (15 such queries); ii) all concepts appear in the video but never jointly in a single shot (15 queries), iii) only one of the concepts constituting the query appears in some shots of the video (15 queries), and iv) none of the concepts in the query are present in the video (1 such query). We describe in the Suppl.\ Materials how we obtain the 46 queries to cover the four scenarios. Such queries and their user annotated summaries challenge an intelligent video summarizer from different aspects and extents.

\subsection{Collecting User Annotations}
\label{subsec:GS}

We plan to build a video summarization dataset that offers 1) efficient and automatic evaluation metrics and 2) user summaries in response to different queries about the videos. For the former 1), we collect user annotations about the presence/absence of concepts in each video shot. This is a quite daunting task conditioning on the lengths of the videos and the size of our concept dictionary. We use Amazon Mechanical Turk (MTurk) (\url{http://www.mturk.com/}) for economy and efficiency considerations. For the latter 2), we hire three student volunteers to have better quality control over the labeled video summaries. We uniformly partition the videos to 5-second-long shots.

\vspace{-5pt}
\subsubsection{Shot Tagging: Visual Content to Semantic Vector}
\label{subsec:tag}
\vspace{-5pt}
We ask MTurkers to tag each video shot with all the concepts that are present in it. To save the workers' time from watching the shots, we uniformly extract five frames from each shot. A concept is assumed relevant to the shot as long as it is found in any of the five frames. Figure ~\ref{fig:tags} illustrates the tagging results for the same shot by three different workers. While all the workers captured the prominent concepts like \textsc{Sky}, \textsc{Lady}, \textsc{Street}, \textsc{Tree}, and \textsc{Car},  they missed different subtle ones. The union of all their annotations, however, provides a more comprehensive semantic description about the video shot than that of any individual annotator. Hence, we ask three workers to annotate each shot and take their union to obtain the final semantic vector for the shot. On average, we have acquired $4.13$, $3.95$, $3.18$, and $3.62$ concepts per shot for the four UTE videos, respectively. In sharp contrast, the automatically derived concepts~\cite{sharghi2016query} from the shot captions~\cite{yeung2014videoset} are far from enough; on average, there are only $0.29$, $0.58$, $0.23$, and $0.26$ concepts respectively associated with each shot of the four videos.
\vspace{-13pt}

\paragraph{Evaluating video summaries.} Thanks to the dense concept annotations per video shot, we can conveniently contrast a system generated video summary to user summaries according to the semantic information they entail. We first define a similarity function between any two video shots by intersection-over-union (IOU) of their corresponding concepts. For instance, if one shot is tagged by \{\textsc{Car}, \textsc{Street}\} and another by \{\textsc{Street}, \textsc{Tree}, \textsc{Sign}\}, then the IOU similarity between them is ${1}/{4} = 0.25$.

To find the match between two summaries, it is convenient to execute it by the maximum weight matching of a bipartite graph, where the summaries are on opposite sides of the graph. The number of matched pairs thus enables us to compute precision, recall, and F1 score. Although this procedure has been used in the previous work~\cite{khosla2013large,de2011vsumm}, there the edge weights are calculated by low-level visual features which by no means match the semantic information humans obtain from the videos. In sharp contrast, we use the IOU similarities defined directly over the user annotated semantic vectors as the edge weights.

\vspace{-5pt}
\subsubsection{Acquiring User Summaries}
\label{subsec:user_sums}
In addition to the dense per-video-shot concept tagging, we also ask annotators to label query-focused video summaries for the 46 queries described in Section~\ref{subsec:Dict}.

To ensure consistency in the summaries and better quality control over the summarization process, we switch from MTurk to three student volunteers in our university. We meet and train the volunteers in person. They each summarize all four videos by taking queries into account --- an annotator receives 4 (videos) $\times$ 46 (queries) summarization tasks in total. We thus obtain three user summaries for each query-video pair.
\eat{
\begin{figure*}[t]
	\centering
	\includegraphics[width=\linewidth]{./InterUser}
	\caption{\small{This figure compares and contrasts the summaries labeled by the same user for two queries $\{\textsc{Hat},\textsc{Phone}\}$ and $\{\textsc{Food},\textsc{Drink}\}$, respectively. Pairs that are marked with green margin exactly match in the user summaries while orange margin depicts the query-relevant shots in the summaries.}}
	\label{fig:interuser}
\end{figure*}
}

However, we acknowledge that it is infeasible to have the annotators to summarize all the query-video pairs from scratch --- the UTE videos are each 3--5 hours long. To overcome this issue, we expand each temporal video to a set of static key frames. First, we uniformly extract five key frames to represent each shot in the same way as in Section~\ref{subsec:tag}. Second, we pool all the shots corresponding to the three textual summaries~\cite{yeung2014videoset} as the initial candidate set. Third, for each query, we further include all the shots that are relevant to it into the set. A shot is relevant to the query if the intersection of the concepts  associated with it and the query is nonempty. As a result, we have a set of candidate shots for each query that covers the main story in the video as well as those of relevance to the query. The annotators summarize the video by removing redundant shots from the set. There are $2500$ to $3600$ shots in the candidate sets, and the summaries labeled by the participants contain only $71$ shots on average.
\vspace{-12pt}

\begin{table}[t]\centering
	\small
		\caption{\small{Inter-user agreement evaluated by F1 score (\%) (U1, U2, and U3: the three student volunteers,  O: the oracle summary).}} 	\label{tab:interuser}
		\vspace{-10pt}
	\begin{tabular}{cccccccccccccccccccccc}\toprule
		U1-U2 & U1-U3 & U2-U3 & U1-O & U2-O & U3-O
		\\ \midrule

		 55.27 & 55.85 & 62.67 & 64.97 & 79.75 & 80.07\\
		\bottomrule
	\end{tabular}
	\label{table:InterUser}
	\vspace{-15pt}
\end{table}

\paragraph{Oracle summaries.}
Supervised video summarization methods~\cite{gong2014diverse,gygli2015video,sharghi2016query,zhang2016summary,zhang2016video} often learn from one summary per video, or per query-video pair in query-focused summarization, while we have three user generated summaries per query. We aggregate them into one, called the oracle summary, per query-video pair by a greedy algorithm. The algorithm starts from the common shots in the three user summaries. It then greedily chooses one shot every time such that this shot gives rise to the largest marginal gain over the evaluated F1 score. We leave the details to the Suppl.\ Materials. The oracle summaries achieve better agreements with the users than the inter-user consensus (cf.\ Table~\ref{tab:interuser}).
\vspace{-12pt}

\begin{table}
	\small
			\centering
		\caption{\small{The average lengths and standard deviations of the summaries for different queries. }}
		\vspace{-10pt}
	\label{table:sumstats}
	\begin{tabular}{ccccccccc}
	\toprule
		& User 1 & User 2 & User 3 & Oracle \\ \cmidrule{2-5}
		Vid1 & \small{143.7$\pm$}32.5 & \small{80.2$\pm$}47.1 & \small{62.6$\pm$}15.7 & \small{82.5$\pm$}33.9 \\%&& 54.7$\pm$0.13 & 63.4$\pm$0.11 & 61.8$\pm$0.08 & 64.3$\pm$0.09\\
		Vid2 & 103.0$\pm$45.0 & 49.9$\pm$25.2 & 64.4$\pm$11.7 & 64.1$\pm$11.7 \\%&& 52.0$\pm$0.13 & 65.6$\pm$0.08 & 59.6$\pm$0.1 & 63.2$\pm$0.08\\
		Vid3 & 97.3$\pm$38.9  & 50.1$\pm$9.6  & 58.4$\pm$9.3 & 59.2$\pm$9.6 \\%&& 56.2$\pm$0.13 & 67.4$\pm$0.06 & 66.7$\pm$0.08 & 68.2$\pm$0.06\\
		Vid4 & 79.9$\pm$30.3 & 34.4$\pm$7.3 & 28.9$\pm$8.7 & 35.6$\pm$8.5 \\%&& 42.9$\pm$0.15 & 54.3$\pm$0.08 & 52.4$\pm$0.09 & 54.1$\pm$0.07\\
		\bottomrule
	\end{tabular}
	\vspace{-15pt}
\end{table}

\paragraph{Summaries of the same video differ due to queries.}
Figure~\ref{fig:sumcompare} shows two summaries labeled by the same user for two distinct queries, $\{\textsc{Hat},\textsc{Phone}\}$ and $\{\textsc{Food},\textsc{Drink}\}$. Note that the summaries both track the main events happening in the video while they differ in the query-specific parts. Besides, table~\ref{table:sumstats} reports the means and standard deviations of the lengths of the summaries per video per user. We can see that the queries highly influence the resulting summaries; the large standard deviations attribute to the queries.
\vspace{-22pt}

\paragraph{Budgeted summary.}
For all the summaries thus far, we do not impose any constraints over the total number of shots to be included into the summaries. After we receive the annotations, however, we let the same participants further reduce the lengths of their summaries to respectively 20 shots and 10 shots. We call them \emph{budgeted} summaries and leave them for future research.

\section{Experimental Setup}
\label{subsec:eval}
In this section, we propose a new evaluation metric and contrast it against other existing metrics.
\section{Summary Examples}
\label{Examples}

In addition to the text, we have enclosed two system summaries to provide qualitative results. The first video summary corresponds to query $q$=\textsc{\{Food,Drink\}} (scenario i), and consists of 93 shots (each shot is 5 seconds long), making it less than 8 minutes long while the original video is $\sim 4$ hours. The second, with query of $q$=\textsc{\{Chocolate,Street\}} (scenario ii), is a summary of length $\sim$ 5 minutes (56 shots) generated by our model for a 3 hours long video.
\subsection{A Nice Behavior of Our Evaluation Metric}
\vspace{-5pt}
Our evaluation method for video summarization is mainly motivated by Yeung et al.~\cite{yeung2014videoset}. Particularly, we share the same opinion that the evaluation should focus on the semantic information which humans can perceive, rather than the low-level visual features or temporal overlaps. However, the captions used in \cite{yeung2014videoset} are diverse, making the ROUGE-SU4 evaluation unstable and poorly correlated with human judgments~\cite{chen2015microsoft}, and often missing subtle details (cf.\ Figure~\ref{fig:captionvstags} for some examples).

We rectify those caveats by instead collecting dense concept annotations. Figure~\ref{fig:captionvstags} exhibits a few video shots where the concepts we collected provide a better coverage than the captions about the semantics in the shots. Moreover, we conveniently define an evaluation metric based on the IOU similarity function between any two shots (cf.\ Section~\ref{subsec:tag}) thanks to the concept annotations.

Our evaluation metric has some nice behaviors. If we randomly remove some video shots from the user summaries and compare the corrupted summaries with the original ones, an accuracy-like metric should give rise to linearly decreasing values. This is indeed what happens to our recall as shown in Figure~\ref{fig:Del}. In contrast, the ROUGE-SU4 recall, taking as input the shot captions, exhibits some nonlinearality. More results on randomly replacing some shots in the user summaries are included in the Suppl. Materials.

\vspace{-5pt}
\section{Experimental Results}
\label{sec:expset}
\begin{table*}[t]
	\centering
	\small
	\caption{\small{Comparison results for query-focused video summarization (\%). }}
	\label{table:results}
	\vspace{-10pt}
	\begin{tabular}{@{}rrrrcrrrcrrr@{}}\toprule
		& \multicolumn{3}{c}{SeqDPP~\cite{gong2014diverse}} & \phantom{abc}& \multicolumn{3}{c}{SH-DPP~\cite{sharghi2016query}} &
		\phantom{abc} & \multicolumn{3}{c}{\textbf{Ours}}\\ \cmidrule{2-4} \cmidrule{6-8} \cmidrule{10-12}
		& Precision & Recall & F1 && Precision & Recall & F1 && Precision & Recall & F1\\ \midrule

		Vid1 & \textbf{53.43} & 29.81 & 36.59 && 50.56 & 29.64 & 35.67 && 49.86 & \textbf{53.38} & \textbf{48.68}\\
		Vid2 & \textbf{44.05}& 46.65 & \textbf{43.67} && 42.13& 46.81& 42.72&& 33.71& \textbf{62.09}& 41.66\\
		Vid3 & 49.25& 17.44 & 25.26 && 51.92& 29.24& 36.51&& \textbf{55.16} & \textbf{62.40} & \textbf{56.47} \\
		Vid4 & 11.14 & \textbf{63.49} & 18.15 && 11.51& 62.88& 18.62&& \textbf{21.39}& 63.12& \textbf{29.96}\\
		\bottomrule
		Avg. & 39.47 & 39.35 & 30.92 && 39.03 & 42.14 & 33.38 && \textbf{40.03} & \textbf{60.25} & \textbf{44.19} \\
		\bottomrule
	\end{tabular}
	\vspace{-2pt}
\end{table*}
\eat{
\begin{table*}[t]\centering
	\small
	\caption{\small{Testing effectiveness of individual components in our proposed attention-based query-focused summarizer.}}
	\label{table:elementtest}
	\vspace{-10pt}
	\begin{tabular}{@{}rrrrcrrrcrrr@{}}\toprule
		& \multicolumn{3}{c}{NoAttention} & \phantom{}& \multicolumn{3}{c}{-Emb $D$} &
		\phantom{} & \multicolumn{3}{c}{EmbSize 256}\\ \cmidrule{2-4} \cmidrule{6-8} \cmidrule{10-12}
		& Precision & Recall & F1 && Precision & Recall & F1 && Precision & Recall & F1\\ \midrule

		Vid1 & 39.68 & 61.21 & 45.66 && 34.58 & 66.28 & 42.95 && 44.12 & 59.44 & 47.64\\
		Vid2 & 25.24 & 67.04 & 35.38 && 24.40 & 68.78 & 34.31 && 30.68 & 65.53 & 39.91\\
		Vid3 & 36.67 & 74.27 & 47.51 && 51.26 & 64.60 & 54.46 && 53.88 & 62.58 & 55.61 \\
		Vid4 & 14.32 & 65.58 & 22.38 && 15.36 & 67.31 & 23.52 && 17.01 & 66.44 & 25.61\\
		\bottomrule
		Avg. & 28.98 & 67.03 & 37.73 && 31.4 & 66.74 & 38.81 && 36.42 & 63.5 & 42.19\\
		\bottomrule
	\end{tabular}
\end{table*}
}
We report experimental setup and results in this section.
\vspace{-10pt}
\paragraph{Features.} We extract the same type of features as used in the existing SH-DPP method~\cite{sharghi2016query} in order to have fair comparisons. First, we employ 70 concept detectors from SentiBank~\cite{borth2013sentibank} and use the detection scores for the features of each key frame (8 key frames per 5-second-long shot). However, it is worth  mentioning that our approach is not limited to using concept detection scores and, more importantly unlike SH-DPP, does not rely on the per-shot annotations about the relevance to the query --- the per shot user labeled semantic vectors serve for evaluation purpose only. Additionally, we extract a six dimensional contextual feature vector per shot as the mean-correlations of low-level features (including color histogram, GIST~\cite{oliva2001modeling}, LBP~\cite{ojala2002multiresolution}, Bag-of-Words, as well as an attribute feature~\cite{yu2013designing}) in a temporal window whose size varies from 5 to 15 shots. The six-dimensional contextual features are appended to the key frame features in our experiments.

\vspace{-10pt}
\paragraph{Data split.} We run four rounds of experiments each leaving one video out for testing and one for validation, while keeping the remaining two for training. Since our video summarizer and the baselines are sequential models, the small number (i.e., two) of training videos is not an issue as the videos are extremely long, providing many variations and supervisions at the training stage.

\subsection{Comparison Results}

\eat{
\begin{table}[t]\centering
	\small
	\caption{\small{Comparison results for generic video summarization, i.e., when no video shots are relevant to the query.}}
	\label{table:glob}
	\vspace{-10pt}
	\begin{tabular}{cccc}\toprule
		& SubMod~\cite{gygli2015video} &  Quasi~\cite{zhao2014quasi} & \textbf{Ours}\\
		\midrule
		Vid1 & 49.51 & 53.06 & \textbf{62.66}\\
		Vid2 & 51.03 & \textbf{53.80} & 46.11\\
		Vid3 & 64.52 & 49.91 & \textbf{58.85} \\
		Vid4 & \textbf{35.82} & 22.31 & 33.5\\
		\bottomrule
		Avg. & 50.22 & 44.77 & \textbf{50.29}\\
		\bottomrule
	\end{tabular}
	\vspace{-10pt}
\end{table}
}

\begin{table*}[t]\centering
	\small
\caption{\small{Comparison results for generic video summarization, i.e., when no video shots are relevant to the query.}}
	\label{table:glob}
	\vspace{-10pt}
	\begin{tabular}{@{}rrrrcrrrcrrr@{}}\toprule
		& \multicolumn{3}{c}{SubMod~\cite{gygli2015video}} & \phantom{abc}& \multicolumn{3}{c}{Quasi~\cite{zhao2014quasi}} &
		\phantom{abc} & \multicolumn{3}{c}{\textbf{Ours}}\\ \cmidrule{2-4} \cmidrule{6-8} \cmidrule{10-12}
		& Precision & Recall & F1 && Precision & Recall & F1 && Precision & Recall & F1\\ \midrule

		Vid1 & 47.86 & 51.28 & 49.51 && 57.37 & 49.36 & 53.06 && \textbf{65.88} & 59.75 & \textbf{62.66}\\
		Vid2 & 56.53 & 46.50 & 51.03 && 46.75& 63.34& \textbf{53.80} && 35.07 & \textbf{67.31} & 46.11\\
		Vid3 & 62.46 & 66.72 & 64.52 && 53.93 & 46.44 & 49.91&& \textbf{65.95} & 53.12 & \textbf{58.85} \\
		Vid4 & \textbf{34.49} & 37.25 & \textbf{35.82} && 13.00 & \textbf{77.88} & 22.31 && 22.29 & 67.74 & 33.5\\
		\bottomrule
		Avg. & \textbf{50.34} & 50.44 & 50.22 && 42.76 & 59.25 & 44.77 && 47.3 & \textbf{61.98} & \textbf{50.29}\\
		\bottomrule
	\end{tabular}
\end{table*}

\section{Introduction}
\label{sec:intro}

\begin{figure*}[t]
	\centering
	\includegraphics[width=\linewidth]{./Caption_vs_tags}
	\vspace{-18pt}
	\caption{\small{Comparing the semantic information captured by captions in~\cite{yeung2014videoset} and by the concept tags we collected.}}
	\label{fig:captionvstags}
	\vspace{-10pt}
\end{figure*}

Recent years have witnessed a resurgence of interest in video summarization, probably due to the overwhelming video volumes showing up in our daily life. Indeed,  both consumers and professionals have the access to ubiquitous video acquisition devices nowadays. While the video data is a great asset for information extraction and knowledge discovery, due to its size and variability, it is extremely hard for users to monitor or find the occurrences in it.

Intelligent video summarization algorithms allow us to quickly browse a lengthy video by capturing the essence and removing redundant information. Early video summarization methods were built mainly upon basic visual qualities (e.g., low-level appearance and motion features)~\cite{goldman2006schematic,gygli2015video,laganiere2008video,liu2002optimization,rav2006making,wolf1996key,zhao2014quasi}, while recently more abstract and higher-level cues are leveraged in the summarization frameworks~\cite{gong2014diverse,khosla2013large,kim2014joint,kwon2012unified,sharghi2016query,xiong2014detecting,yaohighlight,zhang2016summary}.

However, one of the main obstacles to the research on video summarization is the user subjectivity --- users have various preferences over the summaries they would like to watch. The subjectiveness causes at least two problems. First, no single video summarizer fits all users unless it interacts with and adapts to the users. Second, it is very challenging to evaluate the performance of a video summarizer.

In an attempt to solve the first problem, we have studied a new video summarization mechanism, query-focused video summarization~\cite{sharghi2016query}, that introduces user preferences in the form of text queries about the video into the summarization process. While this may be a promising direction to \textbf{\em personalize} video summarizers, the experimental study in \cite{sharghi2016query} was conducted on the datasets originally collected for the conventional generic video summarization~\cite{lee2012discovering,yeung2014videoset}. It remains unclear whether the real users would generate distinct summaries for different queries, and if yes, how much the query-focused summaries differ from each other.

In this paper, we explore more thoroughly the query-focused video summarization and build a new dataset particularly designed for it. While we collect the user annotations, we meet the challenge how to define a good evaluation metric to contrast system generated summaries to user labeled ones --- the second problem above-mentioned due to the user subjectivity about the video summaries.

We contend that the pursuit of new algorithms for video summarization has actually left one of the basic problems underexplored, i.e., how to benchmark different video summarizers. User study~\cite{lee2015predicting,lu2013story} is too time-consuming to compare different approaches and their variations at large scale. In the prior arts of automating the evaluation procedure, on one end, a system generated summary has to consist of exactly the same key units (frame or shot)  as in the user summaries in order to be counted as a good one~\cite{chu2015video,song2015tvsum,xu2015gaze}. On the other end, pixels and low-level features are used to compare the system and user summaries~\cite{gong2014diverse,khosla2013large,kim2014joint,zhang2016summary,zhao2014quasi}, whereas it is unclear what features and distance metrics match users' criteria. Some works strive to find a balance between the two extremes, e.g., using the temporal overlap between two summaries to define the evaluation metrics~\cite{gygli2014creating,gygli2015video,potapov2014category,zhang2016video}. However, all such metrics are derived from either the temporal or visual representations of the videos, without explicitly encoding how humans perceive the information --- after all, the system generated summaries are meant to deliver similar information to the users as those directly labeled by the users.

In terms of defining a better measure that closely tracks what humans can perceive from the video summaries,
we share the same opinion as Yeung et al.'s~\cite{yeung2014videoset}: it is key to evaluate how well a system summary is able to retain the semantic information, as opposed to the visual quantities, of the user supplied video summaries. Arguably, the semantic information is best expressed by the concepts  that represent the fundamental characteristics of what we see in the video at multiple grains, with the focus on different areas, and from a variety of perspectives (e.g., objects, places, people, actions, and their finer-grained entities, etc.).

Therefore, as our first contribution, we collect dense per-video-shot concept annotations for our dataset. In other words, we represent the semantic information in each video shot by a binary semantic vector, in which the 1's indicate the presence of corresponding concepts in the shot. We suggest a new evaluation metric for the query-focused (and generic) video summarization based on these semantic vector representations of the video shots\footnote{Both the dataset and the code of the new evaluation metric are publicly available at \url{http://www.aidean-sharghi.com/cvpr2017}.}.

In addition, we propose a memory network~\cite{sukhbaatar2015end} parameterized sequential determinantal point process~\cite{gong2014diverse} for tackling the query-focused video summarization. Unlike the hierarchical model in~\cite{sharghi2016query}, our approach does not rely on the costly user supervision about which queried concept appears in which video shot or any pre-trained concept detectors. Instead, we use the memory network to implicitly attend the user query about the video onto different frames within each shot. Extensive experiments verify the effectiveness of our approach.

The rest of the paper is organized as follows. We discuss some related works in Section \ref{sec:related}. Section \ref{sec:dataset} elaborates the process of compiling the dataset, acquiring annotations, as well as a new evaluation metric for video summarization. In section \ref{sec:approach} we describe our novel  query-focused summarization model, followed by detailed experimental setup and quantitative results in Sections \ref{sec:expset}.
\begin{figure*}
	\centering
	\includegraphics[width=\linewidth]{./stats}
	\vspace{-25pt}
	\caption{\small{The frequencies of concepts showing up in the video shots, counted for each video separately.}}
	\label{fig:stats}
	\vspace{-10pt}
\end{figure*}

\section{Generating Oracle Summaries}
\label{Oracle}
Supervised video summarization methods often learn from one summary per video, or in the case of query-focused summarization per query-video pair. On the other hand, for evaluation purposes, it is better to contrast a system summary against multiple references and report the average. Thus, we collected 3 user summaries per query-video pair to use for evaluation purposes. However, in order to train the model, we obtain \textit{oracle} summaries that have maximum overall agreement with all three reference summaries (per query-video pair).

The algorithm~\cite{kulesza2012determinantal} starts with the set of common shots ($y^0$) in all three reference summaries. Next, at each iteration, it greedily includes the shot that returns the largest marginal gain $G(i)$,
\begin{align}
\centering
G(i) = \sum_u \text{F1-score}(y^0\cup i,y_u) - \sum_u \text{F1-score}(y^0,y_u)
\end{align}
where $u$ iterates over the user summaries (in our case, $u \in \{1,2,3\}$) and F1-score is obtained from our proposed evaluation metric. Table 1 in the main text shows the correlation between the obtained oracle summaries and user summaries, showing the oracle summary has very high agreement with all the user summaries.

\section{Constructing Queries}
\label{Queries}

In this section, we thoroughly describe the process of generating the queries from dense concept annotations. While users often input free text to query videos through search engines, we simulate the real scenarios and construct the queries using the dense concept annotations we have collected (cf.\ Section 3.2 in the main text) to ease benchmarking different algorithms.

By processing the dense user annotation data, we extract various statistics that enable us to have the queries covering a wide range of varieties. Initially, a concept is assumed present in the video if it appears in at least $T$ shots. This is to filter the present noise in annotations acquired from AMT workers and to make sure the concepts really appear together (to steer clear of the pairs that are tagged together as a result of noise or bias).

As described in the main text, when a user enters a query $q$ (for instance, on a video search engine), which is usually more than one word, we have four distinct scenarios; i) all the concepts in the query appear in the same video shots together, ii) all concepts appear in the video, but never jointly in a single shot, iii) only one of the concepts constituting the query appears in the some shots of the video, and iv) none of the concepts in the query are present in the video (1 such query). A robust video summarizer, must be able to maintain good performance under any of the scenarios. Therefore, by including enough samples of all the scenarios, we build a comprehensive and diverse dataset.

For the scenario i, we create a list of pairs that appear together in the same shots and sort it in descending order. There are two approaches to select concept pairs from this list: 1) to employ a random selection process where the probability of selecting a pair from the list is proportional to the number of times the pair appeared together in the video (this gives higher chance to the concepts that tend to happen together in the video while not completely crossing out the concepts that are not dominant in the video), and 2) to pick few top concept pairs. We opt to use the random selection process to better generalize the dataset and remove bias.

For the scenario ii, we are interested in concept pairs that are present in the video but not in the same shots, e.g., concept pairs such as \textsc{Car} and \textsc{Room} that are unlikely to appear in the same shots of the video. To this end, for each pair we compute their harmonic mean of frequencies:
\begin{equation}
	score(f_{c_1},f_{c_2}) = \frac{f_{c_1} \times f_{c_2}}{f_{c_1} + f_{c_2}}
\end{equation}
where $f_{c_1}$ and $f_{c_2}$ are the frequencies of concepts $c_1$ and $c_2$, respectively. This formulation has two interesting features that make it useful in this regard; 1) the resulting combination of numbers fed to it is always smaller than the smallest entry, 2) it is maximized when both inputs are large and identical. By computing the harmonic mean of frequencies for all the pairs in the list and sorting it in descending order, the concept pairs that have high frequencies for both concepts constituting the query are ranked higher. At this point, we employ the same random selection process to randomly choose pairs from this list.

\begin{figure*}[t]
	\centering
	\includegraphics[width=\textwidth]{./Caption_vs_tags2}
	\caption{\small{Comparing semantic information in our dense tags vs captions provided by~\cite{yeung2014videoset}. The figure illustrates that the caption is targeting limited information about the scene, while the dense annotations are able to better explain the characteristics of the scene.}}
	\label{fig:captionvstags}
\end{figure*}

\begin{figure*}[t]
	\centering
	\includegraphics[width=\textwidth]{./Summary_comparison2}
	\caption{\small{This figure compares two user summaries (generated by the same user) for two different queries. Both summaries contain shared segments, that are assumed important in the context of the video, while they disagree on the query-relevant segments.}}
	\label{fig:sumcom}
\end{figure*}

\begin{figure*}[t]
	\centering
	\includegraphics[width=\textwidth]{ROUGEvsIOU_Chng}
	\caption{\small{Studying the effect of randomly \textit{replacing} some video shots in the user summary on the performance. The evaluation by ROUGE-SU4~\cite{lin2004rouge} is included for reference.}}
	\label{fig:ROUGEvsIOU}
\end{figure*}

For the third scenario, we concentrate on pairs that only one concept constituting the query is present in the video, e.g., if there is no \textsc{Car} present in the entire video while there exists shots with \textsc{Computer} appearing in them, the pair \textsc{Car} and \textsc{Computer} is a candidate for this scenario. To make sure that the constructed dataset is comprehensive and benefits from the versatile dictionary, we first exclude the concepts that were used in the first two scenarios, we put the rest in a list and use their frequencies to randomly sample from them.

For the last scenario, where neither of concepts in pairs must be present in the video, we simply use the concepts that never appear in the video.

For scenarios i, ii, and iii, we select 15 queries. For scenario iv, we only choose one query; summarizing based on any such query consisting of concepts that are not present in the entire video must result in about the same summary. In other terms, when a user wants the model to summarize the video based on a query consisting of non-present concepts, the summarizer must only return \textit{contextually} important segments of the video, that is essentially what a conventional generic video summarization approach (as opposed to query-dependent approaches) generates.

Figure~\ref{fig:sumcom} shows that queries play a major role in the summaries that users generate. For a particular video, the same user has selected summaries that have both common (green margin) and uncommon (orange margin) segments.
\section{Related Work}
\label{sec:related}
We discuss some related works in this section.

This work extends our previous efforts~\cite{sharghi2016query} on \emph{personalizing} video summarizers.  Both  works explore the query-focused video summarization, but we study this problem more thoroughly in this paper through a new dataset with dense per-video-shot tagging of concepts. Our memory network based video summarizer requires less supervision for training than the hierarchical model in~\cite{sharghi2016query}.

Unlike our user-annotated semantic vectors for the video shots, Yeung et al.\ asked annotators to caption each video shot using a sentence~\cite{yeung2014videoset}. A single sentence targets only limited information in a video shot and misses many details. Figure~\ref{fig:captionvstags} contrasts the concept annotations in our dataset with the captions for a few video shots. The concept annotations clearly provide a more comprehensive coverage about the semantic information in the shots.

Memory networks~\cite{bahdanau2014neural,sukhbaatar2015end,weston2015towards,weston2014memory,xiong2016dynamic} are versatile in modeling the attention scheme in neural networks. They are widely used to address question answering and visual question answering~\cite{antol2015vqa}. The query focusing in our summarization task is analogous to attending questions to the ``facts'' in the previous works, but  the facts in our context are temporal video sequences. Moreover, we lay a sequential determinantal point process~\cite{gong2014diverse} on top of the memory network in order to promote diversity in the summaries.

A determinantal point process (DPP)~\cite{kulesza2012determinantal} defines a distribution over the power sets of a ground set that encourages diversity among items of the subsets. There have been growing interest in DPP in machine learning and computer vision~\cite{affandi2014learning,agarwal2014notes,batmanghelich2014diversifying,chao2015large,gartrell2016low,gillenwater2014expectation,DBLP:conf/icml/KuleszaT11,kulesza2011learning,kwok2012priors,li2016fast,mariet2015fixed,mariet2016kronecker,snoek2013determinantal}. Our model in this paper extends DPPs' modeling capabilities through the memory neural network.

\section{Evaluation Metric Behavior}
\label{IOUvsROUGE}

As described in Section 5.2 of the main text, we studied the effect of randomly \textbf{removing} some video shots from the user summary on our proposed metric, observing that our metric has a linear drop in recall. Due to the fact that captions only capture limited information about the scene (cf.\ Figure~\ref{fig:captionvstags}), repeating the same experiment and evaluating with ROUGE-SU4 on captions provided by~\cite{yeung2014videoset}, recall showed a non-linear drop. As a side experiment, figure~\ref{fig:ROUGEvsIOU} illustrates the effect of randomly \textbf{replacing} some video shots in the user summary, studying the effect of noise on performance. Here we are swapping some shots with others that might be very similar or different to the original shots. For reference, we include the ROUGE-SU4 metric in our experiments as well.