\section{Introduction}
\label{sec:intro}
Microwave resonators (for example,
Chap.~7 of Ref.~\onlinecite{Poz90})
are one of the key components
in a variety of circuits operated at GHz frequencies,
and their new applications continue to emerge.
A simple example is band-pass filters, which are based on
the fact that the microwave transmission through resonators
is frequency sensitive.
The same idea is also used for more complex devices,
such as oscillators, tuned amplifiers, and frequency meters.
Actually, having high-quality filters and oscillators
is critical in mobile communications,
where available bands keep getting
overcrowded as demand grows rapidly.

Another application of microwave resonators is
radiation detectors,
which often consist of
sensor heads and read-out circuits,
and
resonators can be used
in the readout circuit.
When one would like to
detect at the single-photon level,
one needs to have detectors with
high enough energy resolutions.
In this respect, superconducting
sensor heads\cite{Irw95,Pea96,Gol01}
can be advantageous, and may be the only solution
at present depending on the energy range
of the object.
Once one decides to use
superconducting sensor heads,
it makes sense to fabricate
the read-out circuit with superconducting
materials as well.
Superconducting microwave resonators
allow one to obtain higher quality factors,
which are favorable for frequency multiplexing.
In addition,
one type of photon detector is designed to
probe the change in the kinetic inductance of
superconducting thin-film resonator
due to the absorbed photons.\cite{Maz02}
In this device concept, the resonator works
as a sensor head
rather than a part of the readout circuit.

Recently, superconducting resonators are used
for the nondemolition
readout of superconducting qubits as well.\cite{Wal04}
Since the demonstration by Wallraff {\it et al.},\cite{Wal04}
this type of readout scheme has been one of the main topics
in the field of superconducting qubits, and
we are also developing a similar readout technique.\cite{Ino09}
In superconducting resonators, kinetic inductance,
which is essentially the internal mass of
the current carriers, plays an important role
especially when the superconducting film is thin.
In our circuit,\cite{Ino09}
for example, a Nb $\lambda/4$ coplanar-waveguide
(CPW) resonator is terminated by an Al dc SQUID,
and the total thickness of the Al layers is 0.04~$\mu$m.
In order to avoid a discontinuity at the Al/Nb interface,
we usually choose the Nb thickness to be 0.05~$\mu$m,
which is much thinner than a typical
thickness of $\geq$0.3~$\mu$m for superconducting integrated
circuits fabricated by the standard photolithographic technology.
Fabricating circuits with thinner films is actually important
from the viewpoint of miniaturization as well.
Therefore, for designing resonators, quantitative understanding
of the kinetic inductance in the CPW is important.

There have been a number of reports on kinetic inductance for
a variety of materials.\cite{Mes69,Rau93,Kis93,Wat94,Gub05,Fru05}
In general, however, kinetic inductance is indirectly measured
by assuming a theoretical model, and as a result,
the uncertainties are relatively large.
Thus, although kinetic inductance is a well established
notion and the phenomenon is qualitatively understood,
the quantitative information is not necessarily sufficient
from the point of view of applications, especially at
high frequencies, $>$10~GHz.
When we would like to precisely predict
the resonant frequency, the best solution would be
to characterize the actual CPW in a simple circuit.
Such characterization should also improve the
knowledge of superconducting microwave circuits.
Very recently, G\"{o}ppl {\it et al.}\cite{Gop08}
measured a series of Al CPW resonators with
nominally the same film thickness of 0.2~$\mu$m,
and investigated the relationship between the loaded quality
factor at the base temperature of 0.02~K
and the coupling capacitance.
For this purpose,
it is justified to neglect kinetic inductance
because the kinetic inductance should
be the same in their resonators and estimated\cite{Gop08}
to be about two orders of magnitude smaller than
the usual magnetic inductance determined by the CPW geometry.
In this work, on the other hand, we paid close attention
to the resonant frequency as well, and
characterized Nb CPW resonators as a function of
film thickness rather than a function of
coupling capacitance.
We also looked at the temperature dependence
in order to discuss kinetic inductance in detail.

\section{Experiment}
\label{sec:Ex}
\begin{table}
\caption
{\label{tab:list}
List of resonators.
$t$ is the thickness of Nb film;
$f_r$ is the resonant frequency;
$Q_L$ is the unloaded quality factor;
$C_c$ is the coupling capacitance;
$v_p$ is the phase velocity, and its ratio
to the speed of light $c$ is listed in percent.
The values for $f_r$ and $Q_L$ are obtained
at the base temperatures.
$C_c$ and $v_p$ are evaluated by least-squares fitting
(see Fig.~\ref{fig:S21}) with $C=1.6\times10^{-10}$~F/m,
and their uncertainties are determined by changing the value
of $C$ by $\pm10\%$, where $C$ is the capacitance
per unit length.}
\begin{ruledtabular}
\begin{tabular}{cccccc}

Reso-	&$t$&$f_r$&$Q_L$&$C_c$&$v_p/c$\\
nator &($\mu$m)&(GHz)&($\times$10$^3$)&(fF)&(\%)\\
 \hline
A1& 0.05           & 10.01 & 1.6& 7.0$\pm$0.4 & 39.31$\pm$0.03 \\
A2& 0.1\phantom{0} & 10.50 & 1.4& 7.3$\pm$0.4 & 41.28$\pm$0.04 \\
A3& 0.2\phantom{0} & 10.74 & 1.4& 7.2$\pm$0.4 & 42.21$\pm$0.04 \\
A4& 0.3\phantom{0} & 10.88 & 1.6& 6.6$\pm$0.4 & 42.71$\pm$0.03 \\
B1& 0.05           & 10.06 & 3.4& 4.6$\pm$0.3 & 39.32$\pm$0.02 \\
B2& 0.1\phantom{0} & 10.56 & 3.1& 4.8$\pm$0.3 & 41.26$\pm$0.02 \\
B3& 0.2\phantom{0} & 10.81 & 2.7& 5.0$\pm$0.3 & 42.27$\pm$0.03 \\
B4& 0.3\phantom{0} & 10.94 & 3.3& 4.5$\pm$0.3 & 42.72$\pm$0.02 \\
\end{tabular}
\end{ruledtabular}
\end{table}
\begin{figure}
\begin{center}
\includegraphics[width=0.8\columnwidth,clip]{fig1_CPW_equ_lr.eps}
\caption{\label{fig:CPW_equ}
Schematic diagram of coplanar-waveguide (CPW) resonators.
A CPW of length $l$ is coupled to the microwave lines
through capacitors $C_c$.
}
\end{center}
\end{figure}
\begin{figure}
\begin{center}
\includegraphics[width=0.9\columnwidth,clip]{fig2_CPWshape3_lr.eps}
\caption{\label{fig:CPWshape}
(Color online)
(a) Cross section of a coplanar waveguide.
(b) Top view of coupling capacitors.
}
\end{center}
\end{figure}
We studied two series of Nb $\lambda/2$ CPW resonators
listed in Table~\ref{tab:list}.
Each resonator consists of a section of CPW
and coupling capacitors,
as shown schematically
in Fig.~\ref{fig:CPW_equ}.
The resonators were fabricated on
a nominally undoped Si wafer whose surface had been
thermally oxidized.
On the SiO$_2$/Si substrate, a Nb film was deposited
by sputtering and then patterned by photolithography
and SF$_6$ reactive ion etching.
Figure~\ref{fig:CPWshape}(a) represents the cross section of CPW.
The center conductor has a width of $w=10$~$\mu$m,
and separated from the the ground planes by
$s=5.8$~$\mu$m, so that
the characteristic impedance becomes $\sim50$~$\Omega$.
The thickness of Nb is $t=0.05$, 0.1, 0.2, or
0.3~$\mu$m (see Table~\ref{tab:list}), and
that of SiO$_2$/Si substrate is $h=300$~$\mu$m.
The SiO$_2$ layer, whose thickness is 0.3~$\mu$m, is
not drawn in Fig.~\ref{fig:CPWshape}(a).
We employed interdigital
coupling capacitors as shown
in Fig.~\ref{fig:CPWshape}(b).
The finger width is $w_f = 9$~$\mu$m, the space between the
fingers is $s_f = 2$~$\mu$m, and the finger length is
$l_f = 78$~$\mu$m for Resonators~A1--A4 and
$l_f = 38$~$\mu$m for Resonators~B1--B4.
Here, we quoted designed dimensions for the Nb structures.
The actual dimensions
differ by about 0.2~$\mu$m due to over-etching;
for example, $w$ and $w_f$ are
$\sim0.2$~$\mu$m smaller, whereas
$s$ and $s_f$ are $\sim0.2$~$\mu$m larger.
In this paper, we define the resonator length $l$ as the
distance between the center of the fingers on one side and
that on the other side, and $l=5.8$~mm for all resonators.
Because our chip size is 2.5~mm by 5.0~mm, our CPWs
meander as in Fig.~\ref{fig:Outline}.

\begin{figure}
\begin{center}
\includegraphics[width=0.7\columnwidth,clip]{fig3_Outline_lr.eps}
\caption{\label{fig:Outline}
Schematic diagram of a typical measurement setup.
Boxes represent attenuators.
}
\end{center}
\end{figure}

The resonators were measured in
a $^3$He-$^4$He dilution refrigerator at $T=0.02-5$~K.
A typical measurement setup is shown schematically
in Fig.~\ref{fig:Outline}.
The boxes in the figure represent attenuators.
The amount of attenuation was not the same
because the microwave lines in our refrigerator
had been designed for several different purposes.
The attenuation was $x = 10$~dB for Resonators~A2, B3, and B4,
and $x = 20$~dB for the others;
$y = 10$~dB for all resonators except A1 and A3.
For Resonators~A1 and A3, we used a line
with no attenuators ($y=0$~dB) but with
an isolator and a cryogenic amplifier at 4.2~K.
The gain of the cryogenic amplifier was 40~dB for
Resonator~A1 and 34~dB for A3.
We measured the transmission coefficient
$S_{21}$ by connecting a vector
network analyzer to the ``IN" and ``OUT" ports
in Fig.~\ref{fig:Outline}.
A typical incident power to the resonator was $-40$~dBm.
For each resonator, we confirmed that
the measurements were done in an appropriate power range
in the sense that the results looked power independent.

\section{Results}
\label{sec:R}
\subsection{
{\boldmath $S_{21}$} at the base temperatures}
\label{subsec:baseT}
\begin{figure}
\begin{center}
\includegraphics[width=\columnwidth,clip]{fig4_S21_lr.eps}
\caption{\label{fig:S21}
(Color online)
Amplitude of the transmission coefficient
$S_{21}$ as a function of frequency for
(a) Resonators~A1--A4, and (b) Resonators~B1--B4.
}
\end{center}
\end{figure}

Figure~\ref{fig:S21} shows
the amplitude of $S_{21}$
at the base temperatures, $T=0.02-0.03$~K,
as a function of frequency $f$
for all resonators.
The resonant frequency $f_r$ has a rather large film-thickness
dependence. Our interpretation is that this is due to
the kinetic inductance of the CPW center conductor.
Before discussing the thickness dependence
in detail, let us look at the quality factors.

What we obtain by measuring
$S_{21}$ as a function of $f$
is the loaded quality factor $Q_L$, which is related to
the external quality factor $Q_e$ and
the unloaded quality factor $Q$ by
\begin{equation}
\label{eq:Q}
Q_L^{-1} = Q_e^{-1}+Q^{-1}.
\end{equation}
In general, $Q_e$ is determined mainly by $C_c$,
whereas $Q$ is a measure of the internal loss,
which arises not only from the dielectric but also
from the superconductor in the high-frequency regime.
Our resonators should be highly overcoupled to
the input/output lines at the base temperatures,
that is, $Q \gg Q_e$, and thus, $Q_L\sim Q_e$.
As listed in Table~\ref{tab:list}, $Q_L$
of our resonators is on the order of 10$^3$.
These values are not only reasonable for the designs of
our finger-shaped coupling capacitors
but also much smaller than typical
values of $Q$ below 0.1~K for superconducting
microwave resonators.\cite{Maz02,Fru05,Gop08}
When $Q \gg Q_e$, the maximum $|S_{21}|$ is
expected to be 0~dB.
We have confirmed by taking into account
attenuators, amplifiers, and cable losses,
that our measurements are indeed
consistent within the uncertainties of
gain/loss calculations, 1--2~dB.
Based on this confirmation,
the experimental data in Fig.~\ref{fig:S21}
are normalized so that the peak heights
equal 0~dB.

The solid curves in Fig.~\ref{fig:S21} are calculations
based on the transmission ($ABCD$) matrix (for example,
Sec.~5.5 of Ref.~\onlinecite{Poz90}), and
they reproduce the experimental data well.
The matrix for the resonators is given by
\begin{equation}
\label{eq:ABCD}
\left(
   \begin{array}{cc}
   A & B \\
   C & D
   \end{array}
\right)
= T_{\rm cc}\,T_{\rm cpw}\,T_{\rm cc}\,,
\end{equation}
where
\begin{equation}
\label{eq:Tcc}
T_{\rm cc} =
\left(
   \begin{array}{cc}
   1 & (j\omega C_c)^{-1} \\
   0 & 1
   \end{array}
\right),
\end{equation}
$j$ is the imaginary unit,
\begin{equation}
\label{eq:Tcpw}
T_{\rm cpw} =
\left(
   \begin{array}{cc}
   \cos\beta l & jZ_{\rm cpw}\sin\beta l\\
   j(Z_{\rm cpw})^{-1}\sin\beta l & \cos\beta l
   \end{array}
\right)
\end{equation}
for lossless CPWs,
$\omega = 2\pi f$, $\beta = \omega/v_p$,
\begin{equation}
\label{eq:vp}
v_p=1/\sqrt{LC}
\end{equation}
is the phase velocity, which is strongly related
to $f_r$,
\begin{equation}
\label{eq:Zcpw}
Z_{\rm cpw}=\sqrt{L/C}
\end{equation}
is the characteristic impedance,
$L$ is the inductance per unit length, and
$C$ is the capacitance per unit length.
From these transmission-matrix elements,
the scattering-matrix elements
are calculated, and $S_{21}$ is given by
\begin{equation}
\label{eq:S21}
S_{21} =
2/(A+B/Z_0+CZ_0+D),
\end{equation}
where $Z_0=50$~$\Omega$ is the characteristic impedance
of the microwave lines connected to the resonator.
Unit-length properties of CPW are determined when
two parameters out of $v_p$, $Z_{\rm cpw}$,
$L$, and $C$ are specified.
In the calculations for Fig.~\ref{fig:S21},
we employed $C=1.6\times10^{-10}$~F/m based on the considerations
described in the following paragraph, and evaluated $C_c$ and
$v_p$ by least-squares fitting.

Wen\cite{Wen69} calculated CPW parameters
using conformal mapping.
Within the theory, $C$ does not depend on $t$,
and it is given by
\begin{equation}
\label{eq:Wen_C}
C = (\epsilon_r+1)\epsilon_0\,2K(k)/K(k'),
\end{equation}
where $\epsilon_r$ is the relative dielectric constant
of the substrate,
$\epsilon_0 = 8.85\times 10^{-12}$~F/m is the
permittivity of free space,
$K(k)$ is the complete elliptical integral
of the first kind, the argument $k$ is given by
\begin{equation}
\label{eq:Wen_k}
k = w/(w+2s)\,,
\end{equation}
and $k'= \sqrt{1-k^2}.$
For our CPWs,
we obtain $C=1.6\times10^{-10}$~F/m
when we employ $\epsilon_r=11.7$ for Si
(p.~223 of Ref.~\onlinecite{Kit96})
neglecting the contribution from the SiO$_2$ layer,
which is much thinner compared to $w$, $s$, or $h$.
Circuit simulators [Microwave Office from AWR (\#1) and
AppCAD from Agilent (\#2)] also predict similar values of $C$.
The simulators calculate CPW parameters from the
dimensions and the material used for the substrate.
The predictions by the simulators have
$t$ dependence, but in the relevant $t$ range,
the variations are on the order of 1\% or smaller
as summarized in Table~\ref{tab:LC},
and the values of $C$ are between
$1.6\times10^{-10}$~F/m
and $1.7\times10^{-10}$~F/m.
Thus, partly for simplicity,
we used
$C=1.6\times10^{-10}$~F/m
for all of our resonators.

\begin{table}
\caption
{\label{tab:LC}
Dependence of coplanar-waveguide parameters on
the film thickness $t$.
For capacitance $C$ and inductance $L$ per unit length,
the normalized variations
$\Delta C(t)/C^*$ and $\Delta L(t)/L^*$
are listed in percent, where
$\Delta C(t) = C(t) - C^*$,
$C^* \sim 1.6 \times 10^{-10}$~F/m
is the value at $t=0.3$~$\mu$m,
and the definitions of
$\Delta L(t)$ and $L^*\sim4\times10^{-7}$~H/m are similar.
The predictions by circuit simulators \#1 and \#2
are compared.
Regarding $L$,
experimental values
for ``A"=Resonators~A1--A4
and for ``B"=Resonators~B1--B4 are also given,
and they are obtained from the values of $v_p$ in
Table~\ref{tab:list} using Eq.~(\ref{eq:vp})
and by neglecting the $t$ dependence of $C$.
}
\begin{ruledtabular}
\begin{tabular}{ccc|cccc}
$t$ & \multicolumn{2}{c|}{$\Delta C(t)/C^*$ (\%)} &
      \multicolumn{4}{c}{$\Delta L(t)/L^*$ (\%)}\\
($\mu$m) & \#1 & \#2 & \#1 & \#2 & A & B \\
 \hline
0.05           & $-0.6$ & 1.7 & 3.9 & 2.9 & 18.0 & 18.0 \\
0.1\phantom{0} & $-0.5$ & 1.4 & 3.0 & 2.2 & \phantom{1}7.1
					  & \phantom{1}7.2\\
0.2\phantom{0} & $-0.2$ & 0.8 & 1.4 & 1.2 & \phantom{1}2.4 		                                  & \phantom{1}2.1\\
\end{tabular}
\end{ruledtabular}
\end{table}

With $C=1.6\times10^{-10}$~F/m, the values of $v_p$
in Table~\ref{tab:list} correspond to
$Z_{\rm cpw}=49-53$~$\Omega$, which agrees with
our design of $\sim50$~$\Omega$.
We have done the same fitting by changing the value
of $C$ by $\pm10\%$ as well
in order to estimate the uncertainties, which are also
listed in Table~\ref{tab:list}.
Within the uncertainties, the values
of $C_c$ from the same coupling-capacitor design agree,
and $C_c\sim7$~fF for Resonators~A1--A4 with $l_f=78$~$\mu$m
and $C_c\sim5$~fF for Resonators~B1--B4 with $l_f=38$~$\mu$m.
The uncertainties for $v_p$ is much smaller, $<0.1\%$,
and again within the uncertainties, the values of $v_p$
for the same $t$ agree.

For the rest of this paper, let us
assume that $t$ dependence of $C$ is negligible.
This assumption is consistent with the fact that
the experimental $C_c$ vs.\ $t$ in Table~\ref{tab:list}
does not show any obvious trend.
Moreover, according to
the circuit simulators in Table~\ref{tab:LC},
$t$ dependence of $C$ is smaller than that of $L$.
Below,
we look at $L$ mainly
instead of $v_p$ or other CPW parameters
so that we will be able to discuss
the kinetic inductance.
As long as we deal with
a normalized inductance
such as
the ratio of $L(t)$ to $L^*\equiv L(\mbox{0.3~$\mu$m})$,
what we choose for the value of $C$ does not matter
very much because $v_p$ obtained from the fitting
was not so sensitive to $C$.
Hence, we analyze the quantities
obtained with $C=1.6\times10^{-10}$~F/m only hereafter.
In Table~\ref{tab:LC},
we list the variations of $L$
in our two series of resonators
as well. For both series, the magnitude of
the variations are much larger than the
predictions by circuit simulators.
We will discuss this large $t$ dependence in terms of
kinetic inductance in Sec.~\ref{sec:D} after
examining the temperature dependence in Sec.~\ref{subsec:Tdep}.

\subsection{Temperature dependence of {\boldmath $S_{21}$}}
\label{subsec:Tdep}

\begin{figure}
\begin{center}
\includegraphics[width=0.95\columnwidth,clip]{fig5_S21_temp_lr.eps}
\caption{\label{fig:S21_temp}
(Color online)
Amplitude of the transmission coefficient
$S_{21}$ as a function of frequency for
Resonators~A1 at different temperatures.
}
\end{center}
\end{figure}
\begin{figure}
\begin{center}
\includegraphics[width=0.7\columnwidth,clip]{fig6_Q2_lr.eps}
\caption{\label{fig:Q2}
(Color online)
Normalized quality factors,
(a) $Q/Q_e$ and (b) $Q_L/Q_e$,
as functions of temperature for
Resonators~A1--A4
(Nb thickness $t=0.05,$ 0.1, 0.2, and 0.3~$\mu$m),
where $Q_L$, $Q_e$, and $Q$ are
loaded, external, and unloaded quality factors, respectively,
and $Q_e$ is assumed to be temperature independent.
The markers are data points, whereas the curves are
guides to the eyes.
}
\end{center}
\end{figure}

We also measured $S_{21}$ vs.\ $f$
at various temperatures
up to $T=4-5$~K for Resonators~A1--A4.
We show the results
for Resonator~A1 in Fig.~\ref{fig:S21_temp}.
With increasing temperature,
$f_r$, $Q_L$, and the peak height
decrease.
As in Sec.~\ref{subsec:baseT},
let us look at the quality factors first.
In our resonators, $Q_e\sim Q_L$
at the base temperatures as we pointed out
in Sec.~\ref{subsec:baseT}.
Thus, when we assume that $Q_e$ is
temperature independent,
we can calculate $Q$
from measured $Q_L$
using Eq.~(\ref{eq:Q}).
We plot $Q_L(T)/Q_e$ and $Q(T)/Q_e$
vs.\ $T$ in Fig.~\ref{fig:Q2}
for all of the four resonators.
With increasing temperature,
$Q$ decreases in all resonators.
A finite $Q^{-1}$ means that the resonator
has a finite internal loss, which is
consistent with a peak height smaller than unity
in Fig.~\ref{fig:S21_temp}.
The internal loss at high temperatures must be
due to quasiparticles in the superconductor,
as discussed in Ref.~\onlinecite{Maz02}.
The reduction of quality factors becomes larger
as the Nb thickness is decreased.
At $T<1$~K, however, the reduction
is negligibly small, and thus, in this sense,
it should be fine to choose
any thickness in the range of $t=0.05-0.3$~$\mu$m
for the study of superconducting qubits
that we mentioned in Sec.~\ref{sec:intro}
because qubit operations are almost
always done at the base temperatures.

When CPWs are no longer lossless, $\beta$
in Eq.~(\ref{eq:Tcpw}) has to be replaced by
$(\alpha+j\beta)/j$.
This $\alpha$ characterizes
the internal loss, and $\beta/(2\alpha)$ is
equal to $Q$ (for example, Sec.~7.2 of
Ref.~\onlinecite{Poz90}).
From similar calculations to those
in Sec.~\ref{subsec:baseT}, we evaluated
$L$ at higher temperatures as well
by neglecting the $T$ dependence of $C$ and $C_c$.
Because we are interested in the temperature variation of $L$,
we show $\Delta L(t,T)/L^*$ vs.\ $T$ in Fig.~\ref{fig:L_delta},
where $\Delta L(t,T) \equiv L(t,T) - L(t,T^*)$,
$T^*$ is the base temperature, and
$L^* \equiv L(\mbox{0.3~$\mu$m},T^*)$.
The variation becomes larger
as the Nb thickness is decreased.
This trend also suggests that we should take into account
the kinetic inductance.

\begin{figure}
\begin{center}
\includegraphics[width=0.7\columnwidth,clip]{fig7_L_delta_lr.eps}
\caption{\label{fig:L_delta}
(Color online)
Temperature variations of
inductance $L$ per unit length
for Resonators~A1--A4, whose Nb thickness
is $t=0.05,$ 0.1, 0.2, and 0.3~$\mu$m.
The unit of the vertical axis is percent.
See text for the definition of $\Delta L(t,T)/L^*$.
}
\end{center}
\end{figure}

\section{Discussion}
\label{sec:D}

The film-thickness and temperature dependence that we have
examined in
Sec.~\ref{sec:R}
is explained by
the model,
\begin{equation}
\label{eq:L}
L(t,T) = L_g(t) + L_k(t,T),
\end{equation}
where
$L_g$ is the usual magnetic inductance per unit length
determined by the CPW geometry and
$L_k$ is the kinetic inductance of the CPW center conductor
per unit length.
We neglect the contribution of the ground planes to $L_k$
because the ground planes are much wider than the center
conductor in our resonators [see Eq.~(\ref{eq:Lk})].
We also assume that $L_g$ depends on $t$ only,
whereas $L_k$ does on both $t$ and $T$.
This type of model has been employed
in earlier works\cite{Rau93,Fru05,Gop08} as well.
The $T$ dependence of $L_k$ arises from the fact that
$L_k$ is determined not only by the geometry
but also by the penetration depth $\lambda$, which
varies with $T$.
Meservey and Tedrow\cite{Mes69} calculated $L_k$ of
a superconducting strip, and when the strip has a rectangular
cross section like our CPWs,
$L_k$ is written as
\begin{equation}
\label{eq:Lk}
L_k = \frac{\mu_0}{\,\pi^2\,}(\lambda/w)\ln(4w/t)
\frac{\sinh(t/\lambda)}{\,\cosh(t/\lambda)-1\,},
\end{equation}
where $\mu_0=4\pi\times10^{-7}$~H/m
is the permeability of free space.
The relationship between $L_k$ and $\lambda$
is expressed in a much simpler form
in the thick- and thin-film limits;
$L_k\propto\lambda$ for
$t\gg\lambda$,
and
$L_k\propto\lambda^2$ for
$t\ll\lambda$.
When we assume Eqs.~(\ref{eq:L}) and (\ref{eq:Lk}),
we obtain $\lambda(t,T)$ numerically,
once $L_g(t)$ is given.
Below, we
discuss $\lambda(t,T)$ in our Nb films
in order to
confirm that the model represented
by Eq.~(\ref{eq:L}) is indeed appropriate.

\begin{figure}
\begin{center}
\includegraphics[width=0.8\columnwidth,clip]{fig8_lambda_tdep_lr.eps}
\caption{\label{fig:lambda_tdep}
(Color online)
(a) Temperature dependence of the penetration depth $\lambda$
for Resonators~A1--A4, whose Nb thickness
is $t=0.05,$ 0.1, 0.2, and 0.3~$\mu$m.
The solid and broken curves are theoretical predictions
expressed by Eqs.~(\ref{eq:2fluid}) and (\ref{eq:Rau}),
respectively.
As in Ref.~\onlinecite{Tin96},
$[\lambda(t,0)/\lambda(t,T)]^2$ is plotted vs.\ $T/T_c(t)$.
(b) Superconducting transition temperature $T_c(t)$ used
in (a), and $\lambda(t,0)$.
The curves are from Ref.~\onlinecite{Gub05},
that is, not fitted to our experimental data.
Both in (a) and (b),
$\lambda(T^*)\sim\lambda(0)$ is assumed, where
$T^*$ is the base temperature.
}
\end{center}
\end{figure}
\begin{table}
\caption
{\label{tab:Lm}
Inductance per unit length
at the base temperatures
in Resonators~A1--A4.
$t$ is the thickness of Nb film;
$\Delta L_g(t) = L_g(t) - L_g^*$,
where $L_g$ is the
usual magnetic inductance per unit length
determined by the CPW geometry,
and $L_g^*\equiv L_g(\mbox{0.3~$\mu$m})
=3.75\times10^{-7}$~H/m;
$L_k$ is the kinetic inductance per unit length,
and $L=L_g+L_k$.
}
\begin{ruledtabular}
\begin{tabular}{ccc}
$t$ ($\mu$m) & $\Delta L_g(t)/L_g^*$ (\%)&$L_k/L$ (\%)\\
 \hline
0.05           & 4.3 & 13.1\\
0.1\phantom{0} & 3.4 & \phantom{1}4.9\\
0.2\phantom{0} & 1.7 & \phantom{1}2.2\\
0.3\phantom{0} & -- & \phantom{1}1.6\\

\end{tabular}
\end{ruledtabular}
\end{table}

In Fig.~\ref{fig:lambda_tdep}(a), we plot
$[\lambda(t,T^*)/\lambda(t,T)]^2$ vs.\ $T/T_c(t)$
for Resonators~A1--A4,
where $T_c(t)$
is the superconducting transition temperature,
which is assumed to be also $t$ dependent
in this paper.
We have found that with a reasonable set of
parameters, $L_g(t)$ and $T_c(t)$,
the experimental data for all resonators
are described by a single curve.
This kind of scaling is expected theoretically
in the limits of $\xi_0/\lambda_L\gg 1$ and
$\xi_0/\lambda_L\ll 1$, where $\xi_0$ is the
coherence length and $\lambda_L$ is
the London penetration depth.\cite{Tin96}
Although $\xi_0/\lambda_L\sim1$ in Nb
(p.~353 of Ref.~\onlinecite{Kit96})
at temperatures well below $T_c$,
it would be still reasonable to expect a scaling
in our Nb resonators
because at a given
normalized temperature $T/T_c(t)$, the relevant
quantities should be on the same order of magnitude
in all resonators, and thus, two parameters,
$\lambda(t,T^*)$ and $T_c(t)$, are probably
enough for characterizing $\lambda(t,T)$ of
our resonators.
The values of $L_g(t)$ and $T_c(t)$ employed
in Fig.~\ref{fig:lambda_tdep}(a)
are summarized in Table~\ref{tab:Lm} and
Fig.~\ref{fig:lambda_tdep}(b), respectively.
The relative change of $L_g(t)$
in Table~\ref{tab:Lm} is
similar to the predictions by circuit simulators
in Table~\ref{tab:LC},
which do not take into account the kinetic inductance.
The magnitude of $L_g(t)$ is also reasonable because
$\sqrt{L_g(t)/C}\sim49$~$\Omega$ for all thickness.
In Table~\ref{tab:Lm}, we also list the ratio of
kinetic inductance $L_k$ to the total inductance $L$.
With decreasing thickness,
$L_k/L$ indeed increases rapidly.
In Fig.~\ref{fig:lambda_tdep}(b), $T_c(t)$ and
$\lambda(t,T^*)$ are plotted together with
the theoretical curves in Figs.~1 and 6 of
Ref.~\onlinecite{Gub05}, where
Gubin {\it et al.}\cite{Gub05} determined
some parameters of the curves
by fitting to their experimental data.
The values of $T_c(t)$ are reasonable,
and $\lambda(t,T^*)$ is on the right order of
magnitude.

The solid curve in Fig.~\ref{fig:lambda_tdep}(a)
is the theoretical $T$ dependence
based on the two-fluid approximation,\cite{Tin96}
\begin{equation}
\label{eq:2fluid}
[\lambda(0)/\lambda(T)]^2 = 1-(T/T_c)^4.
\end{equation}
This theoretical curve reproduces
the experimental data
at $T/T_c < 0.4$, when we assume that
$\lambda(t,T^*)\sim\lambda(t,0)$
in Resonators~A1--A4.
At $T/T_c \geq 0.4$, on the other hand,
the experimental data deviate from
Eq.~(\ref{eq:2fluid}), but
according to Ref.~\onlinecite{Tin96},
the expression for $\lambda$ vs.\ $T$ depends on
the ratio of $\xi_0/\lambda_L$, and thus,
Eq.~(\ref{eq:2fluid})
cannot be expected to apply to all materials equally well.
Indeed,
although the temperature dependence of Eq.~(\ref{eq:2fluid})
has been observed in the classic pure superconductors,\cite{Tin96}
such as Al with $\xi_0/\lambda_L \gg 1$ at temperatures
well below $T_c$, it does not seem to be the case in
the high-$T_c$ materials, whose typical $\xi_0/\lambda_L$
is in the opposite limit,\cite{Tin96} $\xi_0/\lambda_L \ll 1$,
and for example,
Rauch {\it et al.}\cite{Rau93} employed
for a high-$T_c$ material YBa$_2$Cu$_3$O$_{7-x}$,
an empirical expression of
\begin{equation}
\label{eq:Rau}
[\lambda(0)/\lambda(T)]^2 = 1-0.1(T/T_c)-0.9(T/T_c)^2,
\end{equation}
which is the broken curve in Fig.~\ref{fig:lambda_tdep}(a),
instead.
Because $\xi_0/\lambda_L\sim1$ in Nb
even at $T/T_c\ll 1$, and because
the experimental data at $T/T_c \geq 0.4$
are between Eqs.~(\ref{eq:2fluid}) and (\ref{eq:Rau}),
we believe that the deviation from Eq.~(\ref{eq:2fluid})
at $T/T_c \geq 0.4$ is reasonable.

From the discussion in this section,
we conclude that the model represented
by Eq.~(\ref{eq:L}) explains
the film-thickness and temperature dependence of
our resonators.

\section{Conclusion}
We investigated two series of
Nb $\lambda/2$ CPW resonators
with
resonant frequencies in the range of $10-11$~GHz
and with
different Nb-film thicknesses, $0.05-0.3$~$\mu$m.
We measured the transmission coefficient $S_{21}$
as a function of frequency at low temperatures, $T=0.02-5$~K.
For each film thickness,
we determined the phase velocity
in the CPW with an accuracy better than 0.1\%
by least-squares fitting of a theoretical
$S_{21}$ curve based on the transmission matrix
to the experimental data at the base temperatures.
Not only the film-thickness dependence but also the
temperature dependence of the resonators are
explained by taking into account the kinetic inductance
of the CPW center conductor.

\section*{Acknowledgment}
The authors would like to thank
Y. Kitagawa for fabricating the resonators,
and T. Miyazaki for fruitful discussion.
T.\ Y., K.\ M., and J.-S.\ T.\
would like to thank CREST-JST, Japan
for financial support.


\section{Introduction}

Artificial Intelligence has made tremendous progress in recent years due to the development of deep neural networks.
Its deployment at the edge, however, is currently limited by the high power consumption of the associated algorithms \cite{xu2018scaling}.
Low precision neural networks are currently emerging as a solution, as they allow the development of low power consumption hardware specialized in deep learning inference \cite{hubara2017quantized}.
The most extreme case of low precision neural networks, the Binarized Neural Network (BNN), also called XNOR-NET, is receiving particular attention as it is especially efficient for hardware implementation: both synaptic weights and neuronal activations assume only binary values \cite{courbariaux2016binarized,rastegari2016xnor}.
Remarkably, this type of neural network can achieve high accuracy on vision tasks \cite{lin2017towards}.
One particularly investigated lead is to fabricate hardware BNNs with emerging memories such as resistive RAM or memristors
\cite{bocquet2018,yu2016binary,giacomin2019robust,sun2018xnor,zhou2018new,natsui2018design,tang2017binary, lee2019adaptive}.
The low memory requirements of BNNs, as well as their reliance on simple arithmetic operations, make them indeed particularly adapted for ``in-memory'' or ``near-memory'' computing approaches, which achieve superior energy-efficiency by avoiding the von~Neumann bottleneck entirely.

Ternary neural networks \cite{alemdar2017ternary} (TNN, also called Gated XNOR-NET, or GXNOR-NET \cite{deng2018gxnor}), which add the value $0$ to synaptic weights and activations,
are also considered for hardware implementations \cite{ando2017brein,prost2017scalable,li2017design,pan2019skyrmion}.
They are comparatively receiving less attention than binarized neural networks, however. In this work, we highlight that implementing TNNs does not necessarily imply considerable overhead with regards to BNNs.
We introduce a two-transistor/two-resistor  memory architecture for TNN implementation. The array uses a precharge sense amplifier for reading weights, and the ternary weight value can be extracted in a single sense operation, by exploiting the fact that latency of the sense amplifier depends on the resistive states of the memory devices.
This work extends a  hardware developed for the energy-efficient
implementation of BNNs \cite{bocquet2018}, where the synaptic weights are implemented in a differential fashion.
We, therefore, show that it can be extended to TNNs without overhead on the memory array.

The contribution of this work is as follows. After presenting the background of the work (section~\ref{sec:background}):
\begin{itemize}
\item We demonstrate experimentally, on a fabricated 130~nm RRAM/CMOS hybrid chip, a  strategy for implementing ternary weights using a precharge sense amplifier, which is particularly appropriate when the sense amplifier is operated at low supply voltage
(section~\ref{sec:circuit}).
\item We analyze  the bit errors of this scheme experimentally and their dependence on the RRAM programming conditions (section~\ref{sec:programmability}).
\item We verify the robustness of the approach to process, voltage, and temperature variations (section~\ref{sec:PVT}).
\item We carry simulations that show the superiority of TNNs over BNNs on the canonical CIFAR-10 vision task, and evidence the error resilience of hardware TNNs (section~\ref{sec:network}).
\item We discuss the results, and compare our approach with the idea of storing three resistance levels per device.
\end{itemize}

Partial and preliminary results of this work have been presented at a conference \cite{laborieux2020low}.
This journal version adds the experimental characterization of bit errors in our architecture, supported by a comprehensive analysis of the impact on process, voltage, and temperature variations, and their impact at the neural network level, together with a detailed analysis of the use of ternary networks over binarized ones.

\section{Background}
\label{sec:background}

 \begin{figure}[t]
	\centering
	\includegraphics[width=3.0in]{fig1.pdf}
	\caption{(a) Electron microscopy image of a hafnium oxide resistive memory cell (RRAM) integrated in the backend-of-line of a $130\nano\meter$ CMOS process. (b) Photograph and (c) simplified schematic of the hybrid CMOS/RRAM test chip characterized in this work.
	The white rectangle in (b) materializes a single PCSA.}
	\label{fig:testchip}
\end{figure}

The main equation in conventional neural networks is the computation of the neuronal activation $A_j =  f \left( \sum_i W_{ji}X_i \right),$ where $A_j$, the synaptic weights $W_{ji}$, and input neuronal activations $X_i$ assume real values, and $f$ is a non-linear activation function.
Binarized neural networks (BNNs) are a considerable simplification of conventional neural networks, in which all neuronal activations ($A_j$, $X_i$) and synaptic weights $W_{ji}$  can only take binary values meaning $+1$ and $-1$.
Neuronal activation then becomes:
\begin{equation}
\label{eq:activ_BNN}
    A_j = \mathrm{sign} \left( \sum_i  XNOR \left( W_{ji},X_i \right) -T_j \right),
\end{equation}
where $\mathrm{sign}$ is the sign function,   $T_j$ is a threshold associated with the neuron, and the $XNOR$ operation is defined in  Table~\ref{tab:gates}.
Training BNNs is a relatively sophisticated operation,  during which each synapse needs to be associated with a real value in addition to its binary value (see Appendix). Once training is finished, these real values can be discarded, and the neural network is entirely binarized.
Due to their reduced memory requirements, and reliance on simple arithmetic operations, BNNs are especially appropriate for in- or near- memory implementations.
In particular, multiple groups investigate the implementation of BNN inference with resistive memory tightly integrated at the core of CMOS \cite{bocquet2018,yu2016binary,giacomin2019robust,sun2018xnor,zhou2018new,natsui2018design,tang2017binary, lee2019adaptive}. Usually, resistive memory stores the synaptic weights $W_{ji}$.
However, this comes with a significant  challenge: resistive memory is prone to bit errors, and in digital applications, is typically used with strong error-correcting codes (ECC).
ECC, which requires large decoding circuits \cite{gregori2003chip}, goes against the principles of in- or near- memory computing.
For this reason,  \cite{bocquet2018} proposes a two-transistor/two-resistor (2T2R) structure, which reduces resistive memory bit errors, without the need for ECC decoding circuit, by storing synaptic weights in a differential fashion.
This architecture allows  the extremely efficient implementation of BNNs, and  using the resistive memory devices in very favorable programming conditions (low energy, high endurance).
 It should be noted that systems using this architecture function with row-by-row read operations, and do not use the in-memory computing technique of using the Kirchhoff current law to perform the sum operation  of neural networks, while reading all devices at the same time \cite{prezioso2015training,ambrogio2018equivalent}. This choice limits the parallelism of such architectures, while at the same time avoiding the need of analog-to-digital conversion and analog circuits such as operational amplifiers, as discussed in detail in \cite{hirtzlin2019digital}.

In this work, we show that the same architecture can be used for a generalization of BNNs -- ternary neural networks (TNNs)\footnote{
In the literature, the name  ``Ternary Neural Networks'' is sometimes  also used to refer to neural networks where the synaptic weights are ternarized, but the neuronal activations remain real or integer \cite{mellempudi2017ternary,nurvitadhi2017can}.},
where neuronal activations and synaptic weights
$A_j$, $X_i$, and $W_{ji}$ can now assume three  values: $+1$, $-1$, and $0$.
Equation~\eqref{eq:activ_BNN} now becomes:
\begin{equation}
\label{eq:activ_TNN}
    A_j = \phi \left( \sum_i  GXNOR \left( W_{ji},X_i \right) -T_j \right).
\end{equation}
$GXNOR$ is the ``gated'' XNOR operation that realizes the product between numbers with values $+1$, $-1$ and $0$ (Table~\ref{tab:gates}).
$\phi$ is an activation function that outputs $+1$ if its input is greater than a threshold $\Delta$, $-1$ if the input is lesser than $-\Delta$ and $0$ otherwise.
We show experimentally and by circuit simulation in sec.~\ref{sec:circuit} how the 2T2R BNN architecture can be extended to TNNs with practically no overhead, in sec.~\ref{sec:programmability} its bit errors, and in sec.~\ref{sec:network} the corresponding benefits in terms of neural network accuracy.

\begin{table}[tbp]
\caption{Truth Tables of the XNOR and GXNOR Gates}
\begin{center}
\begin{tabular}{|c|c|c|}
\hline
$W_{ji}$ & $X_i$ & $XNOR$  \\
\hline
$-1$ & $-1$ & $1$  \\
$-1$ & $1$ & $-1$  \\
$1$ & $-1$ & $-1$  \\
$1$ & $1$ & $1$  \\
\hline
\end{tabular}
\begin{tabular}{|c|c|c|}
\hline
$W_{ji}$ & $X_i$ & $GXNOR$  \\
\hline
$-1$ & $-1$ & $1$  \\
$-1$ & $1$ & $-1$  \\
$1$ & $-1$ & $-1$  \\
$1$ & $1$ & $1$  \\
$0$ & $X$ & $0$  \\
$X$ & $0$ & $0$  \\
\hline
\end{tabular}
\label{tab:gates}
\end{center}
\end{table}

\section{The Operation of A Precharge Sense Amplifier Can Provide Ternary Weights}
\label{sec:circuit}

\begin{figure}[htbp]
	\centering
	\includegraphics[width=3.4in]{fig2.pdf}
	\caption{Schematic of the precharge sense amplifier fabricated in the test chip.}
	\label{fig:PCSA}
	\vspace{0.5cm}
\end{figure}

\begin{figure}[htbp]
	\centering
	\includegraphics[width=\linewidth]{fig3.pdf}
	\caption{Circuit simulation of the precharge sense amplifier of Fig.~\ref{fig:PCSA} with a supply voltage of $1.2\volt$, using thick oxide transistors (nominal voltage of $5 \volt$)}, if the two devices are programmed in an (a) LRS / HRS ({$5\kilo\ohm$/$350\kilo\ohm$)} or (b) HRS/HRS ({$320\kilo\ohm$/$350\kilo\ohm$}) configuration.

	\label{fig:SPICE}
\end{figure}
In this work, we use the architecture of \cite{bocquet2018}, where synaptic weights are stored in a differential fashion.
Each bit is implemented using two devices programmed either as low resistance state (LRS) / high resistance state (HRS) to mean weight $+1$ or HRS/LRS to mean weight $-1$. Fig.~\ref{fig:testchip} presents the test chip used for the experiments.
This chip cointegrates $130\nano\meter$ CMOS and resistive memory in the back-end-of-line, between levels four and five of metal. The resistive memory cells are based on $10\nano\meter$ thick hafnium oxide (Fig.~\ref{fig:testchip}(a)).
All devices are integrated with a series NMOS transistor.
After an initial forming step (consisting in the application of a voltage ramp from zero volts to $3.3\volt$ at a rate of $1000\volt\per\second$, and with a current limited to a compliance of $200\micro\ampere$),
the devices can switch between high resistance state (HRS) and low resistance state (LRS), through the dissolution or creation of conductive filaments of oxygen vacancies.
Programming into the HRS is obtained by the application of a negative RESET voltage pulse (typically between $1.5\volt$ and $2.5\volt$ during $1\micro\second$). Programming into the LRS is obtained by the application of a positive SET pulse (also typically between $1.5\volt$ and $2.5\volt$ during $1 \micro\second$), with current limited to a compliance current through the choice of the voltage applied on the transistor gate through the word line.
This test chip is designed with highly conservative sizing, allowing the application of a wide range of voltages and electrical currents to the RRAM cells. The area of each bit cell is $6.6 \times 6.9 \micro\meter ^2$.
More details on the RRAM technology are provided in \cite{hirtzlin2019digital}.

Our experiments are based on a $2,048$ devices array incorporating all sense and periphery circuitry, illustrated in Fig.~\ref{fig:testchip}(b-c).
The ternary synaptic weights are read using  on-chip precharge sense amplifiers (PCSA), presented in Fig.~\ref{fig:PCSA}, and initially proposed  in \cite{zhao2009high} for reading spin-transfer magnetoresistive random access memory.
Fig.~\ref{fig:SPICE}(a) shows an electrical simulation of this circuit to explain its working principle, using the Mentor Graphics Eldo simulator.
These first simulations are presented in the commercial $130\nano\meter$ ultra-low leakage technology,
used in our test chip, with a low supply voltage of $1.2 \volt$ \cite{nearth}, with thick oxide transistors (the nominal voltage in this process for thick oxide transistor is $5 \volt$). Since the technology targets ultra-low leakage applications the threshold voltages are significantly high (around $0.6 \volt$), thus a supply voltage of $1.2 \volt$ significantly reduces the overdrive of the transistors ($V_{GS} - V_{TH}$).

In the first phase (SEN=0), the outputs Q and Qb are precharged to the supply voltage $V_{DD}$.
In the second phase (SEN=$V_{DD}$), each branch starts to discharge to the ground. The branch that has the resistive memory (BL or BLb) with the lowest  electrical resistance discharges faster and causes its associated inverter to drive the output of the other inverter to the supply voltage. At the end of the process, the two outputs are therefore complementary and can be used to tell which resistive memory has the highest resistance and therefore the synaptic weight.
We observed that the convergence speed of a PCSA depends heavily on the resistance state of the two resistive memories.
This effect is particularly magnified when the PCSA is used with a reduced overdrive,  as presented here:
the operation of the sense amplifier is slowed down, with regards to nominal voltage operation, and the convergence speed differences between resistance values become more apparent.
Fig.~\ref{fig:SPICE}(b) shows a simulation where the two devices, BL and BLb, were programmed in the HRS. We see that the two outputs converge to complementary values in more than $200\nano\second$, whereas less than  $50 \nano\second$ were necessary in Fig.~\ref{fig:SPICE}(a), where the devices are programmed in complementary LRS/HRS states.

These first simulations suggest a technique for implementing ternary weights using the memory array of our test chip. Similarly to when this array is used to implement BNN,  we propose to program the devices in the LRS/HRS configuration to mean the synaptic weight $1$, and HRS/LRS  to mean the synaptic weight $-1$. Additionally, we use the HRS/HRS configuration to mean synaptic weight $0$, while the LRS/LRS configuration is avoided.  The sense operation is performed during a duration of $50\nano\second$. If at the end of this period, outputs Q and Qb have differentiated, causing the output of the XOR gate to be 1, output Q determines the synaptic weight ($1$ or $-1$). Otherwise,  the output of the XOR gate is 0, and the weight is  determined to be $0$.

This type of coding is reminiscent to the one used by the 2T2R ternary content-addressable memory (TCAM) cell of  \cite{yang2019ternary}, where the LRS/HRS combination is used for coding $0$, the HRS/LRS combination for coding $1$, and the HRS/HRS combination for coding ``don't care'' (or X).

\begin{figure}[tbp]
\centering
\includegraphics[width=3.4in]{fig4.pdf}
\caption{
Two devices have been programmed in four distinct programming conditions, presented in (a), and measured using an on-chip sense amplifier. (b) Proportion of read operations that have converged in $50\nano\second$, over 100 trials.
} 	\label{fig:SenseVsR} \end{figure}

\begin{figure}[tbp]
	\centering
	\includegraphics[width=2.8in]{fig5.pdf}
	\caption{For 109 device pairs programmed with multiple $R_{BL}/R_{BLb}$ configuration, value of the synaptic weight measured by the on-chip sense amplifier using the  strategy described in  body text and $50\nano\second$ reading time.
	} 	\label{fig:multiniveau_2}
\end{figure}

Experimental measurements on our test chip confirm that the PCSA can be used in this fashion.
We first focus on one synapse of the memory array. We program one of the two devices (BLb) to a resistance of $100\kilo\ohm$. We then program its complementary device BL to several resistance values, and for each of them perform 100 read operations of duration $50\nano\second$, using on-chip PCSAs.

These PCSAs are fabricated using thick-oxide transistors, designed for a nominal supply voltage of $5V$, and here used with a supply voltage of $1.2V$, close to their threshold voltage ($0.6V$), to reduce their overdrive, and thus to exacerbate the PCSA delay variations.
In the test chip, they are sized conservatively with a total area of $290 \micro\meter ^2$.
The use of thick oxide transistors in this test chip allows us to investigate the behavior of the devices at high voltages, without the concern of damaging the CMOS periphery circuits.
Fig.~\ref{fig:SenseVsR}
plots the probability that the sense amplifier has converged during the read time.
In  $50\nano\second$, the read operation is only converged if the resistance of the BL device is significantly lower than $100\kilo\ohm$.

To evaluate this behavior in a broader range of programming conditions, we repeated the experiment on 109 devices and their complementary devices of the memory array programmed, every 14 times, with various resistance values in the resistive memory, and performed a read operation  in $50\nano\second$ with an on-chip PCSA.
The memory array of our test chip features one separate PCSA per column. Therefore, 32 different PCSAs are used in our results.
Fig.~\ref{fig:multiniveau_2}(a) shows, for each couple of resistance values $R_{BL}$ and $R_{BLb}$ if the read operation was converged with $Q=V_{DD}$ (blue), meaning a weight of $1$, converged with $Q=0$ (red), meaning a weight of $-1$, or not converged (grey) meaning a weight of $0$.

The results confirm that  LRS/HRS or HRS/LRS configurations may be used to mean weights $1$ and $-1$, and HRS/HRS for weight $0$.
When both devices are in HRS (resistance higher than $100\kilo\ohm$, the PCSA never converges within $50\nano\second$ (weight of $0$). When one device is in LRS (resistance lower than $10\kilo\ohm$, the PCSA always converges within $50\nano\second$ (weight of $\pm1$).
The separation between the $1$ (or $-1$) and $0$ regions is not strict, and for intermediate resistance values, we see that the read operation may or may not converge in $50\nano\second$.
Fig.~\ref{fig:multiniveau_2}(b) summarizes the different operation regimes of the PCSA.

\begin{table}[h]
\begin{center}
	\vspace{0.5cm}
\caption{Error Rates on Ternary Weights Measured Experimentally}
\begin{tabular}{|l|c|c|c|}
\hline
 Programming & Type 1  & Type 2 & Type 3
 \\ Conditions& ($1\longleftrightarrow -1$)& ($\pm 1\rightarrow0$) &   ($0 \rightarrow \pm 1$)
  \\ \hline
Fig.~\ref{fig:distrib}(a) & $<10^{-6}$ &  $<1\%$ &  $6.5\%$
\\Fig.~\ref{fig:distrib}(b) & $<10^{-6}$ &  $<1\%$ &  $18.5\%$
 \\ \hline
\end{tabular}
	\vspace{0.5cm}
\label{table:exp_errors}
\end{center}
\end{table}

\section{
Impact of Process, Voltage, and Temperature Variations}
\label{sec:PVT}

\begin{figure*}[tbp]
	\centering
	\includegraphics[width=1.0\textwidth]{fig6.pdf}
    \caption{
    Three Monte Carlo SPICE-based simulation of the experiments of Fig.~\ref{fig:multiniveau_2}, in three situations: (a) slow transistors ($0 \celsius$ temperature,  $1.1\volt$ supply voltage), (b) experimental conditions  ($27 \celsius$ temperature,  $1.2\volt$ supply voltage), (c) fast transistors ($60 \celsius$ temperature,  $1.3\volt$ supply voltage). The simulations include local and global process variations, as well as transistor mismatch, in a way that each point in the Figure is obtained using different transistor parameters. All results are plotted in the same manner and with the same conventions as Fig.~\ref{fig:multiniveau_2}.
	} 	\label{fig:PVT}
\end{figure*}

We now verify the robustness of the proposed scheme to process, voltage, and temperature variation.
For this purpose, we performed extensive circuit simulations of the operation of the sense amplifier, reproducing the conditions of the experiments of Fig.~\ref{fig:multiniveau_2}, using the same resistance values for the RRAM devices, and including process, voltage, and temperature variations. The results of the simulations are processed and plotted using the same format as the experimental results of Fig.~\ref{fig:multiniveau_2}, to ease comparison.

These simulations are obtained using the Monte Carlo simulator provided by the Mentor Graphics Eldo tool with parameters validated on silicon, provided by the design kit of our commercial CMOS process. Each point in the graphs of Fig.~\ref{fig:PVT} therefore features different transistor parameters. We included global and local process variations, as well as transistor mismatch, in order to capture the whole range of transistor variabilities observed in silicon.
In order to assess the impact of voltage and temperature variations, these simulations are presented in three conditions:
slow transistors ($0 \celsius$ temperature, and $1.1\volt$ supply voltage, Fig.~\ref{fig:PVT}(a)), experimental conditions ($27 \celsius$ temperature, and $1.2\volt$ supply voltage, Fig.~\ref{fig:PVT}(b)), and fast transistors ($60 \celsius$ temperature, and $1.3\volt$ supply voltage, Fig.~\ref{fig:PVT}(c)).
The RRAM devices are modeled by resistors. Their process variations are naturally included through the use of different resistance values in Fig.~\ref{fig:PVT}. The impact of voltage variation on RRAM is naturally included through Ohm's law, and the impact of temperature variation, which is smaller than on transistors, is neglected.

In all three conditions, the simulation results appear very similar to the experiments. Three clear regions are observed: non-convergence of the sense amplifier within $50ns$ for devices in HRS/HRS, and convergence within this time to a $+1$ or $-1$ value for devices in LRS/HRS and HRS/LRS, respectively. However, the frontier between these regimes is much sharper in the simulations than in the experiments. As the different data points in Fig.~\ref{fig:PVT} differ by process and mismatch variations, this suggests that process variation does not cause the stochasticity observed in the experiments of Fig.~\ref{fig:multiniveau_2}, and that they have little impact in our scheme.

We also see that the frontier between the different sense regimes in all three operating conditions remains firmly within the $10-100 k\Omega$ range, suggesting that even high variations of voltage ($\pm 0.1V$) and temperature ($\pm 30 ^\circ C$) do not endanger the functionality of our scheme. Logically, in the case of fast transistors, the frontier is shifted toward higher resistances, whereas in the case of slow transistors, it is shifted toward lower resistances. Independent simulations allowed  verifying that this change is mostly due to the voltage variations: the temperature variations have an almost negligible impact on the proposed scheme.

We also observed that the impact of voltage variations increased importantly when reducing the supply voltage. For example, with a supply voltage of $0.7V$ instead of the $1.2V$ value considered here, variations of the supply voltage of $\pm0.1V$ can impact the mean switching delay of the PCSA, by a factor two. The thick oxide transistors used in this work have a nominal voltage of $5V$, and a typical threshold voltage of approximately $0.6V$. Therefore, although our scheme is especially appropriate for supply voltages far below the nominal voltage, it is not necessarily appropriate for voltages in the subthreshold regime, or very close to the threshold voltage.

\section{Programmability of Ternary Weights}
\label{sec:programmability}

\begin{figure}[h]
	\centering
	\includegraphics[width=3.4in]{fig7.pdf}
	\caption{Distribution of the LRS and HRS states programmed with a SET compliance of $200\micro\ampere$, RESET voltage of $2.5\volt$ and programming pulses of (a) $100\micro\second$ and (b) $1\micro\second$. Measurements are performed $2,048$ RRAM devices, separating bit line (full lines) and bit line bar (dashed lines) devices.}
	\label{fig:distrib}
\end{figure}

To ensure reliable functioning of the ternary sense operation,
we have seen that devices in LRS should be programmed to electrical resistance below $10\kilo\ohm$, and devices in HRS to resistances above $100\kilo\ohm$ (Fig.~\ref{fig:multiniveau_2}(b)). The electrical resistance of resistive memory devices depends considerably on their programming conditions \cite{hirtzlin2019digital,grossi2016fundamental}.
Fig.~\ref{fig:distrib} shows the distributions of LRS and HRS resistances using two programming conditions, over the $2,048$ devices of the array, differentiating devices connected to bit lines and to bit lines bar.
We see that in all cases, the LRS features a tight distribution. The SET process is indeed controlled by a compliance current that naturally stops the filament growth at a targeted resistance  value \cite{bocquet2014robust}. An appropriate choice of the compliance current can ensure LRS below $10\kilo\ohm$ in most situations.

On the other hand, the HRS shows a broad statistical distribution. In the RESET process, the filament indeed breaks in a random process, making it extremely hard to control the final state \cite{bocquet2014robust,ly2018role}. The use of stronger programming conditions leads to higher values of the HRS.

This asymmetry between the variability of LRS and HRS means that in our scheme, the different ternary weight values feature different error rates naturally.
The ternary error rates in the two programming conditions of Fig.~\ref{fig:distrib}(a) are listed in Table~\ref{table:exp_errors}.
Errors of Type~1,  where weight values of $1$ and $-1$ are inverted are the least frequent.
Errors of Type~2, where a weight value of $1$ or $-1$ is replaced by a weight value of $0$ are infrequent as well.
On the other hand, due to the large variability of the HRS, weight values $0$ have a significant probability to be measured as $1$ or $-1$ (Type~3 errors):  $6.5\%$ in the conditions of Fig.~\ref{fig:distrib}(a), and $18.5\%$ in the conditions of Fig.~\ref{fig:distrib}(b).

Some resistive memory technologies with large memory windows, such as specifically optimized conductive bridge memories \cite{vianello2014resistive}, would feature lower Type~3 error rates.
Similarly, program-and-verify strategies \cite{lee2012multi,alibart2012high,xu2013understanding} may reduce this error rate.
Nevertheless, the higher error rate for zeros than for $1$ and $-1$ weights is an inherent feature of our architecture.
Therefore, in the next section, we  assess the impact of these errors on the accuracy of neural networks.

\begin{figure}[bp]
	\centering
	\includegraphics[width=3.4in]{fig8.pdf}
	\caption{Simulation of the
	maximum test accuracy reached during one training procedure, averaged over five trials, for BNNs and TNNs with various model sizes on the CIFAR-10 dataset. Error bar is one standard deviation.}
	\label{fig:CIFAR10}
\end{figure}

\section{Network-Level Implications}
\label{sec:network}

We first investigate the accuracy gain when using ternarized instead of binarized networks.
We trained BNN and TNN versions of networks with Visual Geometry Group (VGG) type architectures
\cite{simonyan2014very}
on the CIFAR-10 task of image recognition, consisting in classifying 1,024 pixels color images among ten classes (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck) \cite{krizhevsky2009learning}.
Simulations are performed using PyTorch 1.1.0 \cite{paszke2017automatic} on  a cluster of eight  Nvidia GeForce RTX 2080 GPUs.

The architecture of our networks consists of six convolutional layers with kernel size three. The number of filters  at the first layer is called $N$ and is multiplied by two every two layers. Maximum-value pooling with kernel size two is used every two layers and batch-normalization \cite{ioffe2015batch} every layer.
The classifier consists of one hidden layer of 512 units. For the TNN, the activation function has a threshold $\Delta = 5\cdot10^{-2}$ (as defined in section~\ref{sec:background}).
The training methods for both the BNN and the TNN are described in the Appendix.
The training is performed using the AdamW optimizer \cite{kingma2014adam, loshchilov2017fixing}, with minibatch size $128$.
The initial learning rate is set to $0.01$, and the learning rate schedule from  \cite{loshchilov2017fixing, loshchilov2016sgdr} (Cosine annealing with two restarts, for respectively $100$, $200$, $400$ epochs) is used, resulting in a total of $700$ epochs.
Training data is augmented using random horizontal flip, and random choice between cropping after padding and random small rotations.

No error is added during the training procedure, as our device is meant to be used for inference. The synaptic weights encoded by device pairs would be set after the model has been trained on a computer.

Fig.~\ref{fig:CIFAR10} shows the maximum test accuracy resulting from these training simulations, for different sizes of the model. The error bars represent one standard deviation of the training accuracies.
TNNs always outperform BNNs with the same model size (and, therefore, the same number of synapses).
The most substantial difference is seen for smaller model size, but a significant gap remains even for large models.
Besides, the difference in the number of parameters required to reach a given accuracy for TNNs and BNNs increases with higher accuracies. There is, therefore, a definite advantage to use TNNs instead of BNNs.

\begin{figure}[tbp]
	\centering
	\includegraphics[width=3.4in]{fig9.pdf}
	\caption{Simulation of the
	impact of Bit Error Rate on the test accuracy at inference time for model size $N = 128$ TNN in (a) and BNN in (b). Type 1 errors are sign switches (e.g. $+1$ mistaken for $-1$), Type 2 errors are
	$\pm 1$ mistaken for $0$, and Type 3 errors are $0$ mistaken for $\pm 1$, as described in the inset schematics. Errors are sampled at each mini batch and the test accuracy is averaged over five passes through the test set. Error bars are one standard deviation. The bit error rate is given as an absolute rate.}
	\label{fig:BT_errors}
\end{figure}

Fig.~\ref{fig:CIFAR10} compared fully ternarized (weights and activations) with regards to fully binarized (weights and activations) ones. Table~\ref{table:acc_gain} lists the impact of weight ternarization for different types of activations (binary, ternary, and real activation).
All results are reported  on a model of size $N=128$, trained  on CIFAR-10, and are averaged over five training procedures.
We observe that for BNNs and TNNs with quantized activations, the accuracy gains provided by ternary weights over binary weights are $0.84$ and $0.86$ points and are statistically significant over the standard deviations. This accuracy gain is more important than the  gain provided by ternary activations over binary activations, which is about $0.3$ points.
This bigger impact of weight ternarization over ternary activation may come from the ternary kernels having a better expressing power over binary kernels, which are often redundant in practical settings \cite{courbariaux2016binarized}.
The gain of ternary weights drops to $0.26$ points if real activation is allowed (using rectified linear unit, or ReLU, as activation function, see appendix), and is not statistically significant considering the standard deviations.

Quantized activations are vastly more favorable in the context of hardware implementations, and in this situation, there is thus a statistically significant benefit provided by ternary weights over binary weights.

\begin{table}[!h]
\begin{center}
\vspace{0.5cm}
\caption{Comparison of the gain in test accuracy for a $N=128$ model size on CIFAR-10 obtained by weight ternarization instead of binarization for three types of activation quantization. }
\begin{tabular}{|l|c|c|c|}
\hline
& \multicolumn{3}{c|}{\textbf{Activations}}   \\
 & Binary & Ternary & Full Precision \\ \hline
 \textbf{Weights} & & & \\
Binary                             & $91.19\pm0.08$   & $91.51 \pm 0.09$    & $93.87 \pm 0.19$           \\
Ternary                            & $92.03\pm 0.12$   & $92.35 \pm 0.05$    & $94.13 \pm 0.10$           \\
\textit{Gain of ternarization}
& \textit{0.84 }
& \textit{0.86}
& \textit{0.26}            \\ \hline
\end{tabular}
\vspace{0.5cm}
\label{table:acc_gain}
\end{center}
\end{table}

We finally investigate the impact of bit errors in BNNs and TNNs to see if the advantage provided by using TNNs in our approach remains constant when errors are taken into account. Consistently with the results reported in section~\ref{sec:programmability}, three types of errors are investigated: Type 1 errors are sign switches,
e.g., $+1$ mistaken for $-1$,
Type 2 errors are only defined for TNNs and correspond to  $\pm1$ mistaken for $0$, and Type 3 errors are $0$ mistaken for $\pm 1$, as illustrated in the inset schematic of Fig.~\ref{fig:BT_errors}(a).

Fig.~\ref{fig:BT_errors}(a) shows the impact of these  errors on the test accuracy for different values of the  error rate at inference time. These simulation results are presented on CIFAR-10    with a model size of $N=128$. Errors are randomly and artificially introduced in the weights of the neural network .
Bit errors are included at the layer level and sampled at each mini-batch of the test set. Type 1 errors switch the sign of a synaptic weight with a probability equal to the rate of type 1 errors. Type 2 errors set a non-zero synaptic weight to 0 with a probability equal to the type 2 error rate. Type 3 errors set a synaptic weight of 0 to $\pm1$ with a probability equal to the type 3 error rate, the choice of the sign ($+1$ or $-1$) is made with 0.5 probability. Fig.~\ref{fig:BT_errors} is obtained by averaging the test accuracy obtained for five passes through the test set for increasing bit error rate.

Type~1 errors have the most impact on neural network accuracy. As seen in Fig.~\ref{fig:BT_errors}(b), the impact of these errors is similar to the impact of weight errors in a BNN.
On the other hand, Type~3 errors have the least impact, with bit error rates as high as $20\%$ degrading surprisingly little the accuracy.
This result is fortunate, as we have seen in section~\ref{sec:programmability} that Type~3 errors are the most frequent in our architecture.

We also performed simulations considering all three types of error at the same time, with error rates reported in Table~\ref{table:exp_errors} corresponding to the programming conditions of Fig.~\ref{fig:distrib}(a) and \ref{fig:distrib}(b).
For Type~1 and Type~2 errors, we considered the upper limits listed in Table~\ref{table:exp_errors}.
For the conditions of Fig.~\ref{fig:distrib}(a) (Type~3 error rate of 6.5\%), the test accuracy was degraded from 92.2\% to $92.05 \pm 0.14 \%$, and to $92.02 \pm 0.17 \%$ for the conditions of \ref{fig:distrib}(b) (Type~3 error rate of 18.5\%), where the average and standard deviation is performed over 100 passes through the test set.
We found that the slight degradation on CIFAR-10 test accuracy was mostly due to the Type~2 errors, although Type 3 errors are much more frequent.

The fact that mistaking a $0$ weight for a $\pm1$ weight (Type~3 error) has much less impact than mistaking a $\pm1$ weight for a $0$ weight (Type~2 error) can seem surprising.
However, it is known, theoretically and practically, that in BNNs, some weights have  little importance to the accuracy of the neural networks \cite{laborieux2020synaptic}. They typically correspond to synapses that feature a $0$ weight in a TNN, whereas synapses with $\pm1$ weights in a TNN correspond to ``important'' synapses of a BNN. It is thus understandable that errors on such synapses have more impact on the final accuracy of the neural network.

\section{
Comparison with Three-Level Programming}
\label{sec:three-level}

An alternative approach to implementing ternary weights with resistive memory can be to program the individual devices into there separate levels. This idea is feasible, as the resistance level of the LRS can to a large extent be controlled through the choice of the compliance current during the SET operation in many resistive memory technologies \cite{bocquet2014robust,hirtzlin2019digital}.

The obvious advantage of this approach is that it requires a single device per synapse. This idea also brings several challenges. First, the sense operation has to be more complex. The most natural technique is to perform two sense operations, comparing the resistance of a device under test to  two different thresholds. Second, this technique is much more prone to bit errors than our technique, as states are not programmed in a differential fashion \cite{hirtzlin2019digital}. Additionally, this approach does not feature the natural resilience to Type~1 and Type~2 errors, and Type~2 and Type~3 errors will typically feature similar rates. Finally, unlike ours, this approach is prone to resistive drift, inherent to some resistive memory technologies \cite{li2012resistance}.

These comments suggest that the choice of a technique for storing ternary weights should be dictated by technology. Our technique is especially appropriate for resistive memories not supporting single-device multilevel storage, with high error rates, or resistance drift. The three-levels per devices approach would be the most appropriate with devices with well controlled analog storage properties.

\section{Conclusion}

In this work, we revisited a differential memory architecture for BNNs.
We showed experimentally on a hybrid CMOS/RRAM chip that,
its sense amplifier can differentiate not only the LRS/HRS and HRS/LRS states, but also the HRS/HRS states in a single sense operation.
This feature allows the architecture to store ternary weights, and to provide a building block for ternary neural networks.
We showed by neural network simulation on the CIFAR-10 task the benefits of using ternary instead of binary networks, and the high resilience of TNNs to weights errors, as the type of errors observed experimentally in our scheme is also the type of errors to which TNNs are the most immune.
This resilience allows the use of our architecture without relying on any formal error correction.
Our approach also appears resilient to process, voltage, and temperature variation if the supply voltage remains reasonably higher than the threshold voltage of the transistors.

As this behavior of the sense amplifier is exacerbated at supply voltages below the nominal voltage, our approach especially targets extremely energy-conscious applications such as uses within wireless sensors or medical applications. This work opens the way for increasing the edge intelligence in such contexts, and also highlights that the low voltage operation of circuits may sometimes provide opportunities for new functionalities.

\section*{Acknowledgments}
The authors would like to thank M.~Ernoult and L.~Herrera Diez for fruitful discussions.

\section*{Appendix: Training Algorithm of Binarized and Ternary Neural Networks}

\begin{algorithm}[ht!]{\emph{Input}: $W^{\rm h}$, $\theta^{\rm BN} = (\gamma_l, \beta_l)$, $U_W$, $U_{\theta}$, $(X, y)$, $\eta$. \\
\emph{Output}: $W^{\rm h}$, $\theta^{\rm BN}$, $U_W$, $U_{\theta}$.}
\caption{Training procedure for binary and ternary neural networks. $W^{\rm h}$ are the hidden weights, $\theta^{\rm BN} = (\gamma_l, \beta_l)$ are Batch Normalization parameters, $U_W$ and $U_{\theta}$ are the parameter updates prescribed by the Adam algorithm \cite{kingma2014adam}, $(X, y)$ is a batch of labelled training data, and $\eta$ is the learning rate. ``cache'' denotes all the intermediate layers computations needed to be stored for the backward pass. ${\rm Quantize}$ is either $\phi$ or ${\rm sign}$ as defined in section \ref{sec:background}. ``~$\cdot$~'' denotes the element-wise product of two tensors with compatible shapes.}\label{algo}
\begin{algorithmic}[1]
\State $W^{\rm Q} \leftarrow {\rm Quantize} ( W^{\rm h})$ \Comment{Computing quantized weights}
\State $A_0 \leftarrow X$ \Comment{Input is not quantized}
\For{$l=1$ to $L$} \Comment{For loop over the layers}
\State $z_l \leftarrow W^{\rm Q}_l A_l$ \Comment{Matrix multiplication}
\State $A_l \leftarrow \gamma_l \cdot \frac{z_l - {\rm E}(z_l)}{\sqrt{{\rm Var}(z_l) + \epsilon}} + \beta_l$ \Comment{Batch~Normalization~\cite{ioffe2015batch}}
\If{$l < L$} \Comment{If not the last layer}
\State $A_l \leftarrow {\rm Quantize}(A_l)$  \Comment{Activation is quantized}
\EndIf
\EndFor
\State $\hat{y} \leftarrow A_L$
\State $C \leftarrow {\rm Cost}(\hat{y}, y)$ \Comment{Compute mean loss over the batch}
\State $(\partial_W C, \partial_{\theta} C) \leftarrow {\rm Backward}(C, \hat{y}, W^{\rm Q}, \theta^{\rm BN}, {\rm cache}) $ \Comment{Cost gradients}

\State $ (U_W, U_{\theta}) \leftarrow {\rm Adam}(\partial_W C, \partial_{\theta} C, U_W, U_{\theta} )$
\State $W^{\rm h} \leftarrow W^{\rm h} - \eta  U_W $
\State $\theta^{\rm BN} \leftarrow \theta^{\rm BN} - \eta U_{\theta}$ \\
\Return $W^{\rm h}$, $\theta^{\rm BN}$, $U_W$, $U_{\theta}$
\end{algorithmic}
\end{algorithm}

During the training of BNNs and TNNs, each quantized (binary or ternary) weight is associated with a real hidden weight. This approach to training quantized  neural network was introduced  in \cite{courbariaux2016binarized} and is presented in Algorithm~\ref{algo}.

The quantized weights are used for computing neuron values (equations~\eqref{eq:activ_BNN} and~\eqref{eq:activ_TNN}), as well as the gradients values in the backward pass.
However, training steps are achieved  by updating the real hidden weights. The quantized weight is then determined by applying to the real value the quantizing function ${\rm Quantize}$, which is $\phi$ for ternary or ${\rm sign}$ for binary as defined in section \ref{sec:background}. The quantization of activations is done by applying the same function ${\rm Quantize}$, except for real activation, which is done by applying a rectified linear unit (${\rm ReLU}(x) = {\rm max}(0, x)$).

Quantized activation functions ($\phi$ or ${\rm sign}$) have zero derivatives almost everywhere,
which is an issue for backpropagating the error gradients through the network.
A way around this issue is the use of a straight-through estimator \cite{bengio2013estimating}, which consists in taking the derivative of another function instead of the almost everywhere zero derivatives.
Throughout this work, we take the derivative of $\rm Hardtanh$, which is 1 between -1 and 1 and 0 elsewhere, both for binary and ternary activations.

The simulation code used in this work is available publicly in the Github repository: \url{https://github.com/Laborieux-Axel/Quantized_VGG}

\FloatBarrier


\section{The Dataset}

\subsection{The game}
The dataset contains practically all actions of all players of the MMOG \verb|Pardus| since  the game went online in 2004  \cite{pardus}.
\verb|Pardus| is an open-ended online game with a worldwide player base of currently more than 370,000 people. Players live in a virtual,
futuristic universe in which they interact with others in a multitude of ways to achieve their self-posed goals \cite{castra2005}.
Most players engage in various economic activities typically with the (self-posed) goal to accumulate wealth and status.
Social and economical decisions of players are often strongly influenced and driven by social factors such as
friendship, cooperation, and conflict. Conflictual relations may result in aggressive acts such as attacks, fights, punishment, or even
destruction of another player's means of production or transportation.
The dataset contains longitudinal and relational data allowing for a complete and dynamical mapping of multiplex relations of the entire
virtual society, over 1238 days. The behavioral data are free of `interviewer-bias' or laboratory effects since users are not reminded of their
actions being logged during playing. The longitudinal aspect of the data allows for the analysis of dynamical aspects such as the emergence and evolution of
network structures. It is possible to extract multiple social relationships between a fixed set of humans \cite{PNAS}.

\subsection{Human behavioral sequences}
  We consider eight different actions every player can execute at any time. These are
communication (C), trade (T), setting a friendship link  (F),  removing an enemy link (forgiving) (X),
attack (A), placing a bounty on another player (punishment) (B),  removing a friendship link (D), and setting an enemy link (E).
While C, T, F and X can be associated with {\em positive} (good) actions,   A, B, D and E are hostile or
{\em negative} (bad) actions. We classify communication as positive because only a negligible part of communication takes place between enemies \cite{SN}.
Segments of action sequences of three players (146, 199 and 701) are shown in the first three lines of Fig.~\ref{fig:seq}~(a).

\begin{table*}[b]
\caption*{Table 1. First row: total number of actions by all  players (with at least 1000 actions) in the Artemis universe of the Pardus game.
Further rows: first 4 moments of $r_Y(d)$, the distribution of the log-increments of the $N_Y$ processes (see text).  Approximate
log-normality is indicated. The large values of kurtosis for $T$ and $A$ result from a few extreme outliers.
}
\begin{tabular}{ l|c c c c c c c c}
		& D 			& B 		&  A 		& E 		&  F 		&  C 		& T 		& X \\
		\hline
$\sum_{d=1}^{1238} N_Y(d)$	& 26,471     	&   9,914   & 558,905 &  64,444&   82,941 & 5,607,060& 393,250&  20,165\\
\hline
mean	&  	0.002   	& 0.001	& 0.004   	& -0.002	& -0.002   & 0.000    	& 0.003    & 0.002 \\
std		&    	1.13    	& 0.79   	& 0.54    	& 0.64    	& 0.35    	& 0.12   	& 0.28    	& 0.94 \\
skew		&	 0.12   	& 0.26   	& 0.35   	 & 0.08   	& 0.23    	& 0.11   	& 1.00   	& -0.01\\
kurtosis		&      3.35    	& 3.84   	& 6.23   	 & 3.67   	& 3.41    	& 3.76  	& 13.89    	& 3.72\\
\end{tabular}
\label{tab1}
\end{table*}

We consider three types of sequences  for any particular player.
The first  is the stream of $N$ consecutive actions $A^i=\{a_n | n=1, \cdots, N \}$
which player $i$ performs during his `life' in the game.
The second sequence is the (time-ordered) stream of actions that player $i$ receives from  all the other players in the game, i.e. all the actions
which are directed towards player $i$.  We denote by  $R^i=\{r_n | n=1, \cdots, M \}$  received-action sequences.
Finally, the third sequence is the time-ordered combination of player $i$'s actions and received-actions, which is
a chronological sequence from the elements of $A^i$ and $R^i$ in the order of occurrence. The combined sequence we denote by $C^i$; its length is $M+N$, see
also Fig.~\ref{fig:seq}~(a).  The $n$th element of one of these  series is denoted by $A^i(n)$, $R^i(n)$, or $C^i(n)$.
We do not consider the actual time between two consecutive actions which can range from milliseconds to weeks,
rather we work in `action-time'.

If we assign $+1$ to any positive action C, T, F or X, and $-1$ to the negative actions A, B, D and E, we can translate a
sequence $A^i$ into a symbolic binary sequence $A_{\rm bin}^i $. From the  cumulative sum of this
binary sequence a `world line' or `random walk' for player $i$ can be generated,
$W_{\rm good-bad}^i(t) = \sum_{n=1}^{t} A^i_{\rm bin} (n)$, see Fig.~\ref{fig:seq}~(b).
Similarly, we define a binary sequence from the combined sequence $C^i$, where we assign $+1$  to an executed action and $-1$ to a received-action.
This sequence we call $C^i_{\rm bin}$, its cumulative sum, $W_{\rm act-rec}^i(t) = \sum_{n=1}^{t} C^i_{\rm bin} (n)$ is the `action-receive' random-walk or world line.
Finally, we denote the number of actions which occurred during a day in the game by $N_Y(d)$, where $d$ indicates the day and $Y$ stands for one of the eight actions.

\section{Results}
The number of occurrences of the  various actions of all players over the entire time period is summarized in Tab. \ref{tab1} (first line).
Communication is the most dominant action, followed by attacks and trading which are each about an order of magnitude less frequent.
The daily number of all communications, trades and attacks, $N_C(d)$, $N_T(d)$ and $N_A(d)$ is shown in Fig.  \ref{fig:actions} (a), (b) and (c), respectively.
These processes are reverting around a mean, $R_Y$.
All processes of actions show an approximate Gaussian statistic of its log-increments, $r_Y(d)=\log \frac{N_Y(d)}{N_Y(d-1)}$.
The first 4 moments of the $r_Y$ series are listed in Tab. \ref{tab1}.
The relatively large kurtosis for $T$ and $A$ results from a few extreme outliers.
The distribution of log-increments for the $N_C$, $N_T$ and $N_A$ timeseries are shown in  Fig.  \ref{fig:actions} (d).
The lines are Gaussians for the respective mean and standard deviation from Tab. \ref{tab1}.
As maybe the simplest mean-reverting model with approximate lognormal distributions, we propose
\begin{equation}
N_Y(d)= N_Y(d-1)^{\rho_Y} \, e^{\xi(d)} \, R_Y^{(1-\rho_Y)} \quad,
\label{mod}
\end{equation}
where $\rho_Y$ is the mean reversion coefficient, $ \xi(d)$ is a realization of a zero mean
Gaussian random number with standard deviation $\sigma_Y$, and
$R_Y$ is the value to which the process $N_Y(t)$ reverts to. $\sigma$ is given by the third line in Tab. \ref{tab1}.

\begin{figure}[t]
    \begin{center}
            \includegraphics{fig02.eps}
    \end{center}
    \caption{Timeseries of the daily number of (a) trades, (b) attacks, (c) communications in the first 1238 days in the game.
    Clearly a  mean reverting tendency of three processes can be seen. (d) Simulation of a model timeseries, Eq. (\ref{mod}), with $\rho=0.94$. We use the values  from the $N_C$
    timeseries, $R=4000$, and standard deviation $\sigma=0.12$. Compare with the  actual $N_C$ in (c). The only free parameter in the model is $\rho$. Parameters are from Tab. \ref{tab1}. Mean reversion and log-normality
    motivate the model presented in Eq. (\ref{mod}).
    (e) The distributions of log-increments $r_Y$ of the processes and the model. All follow approximate Gaussian distribution functions.
    \label{fig:actions}
    }

\end{figure}

\subsection{Transition probabilities}
With $p( Y | Z )$ we denote the probability that an action of type $Y$ follows an action of type $Z$ in the behavioral sequence of a player. $Y$ and $Z$  stand for any of the eight actions, executed or received (received is indicated by a subscript $r$). In Fig.~\ref{fig:trans}~(a) the transition probability matrix $p\left( Y|Z \right)$ is shown. The $y$ axis of the matrix indicates the action (or received-action) happening at a time $t$, the probabilities for the actions (or received-actions) that immediately follow are given in the corresponding horizontal place.

This transition matrix specifies to which extent an action or a received action of a player is influenced by the action that was
done or received at the previous time-step. In fact, if the behavioral sequences of players had no correlations, i.e.
the probability of an action, received or executed, is independent of the history of the player's actions, the transition
probability $p\left( Y | Z \right)$  simply is  $p \left( Y \right)$, i.e. to the probability that an action or received action $Y$
occurs in the sequence is determined by its relative frequency only.
Therefore, deviations of the ratio $\frac{p\left( Y | Z \right)}{p(Y)}$ from $1$ indicate correlations in sequences.
In Fig.~\ref{fig:trans}~(b) we report the values of $\frac{p\left( Y | Z \right)}{p(Y)}$ for actions and received actions
(received actions are indicated with the subscript $r$) classified only according to their positive (+) or negative (-) connotation.
In brackets we report the \emph{Z}-score with respect to the uncorrelated case. We find that the probability to perform a good action
is significantly higher if at the previous time-step a positive action has been received. Similarly, it is more likely that a player is the
target of a positive action if at the previous time-step he executed a positive action. Conversely, it is highly unlikely that after a good action,
executed or received, a player acts negatively or is the target of a negative action. Instead, in the case a player acts negatively, it is
most likely that he will perform another negative action at the following time-step, while it is highly improbable that the following action,
executed or received, will be positive. Finally, in the case a negative action is received, it is likely that another negative action
will be received at the following time-step, while all other possible actions and received actions are underrepresented.
The high statistical significance of the cases $P(-|-)$ and $P(-_r|-_r)$ hints toward a high persistence of negative actions
in the players' behavior, see below.

Another finding is obtained by considering only pairs of received actions followed by performed actions.
This approach allows  to quantify the influence of received actions on the performed actions of players.
For these pairs we measure a probability of
$0.02$
of performing a negative action after a received positive action.
This value is significantly lower compared to the probability of $0.10$ obtained for randomly reshuffled sequences.
Similarly, we measure a probability of $0.27$ of performing a negative action after a received negative action.
Note that this result is not in contrast with the values in Fig.~\ref{fig:trans}~(b), since only pairs made up of received actions and performed actions are taken into account.

\begin{figure}[t]
    \begin{center}
             \includegraphics{fig03.eps}
    \end{center}
    \caption{(a) Transition probabilities $p\left( Y | Z \right)$ for actions (and received-actions) $Y$ at a time $t+1$,
    given that a specific action $Z$ was executed or received in the previous time-step $t$.
    Received-actions are indicated by a subscript $r$.  Normalization is such that rows add up to one.
   The large values in the diagonal signal  that human actions are highly clustered or repetitive.
   Large values for $C \rightarrow C_r$ and $C_r \rightarrow C$ reveal that communication is a tendentially anti-persistent activity -- it is more likely to receive a message after one sent a message
   and vice versa, than to send or to receive two consecutive messages.
   (b) The ratio $\frac{p\left( Y | Z \right)}{p(Y)}$, shows the influence of an action $Z$ at a previous time-step $t$ on a following action $Y$ at a time $t+1$,
   where $Y$ and $Z$ can be positive or negative actions, executed or received (received actions are indicated by the subscript $r$).
   In brackets, we report the \emph{Z}-score (significance in number of standard deviations) in respect to a sample of 100 randomized
   versions of the dataset. The cases for which the transition probability is significantly higher (lower) than expected in uncorrelated
   sequences are highlighted in red (green). Receiving a positive action after performing a positive action is highly overrepresented,
   and vice versa. Performing (receiving) a negative action after performing (receiving) another negative one is also highly overrepresented.
   Performing a negative action has no influence on receiving a negative action next. All other combinations are strongly underrepresented,
   for example after performing a negative action it is very unlikely to perform a positive action with respect to the uncorrelated case. \label{fig:trans}
   	}
\end{figure}

\begin{figure*}[t]
    \begin{center}
            \includegraphics{fig04.eps}
    \end{center}
    \caption{
    (a) World lines of good-bad action random walks of the 1,758 most active players, (b) distribution of their slopes $k$ and (c) of their scaling exponents $\alpha$. By definition, players who perform more good (bad) than bad (good) actions have the endpoints of their world lines above (below) 0 in (a) and only fall into the $k>0$ ($k<0$) category in (b). (d) World lines of action-received random walks, (e) distribution of their slopes $k$ and (f) of their scaling exponents $\alpha$. The inset in (d) shows only the world lines of bad players. These players are typically dominant, i.e. they perform significantly more actions than they receive. In total the players perform many more good than bad actions and are strongly persistent with good as well as with bad behavior, see (c), i.e. actions of the same type are likely to be repeated.  \label{fig:world} }
\end{figure*}

\subsection{World lines}

The world lines $W_{\rm good-bad}^i$ of good-bad action sequences  are shown in
Fig. \ref{fig:world} (a), the action-reaction world lines in Fig.  \ref{fig:world} (b).

As a  simple measure to characterize these world lines we define
the slope $k$ of the line connecting the origin of the world line to its end point (last action of the player).
A slope of $k=1(-1)$ in the good-bad world lines $W_{\rm good-bad}$ indicates that the player performed only positive (negative) actions.
The slope $k^i$ is  an approximate measure of `altruism' for  player $i$.
The histogram of the slopes for all players is shown in Fig.~\ref{fig:world}~(b), separated into good (blue) and bad (red) players,
i.e. players who have performed more good than bad actions and vice versa. The mean and standard deviation of slopes of good, bad, and all players are
$\bar k^{\rm good} = 0.81 \pm 0.19$, $\bar k^{\rm bad} = -0.40 \pm 0.28$, and $\bar k^{\rm all} = 0.76 \pm 0.31$, respectively.
Simulated random walks with the same probability $0.90$ of performing a positive action yield
a much lower variation, $\bar k^{\rm sim} = 0.81 \pm 0.01$, pointing at  an inherent heterogeneity of human behavior.
For the combined action--received-action  world line $W_{\rm act-rec}$ the slope is a measure of
how well a person is integrated in her social environment. If $k=1$ the person only acts
and receives no input, she is `isolated' but dominant. If the slope is $k=-1$ the person is driven by the actions of others and does never act nor react. The
histogram of slopes for all players is shown in  Fig. \ref{fig:world} (e). Most players are well within the $\pm45$ degree cone.
Mean and standard deviation of slopes of good, bad, and all players are $\bar k^{\rm good} = 0.02 \pm 0.10$, $\bar k^{\rm bad} = 0.30 \pm 0.19$,
and $\bar k^{\rm all} = 0.04 \pm 0.12$, respectively. Bad players are tendentially dominant, i.e. they perform significantly more actions than they receive.
Simulated  random walks with equal probabilities for up and down moves for a sample of the same sequence lengths, we find again
a much narrower distribution with slope $\bar k^{\rm sim} = 0.00 \pm 0.01$.

As a second measure we use the mean square displacement of  world lines to quantify the persistence of action sequences,
\begin{equation}
M^2(\tau) = \langle  \Delta W (\tau) -  \langle \Delta W (\tau) \rangle  \rangle^2  \propto \tau^{2\alpha} \quad,
\end{equation}
(see Materials and Methods).

 The histogram of exponents  $\alpha$
 for the good-bad random walk, separated into good (blue) and bad (red) players, is shown in  Fig.~\ref{fig:world}~(c), for
 the action--received-action world line in (f). In the first case strongly persistent behavior is obvious, in the second there
 is a slight tendency towards persistence. Mean and standard deviation for the good-bad world lines
 are $\alpha_{\rm good-bad}= 0.87\pm  0.06$,  for the action-received actions   $\alpha_{\rm act-rec}=0.59 \pm 0.10$.
Simulated sequences of random walks have -- as expected by definition -- an exponent of $\alpha_{\rm rnd} = 0.5$, again with a very small standard deviation of about $0.02$.
Figure  \ref{fig:world} (a) also  indicates that the lifetime of players who use negative actions  frequently is short.
The average lifetime for players with a slope $k<0$  is $2528  \pm  1856$ actions, compared to
players with a slope $k>0$ with $3909 \pm  4559$ actions. The average
lifetime of the whole sample of players is $3849 \pm 4484$  actions.

\subsection{Motifs, Entropy and Zipf law}

By considering all the sequences of actions $A^i$ of all possible players $i$, we have an
ensemble which allows to perform a motif analysis \cite{sinatra_motifs}. We define a $n$-string
as a subsequence of $n$ contiguous actions. An $n$-motif is an $n$-string which appears in the
sequences with a probability higher than expected, after lower-order correlations have been
properly  removed (see Materials and Methods).

We computed the observed and expected probabilities $p^{\rm obs}$ and $p^{\rm exp}$ for all
$8^2=64$ 2-strings and for all  $8^3=512$ 3-strings, focusing on those $n$-strings with the highest ratio $\frac{p^{\rm obs}}{p^{\rm exp}}$.
Higher orders are statistically not feasible due to combinatorial explosion. We find that the $2$-motifs in the sequences of
actions $A$ are clusters of same letters: BB, DD, XX, EE, FF, AA with ratios $\frac{p(\rm obs)}{p(\rm exp)} \approx 169$, $136$, $117$, $31$, $15$, $10$, respectively.
This observation is consistent with the previous first-order observation that actions cluster. The most significant $3$-motifs however are
(with two exceptions) the palindromes: EAX, DAF, DCD, DAD, BGB, BFB, with ratios $\frac{p(\rm obs)}{p(\rm exp)} \approx 123$, $104$, $74$, $62$, $33$, $32$, respectively.
The exceptions disappear when one considers actions executed on the same screen in the game as equivalent, i.e. setting or removing friends or enemies:
F, D, E, X. This observation hints towards  processes where single actions of one type tend to disrupt a flow of actions of another type.

Finally, we partition the action sequences  into $n$-strings (`words').
Fig. \ref{fig:entropy}  shows the rank distribution of word occurrences of different lengths $n$.
The distribution shows an approximate  Zipf law \cite{zipf} (slope of $\kappa=-1$) for ranks below 100. For ranks
between 100 and 25,000 the scaling exponent approaches a smaller value of about $\kappa \sim -1.5$.
The Shannon $n$-tuple redundancy (see e.g. \cite{dna, dna2, dna3})
for  symbol sequences composed of 8  symbols (our action types) is defined as
\begin{equation}
 R^{(n)} = 1+ \frac{1}{3n} \sum_{i}^{8^n} P_i^{(n)} \log_2 P_i^{(n)} \quad ,
\end{equation}
where $P_i^{(n)}$ is the probability of finding a specific $n$-letter word. Uncorrelated sequences
 yield  an equi-distribution, $P_i= 8^{-n}$, i.e. $R^{(n)} =0$. In the other extreme of only one letter being used, $R^{(n)} =1$.
In Fig. \ref{fig:entropy} (inset)  $R^{(n)}$ is shown as a function of sequence length $n$.
Shannon redundancy is not a constant but increases with $n$.
This  indicates that Boltzmann-Gibbs entropy might not be an extensive quantity for
action sequences \cite{hanel2011}.

\section{Discussion}

The analysis of human behavioral sequences as recorded in a massive multiplayer online game shows that communication is by far the most dominant activity
followed by aggression and trade. Communication events are about an order of magnitude more frequent than attacks and trading events, showing the importance of information exchange between humans.
It is possible to understand the collective timeseries of human actions of a particular type ($N_Y$) with a simple mean-reverting
log-normal model.
On the individual level we are able to identify organizational patterns of the emergence of good overall behavior.
Transition rates  of actions of individuals show that positive actions strongly induces  positive \emph{reactions}.
Negative behavior on the other hand has a high tendency of being repeated instead of being reciprocated, showing the `propulsive' nature of negative actions.
However, if we consider only reactions to negative actions, we find that negative reactions are highly overrepresented.
The probability of acting out negative actions is about 10 times higher if a person received a negative action at the previous timestep than if she received a positive action.
The action of communication is found to be of highly reciprocal `back-and-forth' nature.
The analysis of binary timeseries of players (good-bad) shows that the behavior of almost all players is `good' almost all the time.
Negative actions are balanced to a large extent by good ones.
Players with a high fraction of negative actions tend to have  a significantly shorter life.
This may be due to two reasons: First because they are hunted down by others and give up playing, second because they are unable to maintain a social life and quit the game
because of loneliness or frustration.
We interpret these findings as empirical evidence for self organization towards reciprocal, good conduct
within a human society. Note that the game allows bad behavior in the same way as good behavior
but the extent of punishment of bad behavior is freely decided by the players.

 \begin{figure}[t]
    \begin{center}
            \includegraphics{fig05.eps}
    \end{center}
      \caption{ Rank ordered
    probability distribution  of 1 to 6 letter words. Slopes of $\kappa = -1$ and $\kappa = -1.5$ are indicated for reference.
    The inset shows the Shannon $n$-tuple redundancy as a function of word length $n$.
    \label{fig:entropy}
    }

\end{figure}

Behavior is highly persistent in terms of good and bad, as seen in the scaling exponent ($\alpha \sim 0.87$) of the mean square displacement of
the good-bad world lines. This high persistence means that good and bad actions are carried out in clusters. Similarly high levels of persistence were
found in a recent study of human behavior \cite{fan}. A smaller exponent ($\alpha \sim 0.59$) is found for the action--received-action timeseries.

Finally we split behavioral sequences of individuals into subsequences (of length 1-6) and interpret these as behavioral `words'.
In the ranking distribution of these words we find a Zipf law to about ranks of 100. For less frequent words the exponent in the rank distribution
approaches a somewhat smaller exponent of about $\kappa \sim -1.5$.
From word occurrence probabilities we further compute the Shannon $n$-tuple redundancy which yields relatively  large values when compared  for
example  to those of  DNA sequences \cite{dna, dna2, dna3}. This reflects the dominance of communication over all the other actions.
The $n$-tuple redundancy is clearly not a constant, reflecting again the non-trivial statistical structure of behavioral sequences.

\section{Materials and Methods}
The game \verb|Pardus| \cite{pardus} is sectioned into three independent `universes'.
Here we focus on the `Artemis' universe, in which we recorded player actions  over the first
1,238 consecutive days of the universe's existence. Communication between any two players can
take place directly, by using a one-to-one, e-mail-like private messaging system, or indirectly, by meeting in built-in chat channels
or online forums. For the player action sequences analyzed we focus on one-to-one interactions  between players only,
and discard indirect interactions such as e.g.  participation in chats or forums.
Players can express their sympathy (distrust) toward other players by establishing so-called friendship (enmity) links. These links are
only seen by the player marking another as a friend (enemy) and the respective recipient of that link.
For more details on the game, see \cite{SN,pardus}.
From all sequences of all 34,055 Artemis players who performed or received an action at least once within 1,238 days,
we removed players with a life history of less than 1000 actions, leading to the set of the most active 1,758 players which are considered throughout this work.

All data used in this study is fully anonymized; the authors have the written consent to publish from the legal department of the Medical
University of Vienna.

\subsection{Mean square displacement}
The mean square displacement $M^2$ of a world line $W$ is defined as $M^2(\tau) = \langle  \Delta W (\tau) -  \langle \Delta W (\tau) \rangle  \rangle^2, $
where $\Delta W(\tau) \equiv W(t +\tau)- W(t)$ and $\langle . \rangle$ is the average over all  $t$.
 The asymptotic behavior of  $M(\tau)$ yields information about the `persistence' of a  world line. $M(\tau)\propto \tau^{\frac12}$  is the
 pure diffusion case,  $M(\tau)\propto \tau^{\alpha}$ with scaling exponent $\alpha \neq \frac12$ indicates  persistence for $\alpha>\frac12$, and anti-persistence
 for $\alpha < \frac12$.
 Persistence means that the probability of making an up(down)  move at time $t+1$ is larger(less) than $p=1/2$, if   the
 move at time $t$ was an up move. For calculating the exponents $\alpha$ we use a fit range of $\tau$ between 5 and 100. We checked
 from the mean square displacements of single world lines that this fit range is indeed reasonable.

\subsection{Motifs}
We define $n$-strings a subsequence of $n$ contiguous actions. Across the entire ensemble, $8^n$ different $n$-strings can
appear, each of them occurring with a different probability. The frequency, or observed probability, of each $n$-string can be
compared to its expected probability of occurrence, which can be estimated on the basis of the observed probability of lower
order strings, i.e. on the frequency of $(n-1)$-strings.
For example, the expected probability of occurrence of a 2-string $(A_t, A_{t+1})$
is estimated as the product of the observed probability of the single actions $A_t$ and
$A_{t+1}$, $p^{\rm exp}(A_{t},A_{t+1})=p^{\rm obs}(A_{t})p^{\rm obs}(A_{t+1})$. Similarly, the probability of a 3-string
$(A_{t}, A_{t+1}, A_{t+2})$ to occur can be estimated as $p^{\rm exp}(A_{t}, A_{t+1}, A_{t+2})=p^{\rm obs}(A_{t}, A_{t+1})p^{\rm obs}(A_{t+2}|A_{t+1})$, where $p^{\rm obs}(A_{t+2}|A_{t+1})$ is the conditional probability to have action $A_{t+2}$ following action $A_{t+1}$. By definition of conditional probability, one has $p^{\rm obs}(A_{t+2}|A_{t+1})=\frac{p^{\rm obs}(A_{t+1},A_{t+2})}{p^{\rm obs}(A_{t+1})}$ (see \cite{sinatra_motifs} for details). A $n$-motif in the ensemble is then defined as a $n$-string whose observed probability of occurrence is significantly higher than its expected probability.

\begin{acknowledgments}
We thank Werner Bayer for compiling \verb|Pardus| data. Supported in part by the Austrian Science Fund, FWF P23378, and the European Cooperation in Science and Technology Action, COST MP0801.
\end{acknowledgments}