\section{Introduction}
\label{sec:intro}
Given large amounts of training data (such as ImageNet~\cite{deng2009imagenet}) and compute power (powerful GPUs~\cite{dl_gpu}), deep models are highly accurate on the respective tasks for which it is trained. However, these models can easily get fooled when they encounter adversarial images as input that are crafted using an adversarial attack~\cite{szegedy2013intriguing}.
A lot of efforts have been made in the literature to secure the models against adversarial attacks. Adversarial training~\cite{madry2017towards,goodfellow2014explaining} is one of the most common ways to provide empirical defense where the models are trained to minimize the maximum training loss induced by adversarial samples. Such defenses are heuristic-based and are only robust to known or specific adversarial attacks. Powerful adversaries easily break them, hence are not truly robust against adversarial perturbations~\cite{athalye2018robustness,uesato2018adversarial,croce2020reliable}. This motivated the researchers to develop methods where a trained model can be guaranteed to have a constant prediction in the input neighborhood. Such methods that provide formal guarantees are called certified defense methods~\cite{raghunathan2018certified,wong2018provable,ehlers2017formal,katz2017reluplex,mirman2018differentiable}.

Randomized smoothing~\cite{cao2017mitigating, liu2018towards,lecuyer2019certified,li2019certified,cohen2019certified} is a certified defense technique used to provide provable robustness against $l_{2}$ adversarial perturbations. It outperforms other certification methods and is also scalable to deep neural networks due to its architectural independence. Using randomized smoothing, any base classifier can be converted to a smoothed classifier, which is certifiably robust against $l_{2}$ attacks as it has the desirable property of constant predictions within the $l_{2}$ ball around the input. The prediction by the smoothed classifier on an input image is simply the most probable class predicted by the base classifier on random gaussian perturbations of the input. Note that the higher the probability with which the base classifier predicts the most probable class as the correct class, the higher the certified radius~\cite{cohen2019certified}. However, any vanilla trained / off-the-shelf classifier would not be robust to the input corrupted by gaussian noise. The predicted most probable class can be incorrect or may be predicted with very low confidence leading to poor certification. Thus, models are often trained from scratch using the gaussian perturbed samples~\cite{cohen2019certified} as augmentation, and also with adversarial training~\cite{salman2019provably}

Training from scratch on gaussian-noise augmentation is not always a viable option, especially when the large pretrained models are shared as an API, either as a white box or a black box. Also, retraining these large cumbersome models adds a lot of computational overload. To avoid this, Salman \etal~\cite{salman2020denoised} proposed a “denoised smoothing” technique where a custom-trained denoiser is prepended before the pretrained classifier. Though their approach provides certified robustness to pretrained models, they use the entire training data for training the denoiser. In fact, all the previous certification methods also assume the availability of entire training data. This assumption is unrealistic as the API provider may not share the whole large-scale training set. Due to heavy transmission costs associated with complete training data or proprietary reasons, they may provide access to only a few training samples. In such cases, when we perform denoised smoothin
~\cite{salman2020denoised} directly, the certified accuracy decreases significantly as the number of training samples reduces from $100\%$ to $1\%$ as shown in Fig.~\ref{fig:motivation}.
\begin{figure}[htp]
\centering
\centerline{\includegraphics[width=0.49\textwidth]{figures/motivation_fig.pdf}}
\caption{The certified accuracy (both standard and robust) of existing method drops significantly as the training set size of CIFAR-$10$ reduces. Our method (\textit{DE-CROP}) overcomes this issue
by obtaining significant gains in certified performance against $l_2$ perturbations within radius $0.25$, across different training data budgets. We use gaussian noise with $\sigma$ as $0.5$ for certification
}
\label{fig:motivation}
\end{figure}

In this work, we attempt to provide certified robustness to a pretrained non-robust model in the challenging setting of limited training data. We first explore whether the performance of denoised smoothing can be improved using different approaches such as weight decay as a regularization scheme or even generate extra data using augmentation and mix-up strategies \cite{zhang2017mixup} to avoid overfitting. However, these traditional approaches only show a marginal improvement (refer to Tables~\ref{tab:table1},~\ref{tab:table2}). Hence, we propose our approach of \textbf{d}ata-\textbf{e}fficient \textbf{c}ertified \textbf{ro}bustness (dubbed as `\textit{DE-CROP}’)  (refer to Sec.~\ref{sec:proposed} and Fig.~\ref{fig:tsne},~\ref{fig:approach}) that generates better samples, yielding diverse features on the pretrained classifier similar to complete training data.

Since generative models are hard to train and perform poorly (on downstream tasks) in the presence of limited data, we adopt a simple and intuitive approach to generate additional data to minimize overfitting. We achieve this by generating adversarial samples (corresponding to our limited data) that we term `boundary' samples. Adversarial samples act as an upper bound to the decision boundary~\cite{nanda2021fairness} while maintaining the semantic content of the image, allowing the denoiser to learn from samples in its neighborhood. Moreover, we also generate interpolated samples (lying between original and boundary samples) that further increase the data diversity in the feature space. We train the denoiser by maximizing the similarity between the pre-trained classifier’s features on the original data and denoised perturbed generated/original data. We achieve this by performing orthogonal modifications at two granularities: a) instance-level (using cosine-similarity) and b) distribution-level (using Maximum Mean Desprecancy \cite{gretton2012kernel} and negative gradients from a Domain-discriminator \cite{ganin2015unsupervised}). In our experiments, we observed that although cosine-similarity improved denoised performance by exploiting the discrimnativeness of the pretrained classifier, its benefits were limited as it only operates at an instance level. Thus, inspired by fundamental ideas (such as maximum mean discrepancy and domain discriminator) in domain adaptation, we formulate the objective of obtaining correct predictions on denoised images by reducing the distribution discrepancy between feature representations of the original clean inputs and the denoised gaussian perturbed output. As shown in Fig.~\ref{fig:motivation}, our
method DE-CROP significantly improves certified performance across different sample budgets ($100\%$, $20\%$, $10\%$, $5\%$, $1\%$) on CIFAR-$10$ compared to ~\cite{salman2020denoised}.

Our contributions are summarized as follows:
\iffalse
\begin{itemize}
\item \textbf{Formulation}: We formulate a novel problem of certified defense in limited data settings. Given only a few training samples, we provide robustness guarantees for a non-robust pretrained classifier against $l_{2}$ perturbations for both white-box and black-box setups. To the best of our knowledge, certified defense using a few training samples hasn't been explored yet.
\item \textbf{Sample Generation}: To avoid overfitting with limited data, our method generates class-boundary samples using an adversarial attack. We also interpolate between original training sample and its respective class-boundary sample in feature space to obtain diverse features of that class. These features are used as ground truth corresponding to which `interpolated’ sample is generated by perturbing the original sample (Sec.~\ref{subsec:generation}).
\item \textbf{Training Loss}: We regularize the denoiser network through our proposed losses (Sec.~\ref{subsec:denoiser_training}) that align the feature representations of original and denoised gaussian perturbed generated/original samples at multiple granularities (both instance and distribution levels).
\item \textbf{Experiments}: We show the contribution of our boundary and interpolated samples (Sec.~\cite{}) along with individual benefits of each of the proposed losses (Sec.~\cite{}), which leads to significant improvements in certified accuracy over the baselines across different sample budgets and noise levels in both white-box (Sec.~\cite{}) and black-box settings (Sec.~\cite{}).
\end{itemize}
\fi
\begin{itemize}
\item
Given only
limited training data, we provide robustness guarantees for a non-robust pretrained classifier against $l_{2}$ perturbations on both white-box and black-box setups. To the best of our knowledge, we are the first to provide certified adversarial defense using only the few training samples
\item
To mitigate overfitting on limited training data,
we propose a novel sample-generation strategy that synthesize
`boundary' and `interpolated' samples (Sec.~\ref{subsec:generation}) to augment the limited training data,
leading to improved
feature-diversity on the pretrained classifier.
\item The denoiser network trained with regular cross entropy loss provides limited benefit. To enhance the performance further, we proposed additional losses (Sec.~\ref{subsec:denoiser_training}) that align the feature representations of original and denoised gaussian perturbed generated/original samples at multiple granularities (both instance and distribution levels).
\item We show benefit of our
generated samples (Sec.~\ref{gen_data}) along with contributions from each of the proposed losses (Sec.~\ref{dist_align}), by reporting significant improvements observed across
diverse sample budgets and noise levels in both white-box
and black-box settings.
\end{itemize}
\vspace{-0.1in}
\section{Related Works}
\label{sec:related}
We broadly categorize the relevant works that provides adversarial robustness
and briefly discuss them below:

\textbf{Empirical Robustness}: Empirically motivated adversarial robustness defenses can be broadly classified into: a) adversarial training (AT) and b) non-adversarial training regularizations. AT  \cite{madry2017towards, goodfellow2014explaining, szegedy2013intriguing} improves robustness by augmenting the training data with adversarial samples generated by a particular threat model. Although AT is widely regarded as the best empirical defense, it suffers from high-computational costs due to generation of adversarial samples at training time. Non-AT based approaches \cite{chan2019jacobian} attempt to reduce the computational burden (usually at the cost of drop in adversarial robustness) by mimicking properties observed in robust networks explicitly.
AT is also highly dependent on the quantity of training data. Aditi \etal \cite{carmon2019unlabeled} demonstrated that using additional unlabeled data in a pseudo-labelling setup led to a significant increase in adversarial robustness.
However, since performance of pseudo-labelling itself  depends on the amount of labeled data, the performance of their technique drops considerably as labeled data is reduced. To alleviate this problem, Sehwag \etal \cite{sehwag2021robust} illustrated the benefit of using additional data generated from generative models for improving adversarial robustness.

Since the empirical defenses are designed based on heuristics, they can be easily fooled as stronger adversarial attacks are developed in the future. In contrast, we attempt to provably robustify pretrained classifier against $l_{2}$ adversarial attacks under the challenging constraint of limited training samples.

\textbf{Certified Robustness}: Unlike empirical defense, here the model predictions on the neighborhood region lying within ball of small radius around the input sample, is guaranteed to not vary and remain constant. The methods that provide certification are either ‘exact’~\cite{fischetti2017deep,ehlers2017formal,katz2017reluplex,tjeng2019evaluating} or ‘conservative’~\cite{wong2018provable,zhang2018efficient}. The former  is not scalable to large architectures, is compute-intensive, and often uses less expressive networks but can verify with guarantees about the existence of any adversarial samples if lying within the radius of input. The latter is more scalable and requires less computation but can incorrectly decline the certification even if no adversarial sample is present. Both techniques require either customized or specific architectures, hence are not suitable for modern deep architectures.

Randomized smoothing is a popular technique that does not have any architectural dependency and was initially used as heuristic defense~\cite{cao2017mitigating, liu2018towards}. It was first shown to provide certified guarantees by Lecuyer \etal~\cite{lecuyer2019certified} where techniques from `differential privacy’ were used for certification. It was later improved by Li \etal~\cite{li2019certified} who provided better guarantees using ideas from `information theory’. Both these methods have lower guarantees on the smoothed classifier. Cohen \etal~\cite{cohen2019certified} provided a tight certified guarantee against $l_{2}$ norm adversarial perturbations. After that, the certified accuracy was further improved by using adversarial training techniques in the randomized smoothing framework~\cite{salman2019provably}. However, all these techniques trained the classifier from scratch while providing certified robustness. Recently, Salman \etal~\cite{salman2020denoised} provided provable robustness to pretrained models by appending a custom trained denoiser before it. We also train the custom denoiser but only using few training samples. Unlike \cite{salman2020denoised}, our limited data setup is more challenging and directly using their method yields poor certification results. Our generated samples with the added domain discriminator optimized using our proposed losses,
handles the overfitting on the denoiser quite well and gives significant improvements on certified accuracy.

We now discuss the necessary preliminaries to give required background before explaining  our approach.
\vspace{-0.02in}
\section{Preliminaries}
\label{sec:prelims}
\textbf{Notations}: The complete original training dataset is denoted by $D_o = \{D_{train}, D_{test}\}$, where $D_{train}$ and $D_{test}$ are the training set and the test set respectively. The base classifier $B_c$ is pretrained on the entire $D_{train}$ which consists of $N$ training samples. The API provider has granted public access to the trained $B_c$ which can be used by clients to obtain predictions. However, only a limited amount of $D_{train}$ (denoted by $D_{train}^{lim}$) is shared to the clients. $D_{train}^{lim}$ is only $k\%$ of $D_{train}$ containing $N^k$ training samples such that $N^k \ll N$). The same relationship holds for each class of the classifier $B_c$ i.e. for any class $c$: $N_c^k \ll N_c$ and $N_c^k$ is $k\%$ of $N_c$. An $i^{th}$ sample of $D_{train}^{lim}$ (i.e. $x_o^i$)  with label $y_o^i$ is perturbed by gaussian noise which is denoted by $\bar{x}_o^i = x_o^i + \epsilon$ where $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$, constituting the collection of perturbed samples as $\bar{D}_{train}^{lim}$.

The penultimate features, logits and label predictions from the pretrained classifier $B_c$ on $x_o^i$ are denoted by $F_{B_c}(x_o^i)$, $label(B_c(x_o^i))$ and $B_c(x_o^i)$ respectively. Similarly, $label(S_c(x_o^i))$ is the label predicted by smoothed classifier $S_c$ on input $x_o^i$. The classifier $B_c$ is converted to smoothed classifier $S_c$, which is used for certified defense via randomized smoothing. The certified radius of an $i^{th}$ test sample is $R_c^i$. The evaluations are performed on $l_{2}$ perturbations at a radius $r$ denoted by $l_2^r$. The denoiser network $D_n$ and domain discriminator $D_d$, are parameterized by $\theta$ and $\phi$ respectively. The boundary sample and interpolated sample generated using an $i^{th}$ training sample of $D_{train}^{lim}$ (i.e. $x_o^i$) are represented by $x_b^i$ and $x_{int}^i$ respectively.

\textbf{Randomized Smoothing (RS)}: This technique is used to build a new smoothed classifier $S_c$ from the given base classifier $B_c$. For any $i^{th}$ sample of $D_{train}^{lim}$ (i.e. $x_o^i$), the output of the classifier $S_c$ corresponding to input $x_o^i$ is the most likely class that has the highest probability to be get predicted by $B_c$ on $\bar{x}_o^i$
\begin{equation}
\begin{gathered}
\label{eq1}
label(S_c(x_o^i)) = \underset{c \in C}{\mathrm{argmax}}\hspace{0.1in} Prob(label(B_c(x_o^i + \epsilon)) = c)\\ \text{where} \hspace{0.1in} \epsilon \sim \mathcal{N}(0, \sigma^2 I)
\end{gathered}
\end{equation}
Here, $\sigma$ is a hyperparameter which controls the noise level and $C$ is the set of unique target labels in $D_{train}^{lim}$. The process of RS does not assume $B_c$'s architecture and thus permits the $B_c$ to be any arbitrary large deep neural network.

\pagebreak
\textbf{Certified Robustness using RS}: Lecuyer \etal~\cite{lecuyer2019certified} and Li \etal~\cite{li2019certified} gave robustness guarantees for the smoothed classifier $S_c$ using RS, but they were loose as $S_c$ was provably more robust than the obtained guarantees. Cohen~\etal~\cite{cohen2019certified} gave a tight bound on $l_{2}$ robust guarantees using RS.

If the prediction on the base classifier $B_c$ for the gaussian perturbed copies of $x_o^i$ i.e. $\mathcal{N}(x_o^i, \sigma^2 I)$ is $c_1$ as the ‘most probable class’ with probability $p_1$ and $c_2$ as “runner-up” class with $p_2$, then the smoothed classifier $S_c$ is provably robust around the input $x_o^i$ within the radius $R_c = \sigma/2 (\phi^{-1}(p_1)-\phi^{-1}(p_2))$. Here $R_c$ is the certified radius as the predictions are guaranteed to remain constant inside the radius and $\phi^{-1}$ denotes the inverse of standard gaussian CDF. It is impossible to calculate $p_1$ and $p_2$ exactly when $B_c$ is a deep neural network. Hence, Cohen \etal, estimates lower bound on $p_1$ ($\underline{p_1}$) and upper bound on $p_2$ ($\overline{p_2}$) using Monte Carlo technique.

\textbf{Theorem} [Cohen \etal~\cite{cohen2019certified}]: \textit{Let $B_c$ be any function that maps the input to one of the output class labels. Let $S_c$ be defined as in eq.~\ref{eq1}. The noise $\epsilon$ is sampled from the normal distribution i.e. $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$. If $c_1 \in C$ and $\underline{p_1}$, $\overline{p_2} \in [0,1]$ holds the below inequality}:
\begin{equation}
\begin{gathered}
Prob(label(B_c(x_o^i+\epsilon))=c_1) \geq \underline{p_1} \geq \overline{p_2} >= \\\underset{c \neq c_1}{max} Prob(label(B_c(x_o^i+\epsilon)=c)
\end{gathered}
\end{equation}
\textit{Then $label(S_c(x_o^i+\epsilon)=c_1 \forall \left\lVert\epsilon\right\rVert_{2} < R_c$, where}
\begin{equation}
\begin{gathered}
\label{eq3}
R_c =   \sigma/2 ( \phi^{-1}(\underline{p_1}) - \phi^{-1}(\overline{p_2}) )
\end{gathered}
\end{equation}
In practice, Cohen \etal used $R_c = \sigma \phi^{-1}(\underline{p_1})$ for $\underline{p_1}>1/2$, assuming $\overline{p_2} = 1-\underline{p_1}$, otherwise abstained with $R_c =0$. These above expressions can be derived using the “Neyman-Pearson” lemma for which we refer the reader to~\cite{cohen2019certified}. Now, we discuss our proposed approach in detail in the next section.

\section{Proposed Approach}
\label{sec:proposed}
We aim to provide certified robustness to the given pretrained base classifier $B_c$. However, obtaining certification using randomized smoothing expects the model $B_c$ to be robust against the input perturbations with the random Gaussian noise, which may not be the case with the model $B_c$ supplied by the API provider. In order to make the base model $B_c$ appropriate for randomized smoothing based certification without modifying/retraining $B_c$, a denoiser network $D_n$ is prepended to $B_c$. Thus, $B_c \circ D_n$ is the new base classifier using which the prediction on smoothed classifier $S_c$ on an $i^{th}$ training sample is defined as follows:
\begin{equation}
\begin{gathered}
\label{eq4}
label(S_c(x_o^i))=\underset{c \in C}{\mathrm{argmax}}\hspace{0.01in}Prob(label(B_c(D_n(x_o^i + \epsilon)))=c)\\ \text{where} \hspace{0.1in} \epsilon \sim \mathcal{N}(0, \sigma^2 I)
\end{gathered}
\end{equation}
The above smoothed classifier $S_c$ is provably robust against the $l_{2}$ perturbations with certified radius $R_c$ (refer to Sec.~\ref{sec:prelims}). For high performance on the certified defense, high $R_c$ is required, which is directly proportional to $\underline{p_1}$ (eq.~\ref{eq3}). The probability ($\underline{p_1}$) of predicting the most probable class ($c_1$) i.e. confidence depends on the performance of the denoiser network $D_n$. However, $D_n$ trained on the given limited training data $D_{train}^{lim}$ using the existing technique~\cite{salman2020denoised} yields poor certification (shown in Fig.~\ref{fig:motivation}). Even when we attempt to minimize the overfitting of $D_n$ on $D_{train}^{lim}$ ($\because$ $|D_{train}^{lim}| \ll |D_{train}|$) using different traditional approaches such as weight decay, augmentation, and mix-up strategies, we observe only a minor improvement (refer to Tables~\ref{tab:table1},~\ref{tab:table2}) in certified accuracy. Hence, we propose our method (`DE-CROP') that overcomes this limitation by crafting boundary and interpolated samples using $D_{train}^{lim}$,  followed by using them to train the denoiser with appropriate losses.
\begin{figure}[htp]
\centering
\centerline{\includegraphics[width=0.49\textwidth]{figures/tsne_multi_big.pdf}}
\caption{t-SNE visualization of the pretrained base classifier's features for training samples of a particular class of CIFAR-$10$. Our generated samples (interpolated and boundary) increases the feature diversity of limited-original training data.
}
\label{fig:tsne}
\end{figure}
\begin{figure*}[htp]
\centering
\centerline{\includegraphics[width=\textwidth]{figures/main_figure.pdf}}
\caption{Different stages involved in the proposed method (\textit{DE-CROP}) that provides robustness guarantees on pretrained classifier ($B_c$) against $l_2$ perturbations using only limited training samples ($x_o$). Samples crafted via adversarial attack act as proxy \textit{boundary} samples ($x_b$). Using $x_o$ and $x_b$, \textit{interpolated} samples ($x_{int}$) are generated in stage $1$. The generated samples along with limited training data are used in stage $2$ to train the denoiser network ($D_n^\theta$) by aligning the feature representation at instance and distribution levels using $L_{lc}$, $L_{cs}$, $L_{mmd}$ and $L_{dd}$ losses. The forward pass to compute these losses are shown by different colors
}
\label{fig:approach}
\end{figure*}
\vspace{-0.15in}
\subsection{Generation of Boundary \& Interpolated sample}
\label{subsec:generation}
In Fig.~\ref{fig:tsne}, we show the visualization of the logit layer features of the pretrained base classifier $B_c$ on a particular class via t-SNE plot. The class-features corresponding to
available limited class samples of $D_{train}^{lim}$ is shown in
green color (on left).
We focus on improving the feature diversity of the limited training samples. For this, we first estimate class-boundary samples that would have respective features in the boundary region of the t-SNE.

Adversarial attacks~\cite{madry2017towards} synthesize samples via an optimization procedure that carefully perturbs the input sample with a small human-imperceptible noise (i.e., adversarial noise). The model gets fooled on such samples (i.e., adversarial samples) as the prediction gets flipped to some other class. As these samples cross the decision boundary, which are constructed by adding a small noise to the input original sample, they often lie very close to the decision boundary. Moreover, they are human-imperceptible and preserve the class semantics of the input class sample. Hence, adversarial samples serve as a good candidate for a proxy of class-boundary samples. For any $i^{th}$ training sample of $D_{train}^{lim}$ (i.e., $x_o^i$), we obtain boundary sample ($x_b^i$) by computing adversarial noise $\delta$ for which the following relation holds:
\begin{equation}
\begin{gathered}
x_b^i=x_o^i+\delta, \\\left\lVert\delta\right\rVert_\infty<\epsilon, \hspace{0.1in}\epsilon>0, \hspace{0.1in}label(B_c(x_b^i)) \ne label(B_c(x_o^i))
\end{gathered}
\end{equation}
Next, we obtain interpolated features by performing a mixup between the features of generated boundary sample $x_b^i$ and original training sample $x_o^i$ as follows:
\begin{equation}
\begin{gathered}
Logit_{int}^i = \alpha \times B_c (x_o^i) + (1-\alpha) \times B_c (x_b^i)
\end{gathered}
\end{equation}
Here $\alpha$ is the mixing coefficient. The interpolated features, the class-boundary features, and the original limited class-sample features are highlighted in yellow, blue, and green colors respectively, in the t-SNE plot (Fig.~\ref{fig:tsne}). These interpolated features being label preserving help to improve the feature-diversity
leading to improved certified accuracy (refer to Sec.~\ref{gen_data}). We craft the input sample corresponding to the interpolated feature $Logit_{int}^i$ (i.e., $x_{int}^i$) by perturbing the original sample $x_o^i$ to match its feature response to $Logit_{int}^{i}$ using mean square error loss ($L_{mse}$). Mathematically, we obtain interpolated sample $x_{int}^i$ as follows:
\begin{equation}
\label{eq7}
x_{int}^i \leftarrow \underset{x}{\mathrm{min}} \hspace{0.1in} L_{mse}(B_c(x), Logit_{int}^i)
\end{equation}
where $x$ is initialized with $x_o^i$ and kept trainable with ground truth as $Logit_{int}^i$. The model $B_c$ is non-trainable, but the gradients are allowed to backpropagate from the model to update the input $x$. Thus, we obtain the boundary and the interpolated samples corresponding to each training sample.

Refer supplementary for visualization of both the boundary and interpolated samples in the input space. All of them preserve the class semantics. Our generated boundary samples $x_b$ and interpolated samples $x_{int}$ are used along with limited training samples $x_o$ to train the denoiser network $D_n$, which we discuss in the subsequent subsection.
\subsection{Training of the denoiser network ($D_n$)}
\label{subsec:denoiser_training}
The denoiser network $D_n$ is attached before the pretrained base classifier $B_c$ to make it suitable for randomized smoothing. Apart from this, we also add a domain discriminator network $D_d$ whose input is the normalized features of the penultimate layer of $B_c$. The discriminator $D_d$ learns to distinguish between distribution of clean samples and distribution of denoised outputs of gaussian perturbed input samples.
Motivated from domain adaptation literature~\cite{ganin2016domain}, we use gradient reversal layer (GRL) before feeding the normalized penultimate layer features to the discriminator that allows normal forward pass but reverses the direction of gradient in the backward pass. As a consequence, this negative gradients backpropagates to the denoiser network $D_n$ that helps it to produce denoised output which yields indistinguishable domain-invariant features on the pretrained $B_c$ classifier.

Refer to supplementary for architectural details of the networks $D_n$ and $D_d$. The overall steps involved in the proposed framework (`DE-CROP’) are also shown in Fig.~\ref{fig:approach}. The network is trained with our generated data and limited training data $D_{train}^{lim}$ by using different losses aimed at different objectives - label consistency to ensure correct predictions on $B_c$ and matching of high-level feature similarity obtained on $B_c$ for denoised output of gaussian perturbed input and clean original training samples both at the sample level and distribution level. The respective losses are described below:\\
\textbf{Ensuring label consistency}:
Similar to~\cite{salman2020denoised}, we use cross entropy loss ($L_{ce}$) to ensure that the label predicted by the pretrained network $B_c$ on original clean data and denoised output of its gaussian perturbed counterpart are same.
\begin{equation}
\begin{split}
L_{lc} = 1/N^k\sum\nolimits_{i=1}^{N^k} L_{ce}(softmax(B_c(D_n(\bar{x}_o^i))),\\ label(B_c(x_o^i)))
\end{split}
\end{equation}
\textbf{Enforcing feature similarity at sample level}: The pretrained base classifier $B_c$ trained on complete training data $D_{train}$ is highly discriminative. To leverage this on our crafted data, we use cosine similarity loss at the logits of the $B_c$ network to encourage logit features on the denoised output of our generated data to be as discriminative as the features of the limited original training samples $D_{train}^{lim}$ by maximizing this loss.
\begin{equation}
\begin{gathered}
L_{cs} = 1/N^{k}   \sum\nolimits_{i=1}^{N^k} (\hspace{0.05in}CS(B_c(D_n(\bar{x}_b^i)), B_c(x_o^i)) +\\
                                          CS(B_c(D_n(\bar{x}_{int}^i)), B_c(x_o^i))\hspace{0.05in})\text{,}
\hspace{0.1in} \text{s.t.} \hspace{0.05in}CS(w, z) = \frac{w^{T}z}{\norm{w}\norm{z}}
\end{gathered}
\end{equation}
\textbf{Enforcing feature similarity at distribution level}: Unlike $L_{cs}$ which is applied at sample level, here we enforce distribution level matching between the set of our denoised generated data and the set of limited original training data by using maximum mean discrepancy (MMD)~\cite{gretton2012kernel} loss on the normalized pre-logit layer of $B_c$ network.
\begin{equation}
\begin{gathered}
L_{mmd} = MMD(F_{B_c}(D_n(\bar{x}_b)),F_{B_c}(x_o)) +\\ MMD(F_{B_c}(D_n(\bar{x}_{int})),F_{B_c}( x_o))
\end{gathered}
\end{equation}
Moreover, we also train the domain discriminator network $D_d$ (parameterized by $\phi$) using binary cross entropy loss ($L_{bce}$) to distinguish the distribution of gaussian perturbed samples and clean samples.
\begin{equation}
\begin{gathered}
L_{dd} = \sum_{x^i \in D_{train}^{lim}\cup \hspace{0.01in}D_n(\bar{D}_{train}^{lim})}  L_{bce}(D_d(F_{B_c}(x^i)),d^i)
\end{gathered}
\end{equation}
Here, if $x^i \in D_{train}^{lim}$ then $d^i=1$ and if $x^i \in D_n(\bar{D}_{train}^{lim})$ then $d^i=0$ . The negative gradients are backpropagated via GRL~\cite{ganin2015unsupervised} (multiplies calculated gradient by $-1$), to update the parameters $\theta$ of denoiser network $D_n$ such that features of limited training data $D_{train}^{lim}$ and its corresponding denoised output of gaussian corrupted data ($D_n(\bar{D}_{train}^{lim})$) on network $B_c$ are domain invariant.

Hence, the total loss can be written as follows:
\begin{equation}
\label{eq12}
L(D_n^\theta, D_d^\phi) =   \beta_1 L_{lc} - \beta_2 L_{cs} +  \beta_3 L_{mmd} + \beta_4 L_{dd}
\end{equation}
At test time, the trained denoiser network $D_n$ with optimal parameters $\theta^{*}$ prepended with base classifier $B_c$ is used for evaluation.
\section{Experiments}
\label{sec:experiments}
We demonstrate the effectiveness of our proposed approach (DE-CROP) on two widely-popular image-classification datasets, namely, CIFAR-10 \cite{krizhevsky2009learning} and Mini-ImageNet \cite{vinyals2016matching}. We limit the training set size of the datasets mentioned above by randomly selecting $1\%$ and $10\%$ samples respectively (ensuring class balance) from the $D_{train}$. Our baseline corresponds to training the denoiser using the $L_{lc}$ loss (similar to~\cite{salman2020denoised}). We fix the selected samples and use a ResNet-$110$ and ResNet-$12$ \cite{he2016deep} network (for CIFAR-$10$ and Mini-ImageNet respectively) as our pre-trained classifiers ($B_c$) with a value of $\sigma$ as $0.25$ for all our ablations and state-of-the-art comparisons, unless otherwise specified.
Refer to supplementary for additional ablations on different values of noise strength $\sigma$ ($0.12$, $0.50$, $1.00$), quantity of limited training data $D_{train}^{lim}$ ($5\%$, $10\%$, $20\%$, $100\%$) and choice of architecture for the pretrained classifier $B_c$.
We set the weights for the final loss-equation (refer eq.~\ref{eq12}) i.e. $\beta_1$, $\beta_2$, $\beta_3$, $\beta_4$ as $1$, $4$, $4$, $1$ respectively. In the subsequent subsections, we first show limited benefit of conventional techniques, followed by benefits of each component in DE-CROP and a comparison with state-of-the-art techniques.

\subsection{Improving Certification on Limited Training Data via Conventional Techniques}
$D_{n}$ trained with the $L_{lc}$ objective proposed by \cite{salman2020denoised} severely overfits in the presence of limited data resulting in poor certification accuracy (refer Fig.~\ref{fig:motivation}). In this section, we explore whether conventional supervised-learning techniques such as explicit regularization and data-augmentation are equipped to meaningfully improve certification accuracy when dealing with limited data.

\begin{table}[htp]
\centering
\scalebox{0.8}{
\begin{tabular}{|c|c|ccc|}
\hline
\multirow{2}{*}{\textbf{Method}} & \textbf{\begin{tabular}[c]{@{}c@{}}Standard\\ Certified\end{tabular}} & \multicolumn{3}{c|}{\textbf{Robust Certified}} \\ \cline{2-5}
             & (r=0.00) & \multicolumn{1}{c|}{(r=0.25)} & \multicolumn{1}{c|}{(r=0.50)} & (r=0.75) \\ \hline
Without Reg. & 20.60    & \multicolumn{1}{c|}{4.60}    & \multicolumn{1}{c|}{0.80}    & 0.00      \\ \hline
$L_{1}$ Reg.      & 22.60    & \multicolumn{1}{c|}{5.80}    & \multicolumn{1}{c|}{0.60}    & 0.00      \\ \hline
$L_{2}$ Reg.      & \textbf{27.80}    & \multicolumn{1}{c|}{\textbf{7.80}}    & \multicolumn{1}{c|}{\textbf{0.80}}    & 0.00      \\ \hline
\end{tabular}
}
\caption{Effect of adding weight decay regularizer ($L_{1}$ and $L_{2}$) on limited data certification against adversarial attacks. $L_{2}$ regularization obtains better certified standard and robust accuracies.
\label{tab:table1}
\end{table}
In Table~\ref{tab:table1}, we regularize the $D_{n}$ by applying $L_{1}$ and $L_{2}$ reg. (regularization) \cite{hanson1988comparing}. We observe that $L_{2}$ reg. improves certified
performance, whereas $L_{1}$ reg. only results in a marginal improvement. Although $L_{2}$ reg. limits the overfitting effect in the presence of limited data, it can’t help in improving the inherent data diversity. Thus, in Table~\ref{tab:table2}, we explore whether traditional affine and specialized augmentation methods (such as mixup \cite{zhang2017mixup} and cutmix \cite{yun2019cutmix}), when combined with $L_{2}$ reg. further boost performance.

Rows $2$-$4$ in Table~\ref{tab:table2} correspond to affine-transformations where the intensity of the augmentation is increased progressively (i.e. policy$1$$<$policy$2$$<$policy$3$). We also experiment with widely-popular augmentation techniques: ‘mixup’ and 'cutmix'. Surprisingly, we observe that policy $1$(row-$2$; the lightest augmentation) performs the best, followed closely by ‘no-aug.’. We hypothesize that if the augmentation-policy is too aggressive, $B_c$ can make incorrect predictions leading to noisy gradients for the $D_n$ to learn from, resulting in poor certification accuracy.

\begin{table}[htp]
\centering
\scalebox{0.8}{
\begin{tabular}{|c|c|ccc|}
\hline
\multirow{2}{*}{\textbf{Method}} & \textbf{\begin{tabular}[c]{@{}c@{}}Standard\\ Certified\end{tabular}} & \multicolumn{3}{c|}{\textbf{Robust Certified}} \\ \cline{2-5}
                                                          & (r=0.00) & \multicolumn{1}{c|}{(r=0.25)} & \multicolumn{1}{c|}{(r=0.50)} & (r=0.75) \\ \hline
No Aug.                                                   & 27.80    & \multicolumn{1}{c|}{7.80}    & \multicolumn{1}{c|}{0.80}    & 0.00      \\ \hline
\begin{tabular}[c]{@{}c@{}}Aug.\\ (policy 1)\end{tabular} & \textbf{29.80}    & \multicolumn{1}{c|}{\textbf{9.20}}    & \multicolumn{1}{c|}{\textbf{1.40}}    & 0.00      \\ \hline
\begin{tabular}[c]{@{}c@{}}Aug.\\ (policy 2)\end{tabular} & 26.40    & \multicolumn{1}{c|}{7.40}    & \multicolumn{1}{c|}{1.00}    & 0.00      \\ \hline
\begin{tabular}[c]{@{}c@{}}Aug\\ (policy 3)\end{tabular}  & 21.00     & \multicolumn{1}{c|}{3.00}     & \multicolumn{1}{c|}{0.20}    & 0.00      \\ \hline
Mixup                                                     & 24.20    & \multicolumn{1}{c|}{5.40}    & \multicolumn{1}{c|}{0.60}    & 0.00      \\ \hline
Cutmix                                                    & 19.80    & \multicolumn{1}{c|}{3.20}    & \multicolumn{1}{c|}{0.00}    & 0.00      \\ \hline
\end{tabular}
}
\caption{Investigating the effect of augmentation at different intensity levels (policies), mixup and cutmix strategies in minimizing overfitting on limited training data.
The augmentation with light intesnity (policy 1) yields marginal improvement against l2 perturbations of different radii compared to no-augmentation strategy.}
\label{tab:table2}
\end{table}

Thus, a combination of policy $1$ and L$2$-reg. improves the certified standard and robust accuracies. We use this combination as a baseline for further experiments upon which we make orthogonal improvements.

\subsection{Effectiveness of our Generated Data}
\label{gen_data}
One of the critical reasons for the drop in certified accuracy is the lack of diversity in the limited data. We address this problem by generating synthetic samples that provide diversity in the feature space. As elaborated in the proposed section (refer Sec.~\ref{subsec:generation}): adversarial samples (termed as boundary-samples i.e. $x_b$) serve as a good candidate for this task since they flip the classifier prediction with minimum possible perturbations, thus allowing the $D_n$ to train on samples from less-dense boundary regions. Moreover, we also generate samples whose features are an interpolation between the original and boundary samples, crafted by minimizing the $L_{mse}$ loss  between the interpolated logits and the generated sample logits (as described by eq.~\ref{eq7}).

We empirically validate our motivation in Table~\ref{tab:table3}, where we observe that using $L_{cs}$ on both $x_{b}^{i}$ and corresponding $x_{int}^{i}$ leads to a massive improvement in performance over the baseline. Interestingly, the gain in performance when using only $x_{b}^{i}$ is comparatively modest. This observation further reinforces our intuition regarding the complementary nature of the information provided by the $x_{b}^{i}$ and $x_{int}^{i}$. Moreover, contrary to adversarial training that generates adversarial samples at every iteration during the training time, we only need to generate the boundary and interpolated samples once (amounting to negligible increase in training time), as pre-trained classifier $B_c$ is fixed and not trainable.
\vspace{-0.15in}
\begin{table}[htp]
\centering
\scalebox{0.8}{
\begin{tabular}{|c|c|ccc|}
\hline
\multirow{2}{*}{\textbf{Method}} &
  \textbf{\begin{tabular}[c]{@{}c@{}}Standard \\ Certified\end{tabular}} &
  \multicolumn{3}{c|}{\textbf{Robust Certified}} \\ \cline{2-5}
         & (r=0.00) & \multicolumn{1}{c|}{(r=0.25)} & \multicolumn{1}{c|}{(r=0.50)} & (r=0.75) \\ \hline
Baseline & 29.80    & \multicolumn{1}{c|}{9.20}    & \multicolumn{1}{c|}{1.40}    & 0.00      \\ \hline
\begin{tabular}[c]{@{}c@{}}Ours (with boundary\\ samples)\end{tabular} &
  31.60 &
  \multicolumn{1}{c|}{7.80} &
  \multicolumn{1}{c|}{1.00} &
  0.20 \\ \hline
\begin{tabular}[c]{@{}c@{}}Ours (with boundary +\\ interpolated samples)\end{tabular} &
  \textbf{48.80} &
  \multicolumn{1}{c|}{\textbf{22.00}} &
  \multicolumn{1}{c|}{\textbf{6.00}} &
  \textbf{0.80} \\ \hline
\end{tabular}
}
\caption{Benefit of our generated samples in improving certification. Using boundary and interpolated samples, we obtain significant boost in certified accuracy on original and l2 perturbed data.}
\label{tab:table3}
\end{table}
\vspace{-0.15in}
\subsection{Distribution Alignment: Enhancing Certification in Limited Data Setup}
\label{dist_align}

In Table~\ref{tab:table4}, we observe that using domain-discriminator ($D_d$) along with the previously introduced $L_{cs}$ loss works quite well as the standard certified accuracy improves by ~6$\%$ and certified robust accuracy improves by 4$\%$ (at r=$0.25$) compared to only using $L_{cs}$.

We also explore whether equipping the $D_n$ with explicit distribution discrepancy losses such as $L_{mmd}$ would also work well in combination with the $D_d$. Intuitively, applying $L_{mmd}$ along with $L_{cs}$ should make the job of $D_d$ harder, leading to a better $D_d$. Consequently, improving $B_c$’s feature representation via the negative gradients of domain-discriminator loss ($L_{dd}$).

We indeed observe this in Table~\ref{tab:table4}, where using $L_{mmd}$ in combination with the $L_{cs}$ and the $L_{dd}$ setup leads to an improvement in performance across both standard and robust accuracies (across radii). Thus, using $L_{mmd}$ and $L_{dd}$ in conjunction with the previously discussed $L_{cs}$ and $L_{lc}$ constitutes our final approach: DE-CROP
\begin{table}[htp]
\centering
\scalebox{0.8}{
\begin{tabular}{|c|c|ccc|}
\hline
\multirow{2}{*}{\textbf{Method}} &
  \textbf{\begin{tabular}[c]{@{}c@{}}Standard\\ Certified\end{tabular}} &
  \multicolumn{3}{c|}{\textbf{Robust Certified}} \\ \cline{2-5}
                      & (r=0.00) & \multicolumn{1}{c|}{(r=0.25)} & \multicolumn{1}{c|}{(r=0.50)} & (r=0.75) \\ \hline
Baseline              & 29.80    & \multicolumn{1}{c|}{9.20}    & \multicolumn{1}{c|}{1.40}    & 0.00      \\ \hline
Ours (instance level) & 48.80    & \multicolumn{1}{c|}{22.00}     & \multicolumn{1}{c|}{6.00}    & 0.80     \\ \hline
\begin{tabular}[c]{@{}c@{}}Ours (instance + \\ distribution level via \\ discriminator)\end{tabular} &
  54.00 &
  \multicolumn{1}{c|}{26.00} &
  \multicolumn{1}{c|}{6.80} &
  1.80 \\ \hline
\begin{tabular}[c]{@{}c@{}}Ours (instance + \\ distribution level via \\ discriminator and MMD)\end{tabular} &
  \textbf{57.60} &
  \multicolumn{1}{c|}{\textbf{27.20}} &
  \multicolumn{1}{c|}{\textbf{9.20}} &
  \textbf{2.20} \\ \hline
\end{tabular}
}
\caption
Besides performance gains with instance-level feature matching, we observe further improvements in certified standard and robust accuracy when distributions of denoised and clean data are aligned in feature space via domain-discriminator and MMD.}
\label{tab:table4}
\end{table}
\subsection{Comparison with state-of-the-art}
In this section, we compare the effectiveness of our approach DE-CROP against
state-of-the-art robustness certification techniques, namely, denoised-smoothing by Salman \etal~\cite{salman2020denoised} and gaussian-augmentation by Cohen \etal~\cite{cohen2019certified}.

Since Salman \etal~\cite{salman2020denoised} don’t report performance on limited data scenarios in their paper, we use the code from their official implementation\footnote{https://github.com/microsoft/denoised-smoothing} for evaluating their proposed method ($D_n$ with $L_{lc}$) in the presence of only $1\%$ (for CIFAR-$10$) and $10\%$ (for Mini-ImageNet) $D_{train}$. Similarly, for Cohen \etal~\cite{cohen2019certified}, we re-train the classifier with gaussian augmentation only on the available limited training data. Although Cohen \etal~\cite{cohen2019certified}’s technique is unfeasible for our problem setup as the API provider may not prefer re-training and replacing the deployed model, we still compare our performance as gaussian-augmentation often outperforms previous denoiser-based approaches in presence of full training data.

As shown in Fig.~\ref{fig:sota}, our proposed approach DE-CROP comfortably outperforms Salman \etal~\cite{salman2020denoised} on CIFAR-$10$, improving the certified standard accuracy by $27\%$ and consistently improving robust accuracy across radii. The performance of Cohen \etal~\cite{cohen2019certified} drops to $10\%$ across all radii, indicating that the network behaves like a random baseline (i.e. predicting each class with equal probability irrespective of the input). We observe similar trends for Mini-ImageNet, where we comfortably outperform both Salman \etal~\cite{salman2020denoised} and Cohen \etal~\cite{cohen2019certified} further demonstrating the wide applicability of our approach.
\begin{figure}[htp]
\centering
\centerline{\includegraphics[width=0.49\textwidth]{figures/sota_result_per.pdf}}
\caption{Performance comparison of our approach (\textit{DE-CROP}) with other methods. We comfortably outperform on both the datasets. We also compared with Cohen \etal where the classifier is trained from scratch unlike ours where retraining is not feasible.
}
\label{fig:sota}
\end{figure}
\vspace{-0.11in}
\subsection{Certification of Black Box Classifiers with \newline Limited Training Data}
In previous sections we assumed white-box access to the pre-trained base classifier $B_c$ i.e. we can backpropagate the gradients through $B_c$ to optimize the denoiser ($D_n$). However, this may not always be the case as the API provider can limit access to only $B_c$’s predictions (i.e. black-box) due to proprietary reasons. Since the black-box setup restricts the gradient information of $B_c$, we first use a black-box model stealing technique \cite{barbalau2020black} to train a surrogate model: $S_m$. We use $S_m$ which allows gradient backpropagation, to train $D_n$ using our proposed approach DE-CROP (refer Fig.~\ref{fig:approach}). Finally, for evaluation we use the denoiser (trained via $S_m$) to certify robustness of the black-box classifier $B_c$.

\begin{figure}[htp]
\centering
\centerline{\includegraphics[width=0.35\textwidth]{figures/black_box_excel.pdf}}
\caption{Investigating the efficacy of our method in black-box scenario (no access to pertrained model weights). We observe a minor drop in performance as compared to white-box setup for both certified standard accuracy ($l_2^r$=$0.00$) and robust accuracy ($l_2^r$=$0.25$).}
\label{fig:blackbox}
\end{figure}
In Fig.~\ref{fig:blackbox}, we compare the certification performance (of $B_c$) using denoisers trained via $S_m$ (`black-box access’) to the denoiser trained directly on $B_c$ (`white-box access’). We take $B_c$ as Alexnet and two different choices for $S_m$, namely ResNet-$18$ and Half-Alexnet. Our method DE-CROP yields very similar performance in the black box setting across different architectures of $S_m$. Also, the performance drop compared to white box setting is marginal, highlighting the suitability of our technique even when the pretrained classifier weights are not shared.
\section{Conclusion}
We presented our approach (DE-CROP), which for the first time, solves the problem of providing provable robustness guarantees to a pretrained classifier in the challenging limited training data settings. Our method comprises a two-step process - a) generation of boundary and interpolated samples ensuring feature diversity and b) effectively utilizing the generated samples along with limited training samples for training the denoiser using the proposed losses to ensure feature similarity between the denoised output and clean data at two different granularities (instance level and distribution level). We validate the efficacy of the generated data as well as the contribution of individual losses by extensive ablations and experiments across CIFAR-$10$ and Mini-ImageNet datasets. Moreover, our method works quite well in the black-box setting as it provides similar certification performance compared to the white-box setup.

\vspace{0.1in}
\noindent{\textbf{Acknowledgements}} This work is partially supported by a Young Scientist Research Award (Sanction no. 59/20/11/2020-BRNS) from DAE-BRNS, India.

{\small
\bibliographystyle{ieee_fullname}


\section*{References}


\section{Introduction}
The first evidence for an accelerated expansion of the universe at the
present epoch was presented in the late 1990s by Riess et
al.~\cite{riess98} and Perlmutter et al.~\cite{perlmutter}.
Fitting a cosmological model including a cosmological constant matched
the supernovae type Ia of the considered data sets
significantly better than a model without such a constant.
In the following years, a lot of effort was put into getting larger
supernova (SN) sets of increasing quality \cite{jha,snls,gold07,essence}.
On the other hand, a variety of
cosmological models have been developed that could be fit to the
newest data sets. Those models are characterized by certain parameters
which are used to derive a redshift-luminosity relation that can
be compared to the observed values.
The problem with this so-called dynamical
approach is that it is impossible to test without assumptions on
the matter and energy content of the universe
if there really is a phase of acceleration in the expansion history of
the universe.

Some authors tried to avoid this
problem by taking a kinematical approach, i.e. they only considered
the scale factor $a$ and its derivatives, such as
the deceleration parameter $q(z)$, without using
any model specific density parameters or a dark energy equation of
state. The first kinematical analysis of SN data was done by
Turner and Riess \cite{turner} who considered averaged values $q_1$ for
redshift $z<z_1$ and $q_2$ for $z>z_1$ concluding that a present acceleration
and a past deceleration is favoured by the data. Other authors tested
a variety of special parameterizations of $q(z)$
\cite{gold04,elgaroy} or used  $a(t)$ \cite{wang} or the Hubble rate $H(z)$
\cite{john}. Instead of considering a special parameterization,
a more general approach has been done by Shapiro and Turner
\cite{shapiro} who expanded the deceleration parameter $q$ into
principal components.
Rapetti et al.~\cite{rapetti} expanded the jerk parameter $j$ into a
series of orthonormal functions.
However, one has to be careful when doing a series expansion in $z$ as
SNe with a redshift larger than 1 are not within the radius of
convergence. This problem can be solved by reparameterizing redshift
\cite{cattoen}.

In the present work we will neither make any assumptions about the content
of the universe, nor about the parameterization of $q(z)$ or another
kinematical quantity. Moreover, we do not need to assume the validity
of Einstein's equations.
Instead, we will only ask the question if the
hypothesis holds that the universe never expanded accelerated
between the time when the light was emitted from a
SN and today. Our assumption for the test is that the universe is
isotropic and homogeneous. This is certainly not true for the real
universe in a strict sense. However, we follow the standard approach
in assuming that the cosmic structure does not modify the observed SN
magnitudes and redshifts apart from random peculiar motion.
The basic idea of this analysis has already been
presented by Visser \cite{visser} and has been applied to SN data by
different groups \cite{santos, gong}.
But while Santos et al.~\cite{santos} made some mistakes in their
analysis (that we will discuss later in section \ref{concl}) and
Gong et al.~\cite{gong} only state that
accelerated expansion is ``evident'', we are able to give a
quantitative value for this evidence. Additionally, we study the size
of systematic effects.

For the fit one usually marginalizes over a function of the absolute
magnitude $M$ and the Hubble constant $H_0$ because these two values
cannot be determined independently by only considering
SNe. Marginalization is not suitable for our analysis because in order
to do so a special cosmological model or at least a parametization of
$q(z)$ has to be inserted, which is what we want to avoid. But the SNe
can be calibrated by cepheid meassurements and thus the values of $M$
and $H_0$ are determined. Using this additional information, no
marginalization is needed. As there is still a controversy
\cite{jackson} about the
appropriate calibration method, we will take two quite different
results of calibration into account \cite{riess, sandage}.

In order to test the robustness of our analysis, we consider two
different SN data sets (the 2007 Gold sample \cite{gold07} and the
ESSENCE set \cite{essence}), where the data of the ESSENCE set are
once obtained by the multicolour light-curve shape (MLCS2k2) \cite{jha}
fitting method and once by the spectral adaptive light-curve template
(SALT) \cite{guy}. Both calibration methods are applied to each
set. The analysis shows that in all cases the data indicate an
accelerated expansion. But the confidence level at which acceleration
can be stated strongly depends on the data set, the fitting method and
the calibration. We begin our analysis by assuming a flat universe,
but later also consider the cases of an open and a closed universe.

\section{Method}
We want to keep our test as model-independent as possible, but still a
few assumptions have to be made. We consider inflationary cosmology to
be correct. This implies large scale homogeneity and isotropy of the
universe as well as spatial flatness. (Yet, we will give up the assumption
of a flat universe later in section \ref{openclosed}.)
We also assume that cosmological
averaging (here along the line of sight) does not modify the result
obtained in a Friedmann-Lema\^{i}tre model.
In such a universe, the
luminosity distance $d_{\mbox{\scriptsize L}}(z)$ is given by
\begin{equation}
d_{\mbox{\scriptsize L}}(z) = (1+z) \int_0^z \frac{\rmd
  \tilde{z}}{H(\tilde{z})}\;.
\end{equation}
As we are interested in the question whether the universe accelerates
or decelerates at the present epoch, we need to examine the
deceleration parameter $q(z)$: If $q$ is positive, the universe
expands decelerated, for a negative $q$ it accelerates.  $q(z)$ can be
expressed in terms of the Hubble parameter $H(z)$:
\begin{equation}
q(z) = \frac{H'(z)}{H(z)} (1+z) -1 \,,
\end{equation}
where the prime denotes the derivative with respect to $z$.
Integrating this equation yields
\begin{equation}
\ln \frac{H(z)}{H_0} =
\int_0^z\frac{1+q(\tilde{z})}{1+\tilde{z}}\,\rmd \tilde{z}\;.
\end{equation}

Our null hypothesis is that the universe never expanded accelerated,
i.e. $q(z)\ge 0$ for all $z$. Under this assumption, the above
equation turns into the inequality
\begin{equation}
\ln \frac{H(z)}{H_0} \ge \int_0^z\frac{1}{1+\tilde{z}}\,\rmd \tilde{z}
= \ln (1+z)
\end{equation}
or $H(z)\ge H_0(1+z)$. Thus, for the luminosity distance we have
\begin{equation}\label{dlinequ}
d_{\mbox{\scriptsize L}}(z) \le (1+z) \frac{1}{H_0} \int_0^z
\frac{\rmd \tilde{z}}{1+\tilde{z}} = (1+z)\frac{1}{H_0}\ln (1+z)\;.
\end{equation}

In order to test our hypothesis, we consider different data sets of
SNe type Ia. If the observed luminosity distance is
significantly larger than the luminosity distance of a universe with a
constant $q=0$, the hypothesis can be rejected at a high confidence
level. Note that the rejection of the hypothesis does not mean that
there was no deceleration between the time the light was emitted from
a SN until now, it only gives evidence that there was at some
time a phase of acceleration. Thus, we cannot determine the transition
redshift between deceleration and acceleration. This restriction to
our analysis comes from the integral over redshift in the calculation
of $d_{\mbox{\scriptsize L}}(z)$.

As a first step, the data in the considered sets have to be
calibrated consistently. The distance modulus $\mu$ is related
to the luminosity distance by
\begin{equation}
\mu=m-M = 5\log d_{\mbox{\scriptsize L}} +25 \,,
\end{equation}
where $d_{\mbox{\scriptsize L}}$ is given in units of Mpc. The
distance modulus of the SNe Ia in those sets is always given in
arbitrary units because the absolute magnitude $M$ cannot be measured
independently of the Hubble constant $H_0$. Only the apparent
magnitude $m$ and thus the relative distance moduli are measured at
high precision. In order to determine the absolute magnitude, the SNe
have to be calibrated by measuring the distance of cepheids in the
host galaxies. Then also the Hubble constant can be determined. But
there is still disagreement between different groups about the correct
calibration analysis. In this work we will consider two results for
$M$ and $H_0$, namely that of Riess et al.~\cite{riess} and that of
Sandage et al.~\cite{sandage}. In the following we will refer to those
results as Riess calibration and Sandage calibration,
respectively. The difference in the calibration comes mainly from the
different assumptions on the evolution of the cepheid
period-luminosity relation with host metallicity \cite{jackson}.
So for preparing the data for our
analysis, first the distance moduli $\mu_i$ have to be adjusted to the
assumed absolute magnitude.

We define the magnitude $\Delta\mu_i$ as the observed distance modulus
$\mu_i$ of
the $i$-th SN minus the distance modulus in a universe with
constant deceleration parameter $q=0$ at the same redshift $z_i$:
\begin{equation}\label{deltamui}
\Delta\mu_i = \mu_i - \mu(q=0) = \mu_i - 5\log\left[
  \frac{1}{H_0}(1+z_i)\ln(1+z_i) \right] -25 \,.
\end{equation}
If the error in redshift and the peculiar velocities of the SNe are
already included in the error $\sigma_{\mu_i}$ given in a certain data
set, then the error $\sigma_i$ of $\Delta\mu_i$ equals
$\sigma_{\mu_i}$. Otherwise, the resulting error of $\Delta\mu_i$ is
calculated by:
\begin{equation}
\sigma_i = \left[\sigma_{\mu_i}^2 + \left( 5\frac{\ln(1+z_i)
    +1}{(1+z_i)\ln(1+z_i)\ln10} \right)^2 \left(\sigma_z^2 +
  \sigma_v^2 \right) \right]^{\frac{1}{2}} \,.
\end{equation}

Let $\mu_{\mbox{\scriptsize th}}(z)$ be the theoretical
distance modulus of a cosmological model, which describes the
expansion of the universe correctly. Then
\begin{equation}
\Delta\mu_{\mbox{\scriptsize th}}(z_i) = \mu_{\mbox{\scriptsize
    th}}(z_i) - \mu(q=0)
\end{equation}
would be the ``true'' value corresponding to the meassured SN value
$\Delta\mu_i$.  The null hypothesis for our test is that the universe
never expanded accelerated, i.e. $\Delta\mu_{\mbox{\scriptsize th}}
\le 0$ for each SN. We reject that hypothesis if the measured
value $\Delta\mu_i$ lies above a certain action limit
$A_{\mbox{\scriptsize a}}$, otherwise
we accept it. We want to keep the risk low that we conclude a late
time acceleration of the universe, when there is indeed no
acceleration at all. Therefore the action limit must be relatively
high. If we want the confidence level for concluding an accelerated
expansion to be 99\%, the action limit must be $A_{\mbox{\scriptsize
    a}}(99\%)=2.326\sigma_i$. For a confidence level of 95\%, the
limit is $A_{\mbox{\scriptsize a}}(99\%)=1.645\sigma_i$. Here we
assume that the measured values $\Delta\mu_i$ at a given redshift
follow a normal distribution.

On the other hand, we can test the hypothesis that the universe
expanded accelerated all the time from light emission of a SN
until today. This hypothesis can be rejected at a 99\% CL if
$\Delta\mu_i$ is below the action limit $A_{\mbox{\scriptsize
    d}}(99\%)=-2.326\sigma_i$. That would mean that there must have
been a phase of deceleration in the late time universe, but it would
not exclude a phase of acceleration.

\section{Data Sets}
The important parameters of the SNe are obtained by fitting
their light-curves. The results depend on the fitter that is used. The
most common fitter is MLCS2k2 \cite{jha}. Here we will also consider
the SALT fitting method \cite{guy}. The main difference between the
two methods is in how the peak luminosity corrections are determined
by the colour of the SNe \cite{conley}. While MLCS2k2 assumes that the
corrections that have to be made are only due to dust, SALT takes an
empirically approach to determine the relation between the colour and
the luminosity correction.

For the analysis we take the following SN Ia data sets:
\begin{description}
\item{Gold 2007 (MLCS2k2):} the Gold sample by Riess et al.~(2007)
  \cite{gold07}, which was obtained by using the MLCS2k2 fitting
  method
\item{ESSENCE (MLCS2k2):} the set given by Wood-Vasey et al.~(2007)
  \cite{essence}, which includes data from ESSENCE, SNLS \cite{snls}
  and nearby SNe \cite{jha}, fitted with MLCS2k2
\item{ESSENCE (SALT):} the same set fitted with SALT
\end{description}
As suggested by Riess et al.~\cite{gold07}, we discarded all SNe with
a redshift of
$z<0.0233$ from the Gold sample, which leaves us 182 SNe with
$0.0233<z<1.755$. In ESSENCE
(MLCS2k2) and ESSENCE (SALT), the SNe with bad light curve fits were
rejected for each fitting method seperately. This leaves 162 SNe for
ESSENCE (MLCS2k2) and 178 for ESSENCE (SALT) with $0.015<z<1.01$. 153
of the SNe are
contained in both sets. Due to the differences of the SNe contained in
ESSENCE (MLCS2k2) and ESSENCE (SALT) we will refer to them as
different sets in the following. Nevertheless, you should keep in mind
that they share a large number of SNe and thus are not
independent sets.

The Riess calibration yields a
value for the V-band magnitude of $M_{\mbox{\scriptsize
    V}}(t_0)=-19.17\pm 0.07$mag, where $t_0$ is the time of the
B-band maximum. For the Gold 2007 set $M_{\mbox{\scriptsize V}}(t_0)$ was
considered to be $-19.44$mag. Therefore, in order to get the
appropriate distance modulus $\mu=m-M$, we need to substract
$0.27$mag from the value given in the Gold sample. From the distance
modulus values in the ESSENCE sets $0.22$mag have to be substracted
because in these sets a B-band magnitude of $M_{\mbox{\scriptsize
B}}=-19.5$mag is assumed and Riess et al.~\cite{riess} give the relation
$M_{\mbox{\scriptsize B}}-M_{\mbox{\scriptsize V}}=-0.11$. For this
calibration one gets a Hubble constant of $H_0=73\pm
4(\mbox{statistical}) \pm 5(\mbox{systematic})$.

The results of Sandage et al.~\cite{sandage} for the SNe absolute
magnitudes are $M_{\mbox{\scriptsize V}}=-19.46$mag and
$M_{\mbox{\scriptsize B}}=-19.49$mag and for the Hubble
constant $H_0=62.3 \pm 1.3$(statistical) $\pm$
5.0(systematic). So considering the Sandage calibration, we have to
add 0.02mag to distance moduli of the Gold sample and substract
0.01mag from the values given in the ESSENCE sets.

\section{Results for a flat universe}

\subsection{Single SNe}
The relevant parameter for our analysis is $\Delta\mu_i$ given by
equation \eref{deltamui}. Its values are plotted in figure
\ref{deltamufig} for the SNe of the three data sets and the two
different calibration methods. The curve for a flat $\Lambda$CDM model
(with $\Omega_{\mbox{\scriptsize m}}=0.3$) is shown in the plots
in order to give the reader a notion of how the redshift-dependency of
$\Delta\mu$ could look like.  Most of the data values are positive
which indicates an accelerated expansion.
A similar diagram has been presented by Gong et al.~\cite{gong} for
single SNe as well as for binned SN data where
they used a combined Gold and ESSENCE data set. They kept the arbitrary
value of the absolute magnitude $M$ given in the set and then determined
the Hubble constant by using SNe with $z\le 0.1$ to be $H_0=66.04$. At
this point they stopped their analysis by concluding that due to the
large number of SNe that lie above the curve of a universe with
$q=0$, accelerated expansion is evident.
But the fact that there are
also several data points below that curve and that the errors
$\sigma_i$ (with a typical value of 0.23mag for MLCS2k2 and 0.18mag
for SALT) are
approximately of the same size as the values $\Delta\mu_i$ give rise
to the question how certain acceleration really is. Thus, we will
provide a more quantitative analysis in this work.

\begin{figure}
\centering \subfloat[Gold 2007 (MLCS2k2), Riess calibration]{
  \includegraphics*[width=7.5cm]{gold_riess_1_a.eps}} \hfill
\subfloat[Gold 2007 (MLCS2k2), Sandage calibration]{
  \includegraphics*[width=7.5cm]{gold_sand_1_b.eps}}
\\ \subfloat[ESSENCE (MLCS2k2), Riess calibration]{
  \includegraphics*[width=7.5cm]{mlcs2k2_riess_1_c.eps}} \hfill
\subfloat[ESSENCE (MLCS2k2), Sandage calibration]{
  \includegraphics*[width=7.5cm]{mlcs2k2_sand_1_d.eps}}
\\ \subfloat[ESSENCE (SALT), Riess calibration]{
  \includegraphics*[width=7.5cm]{salt_riess_1_e.eps}} \hfill
\subfloat[ESSENCE (SALT), Sandage calibration]{
  \includegraphics*[width=7.5cm]{salt_sand_1_f.eps}}
\caption{Magnitude $\Delta\mu_i$, as defined in equation
  \eref{deltamui}, for the three data sets and the two
  calibrations. SNe that indicate acceleration at 99\% CL are
  plotted in red ($\times$), those that indicate deceleration in blue
  ($\hexstar$). Also shown is the curve for a flat $\Lambda$CDM model
  with $\Omega_m=0.3$ (which is not a fit to the data). Note that the
  SALT method leads to a larger spread in $\Delta\mu_i$, whereas the
  Gold set extends to higher redshifts.}
\label{deltamufig}
\end{figure}

We counted the SNe of each set whose values of $\Delta\mu_i$
were above the action limit $A_{\mbox{\scriptsize
    a}}(95\%)=1.645\sigma_i$ and those above $A_{\mbox{\scriptsize
    a}}(99\%)=2.326\sigma_i$ (which indicates acceleration at a 95\%
and a 99\% CL, respectively) and those with values below
$A_{\mbox{\scriptsize d}}(95\%)=-1.645\sigma_i$ and below
$A_{\mbox{\scriptsize d}}(99\%)=-2.326\sigma_i$ (which indicates
deceleration). The results are shown in table \ref{acctab1}. The
clearest evidence for an accelerated expansion is given by the ESSENCE
(SALT) set with the Riess calibration with 64 SNe
indicating acceleration at a 99\% confidence level and none indicating
deceleration. Also in the other sets accelerated expansion is
preferred with the exception of ESSENCE (MLCS2k2) in the Sandage
calibration where none of the two expansion histories is preferred in
the test using the action limit $A(99\%)$.

\begin{table}
\begin{indented}
\lineup
\caption{\label{acctab1} Number of SNe indicating acceleration or
  deceleration at 95\% and 99\% CL for the different data
  sets and calibrations. Also given is the total number of SNe in each set. The most
  different results are highlighted.}
\item[]
\begin{tabular}{@{}lllllll} \br
&\centre{2}{Gold 2007 (MLCS2k2)}
  &\centre{2}{ESSENCE (MLCS2k2)} &\centre{2}{ESSENCE (SALT)} \\
& Riess & Sandage & Riess & Sandage & Riess & Sandage \\\mr
acceleration (95\% CL) & \037 & \027 & \037 & \014 & \096 & \053\\
acceleration (99\% CL) & \013 &\0\07 & \011 &\0\0{\bf 3}&\0{\bf 64}&\030
\\\mr
deceleration (95\% CL) &\0\02 &\0\04 &\0\03 &\0\07 &\0\01 &\0\07\\
deceleration (99\% CL) &\0\00 &\0\01 &\0\02 &\0\0{\bf 3}&\0\0{\bf 0}&\0\02
\\\mr
number of SNe & 182 & 182 & 162 & 162 & 178 & 178 \\ \br
\end{tabular}
\end{indented}
\end{table}

It is noticeable that the two light-curve fitting methods for the
ESSENCE set yield very different
results. (For a discussion on these differences at low redshifts, see
\cite{conley}.)
This could be an indication that at least one of the two
fitting methods does not give the correct result. In order to consider
this possibility we take a look at the differences between
$\Delta\mu_i$ obtained by SALT and $\Delta\mu_i$ obtained by
MLCS2k2. They are shown in figure \ref{mudiff} for the 153 SNe
contained in both ESSENCE sets. For some SNe the difference is very
large (namely up to one magnitude). In order to see if the two sets
are consistent, we need to know how large the difference is in terms
of the error $\sigma_i=\left[\sigma^2_i\mbox{(SALT)} +
  \sigma^2_i\mbox{(MLCS2k2)}\right]^{1/2}$. The result is shown in
figure \ref{weightmudiff}.  We want the systematical error due to the
different fitting methods to be smaller than the statistical error of the
observational data.  Thus, we discard all SNe with a
difference in $\Delta\mu_i$ larger than $1\sigma_i$ and those that
are only contained in one of the two ESSENCE sets, which leaves 129 SNe
in the sets. Again we counted the SNe indicating acceleration or
deceleration for each set and each calibration method. The result is
shown in table \ref{acctab2}. The outcome of this test did not change
qualitatively by discarding suspicious SNe from the sets.
Thus, we will use all the SNe of the ESSENCE sets in the following.

\begin{figure}
\centering \subfloat[]{\label{mudiff}
  \includegraphics*[width=7.5cm]{mu_diff_2_a.eps}} \hfill
\subfloat[]{\label{weightmudiff}
  \includegraphics*[width=7.5cm]{mu_diff_sigma_2_b.eps}}
\caption{Differences of apparent magnitudes obtained by the SALT and the
  MLCS2k2 fitting method \subref{mudiff} and these differences devided
  by $\sigma_i$ \subref{weightmudiff}.}
\end{figure}

\begin{table}
\begin{indented}\lineup
\caption{\label{acctab2} Number of SNe indicating acceleration or
  deceleration for SNe of the ESSENCE sets with
  $|\Delta\mu_i\mbox{(SALT)}-\Delta\mu_i\mbox{(MLCS2k2)}|/\sigma_i \le
  1$. The total number of SNe in each set is 129. Again, the most
  discrepant results are highlighted.}
\item[]\begin{tabular}{@{}lllll} \br
&\centre{2}{ESSENCE (MLCS2k2)} &\centre{2}{ESSENCE (SALT)} \\
& Riess & Sandage & Riess & Sandage \\\mr
acceleration (95\% CL) & 33 & 11 & 69 & 30 \\
acceleration (99\% CL) &\08 &\0{\bf 2} &{\bf 40} & 14 \\\mr
deceleration (95\% CL) &\01 &\02 &\00 &\06 \\
deceleration (99\% CL) &\00 &\0{\bf 1} &\0{\bf 0} &\01 \\ \br
\end{tabular}
\end{indented}
\end{table}

The error of the distance modulus contains the peculiar velocities
$v_{\mbox{\scriptsize pec}}$ of the SNe. In the ESSENCE sets
$\sigma_v$ is assumed to be 400km/s for all SNe. We
wanted to know how sensitive our test is with respect to changes of
the peculiar velocity. Table \ref{acctabvel} shows that varying
$\sigma_v$ almost does not change the result with
the exception of ESSENCE (MLCS2k2) in the
Sandage calibration, where the number of SNe indicating deceleration
at a 95\% CL is 7 for $\sigma_v=400$km/s and
500km/s, but 11 for $\sigma_v=300$km/s.

\begin{table}
\begin{indented}\lineup
\caption{\label{acctabvel} Number of SNe indicating acceleration or
  deceleration for SNe of the ESSENCE sets with different peculiar
  velocity dispersions $\sigma_v$.}
\item[]\begin{tabular}{@{}llllll} \br &&\centre{2}{ESSENCE (MLCS2k2)}
  &\centre{2}{ESSENCE (SALT)} \\ $\sigma_v$[km/s] &&
  Riess & Sandage & Riess & Sandage \\\mr
  300
  &acceleration (95\% CL) & \037 & \014 & \099 & \053 \\
  &acceleration (99\% CL) & \011 &\0\03 & \064 & \030 \\
  &deceleration (95\% CL) &\0\03 & \011 &\0\01 &\0\07 \\
  &deceleration (99\% CL) &\0\02 &\0\03 &\0\00 &\0\02 \\ \mr
  400
  &acceleration (95\% CL) & \037 & \014 & \096 & \053 \\
  &acceleration (99\% CL) & \011 &\0\03 & \064 & \030 \\
  &deceleration (95\% CL) &\0\03 &\0\07 &\0\01 &\0\07 \\
  &deceleration (99\% CL) &\0\02 &\0\03 &\0\00 &\0\02 \\ \mr
  500
  &acceleration (95\% CL) & \036 & \013 & \095 & \053 \\
  &acceleration (99\% CL) & \010 &\0\03 & \063 & \030 \\
  &deceleration (95\% CL) &\0\03 &\0\07 &\0\01 &\0\05 \\
  &deceleration (99\% CL) &\0\02 &\0\03 &\0\00 &\0\02 \\ \mr
  \multicolumn{2}{@{}l}{number of SNe} & 162 & 162 & 178 & 178 \\
\br
\end{tabular}
\end{indented}
\end{table}

\subsection{Averaging over SN data}

Until now, we have only made tests for single SNe. The
problem in combining these data is that the quantity $\Delta\mu$
depends on the redshift $z$. Thus the data from different redshifts do
not have the same mean value. Nevertheless it is possible to average
over $\Delta\mu_i$. Then the result of course depends on how many SNe
from a certain redshift are used to calculate this mean
value. Therefore, the average over all SNe of a set does not
characterize the function
$\Delta\mu(z)$. But still, when combined with its standard deviation,
the mean value can be an evidence for acceleration or
deceleration. Thus, we define
\begin{equation}
\overline{\Delta\mu} =
\frac{\sum_{i=1}^Ng_i\Delta\mu_i}{\sum_{i=1}^Ng_i} \;,
\end{equation}
where $g_i=1/\sigma_i^2$. The mean value is calculated in such a way
that data points with a small error are weighted more than those with
large errors. The standard deviation of the mean value is calculated
by
\begin{equation}
\sigma_{\overline{\Delta\mu}} =
\left[\frac{\sum_{i=1}^Ng_i\left(\Delta\mu_i -
    \overline{\Delta\mu}\right)^2}{(N-1)\sum_{i=1}^Ng_i}\right]^{\frac{1}{2}}
\;.
\end{equation}

We start with averaging $\Delta\mu_i$ over redshift bins of width
0.2. For all data sets the averaged value of $\Delta\mu$ increases
with redshift (see figure \ref{zbinfig}) which could be expected for
an accelerated expansion. The curve for a flat $\Lambda$CDM with
$\Omega_m=0.3$ has its maximum
at $z=1.2$ and then decreases again. Unfortunately, there are not
enough SNe at high redshifts to state a possible decrease of
$\Delta\mu$ at some point. The differences in the data points of the
two ESSENCE sets again seem to be too large. MLCS2k2 gives smaller
values than the SALT fitter for all redshift bins. As $\Delta\mu(z)$
should be close to zero at low redshifts in a
homogeneous and isotropic universe, the MLCS2k2 data look more
consistent with our assumptions in the Riess calibration, whereas SALT
gives the better results
in the Sandage calibration. The Gold sample has sensible values in
both calibrations.
$\overline{\Delta\mu}$ devided by the error
$\sigma_{\overline{\Delta\mu}}$ gives the evidence for acceleration in
each redshift bin. As can be seen in table \ref{zbin}, the strongest
evidence is given in redshifts between 0.4 and 0.8.

\begin{figure}
\centering \subfloat[Riess calibration]{
  \includegraphics*[width=7.5cm]{zbin_riess_3_a.eps}} \hfill
\subfloat[Sandage calibration]{
  \includegraphics*[width=7.5cm]{zbin_sand_3_b.eps}}
\caption{Magnitude $\Delta\mu$ averaged over redshift bins of width 0.2 for
  different data sets.}
\label{zbinfig}
\end{figure}

\begin{table}
\begin{indented}\lineup
\caption{\label{zbin} Statistical evidence
  $\overline{\Delta\mu}/\sigma_{\overline{\Delta\mu}}$
  within the given redshift range for
  a flat and an open universe.}
\item[]\begin{tabular}{@{}lllllll} \br &\centre{2}{Gold 2007 (MLCS2k2)}
  &\centre{2}{ESSENCE (MLCS2k2)} &\centre{2}{ESSENCE (SALT)} \\
z & Riess & Sandage & Riess & Sandage & Riess & Sandage \\ \mr
flat universe &&&&&&\\
0.0 -- 0.2 & 2.1 & 0.4 & 1.6 &\-2.8 & \05.7 & 0.8 \\
0.2 -- 0.4 & 4.2 & 2.9 & 3.9 &  0.7 & \07.3 & 4.3 \\
0.4 -- 0.6 & 9.8 & 7.6 & 7.1 &  3.1 &  12.1 & 7.7 \\
0.6 -- 0.8 & 5.4 & 4.0 & 7.7 &  3.7 & \08.5 & 4.8  \\
0.8 -- 1.0 & 4.8 & 3.5 & 6.8 &  4.1 & \06.0 & 4.2 \\
1.0 -- 1.2 & 2.8 & 2.2 &&&& \\
1.2 -- 1.4 & 2.3 & 1.8 &&&& \\ \mr
open universe &&&&&&\\
0.0 -- 0.2 &  2.1 &  0.4 & 1.5 &\-2.8 &\05.7 & 0.7 \\
0.2 -- 0.4 &  3.5 &  2.2 & 3.2 &\-0.0 &\06.6 & 3.6 \\
0.4 -- 0.6 &  7.4 &  5.2 & 5.5 &  1.4 & 10.2 & 5.8 \\
0.6 -- 0.8 &  2.8 &  1.4 & 4.9 &  0.9 &\05.9 & 2.2 \\
0.8 -- 1.0 &  1.4 &  0.2 & 3.9 &  1.2 &\04.0 & 2.2 \\
1.0 -- 1.2 &  0.9 &  0.4 &&&&\\
1.2 -- 1.4 &\-0.2 &\-0.7\\
\br
\end{tabular}
\end{indented}
\end{table}

Next we average over all SNe of each data set with a redshift $z\ge0.2$.
We discard SNe with a smaller redshift for the following reasons: (a)
$\Delta\mu(z)$ is expected to be relatively close to zero for small
redshifts. Therefore, nearby SNe do not contribute to the evidence for
acceleration. (b) Considering only the nearby universe, local effects can
modify the results for the distance modulus as the cosmological
principal is not valid on small scales. (c)
Another disadvantage of nearby SNe is that they were
observed with many different telescopes and thus the obtained
systematic error is potentially higher.
Table \ref{meantab2} shows the mean values
$\overline{\Delta\mu}$ and their standard deviations
$\sigma_{\overline{\Delta\mu}}$
evaluated by using only SN data with $z\ge0.2$.
$\overline{\Delta\mu}$ is positive for all data sets.
$\overline{\Delta\mu}$ divided by $\sigma_{\overline{\Delta\mu}}$
indicates the confidence level at which an accelerated expansion can
be stated. The weakest evidence for acceleration is given for ESSENCE
(MLCS2k2) in the Sandage calibration. Here the mean value lies $5.2
\sigma$ above 0, i.e. above the value for a universe that neither
accelerates nor decelerates. In the other cases the confidence level is even
larger, up to $17.0\sigma$ for ESSENCE (SALT) in the Riess
calibration.

\begin{table}
\begin{indented}\lineup
\caption{\label{meantab2} Mean values and standard deviations of
  $\Delta\mu$ obtained by using only SNe with $z\ge 0.2$ for a flat
  universe.}
\item[]\begin{tabular}{@{}lllllll} \br &\centre{2}{Gold 2007 (MLCS2k2)}
  &\centre{2}{ESSENCE (MLCS2k2)} &\centre{2}{ESSENCE (SALT)} \\
& Riess & Sandage & Riess & Sandage & Riess & Sandage \\ \mr
  $\overline{z}$ &
\00.63 & 0.63 & \00.54 & 0.54 & \00.51 & \00.51 \\
$\overline{\Delta\mu}$ &
\00.2196 & 0.1655 & \00.2398 & 0.1056 & \00.3457 & \00.2115 \\
$\sigma_{\overline{\Delta\mu}}$ &
\00.0167 & 0.0167 & \00.0201 & 0.0201 & \00.0203 & \00.0203  \\
$\overline{\Delta\mu}/\sigma_{\overline{\Delta\mu}}$ &
13.1 & 9.9 & 11.9 & 5.2 & 17.0 & 10.4 \\ \br
\end{tabular}
\end{indented}
\end{table}

\section{Open and closed universe}\label{openclosed}
Although there are good reasons to believe that the universe is flat
we give up this assumption in the following section. In an open universe
the luminosity distance of a universe that neither accelerates nor
decelerates is given by
\begin{equation}
d_{\mbox{\scriptsize L}}(z) = \frac{1+z}{H_0\sqrt{\Omega_k}}
\sinh\left( \sqrt{\Omega_k}\ln(1+z)\right)
\end{equation}
and thus depends on the density parameter of the scalar curvature
$\Omega_k$. $d_{\mbox{\scriptsize L}}$ increases with increasing
$\Omega_k$ and thus the evidence for acceleration becomes weaker. As
we are interested in the lower limit of this evidence we have to take
the highest possible value for the scalar curvature, i.e. $\Omega_k=1$,
which corresponds to an empty universe. (Here we allow ourself to make
use of the Einstein equation.)
Then equation \eref{deltamui} for $\Delta\mu_i$ changes to
\begin{equation}
\Delta\mu_i = \mu_i - \mu(q=0) = \mu_i - 5\log\left[
  \frac{1}{H_0}(1+z_i)\sinh[\ln(1+z_i)] \right] -25 \,.
\end{equation}

\begin{table}
\begin{indented}\lineup
\caption{\label{opentab} Statistical evidence
  $\overline{\Delta\mu}/\sigma_{\overline{\Delta\mu}}$
  for an open universe (obtained by using SNe within the redshift
  range $0.2\le z <1.2$), a flat and a closed universe ($0.2\le z$).}
\item[]\begin{tabular}{@{}lllllll} \br &\centre{2}{Gold 2007 (MLCS2k2)}
  &\centre{2}{ESSENCE (MLCS2k2)} &\centre{2}{ESSENCE (SALT)} \\
& Riess & Sandage & Riess & Sandage & Riess & Sandage \\ \mr
open universe &
\08.0 & \04.9 & \08.8 & 1.8 & 13.8 & \07.2 \\
flat universe &
 13.1 & \09.9 & 11.9 & 5.2 & 17.0 & 10.4 \\
closed universe &
 13.1 & \09.9 & 11.9 & 5.2 & 17.0 & 10.4 \\\br
\end{tabular}
\end{indented}
\end{table}

The evidence for accelerated expansion is then calculated in the same
way as for a flat universe. The result obtained by averaging over
redshift bins is shown in table \ref{zbin}.
For the overall average we only used SNe between
redshift 0.2 and 1.2 because including higher redshifts would weaken
the evidence. This is due to the fact that the values
of $\Delta\mu_i$ become negative when the phase of deceleration within
the redshift range over which is integrated is large enough. The
result is shown in table \ref{opentab}. As could be expected, the
evidence is now much weaker than for a flat universe. But we still
find a hint of acceleration at 1.8$\sigma$.

For a closed universe we have a different situation: Here
$d_{\mbox{\scriptsize L}}$ decreases with increasing spatial
curvature. Thus the lower limit of the evidence for acceleration is
given in the case of the lowest possible curvature which corresponds
to a flat universe.

\section{Conclusion}\label{concl}
We tested the cosmic expansion without specifying any density
parameters or parameterizing kinematical quantities. This was not
possible without assuming a certain calibration of $M$ and
$H_0$. Therefore, we considered two very different calibrations. We
also considered two different data sets and two different light curve
fitters and varied the peculiar velocity dispersion. Using single SNe
for the test has already given a
clear indication of acceleration in a flat universe although a few SNe
strongly favour deceleration. We find large systematic effects already
at the level of individual SNe.

Similar analyses have already been done by other groups \cite{santos,
  gong}. But note that the work of Santos et al.~\cite{santos}
exhibits two major shortcomings. First, they did not calibrate the SN
data consistently. They took a certain value for $H_0$ but kept the
arbitrary value of the absolute magnitude $M$ given in the set which
led to distance moduli that are too high. Thus, they concluded
erroneously that there was a recent phase of super-acceleration. The
second problem is that they did not realize that due to the integration
over redshift this method is not suitable to determine the transition
redshift $z_{\mbox{\scriptsize t}}$ between deceleration and
acceleration. Thus, their value of $z_{\mbox{\scriptsize
t}}$ does not hold. The shortcoming of the work of Gong et
al.~\cite{gong} is that they have not applied any statistics at all.

Instead of only considering single SNe, we obtained a more significant
result by averaging over the SN
data of each set. Here it is justified to discard nearby SNe in order
to decrease systematics from using different telescopes or from local
effects. We argue that we reject the null hypothesis of no
acceleration for a spatially flat, homogeneous and isotropic universe
at high confidence ($>5\sigma$).
However, the results show large differences for each data set, calibration
and light-curve fitter. E.g. changing the calibration from Sandage to
Riess calibration in a flat universe using ESSENCE(MLCS2k2) increases
the evidence for
acceleration from 5.2$\sigma$ to 11.9$\sigma$. For the two different
light-curve fitters applied to the ESSENCE SNe using the Riess
calibration, we get values of 11.9$\sigma$ and 17.0$\sigma$.

Wood-Vasey et al.~argue that the differences
due to different light curve fitters are not important as they
disappear when marginalization is applied \cite{essence}. This is
true if the data are only used to determine the parameter values of a
certain cosmological model as in this case no calibration of $M$ and
$H_0$ is needed. But nevertheless, the fitters should give the same
result for a given absolute magnitude $M$. If this is not the case,
a systematic error in the determination of the apparent magnitude
$m$ is introduced by at least one of the fitters. Systematics due to
different fitters have also been described in \cite{conley} and those
within the Gold sample in \cite{jassal1,jassal2,nesseris}.

The different results obtained by the two calibrations are very
surprising. Our test only depends on the reduced distance modulus
$\mathcal{M}=M-5\log H_0+25$ and not on the absolute magnitude or the
Hubble constant individually. (Note that this is the quantity which is
marginalized when fitting cosmological parameters.)
Starting from different calibrations $M_1$ and $M_2$ leads to
different values of $H_0$,
determined by observation of SNe, in such a way that $\mathcal{M}$
should in principle be the same for both values of $M$.
Thus our test should not depend on the calibration. However, the fact
that the calibration changes our result significantly can be explained
as follows: $H_0$ is determined by observing SNe. Riess et al.~and
Sandage et al.~use different sets of SNe and different fitting methods
for this determination. The systematic errors due to different sets and
fitters then of course
also influence the result for the Hubble constant leading to different
values $\mathcal{M}_1$ and $\mathcal{M}_2$.

For a conservative conclusion, we need to take the set that gives the
weakest evidence for acceleration, namely ESSENCE (MLCS2k2) in the
Sandage calibration. Using this set, accelerated expansion can be stated
at 5$\sigma$ if we assume a spatially flat or closed universe. In an
open universe the evidence is much weaker, namely 1.8$\sigma$. These
results only hold if the correct analysis of SNe is somewhere in the
range of the cases we considered here. But as we observe enormous
systematic effects in our test, we cannot be sure of this assumption.
Thus, it is a major issue to better
understand how SNe have to be observed and analyzed.

Remember that results of this work are only valid if the universe is
homogeneous and isotropic. The situation changes dramatically if we
give up those assumptions. There exist enormously large structures in
the universe, an example is the Sloan Great Wall with an extension of
$\sim$400 Mpc \cite{gott}. Besides, there are also big voids and
superclusters on a 100 Mpc scale. This roughness of the universe could
spoil the validity of equation (1).
Indeed it is possible to construct
inhomogeneous models that can describe the observational data without
the need of an average acceleration \cite{celerier,havard,enqvist,ishak}.

Some observational evidence for inhomogeneity and anisotropy in SN
Hubble diagrams has recently been presented by \cite{dyer,weinhorst},
probably due to large scale bulk motion and perhaps systematic
effects. A non-trivial scale dependence of the Hubble rate due to the
so-called cosmological backreaction
\cite{buchert,schwarz,rasanen,kolb,vanderveld,wiltshire}
has been shown in \cite{li}. This effect of averaging can also mimic
curvature effects
\cite{li1} and thus strongly influence the reconstruction of the dark
energy equation of state, as has been shown in \cite{bassett}.

Thus we think a major task must be to establish the acceleration of
the universe independently of the assumption of strict homogeneity and
isotropy.

\ack
We thank Dragan Huterer, Thanu Padmanabhan, Aleksandar Raki\'{c}, Adam
Riess and Bastian Weinhorst for discussions, comments and references
to the literature. This work is supported by the DFG under grant GRK 881.

\section*{References}