\subsubsection*{\bibname}}
\usepackage{graphicx}
\usepackage{times}
\usepackage{ifpdf}
\usepackage{amsthm}
\usepackage{algorithm}
\usepackage[algo2e, ruled, vlined]{algorithm2e}
\newcommand{\ignore}[1]{}
\newcommand{\notinproc}[1]{#1}
\newcommand{\onlyinproc}[1]{}

\def\textsf{E}{\textsf{E}}
\def\textsf{Exp}{\textsf{Exp}}
\def\textsf{Lap}{\textsf{Lap}}
\def\textsf{Var}{\textsf{Var}}
\def\textsf{Bias}{\textsf{Bias}}
\def\textsf{MSE}{\textsf{MSE}}
\def\textsf{NRMSE}{\textsf{NRMSE}}
\def\textsf{CDF}{\textsf{CDF}}
\def\textsf{PDF}{\textsf{PDF}}
\def\mathop{\mathrm{key}}{\textsf{key}}
\def\textsf{val}{\textsf{val}}
\def\boldsymbol{\nu}{\boldsymbol{\nu}}
\def\textsf{Zipf}{\textsf{Zipf}}
\def\mathop{\mathrm{poly}}{\mathop{\mathrm{poly}}}
\def\mathop{\mathrm{polylog}}{\mathop{\mathrm{polylog}}}
\def\mathop{\mathrm{key}}{\mathop{\mathrm{key}}}
\newtheorem{thm}{Theorem}[section]
\newtheorem{theorem}[thm]{Theorem}
\newtheorem{cor}[thm]{Corollary}
\newtheorem{lemma}[thm]{Lemma}
\newtheorem{conjecture}[thm]{Conjecture}
\newtheorem{prop}[thm]{Proposition}
\newtheorem{claim}[thm]{Claim}
\newtheorem{goal}[thm]{Goal}
\newtheorem{infprob}[thm]{Informal\,\,Problem}
\newtheorem{problem}[thm]{Problem}
\newtheorem{fact}[thm]{Fact}
\newtheorem{example}[thm]{Example}
\newtheorem{definition}[thm]{Definition}
\newtheorem{rem}[thm]{Remark}
\newtheorem{corollary}[thm]{ Corollary}

\usepackage{xcolor}
\newcommand\mycommfont[1]{{\scriptsize\ttfamily\textcolor{blue}{#1}}}
\SetCommentSty{mycommfont}

\renewcommand{\epsilon}{\varepsilon}

\title{Differentially Private Weighted Sampling}
\author{Edith Cohen \\
Google Research \\ Tel Aviv University
\and Ofir Geri  \\
Stanford University
\and Tamas Sarlos \\
Google Research
\and Uri Stemmer \\
Ben-Gurion University \\ Google Research
}

\date{}
\begin{document}
 \maketitle

\begin{abstract}
    Common datasets have the form of
    {\em elements} with {\em keys} (e.g., transactions and products)
    and the goal is to perform analytics on the aggregated form of {\em key} and {\em frequency} pairs.  A weighted sample of keys by (a function of) frequency is a highly versatile summary that provides a sparse set of representative keys and supports approximate evaluations of query statistics.

    We propose {\em private weighted sampling} (PWS): A method that sanitizes a weighted sample as to ensure element-level differential privacy, while retaining its utility to the maximum extent possible.

    PWS maximizes the reporting probabilities of keys and estimation quality of a broad family of statistics. PWS improves over the state of the art even for the well-studied special case of {\em private histograms}, when no sampling is performed.  We empirically observe significant performance gains of 20\%-300\% increase in key reporting for common Zipfian frequency distributions and accurate estimation with $\times 2$-$ 8$ lower frequencies.
    PWS is applied as a post-processing of a non-private sample, without requiring the original data.  Therefore, it can be a seamless addition to existing implementations, such as those optimizes for distributed or streamed data.

    We believe that due to practicality and performance, PWS may become a method of choice in applications where privacy is desired.

    \end{abstract}

\section{Introduction}

{\em Weighted sampling schemes} are often used to obtain versatile summaries of large datasets. The sample constitutes a representation of the data and also facilitates efficient estimation of many statistics. Motivated by the increasing awareness and demand for data privacy, in this work we construct {\em privacy preserving} weighted sampling schemes. The privacy notion that we work with is that of {\em differential privacy}~\cite{DMNS06}, a strong privacy notion that is considered by many researchers to be a gold-standard for privacy preserving data analysis.

Before describing our new results, we define our setting more precisely. Consider an input dataset containing $m$ elements, where each element contains a key $x$ from some domain $\mathcal{X}$. For every key $x\in \mathcal{X}$ we write $w_x$ to denote the multiplicity of $x$ in the input dataset. (We also refer to $w_x$ as the {\em frequency} of $x$ in the data.) With this notation, it is convenient to represent the input dataset in its {\em aggregated} form $D=\{(x,w_x) \}$   containing pairs of a key and its frequency $w_x\geq1$ in the data.
Examples of such datasets are plentiful: Keys are search query strings and elements are search requests, keys are products and elements are transactions for the products, keys are locations and elements are visits by individuals, or keys are training examples and elements are activities that generate them. We aim here to protect the privacy of data elements. These example datasets tend to be very sparse, where the number of distinct keys in the data is much smaller than the size $|\mathcal{X}|$ of the domain. Yet, the number of distinct keys can be very large and samples serve as small summaries that can be efficiently stored, computed, and transmitted. We therefore aim for our private sample to retain this property and in particular only include keys that are in the dataset.

The (non-private) sampling schemes we consider are specified by a (non-decreasing) sequence
$(q_i)_{i\geq 0}$ of probabilities $q_i\in [0,1]$, where $q_0 := 0$.
Such a sampling scheme takes an
input dataset $D=\{(x,w_x) \}$ and returns a {\em sample} $S\subseteq D$, where each pair $(x,w_x)$ is included in $S$ independently, with probability $q_{w_x}$. Loosely speaking, given a (non-private) sampling scheme $A$, we aim in this paper to design a {\em privacy preserving} variant of $A$ with the goal of preserving its ``utility'' to the extent possible under privacy constraints. We remark that an immediate consequence of the definition of differential privacy is that keys $x\in \mathcal{X}$ with very low frequencies cannot be included in the private sample $S$ (except with very small probability). On the other hand, keys with high frequencies can be included with probability (close to) $1$.
Private sampling schemes can therefore retain more utility when the dataset has many keys with higher frequencies or for tasks that are less sensitive to low frequency keys.

\begin{infprob}\label{infprob}
Given a (non-private) sampling scheme $A$, specified by a sampling function $q$, design a {\em private} sampling scheme that takes a dataset $D=\{(x,w_x)\}$
and outputs a ``sanitized'' sample $S^* =\{(x,w^*_x)\}$. Informally, the goals are:
\begin{enumerate}
    \item Each pair $(x,w_x)\in D$ is sampled with probability ``as close as possible'' to the non-private sampling probability $q_{w_x}$.
    \item The sanitized sample $S^*$

    provides utility that is ``as close as possible'' to that of a corresponding non-private sample $S$.  In our constructions, the sanitized frequencies $w^*_x$ would be random variables from which we can estimate ordinal and linear statistics with (functions of) the frequency $w_x$.

\end{enumerate}
\end{infprob}

Informal Problem~\ref{infprob} generalizes one of the most basic tasks in the literature of differential privacy -- {\em privately computing histograms}. Informally, algorithms for private histograms take a dataset $D=\{(x,w_x)\}$ as input, and return, in a differentially private manner, a ``sanitized'' dataset $D^* =\{(x,w^*_x)\}$.
It is often required that the output $D^*$ is {\em sparse}, in the sense that if $w_x=0$ then $w^*_x=0$. Commonly, we seek to minimize the expected or maximum error of estimators applied to $\boldsymbol{w}^*$ of statistics on  $\boldsymbol{w}$. One well-studied objective is to minimize $\max_{x\in X} \mid w_x-w^*_x|$. The work on private histograms dates all the way back to the paper that introduced differential privacy~\cite{DMNS06}, and it has received a lot of attention since then, e.g.,~\cite{KorolovaKMN09,HardtT10,BeimelBKN14,BeimelNS16,BunS16,BunNS19,BalcerV18,BunDRS18}. Observe that the private histogram problem is a special case of Informal Problem~\ref{infprob}, where $q\equiv1$.

At first glance, one might try to solve Informal Problem~\ref{infprob} by a reduction to the private histogram problem. Specifically, we consider the baseline where the data is first ``sanitized'' using an algorithm for private histograms, and then a (non-private) weighted sampling algorithm is applied to the sanitized data (treating the sanitized frequencies as actual frequencies).  This framework, of first sanitizing the data and then sampling it was also considered in \cite{CormodePST:ICDT2012}. We show that this baseline is sub-optimal, and improve upon it in several axes.

\subsection{Our Contributions}

Our proposed framework, {\em Private Weighted Sampling} (PWS), takes as input a non-private weighted sample $S$ that is produced  by a (non-private) weighted sampling scheme.
We apply a ``sanitizer'' to the sample  $S$ to obtain a respective privacy-preserving sample $S^*$.  Our proposed solution has the following advantages.

\paragraph{Practicality.}
The private version is generated from the sample $S$ as a post-processing step without the need to revisit the original dataset, which might be massive or unavailable. This means that we can augment existing implementations of non-private sampling schemes and retain their scalability and efficiency. This is particularly appealing for sampling schemes designed for massive distributed or streamed data that use small sketches and avoid a resource-heavy aggregation of the data~\cite{GM:sigmod98,EV:ATAP02,flowsketch:JCSS2014,AndoniKO:FOCS11,CCD:sigmetrics12,freqCapfill:TALG2018,JayaramW:Focs2018,CohenGeri:NeurIPS2019,CohenGeriPagh:ICML2020}. Our code is available at {\tiny \url{github.com/google-research/google-research/tree/master/private_sampling}}.

\paragraph{Benefits of end-to-end privacy analysis}
PWS achieves better utility compared to the baseline of first sanitizing the data and then sampling.
In spirit, our gains follow from a well-known result in the literature of differential privacy stating that applying a differentially private algorithm on a random sample from the original data has the effect of boosting the privacy guarantees of the algorithm \cite{KamalikaMishra:Crypto2006,KasiviswanathanLNRS11,BunNSV15}.
Our solution is derived from a precise {\em end-to-end} formulation of the privacy constraints that account for the benefits of the random sampling in our privacy analysis.

\paragraph{Optimal reporting probabilities.}
PWS is optimal in that it maximizes
the probability that each key $x$ is included in the private sample. The private reporting probability of a key $x$ depends on the privacy parameters, frequency, and sampling rate and is at most the non-private sampling probability $q_{w_x}$. The derivation is provided in Section~\ref{basic:sec}.

\paragraph{Estimation of linear statistics.}
Linear statistics according to a function of frequency have the form:
\onlyinproc{
  $\sum_x L(x) g(w_x)$,
}
\notinproc{
  \begin{equation}
  s := \sum_x L(x) g(w_x)\enspace ,
  \end{equation}
}
where $g(w_x) \geq 0$ is a non-decreasing function of frequency with $g(0):=0$.
The most common use case is when $L(x)$ is a predicate and $g(w) := w$ and the statistics is the sum of frequencies of keys that satisfy the selection $L$.
Our PWS sanitizer in Section~\ref{refined:sec} maintains optimal reporting probabilities and provides private information on frequencies of keys.
We show that generally differential privacy does not allow for unbiased estimators for statistics without significant increase in variance. We propose biased but nonnegative and low-variance estimators.
\paragraph{Estimation of ordinal statistics.}
Ordinal statistics, such as (approximate) quantiles and top-$k$ sets, are derived from the order of keys that is induced by their frequencies. This order can be approximated by the order induced by PWS sanitized frequencies.  We show that PWS is optimal, over all DP sanitization schemes, for a broad class of ordinal statistics.  In particular, PWS maximizes the probability that {\em any} pair is concordant and maximizes the expected Kendall-$\tau$ rank correlation between the order induced by sanitized and true frequencies.

\paragraph{Improvement over prior baselines.}
We show analytically and empirically in Section~\ref{experiments:sec} that we obtain orders of magnitude increase in reporting probability in low-frequency regimes. For estimation tasks, both PWS and prior schemes have lower error for higher frequencies but
PWS obtains higher accuracy for frequencies that are $\times 2$-$8$ lower than prior schemes.  This is particularly helpful for datasets/selections with many mid-low frequency keys.

\paragraph{Improvement for private histograms.}
As an important special case of our results, we improve upon the state-of-the-art constructions for private (sparse) histograms~\cite{KorolovaKMN09,BunNS19}.
These existing constructions obtain privacy properties by adding Laplace or Gaussian noise to the frequencies of the keys whereas we directly formulate and solve elementary constraints.
Let $\pi^*_i$ denote the PWS reporting probability of a key with frequency $i$, when applied to the special case of private histograms. Let $\phi_i$ denote the reporting probability of the state-of-the-art solution for private (sparse) histograms of~\cite{KorolovaKMN09,BunNS19}. Clearly $\pi^*_i$ is always at least $\phi_i$. We show that in low-frequency regimes we have $\pi^*_i/\phi_i \approx 2 i$.  Similarly for estimation tasks, PWS provides more accurate estimates in these regimes.
Qualitatively, PWS and private histograms have high reporting probabilities and low estimation error for high frequencies.  But PWS significantly improves on
low to medium frequencies, which is important for distributions with long tails.  We empirically show gains of 20\%-300\% in overall key reporting for Zipf-distributed frequencies.
As private histograms are one of the most important building blocks in the literature of differential privacy, we believe that our improvement is significant (both in theory and in practice).

\onlyinproc{Due to space limitations, we refer the reader to \cite{PWS:arxiv2020} for proofs and further details.}

\section{Related Work}

  The suboptimality of the Laplace mechanism for anonymization was noted by \cite{GhoshRS:sicomp2012}.  In our language, Ghosh et al.\ studied the non-sparse case, where all values, including $0$ values, can be reported with added noise. They did not consider sampling, and studied  pure differential privacy.  Instead of Laplace noise, they propose the use of a symmetric Geometric distribution and establish it is optimal for certain estimation tasks.  This can be viewed as a special case of what we do in that our schemes converge to that when there is no sampling, we use pure differential privacy, and when frequencies are large (so the effect of the sparse case constraint dissipates).
  Ghosh et al.\ establish the optimality of unbiased estimators for some frequency statistics when loss is symmetric. We show that bias is necessary in the sparse case and propose estimators that control the bias and variance.

  Key reporting was formulated and studied as {\em differentially private set union} problem \cite{GopiGJSSY:ICML2020}. They studied it without sampling, in a more general user privacy setting, and proposed a truncated Laplace noise mechanism similar to~\cite{KorolovaKMN09,BunNS19}.

  Recent independent work by \cite{desfontainesVG:2020} derived the optimal scheme for key reporting for sparse private histograms, a special case of our solution when there is no sampling.
\section{Preliminaries}

  We consider data in the form of a set of elements $\mathcal{E}$, where each element
$e\in\mathcal{E}$ has a key $e.\mathop{\mathrm{key}} \in \mathcal{X}$.  The {\em frequency} of a key $x$,  $w_x := \left|\{ e\in\mathcal{E} \mid e.\mathop{\mathrm{key}}=x\}\right|$, is defined as the number of elements with $e.\mathop{\mathrm{key}} = x$.  The {\em aggregated form} of the data, known in the DP literature as its {\em histogram}, is the set of key and frequency pairs $\{(x,w_x)\}$.  We use the vector notation $\boldsymbol{w}$ for the aggregated form.
We will use $m:= |\mathcal{E}|$ for the number of elements and $n$ for the number of distinct keys in the data.

\subsection{Weighted Sampling} \label{weightedsamplingprelim:sec}

We consider a very general form of without-replacement sampling schemes.  Each scheme is  specified by non-decreasing probabilities $(q_i)_{i\geq 1}$. The probability that a key is sampled depends on its frequency -- a key with frequency $i$ is sampled independently with probability $q_i$. Our proposed methods apply with any non-decreasing $(q_i)$.

{\em Threshold sampling} is a popular class of weighted sampling schemes.  We review it for concreteness and motivation and use it in our empirical evaluation of PWS. A threshold sampling scheme (see Algorithm~\ref{alg:threshold}) is specified by  $(\mathcal{D},f,\tau)$, where $\mathcal{D}$ is a distribution, $f$ is a function of frequency, and $\tau$ is a numeric threshold value that specifies the sampling rate.
For each key we draw i.i.d.\ $u_x \sim \mathcal{D}$. The two common choices are
$\mathcal{D} = \textsf{Exp}[1]$ for a probability proportional to size without replacement (ppswor) sample~\cite{Rosen1972:successive}  and $\mathcal{D} = U[0,1]$ for a Poisson Probability Proportional to Size (PPS) sample~\cite{Ohlsson_SPS:1990,Ohlsson_SPS:1998,DLT:jacm07}.
A key $x$ is included in the sample if $u_x \leq  \tau f(w_x)$.
The probability that a key with frequency $i$ is sampled is
\begin{equation} \label{qdef:eq}
    q_i := \Pr_{u\sim \mathcal{D}}[u < f(i)\tau]\enspace .
\end{equation}
\ignore{
A weighted sample can be used to estimate linear statistics of the form
 \[
 s := \sum_x L(x) g(w_x)\enspace .
 \]
 The per-key inverse-probability estimator~\cite{HT52} of $g(w_x)$ is defined as follows:
\begin{equation} \label{invprob:eq}
      a_{w_x} := \begin{cases}
 \frac{g(w_x)}{q_{w_x}} & x\text{ is included in the sample}\\
 0 & \text{otherwise}
 \end{cases}\enspace .
\end{equation}
   Our estimate of a query statistics will be the sum
 \[
 \hat{s} := \sum_{(x,w_x)\in S}  L(x) a_{w_x}\enspace .
 \]
 The estimate is unbiased, i.e., $\textsf{E}[\hat{s}] = s$.
 }
  Threshold sampling is related to bottom-$k$ (order) sampling~\cite{Rosen1997a,Ohlsson_SPS:1990,DLT:jacm07,bottomk07:ds,bottomk:VLDB2008}  but instead of specifying the sample size $k$ we specify an inclusion threshold $\tau$.
Ppswor is equivalent to drawing keys sequentially with probability proportional to $f(w_x)$.  The bottom-$k$ version stops after $k$ keys and the threshold version has a stopping rule that corresponds to the threshold. The bottom-$k$ version of Poisson PPS sampling is known as sequential Poisson or Priority sampling.

\begin{algorithm2e}\caption{Threshold Sampling}\label{alg:threshold}
{\scriptsize
\DontPrintSemicolon
\tcp{{\bf Threshold Sampler:}}
\KwIn{Dataset $\boldsymbol{w}$ of key frequency pairs $(x,w_x)$; distribution $\mathcal{D}$, function $f$, threshold $\tau$}
\KwOut{Sample $S$  of key-frequency pairs from $\boldsymbol{w}$}
\Begin{
$S \gets \emptyset$\;
\ForEach{$(x,w_x) \in \boldsymbol{w}$}
{
Draw independent $u_x\sim \mathcal{D}$\;
\If{$u_x < f(w_x)\tau$}{$S \gets S\cup\{(x,w_x)\}$
}
}
\Return{S}
}
}
\end{algorithm2e}

Since PWS applies a sanitizer to a sample, it inherits the efficiency of the base sampling scheme. Threshold sampling (via the respective bottom-$k$ schemes) can be implemented efficiently using small sketches
(of size expected sample size) on aggregated data that can be distributed or streamed \cite{DLT:jacm07,Rosen1997a,Ohlsson_SPS:1998,bottomk07:ds}.  On unaggregated datasets, it can be implemented
using small sketches  for some functions of frequency including the moments $f(w)=w^p$ for $p\in [0,2]$ \cite{CCD:sigmetrics12,freqCapfill:TALG2018,CohenGeri:NeurIPS2019,CohenPW:NeurIPS2020}.

Our methods apply with a fixed threshold $\tau$. But the treatment extends to when the threshold is privately determined from the data. If we have a private approximation of the total count $\|f(\boldsymbol{w})\|_1 := \sum_x f(w_x)$  we can set $\tau \approx  k/\|f(\boldsymbol{w})\|_1$.  This provides (from the non-private sample that corresponds to the threshold) estimates with additive error $\|f(\boldsymbol{w})\|_1/\sqrt{k}$ for statistics with function of frequency $g=f$ and when $L$ is a predicate.

\subsection{Differential Privacy}

The privacy requirement we consider is
{\em element-level} differential privacy.    Two datasets with aggregated forms
$\boldsymbol{w}$ and $\boldsymbol{w}'$ are neighbors if $\| \boldsymbol{w}-\boldsymbol{w}' \|_1 = 1$, that is, the frequencies of all keys but one are the same and the difference is at most $1$ for that one key.  The privacy requirements are specified using two parameters $\epsilon, \delta \geq 0$.

\begin{definition}[\cite{DMNS06}]
A mechanism $M$ is $(\varepsilon,\delta)$-differentially private if for any two neighboring inputs $\boldsymbol{w}$, $\boldsymbol{w}'$ and set of potential outputs $T$,
\begin{equation} \label{DP:eq}
\Pr[M(\boldsymbol{w}) \in T] \leq e^\varepsilon \Pr[M(\boldsymbol{w}') \in T] + \delta \enspace .
\end{equation}
\end{definition}

\begin{algorithm2e}\caption{Private Weighted Samples}\label{alg:sanitizer}
{\scriptsize
\DontPrintSemicolon
\tcp{{\bf Sanitized Keys:}}
\KwIn{$(\epsilon,\delta)$, weighted sample $S$, taken with non-decreasing probabilities $(q_i)_{i\geq 1}$}
\KwOut{Private sample of keys $S^*$}
Compute $(p_i)_{i\geq 1}$
\tcp*{Reporting probabilities per freq.}
\Begin(\tcp*[h]{Sanitize using scheme}){
$S^* \gets \emptyset$\;
\ForEach{$(x,w_x) \in S$}
{
With probability $p_{w_x}$,
$S^* \gets S^* \cup \{x\}$
}
\Return{$S^*$}
}

\tcp{{\bf Sanitized keys and frequencies:}}
\KwIn{$(\epsilon,\delta)$, weighted sample $S$, taken with non-decreasing probabilities $(q_i)_{i\geq 1}$}
\KwOut{Sanitized sample $S^*$}
Compute probability vectors $(p_{i\bullet})_{i\geq 1}$
\tcp*{Reported values}
\Begin(\tcp*[h]{Sanitize using scheme}){
$S^* \gets \emptyset$\;
\ForEach{$(x,w_x) \in S$}
{
Draw $j \sim p_{w_x \bullet}$ \tcp*{By probability vector}
\If{$j>0$}{$S^* \gets S^* \cup \{(x,j)\}$}
}
\Return{$S^*$}
}

\tcp{{\bf Estimator:}}
\KwIn{Sanitized sample $S^*=\{(x,j_x)\}$, $\{\pi_{i,j}\}$ (where $\pi_{ij} := p_{ij} q_i$ , functions $g(i)$, $L(x)$}
\KwOut{Estimate of the linear statistics $\sum_x L(x) g(x)$}
\Begin{
 Compute $(a_j)_{j\geq 1}$ using $\{\pi_{ij}\}$ and $g(i)$ \tcp*{Per-key estimates for $g()$}
 \Return{$\sum_{(x,j_x)\in S^*} L(x) a_{j_x}$}
}
}
\end{algorithm2e}

\subsection{Private Weighted Samples}

Given a (non-private) weighted sample $S$ of the data in the form of key and frequency pairs and (a representation) of the sampling probabilities $(q_i)_{i\geq 1}$ that guided the sampling, our
goal is to release as much of $S$ as we can without violating element-level differential privacy.

We consider two utility objectives.  The basic objective, {\em sanitized keys}, is to maximize the reporting probabilities of keys in $S$. The private sample in this case is simply a subset of the keys in $S$. The refined objective is to facilitate estimates of linear frequency and order statistics. The private sample includes sanitized keys from $S$ together with information on their frequencies.
The formats of the sanitizers and estimators are provided as Algorithm~\ref{alg:sanitizer}.

\section{Sanitized Keys} \label{basic:sec}

\begin{algorithm2e}\caption{Compute $\pi_{i}$ for  Sanitizing Keys}\label{alg:reportkeys}
{\scriptsize
\DontPrintSemicolon
\KwIn{$(\epsilon,\delta)$, non-decreasing sampling probabilities $(q_i)_{i\geq 1}$, $\text{Max$\_$Frequency}$}
$\pi_0 \gets 0$\;
\ForEach{$i=1,\ldots,\text{Max$\_$Frequency}$}
{
    $\pi_{i} \gets \min\{q_{i}, e^\varepsilon \pi_{i-1} + \delta, 1+ e^{-\varepsilon}(\pi_{i-1}+\delta-1) \}$
    }
    \Return{$(\pi_{i})_{i=1}^{\text{Max$\_$Frequency}}$}
}
\end{algorithm2e}

A sanitizer $C$ uses a representation of the non-decreasing $(q_i)_{i\geq 1}$  and computes respective probabilities $(p_i)_{i\geq 1}$.  A non-private sample $S$ can then be sanitized by considering each pair $(x,w_x)\in S$ and reporting the key $x$ independently with probability $p_{w_x}$.

We find it convenient to express constraints on $(p_i)_{i\geq 1}$ in terms of
the {\em end-to-end} reporting probability of a key $x$ with frequency $i$ (probability that $x$ is sampled and then reported):
\[\pi_i := p_i q_i =\Pr[x\in C(A(\boldsymbol{w}))] \enspace .\]
Keys of frequency $0$ are not sampled or reported and we have $q_0=0$ and $\pi_0 := 0$.  The objective of maximizing $p_i$ corresponds to maximizing $\pi_i$.  We establish the following\notinproc{ (The proof is provided in Appendix~\ref{proofkeybasic:sec})}:
\begin{lemma} \label{basicpi:lemma}
Consider weighted sampling scheme $A$ where keys are sampled independently according to a non-decreasing $(q_i)_{i\geq 1}$ and a key sanitizer $C$ (Algorithm~\ref{alg:sanitizer}) is applied to the sample.  Then the probabilities $p_i \gets \pi_i/q_i$, where $\pi_i$ are the iterates computed in Algorithm~\ref{alg:reportkeys}, are each at the maximum under the DP constraints for $C(A())$.  Moreover, $(\pi_i)_{i\geq 1}$ is  non-decreasing.
\end{lemma}

\subsection{Structure and Properties of ${\boldsymbol{(\pi_i)_{i\geq 1}}}$} \label{keybasic:sec}

The solution as computed in Algorithm~\ref{alg:reportkeys} applies with any non-decreasing $q_i$. We explore properties of the solution that allow us to compute and store it more efficiently and understand the reporting loss (reduction in reporting probabilities) that is due to the privacy requirement. \notinproc{Proofs are provided in Appendix~\ref{proofkeybasic:sec}.}

We provide closed-form expressions of the solution $\pi^*_i$ that corresponds to $q_i=1$ for all $i$ (aka the private histogram problem).  We will use the following definition of $L(\epsilon,\delta)$.  To simplify the presentation, we assume that $\epsilon$ and $\delta$ are such that $L$ is an integer (this assumption can be removed).
\begin{equation}\label{Ldef:eq}
 L(\epsilon,\delta) :=  \frac{1}{\varepsilon} \ln\left(
 \frac{e^\varepsilon -1 +2\delta}{\delta(e^\varepsilon +1)}

  \right) \approx \frac{1}{\varepsilon} \ln\left(\frac{\min\{1,\varepsilon/2\}}{\delta} \right)
\end{equation}
\begin{lemma} \label{privatehist:lemma}
When $q_i=1$ for all $i$, the sequence $(\pi_i)_{i\geq 1}$ computed by Algorithm~\ref{alg:reportkeys} has the form:
\begin{equation} \label{pistar:eq}
    \pi^*_i = \left\{ \begin{array}{ll}
     \delta \frac{e^{\varepsilon i}-1 }{e^\varepsilon -1} &\; \text{\small $i\leq L+1$  }\\
     1- \delta  \frac{e^{\varepsilon (2L+2-i)} -1}{e^{\varepsilon} -1} &\; \text{\small $L+1 \leq i \leq 2L+1$  }\\
    1 &\; \text{\small $i \geq 2L+2$  }
    \end{array}\right.
\end{equation}

\end{lemma}

For the general case where the $q_i$'s can be smaller than 1, we bound the number of frequency values for which $\pi_i < q_i$. On these frequencies, the private reporting probability is strictly lower than that of the original non-private sample, and hence there is reporting loss due to privacy.
\begin{lemma} \label{totaltwothree:lemma}
There are at most $2L(\varepsilon,\delta)+1$ values $i$ such that $\pi_i < q_i$, where
$L$ is as defined in \eqref{Ldef:eq}.
\end{lemma}

 We now consider the structure of the solution for threshold sampling.  The solution has a particularly simple form that can be efficiently computed and represented.
 \begin{lemma} \label{niceq:lemma}
When the sampling probabilities $(q_i)_{i\geq 1}$ are those of threshold ppswor sampling with $f(i)=i$  then the solution has the form $\pi_i = \pi^*_i$ for $i< \ell$ and $\pi_i = q_i$ for $i\geq \ell$, where $\ell = \min\{i :  \pi^*_i > q_i\}$ is the lowest position with $\pi^*_i > q_i$ and $\pi^*_i$ is as defined in \eqref{pistar:eq}.
\end{lemma}

\section{Sanitized Keys and Frequencies} \label{refined:sec}

\begin{algorithm2e}\caption{Compute $(\pi_{i,j})$ for  Sanitized  Frequencies}\label{alg:reportfreq}
{\scriptsize
\DontPrintSemicolon
\KwIn{($(\epsilon,\delta)$, non-decreasing $(q_i)_{i\geq 1}$), $\text{Max\_Frequency}$}
\KwOut{$(\pi_{i,j})$ for $0\leq j\leq i \leq \text{Max\_Frequency}$}
$\pi_{0,0} \gets 1$, $\pi_0=0$\;
\ForEach(\tcp*[h]{Iterate over rows}){$i= 1,\ldots, \textrm{Max\_Frequency}$}{
$\pi_{i} \gets \min\{q_{i}, e^\varepsilon \pi_{i-1} + \delta, 1+ e^{-\varepsilon}(\pi_{i-1}+\delta-1) \}$\tcp*{End-to-end probability to output a key with frequency $i$}
$\pi_{i,0} \gets 1- \pi_{i}$\;
\ForEach(\tcp*[h]{Set lower bound values; use $[a]_+ :=\max\{a,0\}$}){$j=1,\ldots,i-1$}
{
$\pi_{i,j} \gets e^{-\varepsilon} \left(\sum_{h=1}^j \pi_{i-1,h} - \delta\right) - \sum_{h=1}^{j-1} \pi_{i,h}$ \\ $+ \left[ e^{-\varepsilon} \pi_{i-1,0}-\pi_{i,0}\right]_+$ \;
$\pi_{i,j} \gets [\pi_{i,j}]_+$\;
}
$R \gets \pi_i - \sum_{h=1}^{i-1} \pi_{i,h}$ \tcp*{Remaining probability to assign}\;
\ForEach(\tcp*[h]{Set final values for $\pi_{i,j}$}){$j=i,\ldots,1$}
{
\lIf{$R=0$}{{\bf Break}}
$U \gets e^\varepsilon \sum_{h=j}^{i-1}\pi_{i-1,h} + \delta -\sum_{h=j+1}^i \pi_{i,h}$ \tcp*{Max value allowed for $\pi_{i,j}$}\;
\eIf{$U-\pi_{i,j} \leq R$}{
$R\gets R- (U-\pi_{i,j})$\;
$\pi_{i,j} \gets U$
}
{
$\pi_{i,j} \gets \pi_{i,j}+R$\;
$R \gets 0$
}
}
}
\Return{$(\pi_{i,j})$}
}
\end{algorithm2e}

A frequency sanitizer $C$ returns keys $x$ together with sanitized information on their frequency.
We use $p_{i,j}$ for the probability that $C$ reports $j\in[t]$ for a sampled key that has frequency $i$, with $p_{i,0}$ being the probability that the sampled key is not reported. We have that
$\sum_{j=1}^{t} p_{i,j}$ is the total probability that a sampled key with frequency $i$ is reported by $C$. We use
\[\pi_{i,j} \gets q_i p_{i,j}\]
for the end-to-end probability that a key with frequency $i$ is sampled and reported in the private sample with sanitized value $j$.  For notation convenience, we use
$\pi_{i,0} := 1-\sum_{j=1}^t \pi_{i,t}$ for the probability that a key is not reported, making $\pi_{i,\bullet}$ probability vectors.
The reader can interpret the returned value $j$ as a token from an ordered domain.  The estimators we propose depend only on the order of tokens and not their values and hence are invariant to a mapping of the domain that preserves the order.

We express constraints on $(\pi_{i,j})_{i\geq 0,j\geq 0}$. For a solution to be  {\em realizable}, we must have end-to-end reporting probabilities that do not exceed the sampling probabilities:
\begin{equation} \label{qbound2:eq}
    \forall i,\ \sum_{j=1}^{t} \pi_{i,j} \leq q_i\enspace .
\end{equation}
The DP constraints are provided in the sequel. Note that we must have $\sum_{j=1}^{t} \pi_{i,j}\leq \pi_i$, where $(\pi_i)_{i\geq 1}$ is the solution for sanitized keys (Algorithm~\ref{alg:reportkeys}), this because the sanitized frequencies DP constraints are a superset of the sanitized keys constraints -- we obtain the latter in the former by considering outputs that group together all outputs with a key $x$ with all possible values of $j >0$. For optimality,
we seek solutions that (informally) {\em maximally separate} the distributions of different frequencies (minimize the overlap), over all possible DP frequency reporting schemes.
We will see that maximum separation can (i)~always be achieved by a discrete distribution (when the maximum frequency is bounded) and (ii)~can be simultaneously achieved between any pair of frequencies.  In particular, we maintain optimal reporting, that is, $\sum_{j=1}^{t} \pi_{i,j} = \pi_i$ and $\pi_{i,0} =  \overline{\pi}_i$.  The solutions we express are such that for
$i_1 > i_2$, $\pi_{i_1,\bullet}$ (first-order) stochastically dominates $\pi_{i_2,\bullet}$: That is, for any $h$, the probability of a token $j\geq h$  is non-decreasing with frequency.

  We present two algorithms that express $(\pi_{i,j})$.
    Algorithm~\ref{alg:reportfreq} provides a simplified construction, where
$t$ is equal to the maximum frequency and we always report $j\leq i$ for a key with frequency $i$. The sanitizer satisfies realizability and DP and has optimal key reporting but attains maximum separation only under some restrictions.
The values $\pi_{i,j}$ are specified in order of increasing $i$, where the row $\pi_{i,\bullet}$ is set so that the probability mass of $\pi_i$ is pushed to the extent possible to higher $j$ values.

  Algorithm~\ref{alg:Creport} specifies PDFs $(f_i)$ for a frequency sanitizer. The PDFs have a discrete point mass at $0$ (that corresponds to the probability of not reporting) and are piecewise constant elsewhere. The scheme is a refinement of the scheme of Algorithm~\ref{alg:reportfreq} and, as we shall see, for any $(q_i)$ and $(\varepsilon,\delta)$, it maximally  separates sanitized values for different frequencies. The construction introduces at most $3m$ distinct breakpoints for frequencies up to $m$ and can be discretized to have an equivalent $(\pi_{i,j})$ form with $j\in [3m]$.
 \notinproc{ (More details and proofs are provided in Appendix~\ref{general:sec}.)}

 \begin{theorem}\label{theorem:reported-freq-dist}
  The sanitizer with $(\pi_{i,j})$ expressed in Algorithm~\ref{alg:Creport} satisfies:
  \begin{enumerate}
      \item $\forall i,\, \sum_{j=1}^i \pi_{i,j} = \pi_i$, and in particular,  \eqref{qbound2:eq} holds and the sanitizer is realizable.
      \item $(\varepsilon,\delta)$-DP
      \item \label{separation:cond} Maximum separation:  For each $i$, there is an index $c_i$ so that subject to the above and to row $\pi_{i-1,\bullet}$, for all $j'\leq c_i$, the sum $\sum_{j=1}^{j'} \pi_{ij}$ is at a minimum and for all $j'\geq c_i$, the sum $\sum_{j=j'}^{i} \pi_{ij}$ is at a maximum.

  \end{enumerate}
       \end{theorem}

  The $(\pi_{i,j})$ expressed by Algorithm~\ref{alg:reportfreq}  satisfy maximum separation (Property~\ref{separation:cond}) under the particular restrictions on the reported values (that only $i$ different outputs are possible for frequencies up to $i$).  \notinproc{ (The proofs are provided in Appendix~\ref{proofssanitizedKF:sec})}

   The $(\pi_{i,j})$ expressed by Algorithm~\ref{alg:Creport} and then discretized satisfy maximum separation (Property~\ref{separation:cond}) unconditionally, over all DP frequency sanitization schemes.

For the special case where $q_i=1$ for all $i$, Algorithm~\ref{alg:reportfreq} provides maximum separation. We
provide a closed-form expression for the solution $\pi^*_{i,j}$.

 \begin{lemma} \label{formij:lemma}
 Let the DP parameters $(\varepsilon,\delta)$ be such that
 $L(\varepsilon,\delta)$ as in \eqref{Ldef:eq} is integral.
Let $(\pi^*_{ij})$ be the solution computed in Algorithm~\ref{alg:reportfreq} for $q_i=1$ for all $i$. Then the matrix with entries $\pi^*_{ij}$ for $i,j\geq 1$ has a  lower triangular form, with the non-zero entries as follows:
\[
\text{For } j\in \{\max\{1, i-2L\},\ldots, i\},\ \pi^*_{ij} = \pi^*_{i-j+1} - \pi^*_{i-j}\enspace .
\]
Equivalently,
$
  \pi^*_{i,j} =
\begin{cases}
\delta e^{(i-j)\varepsilon} &\,  \text{if $0\leq i-j\leq L$}\\
\delta e^{(2L-(i-j))\varepsilon}   &\,  \text{if $L+1\leq i-j \leq 2L$ .}
\end{cases}
$
\end{lemma}

\begin{algorithm2e}\caption{Compute $(f_i)$ for  Sanitizing Frequencies}\label{alg:Creport}
{\scriptsize
\DontPrintSemicolon
\KwIn{$(\epsilon,\delta)$, non-decreasing sampling probabilities $(q_i)_{i\geq 1}$, $\text{Max$\_$Frequency}$ }
\KwOut{$(f_i)_{i=0}^{\text{Max$\_$Frequency}} $, where $f_i:[0,i]$
\tcp*{PDF of sanitized frequency for frequency $i$: discrete mass at $f_i(0)$ (probability of not reporting) and density on $(0,i]$}}
$f_0(0) \gets 1$; $\pi_i \gets 0$ \tcp*{Keys with frequency $0$ are never reported}
\For(\tcp*[h]{Specify $f_i:[0,i]$}){$i\gets 1$ \KwTo $\text{Max$\_$Frequency}$}
{
 $\pi_{i} \gets \min\{q_{i}, e^\varepsilon \pi_{i-1} + \delta, 1+ e^{-\varepsilon}(\pi_{i-1}+\delta-1) \}$;  $f_i(0) \gets 1-\pi_i$ \tcp*{Reporting probability for $i$}
 $f_{i}(i-1,i] \gets \min\{\pi_i,\delta\}$\;
\tcp{Represent a function $f_L:(0,i-1]$ that "lower bounds" $f_i$}
\If {$\max\{0,e^{-\varepsilon} f_{i-1}(0) - f_i(0)\} + \int_{0^+}^{i-1} f_{i-1}(x) dx \leq \delta$}
{$f_L(0,i-1]\gets 0$}
\Else
{
$\mathit{b}_i \gets$ $z$ that solves
       $ \max\{0,e^{-\varepsilon} f_{i-1}(0) - f_i(0)\}+\int_{0^+}^z f_{i-1}(x) dx = \delta$ \tcp*{Well defined, as from DP, we always have $\max\{0,e^{-\varepsilon} f_{i-1}(0) - f_i(0)\}\leq \delta$}
$f_L(0,b_i]\gets 0$\;
\lFor{$x\in (\mathit{b}_i,i-1]$}{$f_L(x) \gets e^{-\varepsilon}f_{i-1}(x)$
}}
\tcp{Point where $f_i(x)-f_{i-1}(x)$ switches sign}
        $c_i \gets$ $z$ that solves
$\int_{0}^{z} f_L(x)dx + e^\varepsilon \int_z^{i-1} f_{i-1}(x) dx = \pi_i -\min\{\pi_i,\delta\}$ \tcp*{Any solution $z\in (0,i-1]$}
    \lFor{$x\in (c_i,i-1]$}{$f_i(x)= e^\varepsilon f_{i-1}(x)$}
    \lFor{$x \in (0,c_i]$}{$f_{i}(x) \gets f_L(x)$}

}
}
\Return{$(f_i)_{i=0}^{\text{Max$\_$Frequency}}$}
\end{algorithm2e}

\section{Estimation of Ordinal Statistics}

The sanitized frequencies can be used for estimation of statistics specified with respect to the actual frequencies.  In this section we consider {\em ordinal} statistics, that only depend on the order of frequencies but not on their nominal values.  Ordinal statistics include (approximate) top-$k$ set, quantiles, rank of a key, set of keys with a higher (or lower) rank than a specified key, and more.  We approximate ordinal statistics from the ordering of keys that is induced by sanitized frequencies.  The quality of estimated ordinal statistics is determined by the match between the order induced by exact frequencies and the order induced by sanitized frequencies. We say that the two orders are {\em concordant} on
a subset of keys $\{x_i\}$, when they match on that subset. Since the output of our sanitizer is stochastic, we consider the probability of a subset being concordant.  When sanitized values are discrete and two keys have the same sanitized value, we use probability of $0.5$ that two keys are concordant.

We define \notinproc{(see Appendix~\ref{maxseparation:sec})} a measure of separation between distributions $f_{i_1}$ and $f_{i_2}$ at a certain quantile value $\alpha$ and show that the $(f_i)$ constructed by Algorithm~\ref{alg:Creport} maximize it pointwise for any $i_1,i_2,\alpha$. This measure generalizes and follows from Property~\ref{separation:cond} stated in Theorem~\ref{theorem:reported-freq-dist}.
As a corollary we show\notinproc{ (The proof is provided in Appendix~\ref{maxseparation:sec})}:
\begin{corollary} \label{ordinalstat:coro}
The sanitizing scheme specified by the $(f_i)$ computed by Algorithm~\ref{alg:Creport} maximizes the following:
The probability that a subset of keys is concordant, the probability that a
key is correctly ordered with respect to all other keys, and the expected Kendall-$\tau$ rank correlation.
\end{corollary}
Note that we get optimality in a strong sense -- there is no Pareto front where concordant probability on some pairs of frequencies needs to be reduced in order to get a higher value for other pairs.

\section{Estimation of Linear Frequency Statistics}

The objective is to estimate statistics of the form
 \begin{equation} \label{qstat:eq}
     s := \sum_x L(x) g(w_x)\enspace .
 \end{equation}
We briefly review estimators for the non-private setting where the sample consists of pairs $(x,w_x)$ of keys and their frequency.  We use the per-key inverse-probability estimators~\cite{HT52} (also known as importance sampling).  The estimate $\widehat{g(w_x)}$ of $g(w_x)$ is $0$ if key $x$ is not included in the sample and otherwise the estimate is
\onlyinproc{$a_{w_x} := \frac{g(w_x)}{q_{w_x}}$.}
\notinproc{
  \begin{equation} \label{invprob:est}
    a_{w_x} := \frac{g(w_x)}{q_{w_x}} \enspace .
  \end{equation}
}
 These estimates are nonnegative, a desired property for nonnegative values, and are also unbiased when $q_{w_x}>0$.   The estimate for the query statistics \eqref{qstat:eq} is
 \begin{equation} \label{sumest:eq}
     \hat{s} := \sum_{(x,w_x)} L(x) \widehat{g(w_x)} =  \sum_{(x,w_x)\in S}  L(x) a_{w_x}\enspace .
 \end{equation}
\notinproc{Since the estimate is $0$ for keys not represented in the sample, it can be computed from the sample.
 The variance of a per-key estimate for a key with frequency $i$ is $g(i)^2(\frac{1}{q_i} -1)$ and the variance of the sum estimator \eqref{sumest:eq} is
  \[ \textsf{Var}[\hat{s}] = \sum_x L(x)^2 g(w_x)^2(\frac{1}{q_{w_x}} -1) \enspace .\]
  These inverse-probability estimates are optimal for the sampling scheme in that they minimize the sum of per-key variance under unbiasedness and non-negativity constraints.
   We note that the quality of the estimates depends on the match between $g(i)$ and $q_i$: Probability Proportional to Size (PPS), where $q_i\propto g(i)$ is most effective and minimizes the sum of per-key variance for the sample size. Our aim here is to optimize what we can do privately when $q$ and $g$ are given.}

  \subsection{Estimation with Sanitized Samples} \label{estssanitized:sec}
  We now consider estimation from sanitized samples $S^*$.
We specify our estimators $(a_j)_{j\geq 1}$ in terms of the reported sanitized frequencies $j$.  The estimate is $0$ for keys that are not reported and are $a_j$ when reported with value $j$. The estimate of the statistics is
\begin{equation} \label{partialfest:eq}
        \widehat{s} := \sum_{(x,j)\in S^*}  L(x) a_{j}\ .
\end{equation}

As for choosing $(a_j)_{j\geq 1}$, a first attempt is the unique unbiased estimator:
The unbiasedness constraints
\onlyinproc{$\forall i,\ \sum_{j=1}^i \pi_{ij} a_j = g(i)$}\notinproc{
  \[
 \forall i,\ \sum_{j=1}^i \pi_{ij} a_j = g(i)\enspace
 \]}
 form a triangular system with a unique solution $(a_j)_{j\geq 1}$\onlyinproc{.}\notinproc{:
 \[a_i \gets \frac{g(i) - \sum_{j=1}^{i-1} \pi_{i,j} a_j}{\pi_{i,i}} \enspace .\]}
However, $(a_j)_{j\geq 1}$ may include negative values and estimates have high variance. We argue that bias is unavoidable with privacy:
First, the inclusion probability of keys with frequency $w_x=1$ can not exceed $\delta$. Therefore, the variance contribution of the key to any unbiased estimate is at least $1/\delta$.  Typically, $\delta$ is chosen so that  $1/\delta \gg n k$, where $n$ is the support size and $k$ the sample size, so this error can not be mitigated.  Second, we show \onlyinproc{in the full version}\notinproc{in Appendix~\ref{mustnegative:sec}} that even for the special case of $q=1$, {\em any} unbiased estimator applied to the output of  {\em any} sanitized keys and frequencies scheme with optimal reporting probabilities must assume negative values. That is,
DP schemes do not admit unbiased nonnegative estimators without compromising reporting probabilities. We therefore seek estimators that are biased but balance bias and variance and are nonnegative. In our evaluation we use the following Maximum Likelihood estimator (MLE):
\begin{equation} \label{MLest:eq}
    a_j \gets \frac{g(i)}{\pi_i}\text{, where } i =   \arg\max_h \pi_{hj} \enspace .
\end{equation}

This estimate is "right" for the frequency $i$ for which the probability of reporting $j$ is maximized.\notinproc{ The estimate can be biased up or down.}
\notinproc{Another estimator with desirable properties is proposed in Appendix~\ref{biaseddown:sec}.}

 We express the expected value, bias, Mean Squared Error (MSE), and variance of the per-key estimate for a key with frequency $i$:
 \onlyinproc{
 $\textsf{E}_i := \sum_{j=1}^i  \pi_{i,j} a_j$, $\textsf{Bias}_i := \textsf{E}_i - g(i)$, $\textsf{MSE}_i := \overline{\pi}_i g(i)^2 +  \sum_{j=1}^i  \pi_{i,j} (a_j - g(i))^2$, $\textsf{Var}_i := \textsf{MSE}_i - \textsf{Bias}_i^2$. For the sum estimate \eqref{partialfest:eq} we get:
   $\textsf{Bias}[\widehat{s}] = \sum_x L(x) \textsf{Bias}_{w_x}$, $\textsf{Var}[\widehat{s}] = \sum_x L(x)^2 \textsf{Var}_{w_x}$,
   $\textsf{MSE}[\widehat{s}] = \textsf{Var}[\widehat{s}] + \textsf{Bias}[\widehat{s}]^2$, and $\textsf{NRMSE}[\widehat{s}] = \frac{\sqrt{\textsf{MSE}[\widehat{s}]}}{s}$.
       }
 \notinproc{
 {\small
   \begin{align*}
   \textsf{E}_i :=&\, \sum_{j=1}^i  \pi_{i,j} a_j\\
       \textsf{Bias}_i :=&\, \textsf{E}_i - g(i)\\
       \textsf{MSE}_i :=&\, \overline{\pi}_i g(i)^2 +  \sum_{j=1}^i  \pi_{i,j} (a_j - g(i))^2 \\
       \textsf{Var}_i :=&\, \textsf{MSE}_i - \textsf{Bias}_i^2
    \enspace .
   \end{align*}
   }
 For the sum estimate \eqref{partialfest:eq} we get:
{\small
   \begin{align}
    \textsf{Bias}[\widehat{s}] =&\, \sum_x L(x) \textsf{Bias}_{w_x}\nonumber\\
    \textsf{Var}[\widehat{s}] =&\, \sum_x L(x)^2 \textsf{Var}_{w_x}\nonumber\\
    \textsf{MSE}[\widehat{s}] =&\, \textsf{Var}[\widehat{s}] + \textsf{Bias}[\widehat{s}]^2\nonumber\\
    \textsf{NRMSE}[\widehat{s}] =& \frac{\sqrt{\textsf{MSE}[\widehat{s}]}}{s}\enspace . \label{NRMSE:eq}
 \end{align}
}}
Note that
the variance component of the normalized squared error $\textsf{MSE}[\widehat{s}]/s^2$ decreases linearly with support size whereas the bias component may not.
We therefore consider both the variance and bias of the per-key estimators and
qualitatively seek low bias and ``bounded'' variance. We measure quality of statistics estimators using the Normalized Root Mean Squared Error (NRMSE).

  \section{Performance Analysis} \label{experiments:sec}

    We study the performance of PWS on the key reporting and estimation objectives and compare with a baseline method that provides the same privacy guarantees.  We use precise expressions (not simulations) to compute probabilities, bias, variance, and MSE of the different methods.

    \subsection{Private Histograms Baseline} \label{SbH:sec}

   We review the {\em Stability-based Histograms} (SbH) method of \cite{BunNS19,KorolovaKMN09,VadhanDPsurvey:2017}, which we use as a baseline. SbH, provided as Algorithm~\ref{alg:SBH},  is designed for the special case when $q_i=1$ for all frequencies. The input $S$ is the full data of pairs of  keys and positive frequencies $(x,w_x)$. The private output $S^*$ is a subset of the keys\notinproc{ in the data} with positive sanitized frequencies $(x,w^*_x)$.
   \begin{algorithm2e}\caption{\small Stability-based Histograms (SbH)

   }\label{alg:SBH}
{\small
\DontPrintSemicolon
\KwIn{$(\epsilon,\delta)$ , $S=\{(x,w_x)\}$ where $w_x>0$}
\KwOut{Key value pairs $S'$}
$S^*\gets \emptyset$\tcp*{Initialize private histogram}
$T \gets (1/\varepsilon)\ln(1/\delta)+1$\tcp*{Threshold}
\ForEach{$(x,w_x)\in S$}
{
    $w^*_x \gets w_x + \textsf{Lap}[\frac{1}{\varepsilon}]$\tcp*{Add Laplace random variable}
    \If{$w^*_x \geq T$}{$S^*\gets S^* \cup (x,w^*_x)$ }
    }
}
\end{algorithm2e}

The SbH method is considered the state of the art for sparse histograms (only keys with $w_x > 0$ can be reported). The method returns non-negative $w_x^*>0$  sanitized frequencies.
For the case of no sampling, we compare PWS (with $q\equiv 1$) with SbH. We use the SbH sanitized frequencies directly for estimation.
For sampling, our baseline is {\em sampled-SbH}: The data is first sanitized using SbH and then sampled, using a weighted sampling algorithm with $q$, while treating the sanitized frequencies as actual frequencies. For estimation, we apply the estimator \eqref{sumest:eq} (which in this context is biased).
\notinproc{
To facilitate comparison with SbH and sampled-SbH we express the reporting probabilities, bias, and variance in Appendix~\ref{expressSbH:sec}.}

\notinproc{\subsection{Reporting Probabilities: No Sampling}}
\onlyinproc{\paragraph{Reporting Probabilities: No Sampling}}
We start with the case of no sampling and the objective of maximizing the number of privately reported keys.  We compare the PWS
(optimal) probabilities $\pi^*$ \eqref{pistar:eq} to the baseline SbH  \notinproc{\cite{BunNS19,KorolovaKMN09,VadhanDPsurvey:2017}}
reporting probabilities $\phi$\notinproc{ \eqref{SbHphi:eq}}.
Figure~\ref{probfreqhist:plot} shows reporting probability per frequency for selected DP parameters.  We can see that with both private methods the reporting probability reaches $1$ for high frequencies but PWS (Opt) reaches the maximum earlier and is significantly higher than $\phi$ along the way. Analytically from the expressions we can see that
for $i\leq L(\varepsilon,\delta)$,
$\pi^*_i/\phi_i \in [2,2/\varepsilon]$ with $\pi^*_i/\phi_i \approx 2i$ for lower $i$. We can also see that $\pi^*_i =1$ for  $i=2L+2 \approx \frac{2}{\varepsilon}\ln(\varepsilon/\delta)$ whereas $\phi_i > 1-\delta$ for $i\approx \frac{2}{\varepsilon} \ln(1/\delta)$.  The ratio between the frequency values when maximum reporting is reached is $\approx\ln(1/\delta)/\ln(\varepsilon/\delta)$.\notinproc{

} Figure~\ref{Zipfreport:plot} shows the expected numbers of reported keys with PWS (Opt) and SbH for frequency distributions that are $\textsf{Zipf}[\alpha]$ with $\alpha=1,2$ as we sweep the privacy parameter $\delta$. Overall we see that PWS gains 20\%-300\% in the number of keys reported over baseline.
Note that as expected, the optimal PWS reports {\em all} keys when $\delta=1$ (i.e., no privacy guarantees) but SbH incurs reporting loss. \onlyinproc{In the full version, we show similar results on two real-world datasets.}

\notinproc{
We additionally evaluate PWS and compare it to SbH on two real-world datasets:
\begin{enumerate}
    \item ABC: The words of news headlines from the Australian Broadcasting Corporation.  The keys are words and the frequency is the respective number of occurrences  \cite{abcnews}.
    \item SO: The multi-graph of Stack Overflow where keys are nodes in the graph and frequencies are undirected degrees \cite{StackOverflow}.
\end{enumerate}
Figure~\ref{RealWorldNoSampling:plot} shows the expected numbers of reported keys on these datasets. We note that the results are similar to what was observed on the synthetic Zipf datasets.
}

\begin{figure}[ht]
\centering
\includegraphics[width=0.44\textwidth]{plots/probability-frequency-10-100.pdf}
\includegraphics[width=0.44\textwidth]{plots/probability-frequency-100-1000000.pdf}
\caption{Key reporting probability for frequency. No sampling ($q=1$) with PWS (Opt) and SbH for $(\varepsilon,\delta)=(0.1,0.01),(0.01,10^{-6})$}
\label{probfreqhist:plot}
\end{figure}

\begin{figure}[ht]
\centering
\includegraphics[width=0.30\textwidth]{plots/optsbh-zipf-ratio-ieps10.pdf}
\includegraphics[width=0.30\textwidth]{plots/optsbh-zipf-gains-a1-ieps10.pdf}
\includegraphics[width=0.30\textwidth]{plots/optsbh-zipf-gains-a2-ieps10.pdf}
\caption{Expected fraction of keys that are privately reported with PWS (Opt) and SbH for $\textsf{Zipf}[\alpha]$ frequency distributions. For $\alpha=1,2$, \notinproc{privacy parameters }$\varepsilon=0.1$ and sweeping $\delta$ between $1$ and $10^{-8}$. Left: The respective ratio of PWS to SbH.}
\label{Zipfreport:plot}
\end{figure}

\notinproc{
\begin{figure}[ht]
\centering
\includegraphics[width=0.30\textwidth]{plots/real_world_datasets_gain_ratio.pdf}
\includegraphics[width=0.30\textwidth]{plots/gains_ABC.pdf}
\includegraphics[width=0.30\textwidth]{plots/gains_SO.pdf}
\caption{Evaluation of PWS on real-world datasets (without sampling). Left: The ratio of reported keys with PWS to SbH. Center and Right: Fraction of total keys reported with sampled-SbH and PWS as we sweep the parameter $\delta$.}
\label{RealWorldNoSampling:plot}
\end{figure}
}

 \notinproc{\subsection{Reporting Probabilities with Sampling}}
 \onlyinproc{\paragraph{Reporting Probabilities with Sampling}}
 Figure~\ref{reporting_sampling:fig} shows
 reporting probabilities with PWS (optimal reporting probabilities), sampled-SbH, and non-private sampling, for representative sampling rates and privacy parameters.

  \begin{figure}[ht]
\centering
\includegraphics[width=0.30\textwidth]{plots/probability-frequency-tau1000-10-1000000.pdf}
\includegraphics[width=0.30\textwidth]{plots/probability-frequency-tau1000-100-1000000.pdf}
\includegraphics[width=0.30\textwidth]{plots/probability-frequency-tau10000-10-1000000.pdf}
\caption{Reporting probability as a function of frequency. For ppswor sampling with threshold $\tau$ ($q$), PWS private samples (Opt), and sampled-SbH private samples.}
\label{reporting_sampling:fig}
\end{figure}
As expected, for sufficiently large frequencies both private methods have reporting probabilities that match the sampling probabilities $q$ of the non-private scheme.
But PWS reaches $q$ at a lower frequency than sampled-SbH and has significantly higher reporting probabilities for lower frequencies.
Figure~\ref{sweepsamplingZipf:fig} shows the fraction of keys reported for $\textsf{Zipf}$ distributions as we sweep the sampling rate (threshold $\tau$).   PWS reports more keys than sampled-SbH and the gain persists also with low sampling rates.  We can see that with PWS, thanks to end-to-end privacy analysis, the reporting loss due to sampling mitigates the reporting loss needed for privacy -- reporting approaches that of the non-private sampling when the sampling rate $\tau$ approaches $\delta$.  Sampled-SbH, on the other hand, incurs reporting loss due to privacy on top of the reporting loss due to sampling.
\notinproc{Figures \ref{RealWorldWithSampling1:plot} and \ref{RealWorldWithSampling2:plot} show the expected fraction of reported keys on the real-world datasets ABC and SO.}
   \begin{figure}[ht]
\centering
\includegraphics[width=0.30\textwidth]{plots/optsbh-tau-zipf-gains-a05-10-1000.pdf}
\includegraphics[width=0.30\textwidth]{plots/optsbh-tau-zipf-gains-a1-10-1000.pdf}
\includegraphics[width=0.30\textwidth]{plots/optsbh-tau-zipf-gains-a2-10-1000.pdf}
\caption{Fraction of total keys reported with threshold-ppswor, sampled-SbH, and PWS (Opt), as we sweep the sampling rate $\tau$.  For $\textsf{Zipf}[\alpha]$, $\varepsilon=0.1$ and $\delta=0.001$ the gains in reporting of PWS over Sampled-SbH are at least 230\% ($\alpha=0.5$), 97\% ($\alpha=1$) and 37\% ($\alpha=2$).}
\label{sweepsamplingZipf:fig}
\end{figure}

\notinproc{
\begin{figure}[ht]
\centering
\includegraphics[width=0.44\textwidth]{plots/gains_with_ppswor_ABC.pdf}
\includegraphics[width=0.44\textwidth]{plots/gains_with_ppswor_SO.pdf}
\caption{Evaluation on real-world datasets (with sampling): fraction of total keys reported with threshold-ppswor (non-private), sampled-SbH, and PWS, as we sweep the sampling rate $\tau$.}
\label{RealWorldWithSampling1:plot}
\end{figure}
\begin{figure}[ht]
\centering
\includegraphics[width=0.44\textwidth]{plots/real_world_gain_with_sampling_01_3.pdf}
\includegraphics[width=0.44\textwidth]{plots/real_world_gain_with_sampling_01_5.pdf}
\caption{Ratio of expected number of keys that are privately reported with PWS to SbH for the ABC and SO datasets, as we sweep the sampling rate $\tau$.}
\label{RealWorldWithSampling2:plot}
\end{figure}
}

\notinproc{\subsection{Estimation of Linear Statistics}}
\onlyinproc{\paragraph{Estimation of Linear Statistics}}
We evaluate estimation quality for linear statistics \eqref{qstat:eq} when $g(w)=w$ and $L(x)$ is a selection predicate.  The statistics is simply the sum of frequencies of selected keys.
We compare performance of PWS with the MLE estimator \eqref{MLest:eq}, the baseline sampled-SbH, and for reference, the estimator of the respective non-private sample \eqref{sumest:eq}.
Figure~\ref{bias_error:fig} (top) shows normalized bias $\textsf{Bias}_i/i$ as a function of the frequency $i$ for the two private methods (the non-private estimator is unbiased and not shown).
With both methods, the bias decreases with frequency and diminishes for $i \gg 2\varepsilon^{-1}\ln(1/\delta)$.
PWS has lower bias at lower frequencies than SbH, allowing for more accurate estimates on a broader range. We can see that with PWS, the bias decreases
when the sampling rate ($\tau$) decreases
and diminishes when $\tau$ approaches $\delta$. This is a benefit of the end-to-end privacy analysis. The bias of the baseline method does not change with sampling rate.

Figure~\ref{variance:fig} shows the normalized variance $\textsf{Var}_i/i^2$ per frequency $i$ for representative parameter settings.
The private methods PWS and sampled-SbH maintain low variance across frequencies:  The value is fractional with no sampling and is of the order of that of the non-private unbiased estimator with sampling.
In particular this means that the bias is a good proxy for performance and that the improvement in bias of PWS with respect to baseline does not come with a hidden cost in variance. For high frequencies (not shown), keys with all methods are
included with probability (close to) $1$. The non-private method that reports exact frequencies have $0$ variance whereas the private methods maintain a low variance, but the normalized variance diminishes for all methods.

For statistics estimation, the per-key performance suggest that
when the selection has many high frequency keys, the private methods perform well and are similar to non-private sampling.
When the selection is dominated by very low frequencies, the private methods perform poorly and well below the respective non-private sample.  But for low to medium frequencies,  PWS can provide drastic improvements over SbH and the gain increases with lower sampling rates.  Figure~\ref{bias_error:fig} (bottom) shows the $\textsf{NRMSE}$ as a function of sampling rate for estimating the sum of frequencies on a selection of $2\times 10^5$ keys with
frequencies uniformly drawn between $1$ and $200$.
We can see that the error of non-private sampling and of sampled-SbH decreases with higher sampling rate.  Note the perhaps counter-intuitive phenomenon that PWS (MLE) hits its sweet spot midway:  This is due to a balance of the two components of the error, the variance which increases and the bias that decreases when the sampling rate decreases.  Also note that PWS significantly improves over SbH also with no sampling ($\tau=1$).

     \begin{figure}[ht]
\centering
\includegraphics[width=0.44\textwidth]{plots/norm_bias_e0_1_full.pdf}
\includegraphics[width=0.44\textwidth]{plots/norm_bias_e0_5_full.pdf}
\includegraphics[width=0.44\textwidth]{plots/nrmse_on_uniform_range200_mult1000_e0_1.pdf}
\includegraphics[width=0.44\textwidth]{plots/nrmse_on_uniform_range200_mult1000_e0_5.pdf}
\caption{ Top: Normalized bias for PWS (MLE) and sampled-SbH as a function of frequency, for different sampling rates.  The bias of the sampled-SbH estimates (shown once) does not change with sampling rate. Bottom: NRMSE as a function of sampling rate for a selection of $2\times 10^5$ keys with frequencies drawn uniformly $[1,200]$.
}
\label{bias_error:fig}
\end{figure}

\begin{figure}[!pht]
\centering
\includegraphics[width=0.44\textwidth]{plots/norm_variance_AlwaysIncludeSamplingMethod_t1_0_e0_1.pdf}
\notinproc{
\includegraphics[width=0.44\textwidth]{plots/norm_variance_AlwaysIncludeSamplingMethod_t1_0_e0_5.pdf}}
\includegraphics[width=0.44\textwidth]{plots/norm_variance_PrioritySamplingMethod_t0_001_e0_1.pdf}
\notinproc{
\includegraphics[width=0.44\textwidth]{plots/norm_variance_PrioritySamplingMethod_t0_001_e0_5.pdf}
\includegraphics[width=0.44\textwidth]{plots/variance_PrioritySamplingMethod_t0_001_e0_1.pdf}
\includegraphics[width=0.44\textwidth]{plots/variance_PrioritySamplingMethod_t0_001_e0_5.pdf}
}
\caption{ Normalized variance $\textsf{Var}_i/i^2$ \notinproc{and variance $\textsf{Var}_i$} for PWS (MLE) and sampled-SbH as a function of the frequency $i$.
}
\label{variance:fig}
\end{figure}

\vspace{-0.24cm}
\section*{Conclusion}
We presented
Private Weighted Sampling (PWS), a method to post-process a weighted sample and produce a version that is differentially private.  Our private samples maximize the number of reported keys subject to the privacy constraints and support estimation of linear and ordinal statistics.  We demonstrate significant improvement over prior methods for both reporting and estimation tasks, even for the well studied special case of private histograms (when there is no sampling).

An appealing direction for future work is to explore the use of PWS to design \emph{composable} private sketches, e.g., in the context of coordinated samples.
Threshold and bottom-$k$ samples of different datasets are {\em coordinated} when using consistent $\{u_x\}$. Coordinated samples generalize MinHash sketches and support estimation of similarity measures and statistics over multiple datasets \cite{BrEaJo:1972,Saavedra:1995,ECohen6f,Broder:CPM00,sdiff:KDD2014,sorder:PODC2014}.

\subsubsection*{Acknowledgments}
Part of this work was done while Ofir Geri was an intern at Google Research. This work was partially supported by Moses Charikar's Google Faculty Research Award. We also thank Chinmoy Mandayam for bringing the related paper \cite{GhoshRS:sicomp2012} to our attention.

\bibliographystyle{plain}