  Differential privacy (DP) data synthesizers support public release of
sensitive information, offering theoretical guarantees for privacy but limited
evidence of utility in practical settings. Utility is typically measured as the
error on representative proxy tasks, such as descriptive statistics, accuracy
of trained classifiers, or performance over a query workload. The ability for
these results to generalize to practitioners' experience has been questioned in
a number of settings, including the U.S. Census. In this paper, we propose an
evaluation methodology for synthetic data that avoids assumptions about the
representativeness of proxy tasks, instead measuring the likelihood that
published conclusions would change had the authors used synthetic data, a
condition we call epistemic parity. Our methodology consists of reproducing
empirical conclusions of peer-reviewed papers on real, publicly available data,
then re-running these experiments a second time on DP synthetic data, and
comparing the results.
  We instantiate our methodology over a benchmark of recent peer-reviewed
papers that analyze public datasets in the ICPSR repository. We model
quantitative claims computationally to automate the experimental workflow, and
model qualitative claims by reproducing visualizations and comparing the
results manually. We then generate DP synthetic datasets using multiple
state-of-the-art mechanisms, and estimate the likelihood that these conclusions
will hold. We find that state-of-the-art DP synthesizers are able to achieve
high epistemic parity for several papers in our benchmark. However, some
papers, and particularly some specific findings, are difficult to reproduce for
any of the synthesizers. We advocate for a new class of mechanisms that favor
stronger utility guarantees and offer privacy protection with a focus on
application-specific threat models and risk-assessment.
