  As the prevalence of data analysis grows, safeguarding data privacy has
become a paramount concern. Consequently, there has been an upsurge in the
development of mechanisms aimed at privacy-preserving data analyses. However,
these approaches are task-specific; designing algorithms for new tasks is a
cumbersome process. As an alternative, one can create synthetic data that is
(ideally) devoid of private information. This paper focuses on
privacy-preserving data synthesis (PPDS) by providing a comprehensive overview,
analysis, and discussion of the field. Specifically, we put forth a master
recipe that unifies two prominent strands of research in PPDS: statistical
methods and deep learning (DL)-based methods. Under the master recipe, we
further dissect the statistical methods into choices of modeling and
representation, and investigate the DL-based methods by different generative
modeling principles. To consolidate our findings, we provide comprehensive
reference tables, distill key takeaways, and identify open problems in the
existing literature. In doing so, we aim to answer the following questions:
What are the design principles behind different PPDS methods? How can we
categorize these methods, and what are the advantages and disadvantages
associated with each category? Can we provide guidelines for method selection
in different real-world scenarios? We proceed to benchmark several prominent
DL-based methods on the task of private image synthesis and conclude that
DP-MERF is an all-purpose approach. Finally, upon systematizing the work over
the past decade, we identify future directions and call for actions from
researchers.
