  Adaptivity is an important feature of data analysis---the choice of questions
to ask about a dataset often depends on previous interactions with the same
dataset. However, statistical validity is typically studied in a nonadaptive
model, where all questions are specified before the dataset is drawn. Recent
work by Dwork et al. (STOC, 2015) and Hardt and Ullman (FOCS, 2014) initiated
the formal study of this problem, and gave the first upper and lower bounds on
the achievable generalization error for adaptive data analysis.
  Specifically, suppose there is an unknown distribution $\mathbf{P}$ and a set
of $n$ independent samples $\mathbf{x}$ is drawn from $\mathbf{P}$. We seek an
algorithm that, given $\mathbf{x}$ as input, accurately answers a sequence of
adaptively chosen queries about the unknown distribution $\mathbf{P}$. How many
samples $n$ must we draw from the distribution, as a function of the type of
queries, the number of queries, and the desired level of accuracy?
  In this work we make two new contributions:
  (i) We give upper bounds on the number of samples $n$ that are needed to
answer statistical queries. The bounds improve and simplify the work of Dwork
et al. (STOC, 2015), and have been applied in subsequent work by those authors
(Science, 2015, NIPS, 2015).
  (ii) We prove the first upper bounds on the number of samples required to
answer more general families of queries. These include arbitrary
low-sensitivity queries and an important class of optimization queries.
  As in Dwork et al., our algorithms are based on a connection with algorithmic
stability in the form of differential privacy. We extend their work by giving a
quantitatively optimal, more general, and simpler proof of their main theorem
that stability implies low generalization error. We also study weaker stability
guarantees such as bounded KL divergence and total variation distance.
