  Deep learning is moving towards increasingly sophisticated optimization
objectives that employ higher-order functions, such as integration, continuous
optimization, and root-finding. Since differentiable programming frameworks
such as PyTorch and TensorFlow do not have first-class representations of these
functions, developers must reason about the semantics of such objectives and
manually translate them to differentiable code.
  We present a differentiable programming language, $\lambda_S$, that is the
first to deliver a semantics for higher-order functions, higher-order
derivatives, and Lipschitz but nondifferentiable functions. Together, these
features enable $\lambda_S$ to expose differentiable, higher-order functions
for integration, optimization, and root-finding as first-class functions with
automatically computed derivatives. $\lambda_S$'s semantics is computable,
meaning that values can be computed to arbitrary precision, and we implement
$\lambda_S$ as an embedded language in Haskell.
  We use $\lambda_S$ to construct novel differentiable libraries for
representing probability distributions, implicit surfaces, and generalized
parametric surfaces -- all as instances of higher-order datatypes -- and
present case studies that rely on computing the derivatives of these
higher-order functions and datatypes. In addition to modeling existing
differentiable algorithms, such as a differentiable ray tracer for implicit
surfaces, without requiring any user-level differentiation code, we demonstrate
new differentiable algorithms, such as the Hausdorff distance of generalized
parametric surfaces.
