  We present a proof-by-simulation framework for rigorously controlling a trial
design's operating characteristics over continuous regions of parameter space.
We show that proof-by-simulation can achieve two major goals: (1) Calibrate a
design for provable Type I Error control at a fixed level alpha. (2) Given a
fixed design, bound (with high probability) its operating characteristics, such
as the Type I Error, FDR, or bias of bounded estimators. This framework can
handle adaptive sampling, nuisance parameters, administrative censoring,
multiple arms, and multiple testing. These techniques, which we call Continuous
Simu lation Extension (CSE), were first developed in Sklar (2021) to control
Type I Error and FWER for designs where unknown parameters have exponential
family likelihood. Our appendix improves those results with more efficient
bounding and calibration methods, extends them to general operating
characteristics including FDR and bias, and extends applicability to include
canonical GLMs and some non-parametric problems. In the main paper we
demonstrate our CSE approach and software on 3 examples (1) a gentle
introduction, analyzing the z-test (2) a hierarchical Bayesian analysis of 4
treatments, with sample sizes fixed (3) an adaptive Bayesian Phase II-III
selection design with 4 arms, where interim dropping and go/no-go decisions are
based on a hierarchical model. Trillions of simulations were performed for the
latter two examples, enabled by specialized INLA software. Open-source software
is maintained at (1)
