  Inferences about hypotheses are ubiquitous in the cognitive sciences. Bayes
factors provide one general way to compare different hypotheses by their
compatibility with the observed data. Those quantifications can then also be
used to choose between hypotheses. While Bayes factors provide an immediate
approach to hypothesis testing, they are highly sensitive to details of the
data/model assumptions. Moreover it's not clear how straightforwardly this
approach can be implemented in practice, and in particular how sensitive it is
to the details of the computational implementation. Here, we investigate these
questions for Bayes factor analyses in the cognitive sciences. We explain the
statistics underlying Bayes factors as a tool for Bayesian inferences and
discuss that utility functions are needed for principled decisions on
hypotheses. Next, we study how Bayes factors misbehave under different
conditions. This includes a study of errors in the estimation of Bayes factors.
Importantly, it is unknown whether Bayes factor estimates based on bridge
sampling are unbiased for complex analyses. We are the first to use
simulation-based calibration as a tool to test the accuracy of Bayes factor
estimates. Moreover, we study how stable Bayes factors are against different
MCMC draws. We moreover study how Bayes factors depend on variation in the
data. We also look at variability of decisions based on Bayes factors and how
to optimize decisions using a utility function. We outline a Bayes factor
workflow that researchers can use to study whether Bayes factors are robust for
their individual analysis, and we illustrate this workflow using an example
from the cognitive sciences. We hope that this study will provide a workflow to
test the strengths and limitations of Bayes factors as a way to quantify
evidence in support of scientific hypotheses. Reproducible code is available
from https://osf.io/y354c/.
