  In computational social science, researchers often use a pre-trained, black
box classifier to estimate the frequency of each class in unlabeled datasets. A
variety of prevalence estimation techniques have been developed in the
literature, each yielding an unbiased estimate if certain stability assumption
holds. This work introduces a framework to rethink the prevalence estimation
process as calibrating the classifier outputs against ground truth labels to
obtain the joint distribution of a base dataset and then extrapolating to the
joint distribution of a target dataset. We call this framework
"Calibrate-Extrapolate". Visualizing the joint distribution makes the stability
assumption needed for a prevalence estimation technique clear and easy to
understand. In the calibration phase, the techniques assume only a stable
calibration curve between a calibration dataset and the full base dataset. This
allows for the classifier outputs to be used for purposive sampling, thus
improving the efficiency of calibration. In the extrapolation phase, some
techniques assume a stable calibration curve while some assume stable
class-conditional densities. We discuss the stability assumptions from a causal
perspective. By specifying base and target joint distributions, we can generate
simulated datasets, as a way to build intuitions about the impacts of
assumption violations. This also leads to a better understanding of how the
classifier predictive power affects the accuracy of prevalence estimates: the
greater the predictive power, the lower the sensitivity to violations of
stability assumptions in the extrapolation phase. We illustrate the framework
with an application that estimates the prevalence of toxic news comments over
time on Reddit, Twitter, and YouTube, using Jigsaw's Perspective API as a black
box classifier.
