  The earth system is exceedingly complex and often chaotic in nature, making
prediction incredibly challenging: we cannot expect to make perfect predictions
all of the time. Instead, we look for specific states of the system that lead
to more predictable behavior than others, often termed "forecasts of
opportunity". When these opportunities are not present, scientists need
prediction systems that are capable of saying "I don't know." We introduce a
novel loss function, termed "abstention loss", that allows neural networks to
identify forecasts of opportunity for regression problems. The abstention loss
works by incorporating uncertainty in the network's prediction to identify the
more confident samples and abstain (say "I don't know") on the less confident
samples. The abstention loss is designed to determine the optimal abstention
fraction, or abstain on a user-defined fraction via a PID controller. Unlike
many methods for attaching uncertainty to neural network predictions
post-training, the abstention loss is applied during training to preferentially
learn from the more confident samples. The abstention loss is built upon a
standard computer science method. While the standard approach is itself a
simple yet powerful tool for incorporating uncertainty in regression problems,
we demonstrate that the abstention loss outperforms this more standard method
for the synthetic climate use cases explored here. The implementation of
proposed loss function is straightforward in most network architectures
designed for regression, as it only requires modification of the output layer
and loss function.
