  As image datasets become ubiquitous, the problem of ad-hoc searches over
image data is increasingly important. Many high-level data tasks in machine
learning, such as constructing datasets for training and testing object
detectors, imply finding ad-hoc objects or scenes within large image datasets
as a key sub-problem. New foundational visual-semantic embeddings trained on
massive web datasets such as Contrastive Language-Image Pre-Training (CLIP) can
help users start searches on their own data, but we find there is a long tail
of queries where these models fall short in practice. SeeSaw is a system for
interactive ad-hoc searches on image datasets that integrates state-of-the-art
embeddings like CLIP with user feedback in the form of box annotations to help
users quickly locate images of interest in their data even in the long tail of
harder queries. One key challenge for SeeSaw is that, in practice, many
sensible approaches to incorporating feedback into future results, including
state-of-the-art active-learning algorithms, can worsen results compared to
introducing no feedback, partly due to CLIP's high-average performance.
Therefore, SeeSaw includes several algorithms that empirically result in larger
and also more consistent improvements. We compare SeeSaw's accuracy to both
using CLIP alone and to a state-of-the-art active-learning baseline and find
SeeSaw consistently helps improve results for users across four datasets and
more than a thousand queries. SeeSaw increases Average Precision (AP) on search
tasks by an average of .08 on a wide benchmark (from a base of .72), and by a
.27 on a subset of more difficult queries where CLIP alone performs poorly.
