  Ad relevance modeling plays a critical role in online advertising systems
including Microsoft Bing. To leverage powerful transformers like BERT in this
low-latency setting, many existing approaches perform ad-side computations
offline. While efficient, these approaches are unable to serve cold start ads,
resulting in poor relevance predictions for such ads. This work aims to design
a new, low-latency BERT via structured pruning to empower real-time online
inference for cold start ads relevance on a CPU platform. Our challenge is that
previous methods typically prune all layers of the transformer to a high,
uniform sparsity, thereby producing models which cannot achieve satisfactory
inference speed with an acceptable accuracy.
  In this paper, we propose SwiftPruner - an efficient framework that leverages
evolution-based search to automatically find the best-performing layer-wise
sparse BERT model under the desired latency constraint. Different from existing
evolution algorithms that conduct random mutations, we propose a reinforced
mutator with a latency-aware multi-objective reward to conduct better mutations
for efficiently searching the large space of layer-wise sparse models.
Extensive experiments demonstrate that our method consistently achieves higher
ROC AUC and lower latency than the uniform sparse baseline and state-of-the-art
search methods. Remarkably, under our latency requirement of 1900us on CPU,
SwiftPruner achieves a 0.86% higher AUC than the state-of-the-art uniform
sparse baseline for BERT-Mini on a large scale real-world dataset. Online A/B
testing shows that our model also achieves a significant 11.7% cut in the ratio
of defective cold start ads with satisfactory real-time serving latency.
