  Rapid advancements over the years have helped machine learning models reach
previously hard-to-achieve goals, sometimes even exceeding human capabilities.
However, to attain the desired accuracy, the model sizes and in turn their
computational requirements have increased drastically. Thus, serving
predictions from these models to meet any target latency and cost requirements
of applications remains a key challenge, despite recent work in building
inference-serving systems as well as algorithmic approaches that dynamically
adapt models based on inputs. In this paper, we introduce a form of dynamism,
modality selection, where we adaptively choose modalities from inference inputs
while maintaining the model quality. We introduce MOSEL, an automated inference
serving system for multi-modal ML models that carefully picks input modalities
per request based on user-defined performance and accuracy requirements. MOSEL
exploits modality configurations extensively, improving system throughput by
3.6$\times$ with an accuracy guarantee and shortening job completion times by
11$\times$.
