  Deep learning (DL) has been a revolutionary technique in various domains. To
facilitate the model development and deployment, many deep learning frameworks
are proposed, among which PyTorch is one of the most popular solutions. The
performance of ecosystem around PyTorch is critically important, which saves
the costs of training models and reduces the response time of model inferences.
In this paper, we propose TorchBench, a novel benchmark suite to study the
performance of PyTorch software stack. Unlike existing benchmark suites,
TorchBench encloses many representative models, covering a large PyTorch API
surface. TorchBench is able to comprehensively characterize the performance of
the PyTorch software stack, guiding the performance optimization across models,
PyTorch framework, and GPU libraries. We show two practical use cases of
TorchBench. (1) We profile TorchBench to identify GPU performance
inefficiencies in PyTorch. We are able to optimize many performance bugs and
upstream patches to the official PyTorch repository. (2) We integrate
TorchBench into PyTorch continuous integration system. We are able to identify
performance regression in multiple daily code checkins to prevent PyTorch
repository from introducing performance bugs. TorchBench is open source and
keeps evolving.
