  Recent advances in Deep Neural Networks (DNNs) have led to the development of
DNN-driven autonomous cars that, using sensors like camera, LiDAR, etc., can
drive without any human intervention. Most major manufacturers including Tesla,
GM, Ford, BMW, and Waymo/Google are working on building and testing different
types of autonomous vehicles. The lawmakers of several US states including
California, Texas, and New York have passed new legislation to fast-track the
process of testing and deployment of autonomous vehicles on their roads.
  However, despite their spectacular progress, DNNs, just like traditional
software, often demonstrate incorrect or unexpected corner case behaviors that
can lead to potentially fatal collisions. Several such real-world accidents
involving autonomous cars have already happened including one which resulted in
a fatality. Most existing testing techniques for DNN-driven vehicles are
heavily dependent on the manual collection of test data under different driving
conditions which become prohibitively expensive as the number of test
conditions increases.
  In this paper, we design, implement and evaluate DeepTest, a systematic
testing tool for automatically detecting erroneous behaviors of DNN-driven
vehicles that can potentially lead to fatal crashes. First, our tool is
designed to automatically generated test cases leveraging real-world changes in
driving conditions like rain, fog, lighting conditions, etc. DeepTest
systematically explores different parts of the DNN logic by generating test
inputs that maximize the numbers of activated neurons. DeepTest found thousands
of erroneous behaviors under different realistic driving conditions (e.g.,
blurring, rain, fog, etc.) many of which lead to potentially fatal crashes in
three top performing DNNs in the Udacity self-driving car challenge.
