  Automobiles for our roadways are increasingly using advanced driver
assistance systems. The adoption of such new technologies requires us to
develop novel perception systems not only for accurately understanding the
situational context of these vehicles, but also to infer the driver's awareness
in differentiating between safe and critical situations. This manuscript
focuses on the specific problem of inferring driver awareness in the context of
attention analysis and hazardous incident activity. Even after the development
of wearable and compact multi-modal bio-sensing systems in recent years, their
application in driver awareness context has been scarcely explored.
The~capability of simultaneously recording different kinds of bio-sensing data
in addition to traditionally employed computer vision systems provides exciting
opportunities to explore the limitations of these sensor modalities. In this
work, we explore the applications of three different bio-sensing modalities
namely electroencephalogram (EEG), photoplethysmogram (PPG) and galvanic skin
response (GSR) along with a camera-based vision system in driver awareness
context. We assess the information from these sensors independently and
together using both signal processing- and deep learning-based tools. We show
that our methods outperform previously reported studies to classify driver
attention and detecting hazardous/non-hazardous situations for short time
scales of two seconds. We use EEG and vision data for high resolution temporal
classification (two seconds) while additionally also employing PPG and GSR over
longer time periods. We evaluate our methods by collecting user data on twelve
subjects for two real-world driving datasets among which one is publicly
available (KITTI dataset) while the other was collected by us (LISA dataset)
with the vehicle ....
