  This paper presents a novel framework for demystification of convolutional
deep learning models for time-series analysis. This is a step towards making
informed/explainable decisions in the domain of time-series, powered by deep
learning. There have been numerous efforts to increase the interpretability of
image-centric deep neural network models, where the learned features are more
intuitive to visualize. Visualization in time-series domain is much more
complicated as there is no direct interpretation of the filters and inputs as
compared to the image modality. In addition, little or no concentration has
been devoted for the development of such tools in the domain of time-series in
the past. TSViz provides possibilities to explore and analyze a network from
different dimensions at different levels of abstraction which includes
identification of parts of the input that were responsible for a prediction
(including per filter saliency), importance of different filters present in the
network for a particular prediction, notion of diversity present in the network
through filter clustering, understanding of the main sources of variation
learnt by the network through inverse optimization, and analysis of the
network's robustness against adversarial noise. As a sanity check for the
computed influence values, we demonstrate results regarding pruning of neural
networks based on the computed influence information. These representations
allow to understand the network features so that the acceptability of deep
networks for time-series data can be enhanced. This is extremely important in
domains like finance, industry 4.0, self-driving cars, health-care,
counter-terrorism etc., where reasons for reaching a particular prediction are
equally important as the prediction itself. We assess the proposed framework
for interpretability with a set of desirable properties essential for any
method.
