Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Interpretation of Neural Networks is Fragile   In order for machine learning to be deployed and trusted in many
applications, it is crucial to be able to reliably explain why the machine
learning algorithm makes certain predictions. For example, if an algorithm
classifies a given pathology image to be a malignant tumor, then the doctor may
need to know which parts of the image led the algorithm to this classification.
How to interpret black-box predictors is thus an important and active area of
research. A fundamental question is: how much can we trust the interpretation
itself? In this paper, we show that interpretation of deep learning predictions
is extremely fragile in the following sense: two perceptively indistinguishable
inputs with the same predicted label can be assigned very different
interpretations. We systematically characterize the fragility of several
widely-used feature-importance interpretation methods (saliency maps, relevance
propagation, and DeepLIFT) on ImageNet and CIFAR-10. Our experiments show that
even small random perturbation can change the feature importance and new
systematic perturbations can lead to dramatically different interpretations
without changing the label. We extend these results to show that
interpretations based on exemplars (e.g. influence functions) are similarly
fragile. Our analysis of the geometry of the Hessian matrix gives insight on
why fragility could be a fundamental challenge to the current interpretation
approaches.

35
The categories of this paper are: mtrl-th
28
The categories of this paper are: alg-geom
37
The categories of this paper are: supr-con
23
The categories of this paper are: solv-int
27
The categories of this paper are: comp-gas
36
The categories of this paper are: cmp-lg
2
The categories of this paper are: math
31
The categories of this paper are: ao-sci
12
The categories of this paper are: nucl-th
10
The categories of this paper are: q-bio
24
The categories of this paper are: dg-ga
7
The categories of this paper are: hep-th
6
The categories of this paper are: astro-ph
25
The categories of this paper are: patt-sol
22
The categories of this paper are: chao-dyn
9
The categories of this paper are: nlin