  Background: The lack of explanations for the decisions made by algorithms
such as deep learning has hampered their acceptance by the clinical community
despite highly accurate results on multiple problems. Recently, attribution
methods have emerged for explaining deep learning models, and they have been
tested on medical imaging problems. The performance of attribution methods is
compared on standard machine learning datasets and not on medical images. In
this study, we perform a comparative analysis to determine the most suitable
explainability method for retinal OCT diagnosis.
  Methods: A commonly used deep learning model known as Inception v3 was
trained to diagnose 3 retinal diseases - choroidal neovascularization (CNV),
diabetic macular edema (DME), and drusen. The explanations from 13 different
attribution methods were rated by a panel of 14 clinicians for clinical
significance. Feedback was obtained from the clinicians regarding the current
and future scope of such methods.
  Results: An attribution method based on a Taylor series expansion, called
Deep Taylor was rated the highest by clinicians with a median rating of 3.85/5.
It was followed by two other attribution methods, Guided backpropagation and
SHAP (SHapley Additive exPlanations).
  Conclusion: Explanations of deep learning models can make them more
transparent for clinical diagnosis. This study compared different explanations
methods in the context of retinal OCT diagnosis and found that the best
performing method may not be the one considered best for other deep learning
tasks. Overall, there was a high degree of acceptance from the clinicians
surveyed in the study.
  Keywords: explainable AI, deep learning, machine learning, image processing,
Optical coherence tomography, retina, Diabetic macular edema, Choroidal
Neovascularization, Drusen
