  Context: In recent years, leveraging machine learning (ML) techniques has
become one of the main solutions to tackle many software engineering (SE)
tasks, in research studies (ML4SE). This has been achieved by utilizing
state-of-the-art models that tend to be more complex and black-box, which is
led to less explainable solutions that reduce trust and uptake of ML4SE
solutions by professionals in the industry.
  Objective: One potential remedy is to offer explainable AI (XAI) methods to
provide the missing explainability. In this paper, we aim to explore to what
extent XAI has been studied in the SE community (XAI4SE) and provide a
comprehensive view of the current state-of-the-art as well as challenge and
roadmap for future work.
  Method: We conduct a systematic literature review on 24 (out of 869 primary
studies that were selected by keyword search) most relevant published studies
in XAI4SE. We have three research questions that were answered by meta-analysis
of the collected data per paper.
  Results: Our study reveals that among the identified studies, software
maintenance (\%68) and particularly defect prediction has the highest share on
the SE stages and tasks being studied. Additionally, we found that XAI methods
were mainly applied to classic ML models rather than more complex models. We
also noticed a clear lack of standard evaluation metrics for XAI methods in the
literature which has caused confusion among researchers and a lack of
benchmarks for comparisons.
  Conclusions: XAI has been identified as a helpful tool by most studies, which
we cover in the systematic review. However, XAI4SE is a relatively new domain
with a lot of untouched potentials, including the SE tasks to help with, the
ML4SE methods to explain, and the types of explanations to offer. This study
encourages the researchers to work on the identified challenges and roadmap
reported in the paper.
