  Nowadays, Question Answering (QA) tasks receive significant research focus,
particularly with the development of Large Language Model (LLM) such as Chat
GPT [1]. LLM can be applied to various domains, but it contradicts the
principles of information transmission when applied to the Islamic domain. In
Islam we strictly regulates the sources of information and who can give
interpretations or tafseer for that sources [2]. The approach used by LLM to
generate answers based on its own interpretation is similar to the concept of
tafseer, LLM is neither an Islamic expert nor a human which is not permitted in
Islam. Indonesia is the country with the largest Islamic believer population in
the world [3]. With the high influence of LLM, we need to make evaluation of
LLM in religious domain. Currently, there is only few religious QA dataset
available and none of them using Sirah Nabawiyah especially in Indonesian
Language. In this paper, we propose the Question Answering Sirah Nabawiyah
(QASiNa) dataset, a novel dataset compiled from Sirah Nabawiyah literatures in
Indonesian language. We demonstrate our dataset by using mBERT [4], XLM-R [5],
and IndoBERT [6] which fine-tuned with Indonesian translation of SQuAD v2.0
[7]. XLM-R model returned the best performance on QASiNa with EM of 61.20,
F1-Score of 75.94, and Substring Match of 70.00. We compare XLM-R performance
with Chat GPT-3.5 and GPT-4 [1]. Both Chat GPT version returned lower EM and
F1-Score with higher Substring Match, the gap of EM and Substring Match get
wider in GPT-4. The experiment indicate that Chat GPT tends to give excessive
interpretations as evidenced by its higher Substring Match scores compared to
EM and F1-Score, even after providing instruction and context. This concludes
Chat GPT is unsuitable for question answering task in religious domain
especially for Islamic religion.
