  [Context] Study replication is essential for theory building and empirical
validation. [Problem] Despite its empirical vocation, requirements engineering
(RE) research has given limited attention to study replication, threatening
thereby the ability to verify existing results and use previous research as a
baseline. [Solution] In this perspective paper, we -- a group of experts in
natural language processing (NLP) for RE -- reflect on the challenges for study
replication in NLP for RE. Concretely: (i) we report on hands-on experiences of
replication, (ii) we review the state-of-the-art and extract
replication-relevant information, and (iii) we identify, through focus groups,
challenges across two typical dimensions of replication: data annotation and
tool reconstruction. NLP for RE is a research area that is suitable for study
replication since it builds on automated tools which can be shared, and
quantitative evaluation that enable direct comparisons between results.
[Results] Replication is hampered by several factors, including the context
specificity of the studies, the heterogeneity of the tasks involving NLP, the
tasks' inherent hairiness, and, in turn, the heterogeneous reporting structure.
To address these issues, we propose an ID card whose goal is to provide a
structured summary of research papers, with an emphasis on replication-relevant
information. [Contribution] We contribute in this study with: (i) a set of
reflections on replication in NLP for RE, (ii) a set of recommendations for
researchers in the field to increase their awareness on the topic, and (iii) an
ID card that is intended to primarily foster replication, and can also be used
in other contexts, e.g., for educational purposes. Practitioners will also
benefit from the results since replications increase confidence on research
findings.
