  To perform their daily tasks, developers intensively make use of existing
resources by consulting open-source software (OSS) repositories. Such platforms
contain rich data sources, e.g., code snippets, documentation, and user
discussions, that can be useful for supporting development activities. Over the
last decades, several techniques and tools have been promoted to provide
developers with innovative features, aiming to bring in improvements in terms
of development effort, cost savings, and productivity. In the context of the EU
H2020 CROSSMINER project, a set of recommendation systems has been conceived to
assist software programmers in different phases of the development process. The
systems provide developers with various artifacts, such as third-party
libraries, documentation about how to use the APIs being adopted, or relevant
API function calls. To develop such recommendations, various technical choices
have been made to overcome issues related to several aspects including the lack
of baselines, limited data availability, decisions about the performance
measures, and evaluation approaches. This paper is an experience report to
present the knowledge pertinent to the set of recommendation systems developed
through the CROSSMINER project. We explain in detail the challenges we had to
deal with, together with the related lessons learned when developing and
evaluating these systems. Our aim is to provide the research community with
concrete takeaway messages that are expected to be useful for those who want to
develop or customize their own recommendation systems. The reported experiences
can facilitate interesting discussions and research work, which in the end
contribute to the advancement of recommendation systems applied to solve
different issues in Software Engineering.
