text: Network testing plays an important role in the iterative process of developing new communication protocols and algorithms. However, test environments have to keep up with the evolution of technology and require continuous update and redesign. In this article, we propose COINS, a framework that can be used by wireless technology developers to enable CI practices in their testbed infrastructure. As a proof-of-concept, we provide a reference architecture and implementation of COINS for controlled testing of multi-technology 5G MTC networks. The implementation upgrades an existing wireless experimentation testbed with new software and hardware functionalities. It blends web service technology and operating system virtualization technologies with emerging Internet of Things technologies enabling CI for wireless networks. Moreover, we also extend an existing qualitative methodology for comparing similar frameworks and identify and discuss open challenges for wider use of CI practices in wireless technology development.
  @cite_14: The Berlin Open Wireless Lab (BOWL) project at Technische Universitat Berlin (TUB) maintains an outdoor WiFi network which is used both for Internet access and as a testbed for wireless research. From the very beginning of the BOWL project, we experienced several development and operations challenges to keep Internet users and researchers happy. Development challenges included allowing multiple researchers with very different requirements to run experiments in the network while maintaining reliable Internet access. On the operations side, one of the recent issues we faced was authentication of users from different domains, which required us to integrate with various external authentication services. In this paper, we present our experience in handling these challenges on both development and operations sides and the lessons we learned.
  @cite_7: Networking testbeds are playing an increasingly important role in the development of new communication technologies. Testbeds are traditionally built for a particular project or to study a specific technology. An alternative approach is to federate existing testbeds to a) cater for experimenter needs which cannot be fullled by a single testbed, and b) provide a wider variety of environmental settings at different scales. These heterogenous settings allow the study of new approaches in environments similar to what one finds in the real world. This paper presents OMF, a control, measurement, and management framework for testbeds. It describes through some examples the versatility of OMF's current architecture and gives directions for federation of testbeds through OMF. In addition, this paper introduces a comprehensive experiment description language that allows an experimenter to describe resource requirements and their configurations, as well as experiment orchestration. Researchers would thus be able to reproduce their experiment on the same testbed or in a different environment with little changes. Along with the efficient support for large scale experiments, the use of testbeds and support for repeatable experiments will allow the networking field to build a culture of cross verification and therefore strengthen its scientific approach.
  @cite_2: Experimental platforms (testbeds) play a significant role in the evaluation of new and existing technologies. Their popularity has been raised lately as more and more researchers prefer experimentation over simulation as a way for acquiring more accurate results. This imposes significant challenges in testbed operators since an efficient mechanism is needed to manage the testbed's resources and provision them according to the users' needs. In this paper we describe such a framework which was implemented for the management of networking testbeds. We present the design requirements and the implementation details, along with the challenges we encountered during its operation in the NITOS testbed. Significant results were extracted through the experiences of the every day operation of the testbed's management.
summary: While we did not find any other framework addressing wireless experimentation with CI support, we have identified three existing frameworks that are to some extent similar to COINS, i.e., the control and Management Framework (OMF) @cite_7 , the Network Implementation Testbed Laboratory (NITOS) @cite_2 and the Berlin Open Wireless Lab (BOWL) @cite_14 . We performed a feature-wise comparison of these frameworks and summarized it in Table ,.