  AIOps (Artificial Intelligence for IT Operations) solutions leverage the
massive data produced during the operations of large-scale systems and machine
learning models to assist software engineers in their system operations. As
operation data produced in the field are subject to constant evolution from
factors like the changing operational environment and user base, the models in
AIOps solutions need to be constantly maintained after deployment. While prior
works focus on innovative modeling techniques to improve the performance of
AIOps models before releasing them into the field, when and how to maintain
AIOps models remain an under-investigated topic. In this work, we performed a
case study on three large-scale public operation data to assess different model
maintenance approaches regarding their performance, maintenance cost, and
stability. We observed that active model maintenance approaches achieve better
and more stable performance than a stationary approach. Particularly, applying
sophisticated model maintenance approaches (e.g., concept drift detection,
time-based ensembles, or online learning approaches) could provide better
performance, efficiency, and stability than simply retraining AIOps models
periodically. In addition, we observed that, although some maintenance
approaches (e.g., time-based ensemble and online learning) can save model
training time, they significantly sacrifice model testing time, which could
hinder their applications in AIOps solutions where the operation data arrive at
high speed and volume and where instant predictions are required. Our findings
highlight that practitioners should consider the evolution of operation data
and actively maintain AIOps models over time. Our observations can also guide
researchers and practitioners to investigate more efficient and effective model
maintenance techniques that fit in the context of AIOps.
