  Large-scale cloud systems play a pivotal role in modern IT infrastructure.
However, incidents occurring within these systems can lead to service
disruptions and adversely affect user experience. To swiftly resolve such
incidents, on-call engineers depend on crafting domain-specific language (DSL)
queries to analyze telemetry data. However, writing these queries can be
challenging and time-consuming. This paper presents a thorough empirical study
on the utilization of queries of KQL, a DSL employed for incident management in
a large-scale cloud management system at Microsoft. The findings obtained
underscore the importance and viability of KQL queries recommendation to
enhance incident management.
  Building upon these valuable insights, we introduce Xpert, an end-to-end
machine learning framework that automates KQL recommendation process. By
leveraging historical incident data and large language models, Xpert generates
customized KQL queries tailored to new incidents. Furthermore, Xpert
incorporates a novel performance metric called Xcore, enabling a thorough
evaluation of query quality from three comprehensive perspectives. We conduct
extensive evaluations of Xpert, demonstrating its effectiveness in offline
settings. Notably, we deploy Xpert in the real production environment of a
large-scale incident management system in Microsoft, validating its efficiency
in supporting incident management. To the best of our knowledge, this paper
represents the first empirical study of its kind, and Xpert stands as a
pioneering DSL query recommendation framework designed for incident management.
