  Software Log anomaly event detection with masked event prediction has various
technical approaches with countless configurations and parameters. Our
objective is to provide a baseline of settings for similar studies in the
future. The models we use are the N-Gram model, which is a classic approach in
the field of natural language processing (NLP), and two deep learning (DL)
models long short-term memory (LSTM) and convolutional neural network (CNN).
For datasets we used four datasets Profilence, BlueGene/L (BGL), Hadoop
Distributed File System (HDFS) and Hadoop. Other settings are the size of the
sliding window which determines how many surrounding events we are using to
predict a given event, mask position (the position within the window we are
predicting), the usage of only unique sequences, and the portion of data that
is used for training. The results show clear indications of settings that can
be generalized across datasets. The performance of the DL models does not
deteriorate as the window size increases while the N-Gram model shows worse
performance with large window sizes on the BGL and Profilence datasets. Despite
the popularity of Next Event Prediction, the results show that in this context
it is better not to predict events at the edges of the subsequence, i.e., first
or last event, with the best result coming from predicting the fourth event
when the window size is five. Regarding the amount of data used for training,
the results show differences across datasets and models. For example, the
N-Gram model appears to be more sensitive toward the lack of data than the DL
models. Overall, for similar experimental setups we suggest the following
general baseline: Window size 10, mask position second to last, do not filter
out non-unique sequences, and use a half of the total data for training.
