  Next activity prediction aims to forecast the future behavior of running
process instances. Recent publications in this field predominantly employ deep
learning techniques and evaluate their prediction performance using publicly
available event logs. This paper presents empirical evidence that calls into
question the effectiveness of these current evaluation approaches. We show that
there is an enormous amount of example leakage in all of the commonly used
event logs, so that rather trivial prediction approaches perform almost as well
as ones that leverage deep learning. We further argue that designing robust
evaluations requires a more profound conceptual engagement with the topic of
next-activity prediction, and specifically with the notion of generalization to
new data. To this end, we present various prediction scenarios that necessitate
different types of generalization to guide future research.
