  In this paper, we investigate common pitfalls affecting the evaluation of
authentication systems based on touch dynamics. We consider different factors
that lead to misrepresented performance, are incompatible with stated system
and threat models or impede reproducibility and comparability with previous
work. Specifically, we investigate the effects of (i) small sample sizes (both
number of users and recording sessions), (ii) using different phone models in
training data, (iii) selecting non-contiguous training data, (iv) inserting
attacker samples in training data and (v) swipe aggregation. We perform a
systematic review of 30 touch dynamics papers showing that all of them overlook
at least one of these pitfalls. To quantify each pitfall's effect, we design a
set of experiments and collect a new longitudinal dataset of touch interactions
from 515 users over 31 days comprised of 1,194,451 unique strokes. Part of this
data is collected in-lab with Android devices and the rest remotely with iOS
devices, allowing us to make in-depth comparisons. We make this dataset and our
code available online. Our results show significant percentage-point changes in
reported mean EER for several pitfalls: including attacker data (2.55%),
non-contiguous training data (3.8%) and phone model mixing (3.2%-5.8%). We show
that, in a common evaluation setting, the cumulative effects of these
evaluation choices result in a combined difference of 8.9% EER. We also largely
observe these effects across the entire ROC curve. The pitfalls are evaluated
on four distinct classifiers - SVM, Random Forest, Neural Network, and kNN.
Furthermore, we explore additional considerations for fair evaluation when
building touch-based authentication systems and quantify their impacts. Based
on these insights, we propose a set of best practices that, will lead to more
realistic and comparable reporting of results in the field.
