  Intelligent Tutoring Systems (ITSs) have been developed to provide students
with personalized learning experiences by adaptively generating learning paths
optimized for each individual. Within the vast scope of ITS, score prediction
stands out as an area of study that enables students to construct individually
realistic goals based on their current position. Via the expected score
provided by the ITS, a student can instantaneously compare one's expected score
to one's actual score, which directly corresponds to the reliability that the
ITS can instill. In other words, refining the precision of predicted scores
strictly correlates to the level of confidence that a student may have with an
ITS, which will evidently ensue improved student engagement. However, previous
studies have solely concentrated on improving the performance of a prediction
model, largely lacking focus on the benefits generated by its practical
application. In this paper, we demonstrate that the accuracy of the score
prediction model deployed in a real-world setting significantly impacts user
engagement by providing empirical evidence. To that end, we apply a
state-of-the-art deep attentive neural network-based score prediction model to
Santa, a multi-platform English ITS with approximately 780K users in South
Korea that exclusively focuses on the TOEIC (Test of English for International
Communications) standardized examinations. We run a controlled A/B test on the
ITS with two models, respectively based on collaborative filtering and deep
attentive neural networks, to verify whether the more accurate model engenders
any student engagement. The results conclude that the attentive model not only
induces high student morale (e.g. higher diagnostic test completion ratio,
number of questions answered, etc.) but also encourages active engagement (e.g.
higher purchase rate, improved total profit, etc.) on Santa.
