  Human beings always engage in a vast range of activities and tasks that
demonstrate their ability to adapt to different scenarios. Any human activity
can be represented as a temporal sequence of actions performed to achieve a
certain goal. Unlike the time series datasets extracted from electronics or
machines, these action sequences are highly disparate in their nature -- the
time to finish a sequence of actions can vary between different persons.
Therefore, understanding the dynamics of these sequences is essential for many
downstream tasks such as activity length prediction, goal prediction, next
action recommendation, etc. Existing neural network-based approaches that learn
a continuous-time activity sequence (or CTAS) are limited to the presence of
only visual data or are designed specifically for a particular task, i.e.,
limited to next action or goal prediction. In this paper, we present ProActive,
a neural marked temporal point process (MTPP) framework for modeling the
continuous-time distribution of actions in an activity sequence while
simultaneously addressing three high-impact problems -- next action prediction,
sequence-goal prediction, and end-to-end sequence generation. Specifically, we
utilize a self-attention module with temporal normalizing flows to model the
influence and the inter-arrival times between actions in a sequence. In
addition, we propose a novel addition over the ProActive model that can handle
variations in the order of actions, i.e., different methods of achieving a
given goal. We demonstrate that this variant can learn the order in which the
person or actor prefers to do their actions. Extensive experiments on sequences
derived from three activity recognition datasets show the significant accuracy
boost of ProActive over the state-of-the-art in terms of action and goal
prediction, and the first-ever application of end-to-end action sequence
generation.
