  The wide popularity of short videos on social media poses new opportunities
and challenges to optimize recommender systems on the video-sharing platforms.
Users sequentially interact with the system and provide complex and
multi-faceted responses, including watch time and various types of interactions
with multiple videos. One the one hand, the platforms aims at optimizing the
users' cumulative watch time (main goal) in long term, which can be effectively
optimized by Reinforcement Learning. On the other hand, the platforms also
needs to satisfy the constraint of accommodating the responses of multiple user
interactions (auxiliary goals) such like, follow, share etc. In this paper, we
formulate the problem of short video recommendation as a Constrained Markov
Decision Process (CMDP). We find that traditional constrained reinforcement
learning algorithms can not work well in this setting. We propose a novel
two-stage constrained actor-critic method: At stage one, we learn individual
policies to optimize each auxiliary signal. At stage two, we learn a policy to
(i) optimize the main signal and (ii) stay close to policies learned at the
first stage, which effectively guarantees the performance of this main policy
on the auxiliaries. Through extensive offline evaluations, we demonstrate
effectiveness of our method over alternatives in both optimizing the main goal
as well as balancing the others. We further show the advantage of our method in
live experiments of short video recommendations, where it significantly
outperforms other baselines in terms of both watch time and interactions. Our
approach has been fully launched in the production system to optimize user
experiences on the platform.
