  Recommender System (RS) is an important online application that affects
billions of users every day. The mainstream RS ranking framework is composed of
two parts: a Multi-Task Learning model (MTL) that predicts various user
feedback, i.e., clicks, likes, sharings, and a Multi-Task Fusion model (MTF)
that combines the multi-task outputs into one final ranking score with respect
to user satisfaction. There has not been much research on the fusion model
while it has great impact on the final recommendation as the last crucial
process of the ranking. To optimize long-term user satisfaction rather than
obtain instant returns greedily, we formulate MTF task as Markov Decision
Process (MDP) within a recommendation session and propose a Batch Reinforcement
Learning (RL) based Multi-Task Fusion framework (BatchRL-MTF) that includes a
Batch RL framework and an online exploration. The former exploits Batch RL to
learn an optimal recommendation policy from the fixed batch data offline for
long-term user satisfaction, while the latter explores potential high-value
actions online to break through the local optimal dilemma. With a comprehensive
investigation on user behaviors, we model the user satisfaction reward with
subtle heuristics from two aspects of user stickiness and user activeness.
Finally, we conduct extensive experiments on a billion-sample level real-world
dataset to show the effectiveness of our model. We propose a conservative
offline policy estimator (Conservative-OPEstimator) to test our model offline.
Furthermore, we take online experiments in a real recommendation environment to
compare performance of different models. As one of few Batch RL researches
applied in MTF task successfully, our model has also been deployed on a
large-scale industrial short video platform, serving hundreds of millions of
users.
