  Information-seeking conversation systems are increasingly popular in
real-world applications, especially for e-commerce companies. To retrieve
appropriate responses for users, it is necessary to compute the matching
degrees between candidate responses and users' queries with historical dialogue
utterances. As the contexts are usually much longer than responses, it is thus
necessary to expand the responses (usually short) with richer information.
Recent studies on pseudo-relevance feedback (PRF) have demonstrated its
effectiveness in query expansion for search engines, hence we consider
expanding response using PRF information. However, existing PRF approaches are
either based on heuristic rules or require heavy manual labeling, which are not
suitable for solving our task. To alleviate this problem, we treat the PRF
selection for response expansion as a learning task and propose a reinforced
learning method that can be trained in an end-to-end manner without any human
annotations. More specifically, we propose a reinforced selector to extract
useful PRF terms to enhance response candidates and a BERT-based response
ranker to rank the PRF-enhanced responses. The performance of the ranker serves
as a reward to guide the selector to extract useful PRF terms, which boosts the
overall task performance. Extensive experiments on both standard benchmarks and
commercial datasets prove the superiority of our reinforced PRF term selector
compared with other potential soft or hard selection methods. Both case studies
and quantitative analysis show that our model is capable of selecting
meaningful PRF terms to expand response candidates and also achieving the best
results compared with all baselines on a variety of evaluation metrics. We have
also deployed our method on online production in an e-commerce company, which
shows a significant improvement over the existing online ranking system.
