  Conversational search has seen increased recent attention in both the IR and
NLP communities. It seeks to clarify and solve a user's search need through
multi-turn natural language interactions. However, most existing systems are
trained and demonstrated with recorded or artificial conversation logs.
Eventually, conversational search systems should be trained, evaluated, and
deployed in an open-ended setting with unseen conversation trajectories. A key
challenge is that training and evaluating such systems both require a
human-in-the-loop, which is expensive and does not scale. One strategy for this
is to simulate users, thereby reducing the scaling costs. However, current user
simulators are either limited to only respond to yes-no questions from the
conversational search system, or unable to produce high quality responses in
general.
  In this paper, we show that current state-of-the-art user simulation system
could be significantly improved by replacing it with a smaller but advanced
natural language generation model. But rather than merely reporting this new
state-of-the-art, we present an in-depth investigation of the task of
simulating user response for conversational search. Our goal is to supplement
existing works with an insightful hand-analysis of what challenges are still
unsolved by the advanced model, as well as to propose our solutions for them.
The challenges we identified include (1) dataset noise, (2) a blind spot that
is difficult for existing models to learn, and (3) a specific type of
misevaluation in the standard empirical setup. Except for the dataset noise
issue, we propose solutions to cover the training blind spot and to avoid the
misevaluation. Our proposed solutions lead to further improvements. Our best
system improves the previous state-of-the-art significantly.
