Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue
  Response Models   In this work, we study how the finetuning stage in the pretrain-finetune
framework changes the behavior of a pretrained neural language generator. We
focus on the transformer encoder-decoder model for the open-domain dialogue
response generation task. Our major finding is that after standard finetuning,
the model forgets some of the important language generation skills acquired
during large-scale pretraining. We demonstrate the forgetting phenomenon
through a set of detailed behavior analysis from the perspectives of knowledge
transfer, context sensitivity, and function space projection. As a preliminary
attempt to alleviate the forgetting problem, we propose an intuitive finetuning
strategy named "mix-review". We find that mix-review effectively regularizes
the finetuning process, and the forgetting problem is alleviated to some
extent. Finally, we discuss interesting behavior of the resulting dialogue
model and its implications.

3
The categories of this paper are: physics
32
The categories of this paper are: atom-ph
37
The categories of this paper are: supr-con
31
The categories of this paper are: ao-sci
11
The categories of this paper are: hep-lat
25
The categories of this paper are: patt-sol
28
The categories of this paper are: alg-geom
21
The categories of this paper are: adap-org
8
The categories of this paper are: hep-ex
7
The categories of this paper are: hep-th
35
The categories of this paper are: mtrl-th
16
The categories of this paper are: stat
29
The categories of this paper are: funct-an
12
The categories of this paper are: nucl-th
2
The categories of this paper are: math
14
The categories of this paper are: math-ph