We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Failed to Reproduce Human Evaluation of Controllable Dialogue with Pretrained Model
**Bug description**
I failed to reproduce the human evaluation of the controllable dialogue with pre-trained model. The dialogue log in `evaluation_logs/` folder. 

**Reproduction steps**
For the first dialogue in `evaluation_logs/baseline_model.jsonl`
*MetaData*
- model_persona: "i am terrified of clowns due to childhood trauma.\ni enjoy working with my hands and love gardening.\ni just got a job working in a gas station.\ni was adopted at a young age to an older couple.\nmy father keeps trying to set me up on blind dates."
- dialog:[{"speaker":"model","text":"Hello, how are you today?"},{"speaker":"human_evaluator","text":"hello! im doing well. how are you"},{"speaker":"model","text":"I am well, thank you. What do you do for a living?"},{"speaker":"human_evaluator","text":"i work part time at burger king making minimum wage. what about you"},{"speaker":"model","text":"I work in a gas station."},{"speaker":"human_evaluator","text":"what do you do for fun"},{"speaker":"model","text":"I work in a gas station. What about you?"},{"speaker":"human_evaluator","text":"i work at burger king"},{"speaker":"model","text":"What do you do for a living?"},{"speaker":"human_evaluator","text":"i live with my parents but i work part time in fast food"},{"speaker":"model","text":"What city are you from?"},{"speaker":"human_evaluator","text":"i am from boston. where are you"}]

Step 1. Open interactive chat with baseline agent.
```
python projects/controllable_dialogue/interactive.py \
-mf models:controllable_dialogue/convai2_finetuned_baseline
```
Step 2.
Enter your Message: `your persona: i am terrified of clowns due to childhood trauma.\nyour persona: i enjoy working with my hands and love gardening.\nyour persona: i just got a job working in a gas station.\nyour persona: i was adopted at a young age to an older couple.\nyour persona: my father keeps trying to set me up on blind dates.\n__SILENCE__`
Step 3.
Get agent response: `Hi how are you doing?`. Which does not match the evaluation log `Hello, how are you today?`

For all the 111 dialogues in `evaluation_logs/baseline_model.jsonl`, I can only reproduce 44 of them. 

**Expected behavior**
When the pre-trained model is given the same model persona and human response, I expect to reproduce the dialogue in `evaluation_logs`. 

**Logs**
Please paste the command line output:

The command line output is slightly different from the original script, where I compare the model output with the `evaluation_logs/baseline_model.jsonl`. 

The first dialogue in `evaluation_logs/baseline_model.jsonl`:
```
{'text': 'your persona: i am terrified of clowns due to childhood trauma.\nyour persona: i enjoy workingwith my hands and love gardening.\nyour persona: i just got a job working in a gas station.\nyour persona: i was adopted at a young age to an older couple.\nyour persona: my father keeps trying to set me up onblind dates.\n__SILENCE__', 'episode_done': False}
[PERSON_2]: Hi how are you doing?
[REFDATA]: Hello, how are you today?  
Warning! Mismatched
[Human]: hello! im doing well. how are you  
[PERSON_2]: I am good thanks for asking
[REFDATA]: I am well, thank you. What do you do for a living?  
Warning! Mismatched
[Human]: i work part time at burger king making minimum wage. what about you  
[PERSON_2]: I work in a gas station
[REFDATA]: I work in a gas station. 
Warning! Mismatched
[Human]: what do you do for fun
[PERSON_2]: I work at a gas station
[REFDATA]: I work in a gas station. What about you? 
Warning! Mismatched
[Human]: i work at burger king  
[PERSON_2]: What do you do for a living
[REFDATA]: What do you do for a living?  
Warning! Mismatched
[Human]: i live with my parents but i work part time in fast food 
[PERSON_2]: What city are you from?
[REFDATA]: What city are you from?
```

I can reproduce the second dialogue.

The third dialogue in `evaluation_logs/baseline_model.jsonl`:
```
{'text': 'your persona: i am a professional wrestler.\nyour persona: i enjoy being around people.\nyour persona: i like to workout a a times a week.\nyour persona: i volunteer in a homeless shelter.\nyour persona: in my spare time i do volunteer work.\n__SILENCE__', 'episode_done': False}
[PERSON_2]: Hello, how are you today?
[REFDATA]: Hi how are you today  
Warning! Mismatched
[Human]: im good and how are you  
[PERSON_2]: I am doing well. Just got back from a long workout.
[REFDATA]: I am good do you work  
Warning! Mismatched
[Human]: yes i do work and what about you  
[PERSON_2]: I am a professional wrestler.
[REFDATA]: I am a professional wrestler  
Warning! Mismatched
[Human]: thats cool i work online  
[PERSON_2]: What do you do online?
[REFDATA]: What do you do online  
Warning! Mismatched
[Human]: i work for a company to answer phones   
[PERSON_2]: That sounds like a lot of fun.
[REFDATA]: What do you do online  
Warning! Mismatched
[Human]: i  just told you  
[PERSON_2]: What do you do online?
[REFDATA]: What do you do for a living  
Warning! Mismatched
```

**Additional context**
I tried to roll back to different code-base versions and got no luck. 


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `projects/controllable_dialogue/tasks/build.py`
Content:
```
1 #!/usr/bin/env python3
2 
3 # Copyright (c) Facebook, Inc. and its affiliates.
4 # This source code is licensed under the MIT license found in the
5 # LICENSE file in the root directory of this source tree.
6 
7 import os
8 import parlai.core.params as params
9 import parlai.core.build_data as build_data
10 
11 
12 URL_ROOT = 'https://parl.ai/downloads/controllable_dialogue/'
13 FOLDER_NAME = 'controllable_dialogue'
14 
15 
16 def build(opt):
17     dpath = os.path.join(opt['datapath'], FOLDER_NAME)
18     # version 1.0: initial release
19     # version 1.1: add evaluation logs
20     version = '1.1'
21 
22     if not build_data.built(dpath, version_string=version):
23         if build_data.built(dpath):
24             # older version exists, so remove the outdated files.
25             build_data.remove_dir(dpath)
26         build_data.make_dir(dpath)
27 
28         # first download the data files
29         fname_data = 'data_v1.tar.gz'
30         build_data.download(URL_ROOT + fname_data, dpath, fname_data)
31         build_data.untar(dpath, fname_data)
32 
33         # next download the wordstats files
34         fname_wordstats = 'wordstats_v1.tar.gz'
35         build_data.download(URL_ROOT + fname_wordstats, dpath, fname_wordstats)
36         build_data.untar(dpath, fname_wordstats)
37 
38         # next download the evaluation logs
39         fname_evallogs = 'evaluationlogs_v1.tar.gz'
40         build_data.download(URL_ROOT + fname_evallogs, dpath, fname_evallogs)
41         build_data.untar(dpath, fname_evallogs)
42 
43         print("Data has been placed in " + dpath)
44 
45         build_data.mark_done(dpath, version)
46 
47 
48 def make_path(opt, fname):
49     return os.path.join(opt['datapath'], FOLDER_NAME, fname)
50 
51 
52 if __name__ == '__main__':
53     opt = params.ParlaiParser().parse_args(print_args=False)
54     build(opt)
55
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/projects/controllable_dialogue/tasks/build.py b/projects/controllable_dialogue/tasks/build.py
--- a/projects/controllable_dialogue/tasks/build.py
+++ b/projects/controllable_dialogue/tasks/build.py
@@ -17,7 +17,8 @@
     dpath = os.path.join(opt['datapath'], FOLDER_NAME)
     # version 1.0: initial release
     # version 1.1: add evaluation logs
-    version = '1.1'
+    # version 1.2: add reproducible evaluation logs
+    version = '1.2'
 
     if not build_data.built(dpath, version_string=version):
         if build_data.built(dpath):
@@ -40,6 +41,12 @@
         build_data.download(URL_ROOT + fname_evallogs, dpath, fname_evallogs)
         build_data.untar(dpath, fname_evallogs)
 
+        # and the reproducible logs.
+        # for more info see https://github.com/facebookresearch/ParlAI/issues/2855
+        fname_evallogs = 'evaluation_logs_reproducible_v1.tar.gz'
+        build_data.download(URL_ROOT + fname_evallogs, dpath, fname_evallogs)
+        build_data.untar(dpath, fname_evallogs)
+
         print("Data has been placed in " + dpath)
 
         build_data.mark_done(dpath, version)

{"golden_diff": "diff --git a/projects/controllable_dialogue/tasks/build.py b/projects/controllable_dialogue/tasks/build.py\n--- a/projects/controllable_dialogue/tasks/build.py\n+++ b/projects/controllable_dialogue/tasks/build.py\n@@ -17,7 +17,8 @@\n     dpath = os.path.join(opt['datapath'], FOLDER_NAME)\n     # version 1.0: initial release\n     # version 1.1: add evaluation logs\n-    version = '1.1'\n+    # version 1.2: add reproducible evaluation logs\n+    version = '1.2'\n \n     if not build_data.built(dpath, version_string=version):\n         if build_data.built(dpath):\n@@ -40,6 +41,12 @@\n         build_data.download(URL_ROOT + fname_evallogs, dpath, fname_evallogs)\n         build_data.untar(dpath, fname_evallogs)\n \n+        # and the reproducible logs.\n+        # for more info see https://github.com/facebookresearch/ParlAI/issues/2855\n+        fname_evallogs = 'evaluation_logs_reproducible_v1.tar.gz'\n+        build_data.download(URL_ROOT + fname_evallogs, dpath, fname_evallogs)\n+        build_data.untar(dpath, fname_evallogs)\n+\n         print(\"Data has been placed in \" + dpath)\n \n         build_data.mark_done(dpath, version)\n", "issue": "Failed to Reproduce Human Evaluation of Controllable Dialogue with Pretrained Model\n**Bug description**\r\nI failed to reproduce the human evaluation of the controllable dialogue with pre-trained model. The dialogue log in `evaluation_logs/` folder. \r\n\r\n**Reproduction steps**\r\nFor the first dialogue in `evaluation_logs/baseline_model.jsonl`\r\n*MetaData*\r\n- model_persona: \"i am terrified of clowns due to childhood trauma.\\ni enjoy working with my hands and love gardening.\\ni just got a job working in a gas station.\\ni was adopted at a young age to an older couple.\\nmy father keeps trying to set me up on blind dates.\"\r\n- dialog:[{\"speaker\":\"model\",\"text\":\"Hello, how are you today?\"},{\"speaker\":\"human_evaluator\",\"text\":\"hello! im doing well. how are you\"},{\"speaker\":\"model\",\"text\":\"I am well, thank you. What do you do for a living?\"},{\"speaker\":\"human_evaluator\",\"text\":\"i work part time at burger king making minimum wage. what about you\"},{\"speaker\":\"model\",\"text\":\"I work in a gas station.\"},{\"speaker\":\"human_evaluator\",\"text\":\"what do you do for fun\"},{\"speaker\":\"model\",\"text\":\"I work in a gas station. What about you?\"},{\"speaker\":\"human_evaluator\",\"text\":\"i work at burger king\"},{\"speaker\":\"model\",\"text\":\"What do you do for a living?\"},{\"speaker\":\"human_evaluator\",\"text\":\"i live with my parents but i work part time in fast food\"},{\"speaker\":\"model\",\"text\":\"What city are you from?\"},{\"speaker\":\"human_evaluator\",\"text\":\"i am from boston. where are you\"}]\r\n\r\nStep 1. Open interactive chat with baseline agent.\r\n```\r\npython projects/controllable_dialogue/interactive.py \\\r\n-mf models:controllable_dialogue/convai2_finetuned_baseline\r\n```\r\nStep 2.\r\nEnter your Message: `your persona: i am terrified of clowns due to childhood trauma.\\nyour persona: i enjoy working with my hands and love gardening.\\nyour persona: i just got a job working in a gas station.\\nyour persona: i was adopted at a young age to an older couple.\\nyour persona: my father keeps trying to set me up on blind dates.\\n__SILENCE__`\r\nStep 3.\r\nGet agent response: `Hi how are you doing?`. Which does not match the evaluation log `Hello, how are you today?`\r\n\r\nFor all the 111 dialogues in `evaluation_logs/baseline_model.jsonl`, I can only reproduce 44 of them. \r\n\r\n**Expected behavior**\r\nWhen the pre-trained model is given the same model persona and human response, I expect to reproduce the dialogue in `evaluation_logs`. \r\n\r\n**Logs**\r\nPlease paste the command line output:\r\n\r\nThe command line output is slightly different from the original script, where I compare the model output with the `evaluation_logs/baseline_model.jsonl`. \r\n\r\nThe first dialogue in `evaluation_logs/baseline_model.jsonl`:\r\n```\r\n{'text': 'your persona: i am terrified of clowns due to childhood trauma.\\nyour persona: i enjoy workingwith my hands and love gardening.\\nyour persona: i just got a job working in a gas station.\\nyour persona: i was adopted at a young age to an older couple.\\nyour persona: my father keeps trying to set me up onblind dates.\\n__SILENCE__', 'episode_done': False}\r\n[PERSON_2]: Hi how are you doing?\r\n[REFDATA]: Hello, how are you today?  \r\nWarning! Mismatched\r\n[Human]: hello! im doing well. how are you  \r\n[PERSON_2]: I am good thanks for asking\r\n[REFDATA]: I am well, thank you. What do you do for a living?  \r\nWarning! Mismatched\r\n[Human]: i work part time at burger king making minimum wage. what about you  \r\n[PERSON_2]: I work in a gas station\r\n[REFDATA]: I work in a gas station. \r\nWarning! Mismatched\r\n[Human]: what do you do for fun\r\n[PERSON_2]: I work at a gas station\r\n[REFDATA]: I work in a gas station. What about you? \r\nWarning! Mismatched\r\n[Human]: i work at burger king  \r\n[PERSON_2]: What do you do for a living\r\n[REFDATA]: What do you do for a living?  \r\nWarning! Mismatched\r\n[Human]: i live with my parents but i work part time in fast food \r\n[PERSON_2]: What city are you from?\r\n[REFDATA]: What city are you from?\r\n```\r\n\r\nI can reproduce the second dialogue.\r\n\r\nThe third dialogue in `evaluation_logs/baseline_model.jsonl`:\r\n```\r\n{'text': 'your persona: i am a professional wrestler.\\nyour persona: i enjoy being around people.\\nyour persona: i like to workout a a times a week.\\nyour persona: i volunteer in a homeless shelter.\\nyour persona: in my spare time i do volunteer work.\\n__SILENCE__', 'episode_done': False}\r\n[PERSON_2]: Hello, how are you today?\r\n[REFDATA]: Hi how are you today  \r\nWarning! Mismatched\r\n[Human]: im good and how are you  \r\n[PERSON_2]: I am doing well. Just got back from a long workout.\r\n[REFDATA]: I am good do you work  \r\nWarning! Mismatched\r\n[Human]: yes i do work and what about you  \r\n[PERSON_2]: I am a professional wrestler.\r\n[REFDATA]: I am a professional wrestler  \r\nWarning! Mismatched\r\n[Human]: thats cool i work online  \r\n[PERSON_2]: What do you do online?\r\n[REFDATA]: What do you do online  \r\nWarning! Mismatched\r\n[Human]: i work for a company to answer phones   \r\n[PERSON_2]: That sounds like a lot of fun.\r\n[REFDATA]: What do you do online  \r\nWarning! Mismatched\r\n[Human]: i  just told you  \r\n[PERSON_2]: What do you do online?\r\n[REFDATA]: What do you do for a living  \r\nWarning! Mismatched\r\n```\r\n\r\n**Additional context**\r\nI tried to roll back to different code-base versions and got no luck. \r\n\n", "before_files": [{"content": "#!/usr/bin/env python3\n\n# Copyright (c) Facebook, Inc. and its affiliates.\n# This source code is licensed under the MIT license found in the\n# LICENSE file in the root directory of this source tree.\n\nimport os\nimport parlai.core.params as params\nimport parlai.core.build_data as build_data\n\n\nURL_ROOT = 'https://parl.ai/downloads/controllable_dialogue/'\nFOLDER_NAME = 'controllable_dialogue'\n\n\ndef build(opt):\n    dpath = os.path.join(opt['datapath'], FOLDER_NAME)\n    # version 1.0: initial release\n    # version 1.1: add evaluation logs\n    version = '1.1'\n\n    if not build_data.built(dpath, version_string=version):\n        if build_data.built(dpath):\n            # older version exists, so remove the outdated files.\n            build_data.remove_dir(dpath)\n        build_data.make_dir(dpath)\n\n        # first download the data files\n        fname_data = 'data_v1.tar.gz'\n        build_data.download(URL_ROOT + fname_data, dpath, fname_data)\n        build_data.untar(dpath, fname_data)\n\n        # next download the wordstats files\n        fname_wordstats = 'wordstats_v1.tar.gz'\n        build_data.download(URL_ROOT + fname_wordstats, dpath, fname_wordstats)\n        build_data.untar(dpath, fname_wordstats)\n\n        # next download the evaluation logs\n        fname_evallogs = 'evaluationlogs_v1.tar.gz'\n        build_data.download(URL_ROOT + fname_evallogs, dpath, fname_evallogs)\n        build_data.untar(dpath, fname_evallogs)\n\n        print(\"Data has been placed in \" + dpath)\n\n        build_data.mark_done(dpath, version)\n\n\ndef make_path(opt, fname):\n    return os.path.join(opt['datapath'], FOLDER_NAME, fname)\n\n\nif __name__ == '__main__':\n    opt = params.ParlaiParser().parse_args(print_args=False)\n    build(opt)\n", "path": "projects/controllable_dialogue/tasks/build.py"}], "after_files": [{"content": "#!/usr/bin/env python3\n\n# Copyright (c) Facebook, Inc. and its affiliates.\n# This source code is licensed under the MIT license found in the\n# LICENSE file in the root directory of this source tree.\n\nimport os\nimport parlai.core.params as params\nimport parlai.core.build_data as build_data\n\n\nURL_ROOT = 'https://parl.ai/downloads/controllable_dialogue/'\nFOLDER_NAME = 'controllable_dialogue'\n\n\ndef build(opt):\n    dpath = os.path.join(opt['datapath'], FOLDER_NAME)\n    # version 1.0: initial release\n    # version 1.1: add evaluation logs\n    # version 1.2: add reproducible evaluation logs\n    version = '1.2'\n\n    if not build_data.built(dpath, version_string=version):\n        if build_data.built(dpath):\n            # older version exists, so remove the outdated files.\n            build_data.remove_dir(dpath)\n        build_data.make_dir(dpath)\n\n        # first download the data files\n        fname_data = 'data_v1.tar.gz'\n        build_data.download(URL_ROOT + fname_data, dpath, fname_data)\n        build_data.untar(dpath, fname_data)\n\n        # next download the wordstats files\n        fname_wordstats = 'wordstats_v1.tar.gz'\n        build_data.download(URL_ROOT + fname_wordstats, dpath, fname_wordstats)\n        build_data.untar(dpath, fname_wordstats)\n\n        # next download the evaluation logs\n        fname_evallogs = 'evaluationlogs_v1.tar.gz'\n        build_data.download(URL_ROOT + fname_evallogs, dpath, fname_evallogs)\n        build_data.untar(dpath, fname_evallogs)\n\n        # and the reproducible logs.\n        # for more info see https://github.com/facebookresearch/ParlAI/issues/2855\n        fname_evallogs = 'evaluation_logs_reproducible_v1.tar.gz'\n        build_data.download(URL_ROOT + fname_evallogs, dpath, fname_evallogs)\n        build_data.untar(dpath, fname_evallogs)\n\n        print(\"Data has been placed in \" + dpath)\n\n        build_data.mark_done(dpath, version)\n\n\ndef make_path(opt, fname):\n    return os.path.join(opt['datapath'], FOLDER_NAME, fname)\n\n\nif __name__ == '__main__':\n    opt = params.ParlaiParser().parse_args(print_args=False)\n    build(opt)\n", "path": "projects/controllable_dialogue/tasks/build.py"}]}