We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[rllib] rllib train ... --checkpoint-at-end flag is ignored
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?

Ray 0.8.5
Python 3.7
MacOS Catalina
PyTorch 1.4.0

No checkpoint is produced when the `--checkpoint-at-end` flag is used. Adding `--checkpoint-freq 10` does cause checkpoints to be saved.

### Reproduction (REQUIRED)

```shell
rllib train --run PPO --env CartPole-v0 --stop='{"training_iteration": 25}' --ray-address auto --checkpoint-at-end
```

No checkpoint directory exists under `~/ray_results/default/PPO...`

Add the `--checkpoint-freq 10` flag:

```shell
rllib train --run PPO --env CartPole-v0 --stop='{"training_iteration": 25}' --ray-address auto --checkpoint-at-end --checkpoint-freq 10
```

Now there are `checkpoint_10` and `checkpoint_20` directories, but not a `checkpoint_25` at the end. 

Could the choice of stop criteria, `training_iteration`, have something to do with it?

> **Note:** Being persnickety, it bugs me that the final directory path for the checkpoints is `.../checkpoint_20/checkpoint-20` (underscore vs. dash). How about one or the other?

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [ ] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).

[rllib] rllib train ... --checkpoint-at-end flag is ignored
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?

Ray 0.8.5
Python 3.7
MacOS Catalina
PyTorch 1.4.0

No checkpoint is produced when the `--checkpoint-at-end` flag is used. Adding `--checkpoint-freq 10` does cause checkpoints to be saved.

### Reproduction (REQUIRED)

```shell
rllib train --run PPO --env CartPole-v0 --stop='{"training_iteration": 25}' --ray-address auto --checkpoint-at-end
```

No checkpoint directory exists under `~/ray_results/default/PPO...`

Add the `--checkpoint-freq 10` flag:

```shell
rllib train --run PPO --env CartPole-v0 --stop='{"training_iteration": 25}' --ray-address auto --checkpoint-at-end --checkpoint-freq 10
```

Now there are `checkpoint_10` and `checkpoint_20` directories, but not a `checkpoint_25` at the end. 

Could the choice of stop criteria, `training_iteration`, have something to do with it?

> **Note:** Being persnickety, it bugs me that the final directory path for the checkpoints is `.../checkpoint_20/checkpoint-20` (underscore vs. dash). How about one or the other?

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [ ] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `rllib/train.py`
Content:
```
1 #!/usr/bin/env python
2 
3 import argparse
4 import os
5 from pathlib import Path
6 import yaml
7 
8 import ray
9 from ray.cluster_utils import Cluster
10 from ray.tune.config_parser import make_parser
11 from ray.tune.result import DEFAULT_RESULTS_DIR
12 from ray.tune.resources import resources_to_json
13 from ray.tune.tune import _make_scheduler, run_experiments
14 from ray.rllib.utils.framework import try_import_tf, try_import_torch
15 
16 # Try to import both backends for flag checking/warnings.
17 tf = try_import_tf()
18 torch, _ = try_import_torch()
19 
20 EXAMPLE_USAGE = """
21 Training example via RLlib CLI:
22     rllib train --run DQN --env CartPole-v0
23 
24 Grid search example via RLlib CLI:
25     rllib train -f tuned_examples/cartpole-grid-search-example.yaml
26 
27 Grid search example via executable:
28     ./train.py -f tuned_examples/cartpole-grid-search-example.yaml
29 
30 Note that -f overrides all other trial-specific command-line options.
31 """
32 
33 
34 def create_parser(parser_creator=None):
35     parser = make_parser(
36         parser_creator=parser_creator,
37         formatter_class=argparse.RawDescriptionHelpFormatter,
38         description="Train a reinforcement learning agent.",
39         epilog=EXAMPLE_USAGE)
40 
41     # See also the base parser definition in ray/tune/config_parser.py
42     parser.add_argument(
43         "--ray-address",
44         default=None,
45         type=str,
46         help="Connect to an existing Ray cluster at this address instead "
47         "of starting a new one.")
48     parser.add_argument(
49         "--no-ray-ui",
50         action="store_true",
51         help="Whether to disable the Ray web ui.")
52     parser.add_argument(
53         "--local-mode",
54         action="store_true",
55         help="Whether to run ray with `local_mode=True`. "
56         "Only if --ray-num-nodes is not used.")
57     parser.add_argument(
58         "--ray-num-cpus",
59         default=None,
60         type=int,
61         help="--num-cpus to use if starting a new cluster.")
62     parser.add_argument(
63         "--ray-num-gpus",
64         default=None,
65         type=int,
66         help="--num-gpus to use if starting a new cluster.")
67     parser.add_argument(
68         "--ray-num-nodes",
69         default=None,
70         type=int,
71         help="Emulate multiple cluster nodes for debugging.")
72     parser.add_argument(
73         "--ray-redis-max-memory",
74         default=None,
75         type=int,
76         help="--redis-max-memory to use if starting a new cluster.")
77     parser.add_argument(
78         "--ray-memory",
79         default=None,
80         type=int,
81         help="--memory to use if starting a new cluster.")
82     parser.add_argument(
83         "--ray-object-store-memory",
84         default=None,
85         type=int,
86         help="--object-store-memory to use if starting a new cluster.")
87     parser.add_argument(
88         "--experiment-name",
89         default="default",
90         type=str,
91         help="Name of the subdirectory under `local_dir` to put results in.")
92     parser.add_argument(
93         "--local-dir",
94         default=DEFAULT_RESULTS_DIR,
95         type=str,
96         help="Local dir to save training results to. Defaults to '{}'.".format(
97             DEFAULT_RESULTS_DIR))
98     parser.add_argument(
99         "--upload-dir",
100         default="",
101         type=str,
102         help="Optional URI to sync training results to (e.g. s3://bucket).")
103     parser.add_argument(
104         "-v", action="store_true", help="Whether to use INFO level logging.")
105     parser.add_argument(
106         "-vv", action="store_true", help="Whether to use DEBUG level logging.")
107     parser.add_argument(
108         "--resume",
109         action="store_true",
110         help="Whether to attempt to resume previous Tune experiments.")
111     parser.add_argument(
112         "--torch",
113         action="store_true",
114         help="Whether to use PyTorch (instead of tf) as the DL framework.")
115     parser.add_argument(
116         "--eager",
117         action="store_true",
118         help="Whether to attempt to enable TF eager execution.")
119     parser.add_argument(
120         "--trace",
121         action="store_true",
122         help="Whether to attempt to enable tracing for eager mode.")
123     parser.add_argument(
124         "--env", default=None, type=str, help="The gym environment to use.")
125     parser.add_argument(
126         "--queue-trials",
127         action="store_true",
128         help=(
129             "Whether to queue trials when the cluster does not currently have "
130             "enough resources to launch one. This should be set to True when "
131             "running on an autoscaling cluster to enable automatic scale-up."))
132     parser.add_argument(
133         "-f",
134         "--config-file",
135         default=None,
136         type=str,
137         help="If specified, use config options from this file. Note that this "
138         "overrides any trial-specific options set via flags above.")
139     return parser
140 
141 
142 def run(args, parser):
143     if args.config_file:
144         with open(args.config_file) as f:
145             experiments = yaml.safe_load(f)
146     else:
147         # Note: keep this in sync with tune/config_parser.py
148         experiments = {
149             args.experiment_name: {  # i.e. log to ~/ray_results/default
150                 "run": args.run,
151                 "checkpoint_freq": args.checkpoint_freq,
152                 "keep_checkpoints_num": args.keep_checkpoints_num,
153                 "checkpoint_score_attr": args.checkpoint_score_attr,
154                 "local_dir": args.local_dir,
155                 "resources_per_trial": (
156                     args.resources_per_trial and
157                     resources_to_json(args.resources_per_trial)),
158                 "stop": args.stop,
159                 "config": dict(args.config, env=args.env),
160                 "restore": args.restore,
161                 "num_samples": args.num_samples,
162                 "upload_dir": args.upload_dir,
163             }
164         }
165 
166     verbose = 1
167     for exp in experiments.values():
168         # Bazel makes it hard to find files specified in `args` (and `data`).
169         # Look for them here.
170         # NOTE: Some of our yaml files don't have a `config` section.
171         if exp.get("config", {}).get("input") and \
172                 not os.path.exists(exp["config"]["input"]):
173             # This script runs in the ray/rllib dir.
174             rllib_dir = Path(__file__).parent
175             input_file = rllib_dir.absolute().joinpath(exp["config"]["input"])
176             exp["config"]["input"] = str(input_file)
177 
178         if not exp.get("run"):
179             parser.error("the following arguments are required: --run")
180         if not exp.get("env") and not exp.get("config", {}).get("env"):
181             parser.error("the following arguments are required: --env")
182         if args.eager:
183             exp["config"]["framework"] = "tfe"
184         elif args.torch:
185             exp["config"]["framework"] = "torch"
186         else:
187             exp["config"]["framework"] = "tf"
188         if args.v:
189             exp["config"]["log_level"] = "INFO"
190             verbose = 2
191         if args.vv:
192             exp["config"]["log_level"] = "DEBUG"
193             verbose = 3
194         if args.trace:
195             if exp["config"]["framework"] != "tfe":
196                 raise ValueError("Must enable --eager to enable tracing.")
197             exp["config"]["eager_tracing"] = True
198 
199     if args.ray_num_nodes:
200         cluster = Cluster()
201         for _ in range(args.ray_num_nodes):
202             cluster.add_node(
203                 num_cpus=args.ray_num_cpus or 1,
204                 num_gpus=args.ray_num_gpus or 0,
205                 object_store_memory=args.ray_object_store_memory,
206                 memory=args.ray_memory,
207                 redis_max_memory=args.ray_redis_max_memory)
208         ray.init(address=cluster.address)
209     else:
210         ray.init(
211             include_webui=not args.no_ray_ui,
212             address=args.ray_address,
213             object_store_memory=args.ray_object_store_memory,
214             memory=args.ray_memory,
215             redis_max_memory=args.ray_redis_max_memory,
216             num_cpus=args.ray_num_cpus,
217             num_gpus=args.ray_num_gpus,
218             local_mode=args.local_mode)
219     run_experiments(
220         experiments,
221         scheduler=_make_scheduler(args),
222         queue_trials=args.queue_trials,
223         resume=args.resume,
224         verbose=verbose,
225         concurrent=True)
226 
227 
228 if __name__ == "__main__":
229     parser = create_parser()
230     args = parser.parse_args()
231     run(args, parser)
232
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/rllib/train.py b/rllib/train.py
--- a/rllib/train.py
+++ b/rllib/train.py
@@ -149,6 +149,7 @@
             args.experiment_name: {  # i.e. log to ~/ray_results/default
                 "run": args.run,
                 "checkpoint_freq": args.checkpoint_freq,
+                "checkpoint_at_end": args.checkpoint_at_end,
                 "keep_checkpoints_num": args.keep_checkpoints_num,
                 "checkpoint_score_attr": args.checkpoint_score_attr,
                 "local_dir": args.local_dir,

{"golden_diff": "diff --git a/rllib/train.py b/rllib/train.py\n--- a/rllib/train.py\n+++ b/rllib/train.py\n@@ -149,6 +149,7 @@\n             args.experiment_name: {  # i.e. log to ~/ray_results/default\n                 \"run\": args.run,\n                 \"checkpoint_freq\": args.checkpoint_freq,\n+                \"checkpoint_at_end\": args.checkpoint_at_end,\n                 \"keep_checkpoints_num\": args.keep_checkpoints_num,\n                 \"checkpoint_score_attr\": args.checkpoint_score_attr,\n                 \"local_dir\": args.local_dir,\n", "issue": "[rllib] rllib train ... --checkpoint-at-end flag is ignored\n<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->\r\n\r\n### What is the problem?\r\n\r\nRay 0.8.5\r\nPython 3.7\r\nMacOS Catalina\r\nPyTorch 1.4.0\r\n\r\nNo checkpoint is produced when the `--checkpoint-at-end` flag is used. Adding `--checkpoint-freq 10` does cause checkpoints to be saved.\r\n\r\n### Reproduction (REQUIRED)\r\n\r\n```shell\r\nrllib train --run PPO --env CartPole-v0 --stop='{\"training_iteration\": 25}' --ray-address auto --checkpoint-at-end\r\n```\r\n\r\nNo checkpoint directory exists under `~/ray_results/default/PPO...`\r\n\r\nAdd the `--checkpoint-freq 10` flag:\r\n\r\n```shell\r\nrllib train --run PPO --env CartPole-v0 --stop='{\"training_iteration\": 25}' --ray-address auto --checkpoint-at-end --checkpoint-freq 10\r\n```\r\n\r\nNow there are `checkpoint_10` and `checkpoint_20` directories, but not a `checkpoint_25` at the end. \r\n\r\nCould the choice of stop criteria, `training_iteration`, have something to do with it?\r\n\r\n> **Note:** Being persnickety, it bugs me that the final directory path for the checkpoints is `.../checkpoint_20/checkpoint-20` (underscore vs. dash). How about one or the other?\r\n\r\n- [x] I have verified my script runs in a clean environment and reproduces the issue.\r\n- [ ] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).\r\n\n[rllib] rllib train ... --checkpoint-at-end flag is ignored\n<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->\r\n\r\n### What is the problem?\r\n\r\nRay 0.8.5\r\nPython 3.7\r\nMacOS Catalina\r\nPyTorch 1.4.0\r\n\r\nNo checkpoint is produced when the `--checkpoint-at-end` flag is used. Adding `--checkpoint-freq 10` does cause checkpoints to be saved.\r\n\r\n### Reproduction (REQUIRED)\r\n\r\n```shell\r\nrllib train --run PPO --env CartPole-v0 --stop='{\"training_iteration\": 25}' --ray-address auto --checkpoint-at-end\r\n```\r\n\r\nNo checkpoint directory exists under `~/ray_results/default/PPO...`\r\n\r\nAdd the `--checkpoint-freq 10` flag:\r\n\r\n```shell\r\nrllib train --run PPO --env CartPole-v0 --stop='{\"training_iteration\": 25}' --ray-address auto --checkpoint-at-end --checkpoint-freq 10\r\n```\r\n\r\nNow there are `checkpoint_10` and `checkpoint_20` directories, but not a `checkpoint_25` at the end. \r\n\r\nCould the choice of stop criteria, `training_iteration`, have something to do with it?\r\n\r\n> **Note:** Being persnickety, it bugs me that the final directory path for the checkpoints is `.../checkpoint_20/checkpoint-20` (underscore vs. dash). How about one or the other?\r\n\r\n- [x] I have verified my script runs in a clean environment and reproduces the issue.\r\n- [ ] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).\r\n\n", "before_files": [{"content": "#!/usr/bin/env python\n\nimport argparse\nimport os\nfrom pathlib import Path\nimport yaml\n\nimport ray\nfrom ray.cluster_utils import Cluster\nfrom ray.tune.config_parser import make_parser\nfrom ray.tune.result import DEFAULT_RESULTS_DIR\nfrom ray.tune.resources import resources_to_json\nfrom ray.tune.tune import _make_scheduler, run_experiments\nfrom ray.rllib.utils.framework import try_import_tf, try_import_torch\n\n# Try to import both backends for flag checking/warnings.\ntf = try_import_tf()\ntorch, _ = try_import_torch()\n\nEXAMPLE_USAGE = \"\"\"\nTraining example via RLlib CLI:\n    rllib train --run DQN --env CartPole-v0\n\nGrid search example via RLlib CLI:\n    rllib train -f tuned_examples/cartpole-grid-search-example.yaml\n\nGrid search example via executable:\n    ./train.py -f tuned_examples/cartpole-grid-search-example.yaml\n\nNote that -f overrides all other trial-specific command-line options.\n\"\"\"\n\n\ndef create_parser(parser_creator=None):\n    parser = make_parser(\n        parser_creator=parser_creator,\n        formatter_class=argparse.RawDescriptionHelpFormatter,\n        description=\"Train a reinforcement learning agent.\",\n        epilog=EXAMPLE_USAGE)\n\n    # See also the base parser definition in ray/tune/config_parser.py\n    parser.add_argument(\n        \"--ray-address\",\n        default=None,\n        type=str,\n        help=\"Connect to an existing Ray cluster at this address instead \"\n        \"of starting a new one.\")\n    parser.add_argument(\n        \"--no-ray-ui\",\n        action=\"store_true\",\n        help=\"Whether to disable the Ray web ui.\")\n    parser.add_argument(\n        \"--local-mode\",\n        action=\"store_true\",\n        help=\"Whether to run ray with `local_mode=True`. \"\n        \"Only if --ray-num-nodes is not used.\")\n    parser.add_argument(\n        \"--ray-num-cpus\",\n        default=None,\n        type=int,\n        help=\"--num-cpus to use if starting a new cluster.\")\n    parser.add_argument(\n        \"--ray-num-gpus\",\n        default=None,\n        type=int,\n        help=\"--num-gpus to use if starting a new cluster.\")\n    parser.add_argument(\n        \"--ray-num-nodes\",\n        default=None,\n        type=int,\n        help=\"Emulate multiple cluster nodes for debugging.\")\n    parser.add_argument(\n        \"--ray-redis-max-memory\",\n        default=None,\n        type=int,\n        help=\"--redis-max-memory to use if starting a new cluster.\")\n    parser.add_argument(\n        \"--ray-memory\",\n        default=None,\n        type=int,\n        help=\"--memory to use if starting a new cluster.\")\n    parser.add_argument(\n        \"--ray-object-store-memory\",\n        default=None,\n        type=int,\n        help=\"--object-store-memory to use if starting a new cluster.\")\n    parser.add_argument(\n        \"--experiment-name\",\n        default=\"default\",\n        type=str,\n        help=\"Name of the subdirectory under `local_dir` to put results in.\")\n    parser.add_argument(\n        \"--local-dir\",\n        default=DEFAULT_RESULTS_DIR,\n        type=str,\n        help=\"Local dir to save training results to. Defaults to '{}'.\".format(\n            DEFAULT_RESULTS_DIR))\n    parser.add_argument(\n        \"--upload-dir\",\n        default=\"\",\n        type=str,\n        help=\"Optional URI to sync training results to (e.g. s3://bucket).\")\n    parser.add_argument(\n        \"-v\", action=\"store_true\", help=\"Whether to use INFO level logging.\")\n    parser.add_argument(\n        \"-vv\", action=\"store_true\", help=\"Whether to use DEBUG level logging.\")\n    parser.add_argument(\n        \"--resume\",\n        action=\"store_true\",\n        help=\"Whether to attempt to resume previous Tune experiments.\")\n    parser.add_argument(\n        \"--torch\",\n        action=\"store_true\",\n        help=\"Whether to use PyTorch (instead of tf) as the DL framework.\")\n    parser.add_argument(\n        \"--eager\",\n        action=\"store_true\",\n        help=\"Whether to attempt to enable TF eager execution.\")\n    parser.add_argument(\n        \"--trace\",\n        action=\"store_true\",\n        help=\"Whether to attempt to enable tracing for eager mode.\")\n    parser.add_argument(\n        \"--env\", default=None, type=str, help=\"The gym environment to use.\")\n    parser.add_argument(\n        \"--queue-trials\",\n        action=\"store_true\",\n        help=(\n            \"Whether to queue trials when the cluster does not currently have \"\n            \"enough resources to launch one. This should be set to True when \"\n            \"running on an autoscaling cluster to enable automatic scale-up.\"))\n    parser.add_argument(\n        \"-f\",\n        \"--config-file\",\n        default=None,\n        type=str,\n        help=\"If specified, use config options from this file. Note that this \"\n        \"overrides any trial-specific options set via flags above.\")\n    return parser\n\n\ndef run(args, parser):\n    if args.config_file:\n        with open(args.config_file) as f:\n            experiments = yaml.safe_load(f)\n    else:\n        # Note: keep this in sync with tune/config_parser.py\n        experiments = {\n            args.experiment_name: {  # i.e. log to ~/ray_results/default\n                \"run\": args.run,\n                \"checkpoint_freq\": args.checkpoint_freq,\n                \"keep_checkpoints_num\": args.keep_checkpoints_num,\n                \"checkpoint_score_attr\": args.checkpoint_score_attr,\n                \"local_dir\": args.local_dir,\n                \"resources_per_trial\": (\n                    args.resources_per_trial and\n                    resources_to_json(args.resources_per_trial)),\n                \"stop\": args.stop,\n                \"config\": dict(args.config, env=args.env),\n                \"restore\": args.restore,\n                \"num_samples\": args.num_samples,\n                \"upload_dir\": args.upload_dir,\n            }\n        }\n\n    verbose = 1\n    for exp in experiments.values():\n        # Bazel makes it hard to find files specified in `args` (and `data`).\n        # Look for them here.\n        # NOTE: Some of our yaml files don't have a `config` section.\n        if exp.get(\"config\", {}).get(\"input\") and \\\n                not os.path.exists(exp[\"config\"][\"input\"]):\n            # This script runs in the ray/rllib dir.\n            rllib_dir = Path(__file__).parent\n            input_file = rllib_dir.absolute().joinpath(exp[\"config\"][\"input\"])\n            exp[\"config\"][\"input\"] = str(input_file)\n\n        if not exp.get(\"run\"):\n            parser.error(\"the following arguments are required: --run\")\n        if not exp.get(\"env\") and not exp.get(\"config\", {}).get(\"env\"):\n            parser.error(\"the following arguments are required: --env\")\n        if args.eager:\n            exp[\"config\"][\"framework\"] = \"tfe\"\n        elif args.torch:\n            exp[\"config\"][\"framework\"] = \"torch\"\n        else:\n            exp[\"config\"][\"framework\"] = \"tf\"\n        if args.v:\n            exp[\"config\"][\"log_level\"] = \"INFO\"\n            verbose = 2\n        if args.vv:\n            exp[\"config\"][\"log_level\"] = \"DEBUG\"\n            verbose = 3\n        if args.trace:\n            if exp[\"config\"][\"framework\"] != \"tfe\":\n                raise ValueError(\"Must enable --eager to enable tracing.\")\n            exp[\"config\"][\"eager_tracing\"] = True\n\n    if args.ray_num_nodes:\n        cluster = Cluster()\n        for _ in range(args.ray_num_nodes):\n            cluster.add_node(\n                num_cpus=args.ray_num_cpus or 1,\n                num_gpus=args.ray_num_gpus or 0,\n                object_store_memory=args.ray_object_store_memory,\n                memory=args.ray_memory,\n                redis_max_memory=args.ray_redis_max_memory)\n        ray.init(address=cluster.address)\n    else:\n        ray.init(\n            include_webui=not args.no_ray_ui,\n            address=args.ray_address,\n            object_store_memory=args.ray_object_store_memory,\n            memory=args.ray_memory,\n            redis_max_memory=args.ray_redis_max_memory,\n            num_cpus=args.ray_num_cpus,\n            num_gpus=args.ray_num_gpus,\n            local_mode=args.local_mode)\n    run_experiments(\n        experiments,\n        scheduler=_make_scheduler(args),\n        queue_trials=args.queue_trials,\n        resume=args.resume,\n        verbose=verbose,\n        concurrent=True)\n\n\nif __name__ == \"__main__\":\n    parser = create_parser()\n    args = parser.parse_args()\n    run(args, parser)\n", "path": "rllib/train.py"}], "after_files": [{"content": "#!/usr/bin/env python\n\nimport argparse\nimport os\nfrom pathlib import Path\nimport yaml\n\nimport ray\nfrom ray.cluster_utils import Cluster\nfrom ray.tune.config_parser import make_parser\nfrom ray.tune.result import DEFAULT_RESULTS_DIR\nfrom ray.tune.resources import resources_to_json\nfrom ray.tune.tune import _make_scheduler, run_experiments\nfrom ray.rllib.utils.framework import try_import_tf, try_import_torch\n\n# Try to import both backends for flag checking/warnings.\ntf = try_import_tf()\ntorch, _ = try_import_torch()\n\nEXAMPLE_USAGE = \"\"\"\nTraining example via RLlib CLI:\n    rllib train --run DQN --env CartPole-v0\n\nGrid search example via RLlib CLI:\n    rllib train -f tuned_examples/cartpole-grid-search-example.yaml\n\nGrid search example via executable:\n    ./train.py -f tuned_examples/cartpole-grid-search-example.yaml\n\nNote that -f overrides all other trial-specific command-line options.\n\"\"\"\n\n\ndef create_parser(parser_creator=None):\n    parser = make_parser(\n        parser_creator=parser_creator,\n        formatter_class=argparse.RawDescriptionHelpFormatter,\n        description=\"Train a reinforcement learning agent.\",\n        epilog=EXAMPLE_USAGE)\n\n    # See also the base parser definition in ray/tune/config_parser.py\n    parser.add_argument(\n        \"--ray-address\",\n        default=None,\n        type=str,\n        help=\"Connect to an existing Ray cluster at this address instead \"\n        \"of starting a new one.\")\n    parser.add_argument(\n        \"--no-ray-ui\",\n        action=\"store_true\",\n        help=\"Whether to disable the Ray web ui.\")\n    parser.add_argument(\n        \"--local-mode\",\n        action=\"store_true\",\n        help=\"Whether to run ray with `local_mode=True`. \"\n        \"Only if --ray-num-nodes is not used.\")\n    parser.add_argument(\n        \"--ray-num-cpus\",\n        default=None,\n        type=int,\n        help=\"--num-cpus to use if starting a new cluster.\")\n    parser.add_argument(\n        \"--ray-num-gpus\",\n        default=None,\n        type=int,\n        help=\"--num-gpus to use if starting a new cluster.\")\n    parser.add_argument(\n        \"--ray-num-nodes\",\n        default=None,\n        type=int,\n        help=\"Emulate multiple cluster nodes for debugging.\")\n    parser.add_argument(\n        \"--ray-redis-max-memory\",\n        default=None,\n        type=int,\n        help=\"--redis-max-memory to use if starting a new cluster.\")\n    parser.add_argument(\n        \"--ray-memory\",\n        default=None,\n        type=int,\n        help=\"--memory to use if starting a new cluster.\")\n    parser.add_argument(\n        \"--ray-object-store-memory\",\n        default=None,\n        type=int,\n        help=\"--object-store-memory to use if starting a new cluster.\")\n    parser.add_argument(\n        \"--experiment-name\",\n        default=\"default\",\n        type=str,\n        help=\"Name of the subdirectory under `local_dir` to put results in.\")\n    parser.add_argument(\n        \"--local-dir\",\n        default=DEFAULT_RESULTS_DIR,\n        type=str,\n        help=\"Local dir to save training results to. Defaults to '{}'.\".format(\n            DEFAULT_RESULTS_DIR))\n    parser.add_argument(\n        \"--upload-dir\",\n        default=\"\",\n        type=str,\n        help=\"Optional URI to sync training results to (e.g. s3://bucket).\")\n    parser.add_argument(\n        \"-v\", action=\"store_true\", help=\"Whether to use INFO level logging.\")\n    parser.add_argument(\n        \"-vv\", action=\"store_true\", help=\"Whether to use DEBUG level logging.\")\n    parser.add_argument(\n        \"--resume\",\n        action=\"store_true\",\n        help=\"Whether to attempt to resume previous Tune experiments.\")\n    parser.add_argument(\n        \"--torch\",\n        action=\"store_true\",\n        help=\"Whether to use PyTorch (instead of tf) as the DL framework.\")\n    parser.add_argument(\n        \"--eager\",\n        action=\"store_true\",\n        help=\"Whether to attempt to enable TF eager execution.\")\n    parser.add_argument(\n        \"--trace\",\n        action=\"store_true\",\n        help=\"Whether to attempt to enable tracing for eager mode.\")\n    parser.add_argument(\n        \"--env\", default=None, type=str, help=\"The gym environment to use.\")\n    parser.add_argument(\n        \"--queue-trials\",\n        action=\"store_true\",\n        help=(\n            \"Whether to queue trials when the cluster does not currently have \"\n            \"enough resources to launch one. This should be set to True when \"\n            \"running on an autoscaling cluster to enable automatic scale-up.\"))\n    parser.add_argument(\n        \"-f\",\n        \"--config-file\",\n        default=None,\n        type=str,\n        help=\"If specified, use config options from this file. Note that this \"\n        \"overrides any trial-specific options set via flags above.\")\n    return parser\n\n\ndef run(args, parser):\n    if args.config_file:\n        with open(args.config_file) as f:\n            experiments = yaml.safe_load(f)\n    else:\n        # Note: keep this in sync with tune/config_parser.py\n        experiments = {\n            args.experiment_name: {  # i.e. log to ~/ray_results/default\n                \"run\": args.run,\n                \"checkpoint_freq\": args.checkpoint_freq,\n                \"checkpoint_at_end\": args.checkpoint_at_end,\n                \"keep_checkpoints_num\": args.keep_checkpoints_num,\n                \"checkpoint_score_attr\": args.checkpoint_score_attr,\n                \"local_dir\": args.local_dir,\n                \"resources_per_trial\": (\n                    args.resources_per_trial and\n                    resources_to_json(args.resources_per_trial)),\n                \"stop\": args.stop,\n                \"config\": dict(args.config, env=args.env),\n                \"restore\": args.restore,\n                \"num_samples\": args.num_samples,\n                \"upload_dir\": args.upload_dir,\n            }\n        }\n\n    verbose = 1\n    for exp in experiments.values():\n        # Bazel makes it hard to find files specified in `args` (and `data`).\n        # Look for them here.\n        # NOTE: Some of our yaml files don't have a `config` section.\n        if exp.get(\"config\", {}).get(\"input\") and \\\n                not os.path.exists(exp[\"config\"][\"input\"]):\n            # This script runs in the ray/rllib dir.\n            rllib_dir = Path(__file__).parent\n            input_file = rllib_dir.absolute().joinpath(exp[\"config\"][\"input\"])\n            exp[\"config\"][\"input\"] = str(input_file)\n\n        if not exp.get(\"run\"):\n            parser.error(\"the following arguments are required: --run\")\n        if not exp.get(\"env\") and not exp.get(\"config\", {}).get(\"env\"):\n            parser.error(\"the following arguments are required: --env\")\n        if args.eager:\n            exp[\"config\"][\"framework\"] = \"tfe\"\n        elif args.torch:\n            exp[\"config\"][\"framework\"] = \"torch\"\n        else:\n            exp[\"config\"][\"framework\"] = \"tf\"\n        if args.v:\n            exp[\"config\"][\"log_level\"] = \"INFO\"\n            verbose = 2\n        if args.vv:\n            exp[\"config\"][\"log_level\"] = \"DEBUG\"\n            verbose = 3\n        if args.trace:\n            if exp[\"config\"][\"framework\"] != \"tfe\":\n                raise ValueError(\"Must enable --eager to enable tracing.\")\n            exp[\"config\"][\"eager_tracing\"] = True\n\n    if args.ray_num_nodes:\n        cluster = Cluster()\n        for _ in range(args.ray_num_nodes):\n            cluster.add_node(\n                num_cpus=args.ray_num_cpus or 1,\n                num_gpus=args.ray_num_gpus or 0,\n                object_store_memory=args.ray_object_store_memory,\n                memory=args.ray_memory,\n                redis_max_memory=args.ray_redis_max_memory)\n        ray.init(address=cluster.address)\n    else:\n        ray.init(\n            include_webui=not args.no_ray_ui,\n            address=args.ray_address,\n            object_store_memory=args.ray_object_store_memory,\n            memory=args.ray_memory,\n            redis_max_memory=args.ray_redis_max_memory,\n            num_cpus=args.ray_num_cpus,\n            num_gpus=args.ray_num_gpus,\n            local_mode=args.local_mode)\n    run_experiments(\n        experiments,\n        scheduler=_make_scheduler(args),\n        queue_trials=args.queue_trials,\n        resume=args.resume,\n        verbose=verbose,\n        concurrent=True)\n\n\nif __name__ == \"__main__\":\n    parser = create_parser()\n    args = parser.parse_args()\n    run(args, parser)\n", "path": "rllib/train.py"}]}