We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[rllib] Integer entropy coeff cannot be passed in
<!--
General questions should be asked on the mailing list ray-dev@googlegroups.com.
Questions about how to use Ray should be asked on
[StackOverflow](https://stackoverflow.com/questions/tagged/ray).

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Steropes
- **Ray installed from (source or binary)**: pip install -U <latest whl>
- **Ray version**: nightly
- **Python version**: 3.7
- **Exact command to reproduce**: Pass integer value of entropy_coeff into run() with PPO

<!--
You can obtain the Ray version with

python -c "import ray; print(ray.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->
```
2019-09-11 00:11:50,889 ERROR trial_runner.py:552 -- Error processing event.
Traceback (most recent call last):
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 498, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/tune/ray_trial_executor.py", line 347, in fetch_result
    result = ray.get(trial_future[0])
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 2340, in get
    raise value
ray.exceptions.RayTaskError: ray_PPO:train() (pid=11050, host=steropes)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 527, in _apply_op_helper
    preferred_dtype=default_dtype)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1224, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1018, in _TensorTensorConversionFunction
    (dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype int32 for Tensor with dtype float32: 'Tensor("default_policy/Sum_5:0", shape=(?,), dtype=float32)'

During handling of the above exception, another exception occurred:

ray_PPO:train() (pid=11050, host=steropes)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
    Trainer.__init__(self, config, env, logger_creator)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/trainer.py", line 366, in __init__
    Trainable.__init__(self, config, logger_creator)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/tune/trainable.py", line 99, in __init__
    self._setup(copy.deepcopy(self.config))
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/trainer.py", line 486, in _setup
    self._init(self.config, self.env_creator)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/trainer_template.py", line 109, in _init
    self.config["num_workers"])
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/trainer.py", line 531, in _make_workers
    logdir=self.logdir)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/evaluation/worker_set.py", line 64, in __init__
    RolloutWorker, env_creator, policy, 0, self._local_config)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/evaluation/worker_set.py", line 220, in _make_worker
    _fake_sampler=config.get("_fake_sampler", False))
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/evaluation/rollout_worker.py", line 348, in __init__
    self._build_policy_map(policy_dict, policy_config)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/evaluation/rollout_worker.py", line 762, in _build_policy_map
    policy_map[name] = cls(obs_space, act_space, merged_conf)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/policy/tf_policy_template.py", line 143, in __init__
    obs_include_prev_action_reward=obs_include_prev_action_reward)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/policy/dynamic_tf_policy.py", line 196, in __init__
    self._initialize_loss()
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/policy/dynamic_tf_policy.py", line 337, in _initialize_loss
    loss = self._do_loss_init(train_batch)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/policy/dynamic_tf_policy.py", line 349, in _do_loss_init
    loss = self._loss_fn(self, self.model, self._dist_class, train_batch)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/ppo/ppo_policy.py", line 146, in ppo_surrogate_loss
    model_config=policy.config["model"])
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/ppo/ppo_policy.py", line 106, in __init__
    vf_loss_coeff * vf_loss - entropy_coeff * curr_entropy)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 1045, in _run_op
    return tensor_oper(a.value(), *args, **kwargs)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 884, in binary_op_wrapper
    return func(x, y, name=name)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 1180, in _mul_dispatch
    return gen_math_ops.mul(x, y, name=name)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 6490, in mul
    "Mul", x=x, y=y, name=name)
  File "/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 563, in _apply_op_helper
    inferred_from[input_arg.type_attr]))
TypeError: Input 'y' of 'Mul' Op has type float32 that does not match type int32 of argument 'x'.
```

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `rllib/agents/ppo/ppo.py`
Content:
```
1 from __future__ import absolute_import
2 from __future__ import division
3 from __future__ import print_function
4 
5 import logging
6 
7 from ray.rllib.agents import with_common_config
8 from ray.rllib.agents.ppo.ppo_policy import PPOTFPolicy
9 from ray.rllib.agents.trainer_template import build_trainer
10 from ray.rllib.optimizers import SyncSamplesOptimizer, LocalMultiGPUOptimizer
11 from ray.rllib.utils import try_import_tf
12 
13 tf = try_import_tf()
14 logger = logging.getLogger(__name__)
15 
16 # yapf: disable
17 # __sphinx_doc_begin__
18 DEFAULT_CONFIG = with_common_config({
19     # If true, use the Generalized Advantage Estimator (GAE)
20     # with a value function, see https://arxiv.org/pdf/1506.02438.pdf.
21     "use_gae": True,
22     # GAE(lambda) parameter
23     "lambda": 1.0,
24     # Initial coefficient for KL divergence
25     "kl_coeff": 0.2,
26     # Size of batches collected from each worker
27     "sample_batch_size": 200,
28     # Number of timesteps collected for each SGD round
29     "train_batch_size": 4000,
30     # Total SGD batch size across all devices for SGD
31     "sgd_minibatch_size": 128,
32     # Whether to shuffle sequences in the batch when training (recommended)
33     "shuffle_sequences": True,
34     # Number of SGD iterations in each outer loop
35     "num_sgd_iter": 30,
36     # Stepsize of SGD
37     "lr": 5e-5,
38     # Learning rate schedule
39     "lr_schedule": None,
40     # Share layers for value function. If you set this to True, it's important
41     # to tune vf_loss_coeff.
42     "vf_share_layers": False,
43     # Coefficient of the value function loss. It's important to tune this if
44     # you set vf_share_layers: True
45     "vf_loss_coeff": 1.0,
46     # Coefficient of the entropy regularizer
47     "entropy_coeff": 0.0,
48     # Decay schedule for the entropy regularizer
49     "entropy_coeff_schedule": None,
50     # PPO clip parameter
51     "clip_param": 0.3,
52     # Clip param for the value function. Note that this is sensitive to the
53     # scale of the rewards. If your expected V is large, increase this.
54     "vf_clip_param": 10.0,
55     # If specified, clip the global norm of gradients by this amount
56     "grad_clip": None,
57     # Target value for KL divergence
58     "kl_target": 0.01,
59     # Whether to rollout "complete_episodes" or "truncate_episodes"
60     "batch_mode": "truncate_episodes",
61     # Which observation filter to apply to the observation
62     "observation_filter": "NoFilter",
63     # Uses the sync samples optimizer instead of the multi-gpu one. This does
64     # not support minibatches.
65     "simple_optimizer": False,
66 })
67 # __sphinx_doc_end__
68 # yapf: enable
69 
70 
71 def choose_policy_optimizer(workers, config):
72     if config["simple_optimizer"]:
73         return SyncSamplesOptimizer(
74             workers,
75             num_sgd_iter=config["num_sgd_iter"],
76             train_batch_size=config["train_batch_size"],
77             sgd_minibatch_size=config["sgd_minibatch_size"])
78 
79     return LocalMultiGPUOptimizer(
80         workers,
81         sgd_batch_size=config["sgd_minibatch_size"],
82         num_sgd_iter=config["num_sgd_iter"],
83         num_gpus=config["num_gpus"],
84         sample_batch_size=config["sample_batch_size"],
85         num_envs_per_worker=config["num_envs_per_worker"],
86         train_batch_size=config["train_batch_size"],
87         standardize_fields=["advantages"],
88         shuffle_sequences=config["shuffle_sequences"])
89 
90 
91 def update_kl(trainer, fetches):
92     if "kl" in fetches:
93         # single-agent
94         trainer.workers.local_worker().for_policy(
95             lambda pi: pi.update_kl(fetches["kl"]))
96     else:
97 
98         def update(pi, pi_id):
99             if pi_id in fetches:
100                 pi.update_kl(fetches[pi_id]["kl"])
101             else:
102                 logger.debug("No data for {}, not updating kl".format(pi_id))
103 
104         # multi-agent
105         trainer.workers.local_worker().foreach_trainable_policy(update)
106 
107 
108 def warn_about_bad_reward_scales(trainer, result):
109     # Warn about bad clipping configs
110     if trainer.config["vf_clip_param"] <= 0:
111         rew_scale = float("inf")
112     elif result["policy_reward_mean"]:
113         rew_scale = 0  # punt on handling multiagent case
114     else:
115         rew_scale = round(
116             abs(result["episode_reward_mean"]) /
117             trainer.config["vf_clip_param"], 0)
118     if rew_scale > 200:
119         logger.warning(
120             "The magnitude of your environment rewards are more than "
121             "{}x the scale of `vf_clip_param`. ".format(rew_scale) +
122             "This means that it will take more than "
123             "{} iterations for your value ".format(rew_scale) +
124             "function to converge. If this is not intended, consider "
125             "increasing `vf_clip_param`.")
126 
127 
128 def validate_config(config):
129     if config["entropy_coeff"] < 0:
130         raise DeprecationWarning("entropy_coeff must be >= 0")
131     if config["sgd_minibatch_size"] > config["train_batch_size"]:
132         raise ValueError(
133             "Minibatch size {} must be <= train batch size {}.".format(
134                 config["sgd_minibatch_size"], config["train_batch_size"]))
135     if config["batch_mode"] == "truncate_episodes" and not config["use_gae"]:
136         raise ValueError(
137             "Episode truncation is not supported without a value "
138             "function. Consider setting batch_mode=complete_episodes.")
139     if config["multiagent"]["policies"] and not config["simple_optimizer"]:
140         logger.info(
141             "In multi-agent mode, policies will be optimized sequentially "
142             "by the multi-GPU optimizer. Consider setting "
143             "simple_optimizer=True if this doesn't work for you.")
144     if config["simple_optimizer"]:
145         logger.warning(
146             "Using the simple minibatch optimizer. This will significantly "
147             "reduce performance, consider simple_optimizer=False.")
148     elif tf and tf.executing_eagerly():
149         config["simple_optimizer"] = True  # multi-gpu not supported
150 
151 
152 PPOTrainer = build_trainer(
153     name="PPO",
154     default_config=DEFAULT_CONFIG,
155     default_policy=PPOTFPolicy,
156     make_policy_optimizer=choose_policy_optimizer,
157     validate_config=validate_config,
158     after_optimizer_step=update_kl,
159     after_train_result=warn_about_bad_reward_scales)
160
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/rllib/agents/ppo/ppo.py b/rllib/agents/ppo/ppo.py
--- a/rllib/agents/ppo/ppo.py
+++ b/rllib/agents/ppo/ppo.py
@@ -128,6 +128,8 @@
 def validate_config(config):
     if config["entropy_coeff"] < 0:
         raise DeprecationWarning("entropy_coeff must be >= 0")
+    if isinstance(config["entropy_coeff"], int):
+        config["entropy_coeff"] = float(config["entropy_coeff"])
     if config["sgd_minibatch_size"] > config["train_batch_size"]:
         raise ValueError(
             "Minibatch size {} must be <= train batch size {}.".format(

{"golden_diff": "diff --git a/rllib/agents/ppo/ppo.py b/rllib/agents/ppo/ppo.py\n--- a/rllib/agents/ppo/ppo.py\n+++ b/rllib/agents/ppo/ppo.py\n@@ -128,6 +128,8 @@\n def validate_config(config):\n     if config[\"entropy_coeff\"] < 0:\n         raise DeprecationWarning(\"entropy_coeff must be >= 0\")\n+    if isinstance(config[\"entropy_coeff\"], int):\n+        config[\"entropy_coeff\"] = float(config[\"entropy_coeff\"])\n     if config[\"sgd_minibatch_size\"] > config[\"train_batch_size\"]:\n         raise ValueError(\n             \"Minibatch size {} must be <= train batch size {}.\".format(\n", "issue": "[rllib] Integer entropy coeff cannot be passed in\n<!--\r\nGeneral questions should be asked on the mailing list ray-dev@googlegroups.com.\r\nQuestions about how to use Ray should be asked on\r\n[StackOverflow](https://stackoverflow.com/questions/tagged/ray).\r\n\r\nBefore submitting an issue, please fill out the following form.\r\n-->\r\n\r\n### System information\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Steropes\r\n- **Ray installed from (source or binary)**: pip install -U <latest whl>\r\n- **Ray version**: nightly\r\n- **Python version**: 3.7\r\n- **Exact command to reproduce**: Pass integer value of entropy_coeff into run() with PPO\r\n\r\n<!--\r\nYou can obtain the Ray version with\r\n\r\npython -c \"import ray; print(ray.__version__)\"\r\n-->\r\n\r\n### Describe the problem\r\n<!-- Describe the problem clearly here. -->\r\n\r\n### Source code / logs\r\n<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->\r\n```\r\n2019-09-11 00:11:50,889 ERROR trial_runner.py:552 -- Error processing event.\r\nTraceback (most recent call last):\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/tune/trial_runner.py\", line 498, in _process_trial\r\n    result = self.trial_executor.fetch_result(trial)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/tune/ray_trial_executor.py\", line 347, in fetch_result\r\n    result = ray.get(trial_future[0])\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/worker.py\", line 2340, in get\r\n    raise value\r\nray.exceptions.RayTaskError: ray_PPO:train() (pid=11050, host=steropes)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py\", line 527, in _apply_op_helper\r\n    preferred_dtype=default_dtype)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py\", line 1224, in internal_convert_to_tensor\r\n    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py\", line 1018, in _TensorTensorConversionFunction\r\n    (dtype.name, t.dtype.name, str(t)))\r\nValueError: Tensor conversion requested dtype int32 for Tensor with dtype float32: 'Tensor(\"default_policy/Sum_5:0\", shape=(?,), dtype=float32)'\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nray_PPO:train() (pid=11050, host=steropes)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/trainer_template.py\", line 90, in __init__\r\n    Trainer.__init__(self, config, env, logger_creator)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/trainer.py\", line 366, in __init__\r\n    Trainable.__init__(self, config, logger_creator)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/tune/trainable.py\", line 99, in __init__\r\n    self._setup(copy.deepcopy(self.config))\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/trainer.py\", line 486, in _setup\r\n    self._init(self.config, self.env_creator)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/trainer_template.py\", line 109, in _init\r\n    self.config[\"num_workers\"])\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/trainer.py\", line 531, in _make_workers\r\n    logdir=self.logdir)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/evaluation/worker_set.py\", line 64, in __init__\r\n    RolloutWorker, env_creator, policy, 0, self._local_config)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/evaluation/worker_set.py\", line 220, in _make_worker\r\n    _fake_sampler=config.get(\"_fake_sampler\", False))\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/evaluation/rollout_worker.py\", line 348, in __init__\r\n    self._build_policy_map(policy_dict, policy_config)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/evaluation/rollout_worker.py\", line 762, in _build_policy_map\r\n    policy_map[name] = cls(obs_space, act_space, merged_conf)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/policy/tf_policy_template.py\", line 143, in __init__\r\n    obs_include_prev_action_reward=obs_include_prev_action_reward)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/policy/dynamic_tf_policy.py\", line 196, in __init__\r\n    self._initialize_loss()\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/policy/dynamic_tf_policy.py\", line 337, in _initialize_loss\r\n    loss = self._do_loss_init(train_batch)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/policy/dynamic_tf_policy.py\", line 349, in _do_loss_init\r\n    loss = self._loss_fn(self, self.model, self._dist_class, train_batch)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/ppo/ppo_policy.py\", line 146, in ppo_surrogate_loss\r\n    model_config=policy.config[\"model\"])\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/ray/rllib/agents/ppo/ppo_policy.py\", line 106, in __init__\r\n    vf_loss_coeff * vf_loss - entropy_coeff * curr_entropy)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/variables.py\", line 1045, in _run_op\r\n    return tensor_oper(a.value(), *args, **kwargs)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py\", line 884, in binary_op_wrapper\r\n    return func(x, y, name=name)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py\", line 1180, in _mul_dispatch\r\n    return gen_math_ops.mul(x, y, name=name)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py\", line 6490, in mul\r\n    \"Mul\", x=x, y=y, name=name)\r\n  File \"/data/ashwineep/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py\", line 563, in _apply_op_helper\r\n    inferred_from[input_arg.type_attr]))\r\nTypeError: Input 'y' of 'Mul' Op has type float32 that does not match type int32 of argument 'x'.\r\n```\n", "before_files": [{"content": "from __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport logging\n\nfrom ray.rllib.agents import with_common_config\nfrom ray.rllib.agents.ppo.ppo_policy import PPOTFPolicy\nfrom ray.rllib.agents.trainer_template import build_trainer\nfrom ray.rllib.optimizers import SyncSamplesOptimizer, LocalMultiGPUOptimizer\nfrom ray.rllib.utils import try_import_tf\n\ntf = try_import_tf()\nlogger = logging.getLogger(__name__)\n\n# yapf: disable\n# __sphinx_doc_begin__\nDEFAULT_CONFIG = with_common_config({\n    # If true, use the Generalized Advantage Estimator (GAE)\n    # with a value function, see https://arxiv.org/pdf/1506.02438.pdf.\n    \"use_gae\": True,\n    # GAE(lambda) parameter\n    \"lambda\": 1.0,\n    # Initial coefficient for KL divergence\n    \"kl_coeff\": 0.2,\n    # Size of batches collected from each worker\n    \"sample_batch_size\": 200,\n    # Number of timesteps collected for each SGD round\n    \"train_batch_size\": 4000,\n    # Total SGD batch size across all devices for SGD\n    \"sgd_minibatch_size\": 128,\n    # Whether to shuffle sequences in the batch when training (recommended)\n    \"shuffle_sequences\": True,\n    # Number of SGD iterations in each outer loop\n    \"num_sgd_iter\": 30,\n    # Stepsize of SGD\n    \"lr\": 5e-5,\n    # Learning rate schedule\n    \"lr_schedule\": None,\n    # Share layers for value function. If you set this to True, it's important\n    # to tune vf_loss_coeff.\n    \"vf_share_layers\": False,\n    # Coefficient of the value function loss. It's important to tune this if\n    # you set vf_share_layers: True\n    \"vf_loss_coeff\": 1.0,\n    # Coefficient of the entropy regularizer\n    \"entropy_coeff\": 0.0,\n    # Decay schedule for the entropy regularizer\n    \"entropy_coeff_schedule\": None,\n    # PPO clip parameter\n    \"clip_param\": 0.3,\n    # Clip param for the value function. Note that this is sensitive to the\n    # scale of the rewards. If your expected V is large, increase this.\n    \"vf_clip_param\": 10.0,\n    # If specified, clip the global norm of gradients by this amount\n    \"grad_clip\": None,\n    # Target value for KL divergence\n    \"kl_target\": 0.01,\n    # Whether to rollout \"complete_episodes\" or \"truncate_episodes\"\n    \"batch_mode\": \"truncate_episodes\",\n    # Which observation filter to apply to the observation\n    \"observation_filter\": \"NoFilter\",\n    # Uses the sync samples optimizer instead of the multi-gpu one. This does\n    # not support minibatches.\n    \"simple_optimizer\": False,\n})\n# __sphinx_doc_end__\n# yapf: enable\n\n\ndef choose_policy_optimizer(workers, config):\n    if config[\"simple_optimizer\"]:\n        return SyncSamplesOptimizer(\n            workers,\n            num_sgd_iter=config[\"num_sgd_iter\"],\n            train_batch_size=config[\"train_batch_size\"],\n            sgd_minibatch_size=config[\"sgd_minibatch_size\"])\n\n    return LocalMultiGPUOptimizer(\n        workers,\n        sgd_batch_size=config[\"sgd_minibatch_size\"],\n        num_sgd_iter=config[\"num_sgd_iter\"],\n        num_gpus=config[\"num_gpus\"],\n        sample_batch_size=config[\"sample_batch_size\"],\n        num_envs_per_worker=config[\"num_envs_per_worker\"],\n        train_batch_size=config[\"train_batch_size\"],\n        standardize_fields=[\"advantages\"],\n        shuffle_sequences=config[\"shuffle_sequences\"])\n\n\ndef update_kl(trainer, fetches):\n    if \"kl\" in fetches:\n        # single-agent\n        trainer.workers.local_worker().for_policy(\n            lambda pi: pi.update_kl(fetches[\"kl\"]))\n    else:\n\n        def update(pi, pi_id):\n            if pi_id in fetches:\n                pi.update_kl(fetches[pi_id][\"kl\"])\n            else:\n                logger.debug(\"No data for {}, not updating kl\".format(pi_id))\n\n        # multi-agent\n        trainer.workers.local_worker().foreach_trainable_policy(update)\n\n\ndef warn_about_bad_reward_scales(trainer, result):\n    # Warn about bad clipping configs\n    if trainer.config[\"vf_clip_param\"] <= 0:\n        rew_scale = float(\"inf\")\n    elif result[\"policy_reward_mean\"]:\n        rew_scale = 0  # punt on handling multiagent case\n    else:\n        rew_scale = round(\n            abs(result[\"episode_reward_mean\"]) /\n            trainer.config[\"vf_clip_param\"], 0)\n    if rew_scale > 200:\n        logger.warning(\n            \"The magnitude of your environment rewards are more than \"\n            \"{}x the scale of `vf_clip_param`. \".format(rew_scale) +\n            \"This means that it will take more than \"\n            \"{} iterations for your value \".format(rew_scale) +\n            \"function to converge. If this is not intended, consider \"\n            \"increasing `vf_clip_param`.\")\n\n\ndef validate_config(config):\n    if config[\"entropy_coeff\"] < 0:\n        raise DeprecationWarning(\"entropy_coeff must be >= 0\")\n    if config[\"sgd_minibatch_size\"] > config[\"train_batch_size\"]:\n        raise ValueError(\n            \"Minibatch size {} must be <= train batch size {}.\".format(\n                config[\"sgd_minibatch_size\"], config[\"train_batch_size\"]))\n    if config[\"batch_mode\"] == \"truncate_episodes\" and not config[\"use_gae\"]:\n        raise ValueError(\n            \"Episode truncation is not supported without a value \"\n            \"function. Consider setting batch_mode=complete_episodes.\")\n    if config[\"multiagent\"][\"policies\"] and not config[\"simple_optimizer\"]:\n        logger.info(\n            \"In multi-agent mode, policies will be optimized sequentially \"\n            \"by the multi-GPU optimizer. Consider setting \"\n            \"simple_optimizer=True if this doesn't work for you.\")\n    if config[\"simple_optimizer\"]:\n        logger.warning(\n            \"Using the simple minibatch optimizer. This will significantly \"\n            \"reduce performance, consider simple_optimizer=False.\")\n    elif tf and tf.executing_eagerly():\n        config[\"simple_optimizer\"] = True  # multi-gpu not supported\n\n\nPPOTrainer = build_trainer(\n    name=\"PPO\",\n    default_config=DEFAULT_CONFIG,\n    default_policy=PPOTFPolicy,\n    make_policy_optimizer=choose_policy_optimizer,\n    validate_config=validate_config,\n    after_optimizer_step=update_kl,\n    after_train_result=warn_about_bad_reward_scales)\n", "path": "rllib/agents/ppo/ppo.py"}], "after_files": [{"content": "from __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport logging\n\nfrom ray.rllib.agents import with_common_config\nfrom ray.rllib.agents.ppo.ppo_policy import PPOTFPolicy\nfrom ray.rllib.agents.trainer_template import build_trainer\nfrom ray.rllib.optimizers import SyncSamplesOptimizer, LocalMultiGPUOptimizer\nfrom ray.rllib.utils import try_import_tf\n\ntf = try_import_tf()\nlogger = logging.getLogger(__name__)\n\n# yapf: disable\n# __sphinx_doc_begin__\nDEFAULT_CONFIG = with_common_config({\n    # If true, use the Generalized Advantage Estimator (GAE)\n    # with a value function, see https://arxiv.org/pdf/1506.02438.pdf.\n    \"use_gae\": True,\n    # GAE(lambda) parameter\n    \"lambda\": 1.0,\n    # Initial coefficient for KL divergence\n    \"kl_coeff\": 0.2,\n    # Size of batches collected from each worker\n    \"sample_batch_size\": 200,\n    # Number of timesteps collected for each SGD round\n    \"train_batch_size\": 4000,\n    # Total SGD batch size across all devices for SGD\n    \"sgd_minibatch_size\": 128,\n    # Whether to shuffle sequences in the batch when training (recommended)\n    \"shuffle_sequences\": True,\n    # Number of SGD iterations in each outer loop\n    \"num_sgd_iter\": 30,\n    # Stepsize of SGD\n    \"lr\": 5e-5,\n    # Learning rate schedule\n    \"lr_schedule\": None,\n    # Share layers for value function. If you set this to True, it's important\n    # to tune vf_loss_coeff.\n    \"vf_share_layers\": False,\n    # Coefficient of the value function loss. It's important to tune this if\n    # you set vf_share_layers: True\n    \"vf_loss_coeff\": 1.0,\n    # Coefficient of the entropy regularizer\n    \"entropy_coeff\": 0.0,\n    # Decay schedule for the entropy regularizer\n    \"entropy_coeff_schedule\": None,\n    # PPO clip parameter\n    \"clip_param\": 0.3,\n    # Clip param for the value function. Note that this is sensitive to the\n    # scale of the rewards. If your expected V is large, increase this.\n    \"vf_clip_param\": 10.0,\n    # If specified, clip the global norm of gradients by this amount\n    \"grad_clip\": None,\n    # Target value for KL divergence\n    \"kl_target\": 0.01,\n    # Whether to rollout \"complete_episodes\" or \"truncate_episodes\"\n    \"batch_mode\": \"truncate_episodes\",\n    # Which observation filter to apply to the observation\n    \"observation_filter\": \"NoFilter\",\n    # Uses the sync samples optimizer instead of the multi-gpu one. This does\n    # not support minibatches.\n    \"simple_optimizer\": False,\n})\n# __sphinx_doc_end__\n# yapf: enable\n\n\ndef choose_policy_optimizer(workers, config):\n    if config[\"simple_optimizer\"]:\n        return SyncSamplesOptimizer(\n            workers,\n            num_sgd_iter=config[\"num_sgd_iter\"],\n            train_batch_size=config[\"train_batch_size\"],\n            sgd_minibatch_size=config[\"sgd_minibatch_size\"])\n\n    return LocalMultiGPUOptimizer(\n        workers,\n        sgd_batch_size=config[\"sgd_minibatch_size\"],\n        num_sgd_iter=config[\"num_sgd_iter\"],\n        num_gpus=config[\"num_gpus\"],\n        sample_batch_size=config[\"sample_batch_size\"],\n        num_envs_per_worker=config[\"num_envs_per_worker\"],\n        train_batch_size=config[\"train_batch_size\"],\n        standardize_fields=[\"advantages\"],\n        shuffle_sequences=config[\"shuffle_sequences\"])\n\n\ndef update_kl(trainer, fetches):\n    if \"kl\" in fetches:\n        # single-agent\n        trainer.workers.local_worker().for_policy(\n            lambda pi: pi.update_kl(fetches[\"kl\"]))\n    else:\n\n        def update(pi, pi_id):\n            if pi_id in fetches:\n                pi.update_kl(fetches[pi_id][\"kl\"])\n            else:\n                logger.debug(\"No data for {}, not updating kl\".format(pi_id))\n\n        # multi-agent\n        trainer.workers.local_worker().foreach_trainable_policy(update)\n\n\ndef warn_about_bad_reward_scales(trainer, result):\n    # Warn about bad clipping configs\n    if trainer.config[\"vf_clip_param\"] <= 0:\n        rew_scale = float(\"inf\")\n    elif result[\"policy_reward_mean\"]:\n        rew_scale = 0  # punt on handling multiagent case\n    else:\n        rew_scale = round(\n            abs(result[\"episode_reward_mean\"]) /\n            trainer.config[\"vf_clip_param\"], 0)\n    if rew_scale > 200:\n        logger.warning(\n            \"The magnitude of your environment rewards are more than \"\n            \"{}x the scale of `vf_clip_param`. \".format(rew_scale) +\n            \"This means that it will take more than \"\n            \"{} iterations for your value \".format(rew_scale) +\n            \"function to converge. If this is not intended, consider \"\n            \"increasing `vf_clip_param`.\")\n\n\ndef validate_config(config):\n    if config[\"entropy_coeff\"] < 0:\n        raise DeprecationWarning(\"entropy_coeff must be >= 0\")\n    if isinstance(config[\"entropy_coeff\"], int):\n        config[\"entropy_coeff\"] = float(config[\"entropy_coeff\"])\n    if config[\"sgd_minibatch_size\"] > config[\"train_batch_size\"]:\n        raise ValueError(\n            \"Minibatch size {} must be <= train batch size {}.\".format(\n                config[\"sgd_minibatch_size\"], config[\"train_batch_size\"]))\n    if config[\"batch_mode\"] == \"truncate_episodes\" and not config[\"use_gae\"]:\n        raise ValueError(\n            \"Episode truncation is not supported without a value \"\n            \"function. Consider setting batch_mode=complete_episodes.\")\n    if config[\"multiagent\"][\"policies\"] and not config[\"simple_optimizer\"]:\n        logger.info(\n            \"In multi-agent mode, policies will be optimized sequentially \"\n            \"by the multi-GPU optimizer. Consider setting \"\n            \"simple_optimizer=True if this doesn't work for you.\")\n    if config[\"simple_optimizer\"]:\n        logger.warning(\n            \"Using the simple minibatch optimizer. This will significantly \"\n            \"reduce performance, consider simple_optimizer=False.\")\n    elif tf and tf.executing_eagerly():\n        config[\"simple_optimizer\"] = True  # multi-gpu not supported\n\n\nPPOTrainer = build_trainer(\n    name=\"PPO\",\n    default_config=DEFAULT_CONFIG,\n    default_policy=PPOTFPolicy,\n    make_policy_optimizer=choose_policy_optimizer,\n    validate_config=validate_config,\n    after_optimizer_step=update_kl,\n    after_train_result=warn_about_bad_reward_scales)\n", "path": "rllib/agents/ppo/ppo.py"}]}