We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[rllib] PyTorch and SampleAsync validation
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?

PyTorch is supposed to be thread-safe, as long as you don't write a tensor using multiple threads. Please see https://discuss.pytorch.org/t/is-pytorch-supposed-to-be-thread-safe/36540/2 

It might be worth removing the validation of sample_async and use_pytorch for A3C (and maybe others?).

Ray Version 0.9.0dev (but this applies to any ray version actually)

### Reproduction (REQUIRED)
Please provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):

If we cannot run your script, we cannot fix your issue.

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [x] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).

[rllib] PyTorch and SampleAsync validation
<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->

### What is the problem?

PyTorch is supposed to be thread-safe, as long as you don't write a tensor using multiple threads. Please see https://discuss.pytorch.org/t/is-pytorch-supposed-to-be-thread-safe/36540/2 

It might be worth removing the validation of sample_async and use_pytorch for A3C (and maybe others?).

Ray Version 0.9.0dev (but this applies to any ray version actually)

### Reproduction (REQUIRED)
Please provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):

If we cannot run your script, we cannot fix your issue.

- [x] I have verified my script runs in a clean environment and reproduces the issue.
- [x] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `rllib/agents/a3c/a3c.py`
Content:
```
1 import logging
2 
3 from ray.rllib.agents.a3c.a3c_tf_policy import A3CTFPolicy
4 from ray.rllib.agents.trainer import with_common_config
5 from ray.rllib.agents.trainer_template import build_trainer
6 from ray.rllib.execution.rollout_ops import AsyncGradients
7 from ray.rllib.execution.train_ops import ApplyGradients
8 from ray.rllib.execution.metric_ops import StandardMetricsReporting
9 
10 logger = logging.getLogger(__name__)
11 
12 # yapf: disable
13 # __sphinx_doc_begin__
14 DEFAULT_CONFIG = with_common_config({
15     # Should use a critic as a baseline (otherwise don't use value baseline;
16     # required for using GAE).
17     "use_critic": True,
18     # If true, use the Generalized Advantage Estimator (GAE)
19     # with a value function, see https://arxiv.org/pdf/1506.02438.pdf.
20     "use_gae": True,
21     # Size of rollout batch
22     "rollout_fragment_length": 10,
23     # GAE(gamma) parameter
24     "lambda": 1.0,
25     # Max global norm for each gradient calculated by worker
26     "grad_clip": 40.0,
27     # Learning rate
28     "lr": 0.0001,
29     # Learning rate schedule
30     "lr_schedule": None,
31     # Value Function Loss coefficient
32     "vf_loss_coeff": 0.5,
33     # Entropy coefficient
34     "entropy_coeff": 0.01,
35     # Min time per iteration
36     "min_iter_time_s": 5,
37     # Workers sample async. Note that this increases the effective
38     # rollout_fragment_length by up to 5x due to async buffering of batches.
39     "sample_async": True,
40 })
41 # __sphinx_doc_end__
42 # yapf: enable
43 
44 
45 def get_policy_class(config):
46     if config["use_pytorch"]:
47         from ray.rllib.agents.a3c.a3c_torch_policy import \
48             A3CTorchPolicy
49         return A3CTorchPolicy
50     else:
51         return A3CTFPolicy
52 
53 
54 def validate_config(config):
55     if config["entropy_coeff"] < 0:
56         raise DeprecationWarning("entropy_coeff must be >= 0")
57     if config["sample_async"] and config["use_pytorch"]:
58         config["sample_async"] = False
59         logger.warning(
60             "The sample_async option is not supported with use_pytorch: "
61             "Multithreading can be lead to crashes if used with pytorch.")
62 
63 
64 def execution_plan(workers, config):
65     # For A3C, compute policy gradients remotely on the rollout workers.
66     grads = AsyncGradients(workers)
67 
68     # Apply the gradients as they arrive. We set update_all to False so that
69     # only the worker sending the gradient is updated with new weights.
70     train_op = grads.for_each(ApplyGradients(workers, update_all=False))
71 
72     return StandardMetricsReporting(train_op, workers, config)
73 
74 
75 A3CTrainer = build_trainer(
76     name="A3C",
77     default_config=DEFAULT_CONFIG,
78     default_policy=A3CTFPolicy,
79     get_policy_class=get_policy_class,
80     validate_config=validate_config,
81     execution_plan=execution_plan)
82
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/rllib/agents/a3c/a3c.py b/rllib/agents/a3c/a3c.py
--- a/rllib/agents/a3c/a3c.py
+++ b/rllib/agents/a3c/a3c.py
@@ -54,11 +54,6 @@
 def validate_config(config):
     if config["entropy_coeff"] < 0:
         raise DeprecationWarning("entropy_coeff must be >= 0")
-    if config["sample_async"] and config["use_pytorch"]:
-        config["sample_async"] = False
-        logger.warning(
-            "The sample_async option is not supported with use_pytorch: "
-            "Multithreading can be lead to crashes if used with pytorch.")
 
 
 def execution_plan(workers, config):

{"golden_diff": "diff --git a/rllib/agents/a3c/a3c.py b/rllib/agents/a3c/a3c.py\n--- a/rllib/agents/a3c/a3c.py\n+++ b/rllib/agents/a3c/a3c.py\n@@ -54,11 +54,6 @@\n def validate_config(config):\n     if config[\"entropy_coeff\"] < 0:\n         raise DeprecationWarning(\"entropy_coeff must be >= 0\")\n-    if config[\"sample_async\"] and config[\"use_pytorch\"]:\n-        config[\"sample_async\"] = False\n-        logger.warning(\n-            \"The sample_async option is not supported with use_pytorch: \"\n-            \"Multithreading can be lead to crashes if used with pytorch.\")\n \n \n def execution_plan(workers, config):\n", "issue": "[rllib] PyTorch and SampleAsync validation\n<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->\r\n\r\n### What is the problem?\r\n\r\nPyTorch is supposed to be thread-safe, as long as you don't write a tensor using multiple threads. Please see https://discuss.pytorch.org/t/is-pytorch-supposed-to-be-thread-safe/36540/2 \r\n\r\nIt might be worth removing the validation of sample_async and use_pytorch for A3C (and maybe others?).\r\n\r\nRay Version 0.9.0dev (but this applies to any ray version actually)\r\n\r\n### Reproduction (REQUIRED)\r\nPlease provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):\r\n\r\nIf we cannot run your script, we cannot fix your issue.\r\n\r\n- [x] I have verified my script runs in a clean environment and reproduces the issue.\r\n- [x] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).\r\n\n[rllib] PyTorch and SampleAsync validation\n<!--Please include [tune], [rllib], [autoscaler] etc. in the issue title if relevant-->\r\n\r\n### What is the problem?\r\n\r\nPyTorch is supposed to be thread-safe, as long as you don't write a tensor using multiple threads. Please see https://discuss.pytorch.org/t/is-pytorch-supposed-to-be-thread-safe/36540/2 \r\n\r\nIt might be worth removing the validation of sample_async and use_pytorch for A3C (and maybe others?).\r\n\r\nRay Version 0.9.0dev (but this applies to any ray version actually)\r\n\r\n### Reproduction (REQUIRED)\r\nPlease provide a script that can be run to reproduce the issue. The script should have **no external library dependencies** (i.e., use fake or mock data / environments):\r\n\r\nIf we cannot run your script, we cannot fix your issue.\r\n\r\n- [x] I have verified my script runs in a clean environment and reproduces the issue.\r\n- [x] I have verified the issue also occurs with the [latest wheels](https://docs.ray.io/en/latest/installation.html).\r\n\n", "before_files": [{"content": "import logging\n\nfrom ray.rllib.agents.a3c.a3c_tf_policy import A3CTFPolicy\nfrom ray.rllib.agents.trainer import with_common_config\nfrom ray.rllib.agents.trainer_template import build_trainer\nfrom ray.rllib.execution.rollout_ops import AsyncGradients\nfrom ray.rllib.execution.train_ops import ApplyGradients\nfrom ray.rllib.execution.metric_ops import StandardMetricsReporting\n\nlogger = logging.getLogger(__name__)\n\n# yapf: disable\n# __sphinx_doc_begin__\nDEFAULT_CONFIG = with_common_config({\n    # Should use a critic as a baseline (otherwise don't use value baseline;\n    # required for using GAE).\n    \"use_critic\": True,\n    # If true, use the Generalized Advantage Estimator (GAE)\n    # with a value function, see https://arxiv.org/pdf/1506.02438.pdf.\n    \"use_gae\": True,\n    # Size of rollout batch\n    \"rollout_fragment_length\": 10,\n    # GAE(gamma) parameter\n    \"lambda\": 1.0,\n    # Max global norm for each gradient calculated by worker\n    \"grad_clip\": 40.0,\n    # Learning rate\n    \"lr\": 0.0001,\n    # Learning rate schedule\n    \"lr_schedule\": None,\n    # Value Function Loss coefficient\n    \"vf_loss_coeff\": 0.5,\n    # Entropy coefficient\n    \"entropy_coeff\": 0.01,\n    # Min time per iteration\n    \"min_iter_time_s\": 5,\n    # Workers sample async. Note that this increases the effective\n    # rollout_fragment_length by up to 5x due to async buffering of batches.\n    \"sample_async\": True,\n})\n# __sphinx_doc_end__\n# yapf: enable\n\n\ndef get_policy_class(config):\n    if config[\"use_pytorch\"]:\n        from ray.rllib.agents.a3c.a3c_torch_policy import \\\n            A3CTorchPolicy\n        return A3CTorchPolicy\n    else:\n        return A3CTFPolicy\n\n\ndef validate_config(config):\n    if config[\"entropy_coeff\"] < 0:\n        raise DeprecationWarning(\"entropy_coeff must be >= 0\")\n    if config[\"sample_async\"] and config[\"use_pytorch\"]:\n        config[\"sample_async\"] = False\n        logger.warning(\n            \"The sample_async option is not supported with use_pytorch: \"\n            \"Multithreading can be lead to crashes if used with pytorch.\")\n\n\ndef execution_plan(workers, config):\n    # For A3C, compute policy gradients remotely on the rollout workers.\n    grads = AsyncGradients(workers)\n\n    # Apply the gradients as they arrive. We set update_all to False so that\n    # only the worker sending the gradient is updated with new weights.\n    train_op = grads.for_each(ApplyGradients(workers, update_all=False))\n\n    return StandardMetricsReporting(train_op, workers, config)\n\n\nA3CTrainer = build_trainer(\n    name=\"A3C\",\n    default_config=DEFAULT_CONFIG,\n    default_policy=A3CTFPolicy,\n    get_policy_class=get_policy_class,\n    validate_config=validate_config,\n    execution_plan=execution_plan)\n", "path": "rllib/agents/a3c/a3c.py"}], "after_files": [{"content": "import logging\n\nfrom ray.rllib.agents.a3c.a3c_tf_policy import A3CTFPolicy\nfrom ray.rllib.agents.trainer import with_common_config\nfrom ray.rllib.agents.trainer_template import build_trainer\nfrom ray.rllib.execution.rollout_ops import AsyncGradients\nfrom ray.rllib.execution.train_ops import ApplyGradients\nfrom ray.rllib.execution.metric_ops import StandardMetricsReporting\n\nlogger = logging.getLogger(__name__)\n\n# yapf: disable\n# __sphinx_doc_begin__\nDEFAULT_CONFIG = with_common_config({\n    # Should use a critic as a baseline (otherwise don't use value baseline;\n    # required for using GAE).\n    \"use_critic\": True,\n    # If true, use the Generalized Advantage Estimator (GAE)\n    # with a value function, see https://arxiv.org/pdf/1506.02438.pdf.\n    \"use_gae\": True,\n    # Size of rollout batch\n    \"rollout_fragment_length\": 10,\n    # GAE(gamma) parameter\n    \"lambda\": 1.0,\n    # Max global norm for each gradient calculated by worker\n    \"grad_clip\": 40.0,\n    # Learning rate\n    \"lr\": 0.0001,\n    # Learning rate schedule\n    \"lr_schedule\": None,\n    # Value Function Loss coefficient\n    \"vf_loss_coeff\": 0.5,\n    # Entropy coefficient\n    \"entropy_coeff\": 0.01,\n    # Min time per iteration\n    \"min_iter_time_s\": 5,\n    # Workers sample async. Note that this increases the effective\n    # rollout_fragment_length by up to 5x due to async buffering of batches.\n    \"sample_async\": True,\n})\n# __sphinx_doc_end__\n# yapf: enable\n\n\ndef get_policy_class(config):\n    if config[\"use_pytorch\"]:\n        from ray.rllib.agents.a3c.a3c_torch_policy import \\\n            A3CTorchPolicy\n        return A3CTorchPolicy\n    else:\n        return A3CTFPolicy\n\n\ndef validate_config(config):\n    if config[\"entropy_coeff\"] < 0:\n        raise DeprecationWarning(\"entropy_coeff must be >= 0\")\n\n\ndef execution_plan(workers, config):\n    # For A3C, compute policy gradients remotely on the rollout workers.\n    grads = AsyncGradients(workers)\n\n    # Apply the gradients as they arrive. We set update_all to False so that\n    # only the worker sending the gradient is updated with new weights.\n    train_op = grads.for_each(ApplyGradients(workers, update_all=False))\n\n    return StandardMetricsReporting(train_op, workers, config)\n\n\nA3CTrainer = build_trainer(\n    name=\"A3C\",\n    default_config=DEFAULT_CONFIG,\n    default_policy=A3CTFPolicy,\n    get_policy_class=get_policy_class,\n    validate_config=validate_config,\n    execution_plan=execution_plan)\n", "path": "rllib/agents/a3c/a3c.py"}]}