We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
AttributeError: 'TFTrainingArguments' object has no attribute 'args'
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.2.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 -->
 Trainer: @sgugger 
 tensorflow: @jplu 


## Information

Model I am using (Bert, XLNet ...): `distilbert-base-uncased`

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
    IMDB Sequence Classification

## To reproduce

Steps to reproduce the behavior:

1. Follow the [fine-tuning tutorial here and use TensorFlow](https://huggingface.co/transformers/master/custom_datasets.html#fine-tuning-with-trainer)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```python
AttributeError                            Traceback (most recent call last)

<ipython-input-10-c5306faf2c2f> in <module>()
     12 )
     13 
---> 14 with training_args.strategy.scope():
     15     model = TFDistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased")
     16 

4 frames

/usr/local/lib/python3.6/dist-packages/transformers/training_args_tf.py in _setup_strategy(self)
    120         logger.info("Tensorflow: setting up strategy")
    121 
--> 122         if self.args.xla:
    123             tf.config.optimizer.set_jit(True)
    124 

AttributeError: 'TFTrainingArguments' object has no attribute 'args'
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `src/transformers/training_args_tf.py`
Content:
```
1 import warnings
2 from dataclasses import dataclass, field
3 from typing import Tuple
4 
5 from .file_utils import cached_property, is_tf_available, tf_required
6 from .training_args import TrainingArguments
7 from .utils import logging
8 
9 
10 logger = logging.get_logger(__name__)
11 
12 if is_tf_available():
13     import tensorflow as tf
14 
15 
16 @dataclass
17 class TFTrainingArguments(TrainingArguments):
18     """
19     TrainingArguments is the subset of the arguments we use in our example scripts
20     **which relate to the training loop itself**.
21 
22     Using :class:`~transformers.HfArgumentParser` we can turn this class
23     into argparse arguments to be able to specify them on the command line.
24 
25     Parameters:
26         output_dir (:obj:`str`):
27             The output directory where the model predictions and checkpoints will be written.
28         overwrite_output_dir (:obj:`bool`, `optional`, defaults to :obj:`False`):
29             If :obj:`True`, overwrite the content of the output directory. Use this to continue training if
30             :obj:`output_dir` points to a checkpoint directory.
31         do_train (:obj:`bool`, `optional`, defaults to :obj:`False`):
32             Whether to run training or not.
33         do_eval (:obj:`bool`, `optional`, defaults to :obj:`False`):
34             Whether to run evaluation on the dev set or not.
35         do_predict (:obj:`bool`, `optional`, defaults to :obj:`False`):
36             Whether to run predictions on the test set or not.
37         evaluate_during_training (:obj:`bool`, `optional`, defaults to :obj:`False`):
38             Whether to run evaluation during training at each logging step or not.
39         per_device_train_batch_size (:obj:`int`, `optional`, defaults to 8):
40             The batch size per GPU/TPU core/CPU for training.
41         per_device_eval_batch_size (:obj:`int`, `optional`, defaults to 8):
42             The batch size per GPU/TPU core/CPU for evaluation.
43         gradient_accumulation_steps: (:obj:`int`, `optional`, defaults to 1):
44             Number of updates steps to accumulate the gradients for, before performing a backward/update pass.
45 
46             .. warning::
47 
48                 When using gradient accumulation, one step is counted as one step with backward pass. Therefore,
49                 logging, evaluation, save will be conducted every ``gradient_accumulation_steps * xxx_step`` training
50                 examples.
51         learning_rate (:obj:`float`, `optional`, defaults to 5e-5):
52             The initial learning rate for Adam.
53         weight_decay (:obj:`float`, `optional`, defaults to 0):
54             The weight decay to apply (if not zero).
55         adam_epsilon (:obj:`float`, `optional`, defaults to 1e-8):
56             Epsilon for the Adam optimizer.
57         max_grad_norm (:obj:`float`, `optional`, defaults to 1.0):
58             Maximum gradient norm (for gradient clipping).
59         num_train_epochs(:obj:`float`, `optional`, defaults to 3.0):
60             Total number of training epochs to perform.
61         max_steps (:obj:`int`, `optional`, defaults to -1):
62             If set to a positive number, the total number of training steps to perform. Overrides
63             :obj:`num_train_epochs`.
64         warmup_steps (:obj:`int`, `optional`, defaults to 0):
65             Number of steps used for a linear warmup from 0 to :obj:`learning_rate`.
66         logging_dir (:obj:`str`, `optional`):
67             Tensorboard log directory. Will default to `runs/**CURRENT_DATETIME_HOSTNAME**`.
68         logging_first_step (:obj:`bool`, `optional`, defaults to :obj:`False`):
69             Wheter to log and evalulate the first :obj:`global_step` or not.
70         logging_steps (:obj:`int`, `optional`, defaults to 500):
71             Number of update steps between two logs.
72         save_steps (:obj:`int`, `optional`, defaults to 500):
73             Number of updates steps before two checkpoint saves.
74         save_total_limit (:obj:`int`, `optional`):
75             If a value is passed, will limit the total amount of checkpoints. Deletes the older checkpoints in
76             :obj:`output_dir`.
77         no_cuda (:obj:`bool`, `optional`, defaults to :obj:`False`):
78             Whether to not use CUDA even when it is available or not.
79         seed (:obj:`int`, `optional`, defaults to 42):
80             Random seed for initialization.
81         fp16 (:obj:`bool`, `optional`, defaults to :obj:`False`):
82             Whether to use 16-bit (mixed) precision training (through NVIDIA apex) instead of 32-bit training.
83         fp16_opt_level (:obj:`str`, `optional`, defaults to 'O1'):
84             For :obj:`fp16` training, apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. See details
85             on the `apex documentation <https://nvidia.github.io/apex/amp.html>`__.
86         local_rank (:obj:`int`, `optional`, defaults to -1):
87             During distributed training, the rank of the process.
88         tpu_num_cores (:obj:`int`, `optional`):
89             When training on TPU, the mumber of TPU cores (automatically passed by launcher script).
90         debug (:obj:`bool`, `optional`, defaults to :obj:`False`):
91             Whether to activate the trace to record computation graphs and profiling information or not.
92         dataloader_drop_last (:obj:`bool`, `optional`, defaults to :obj:`False`):
93             Whether to drop the last incomplete batch (if the length of the dataset is not divisible by the batch size)
94             or not.
95         eval_steps (:obj:`int`, `optional`, defaults to 1000):
96             Number of update steps before two evaluations.
97         past_index (:obj:`int`, `optional`, defaults to -1):
98             Some models like :doc:`TransformerXL <../model_doc/transformerxl>` or :doc`XLNet <../model_doc/xlnet>` can
99             make use of the past hidden states for their predictions. If this argument is set to a positive int, the
100             ``Trainer`` will use the corresponding output (usually index 2) as the past state and feed it to the model
101             at the next training step under the keyword argument ``mems``.
102         tpu_name (:obj:`str`, `optional`):
103             The name of the TPU the process is running on.
104         run_name (:obj:`str`, `optional`):
105             A descriptor for the run. Notably used for wandb logging.
106         xla (:obj:`bool`, `optional`):
107             Whether to activate the XLA compilation or not.
108     """
109 
110     tpu_name: str = field(
111         default=None,
112         metadata={"help": "Name of TPU"},
113     )
114 
115     xla: bool = field(default=False, metadata={"help": "Whether to activate the XLA compilation or not"})
116 
117     @cached_property
118     @tf_required
119     def _setup_strategy(self) -> Tuple["tf.distribute.Strategy", int]:
120         logger.info("Tensorflow: setting up strategy")
121 
122         if self.args.xla:
123             tf.config.optimizer.set_jit(True)
124 
125         gpus = tf.config.list_physical_devices("GPU")
126 
127         # Set to float16 at first
128         if self.fp16:
129             policy = tf.keras.mixed_precision.experimental.Policy("mixed_float16")
130             tf.keras.mixed_precision.experimental.set_policy(policy)
131 
132         if self.no_cuda:
133             strategy = tf.distribute.OneDeviceStrategy(device="/cpu:0")
134         else:
135             try:
136                 if self.tpu_name:
137                     tpu = tf.distribute.cluster_resolver.TPUClusterResolver(self.tpu_name)
138                 else:
139                     tpu = tf.distribute.cluster_resolver.TPUClusterResolver()
140             except ValueError:
141                 tpu = None
142 
143             if tpu:
144                 # Set to bfloat16 in case of TPU
145                 if self.fp16:
146                     policy = tf.keras.mixed_precision.experimental.Policy("mixed_bfloat16")
147                     tf.keras.mixed_precision.experimental.set_policy(policy)
148 
149                 tf.config.experimental_connect_to_cluster(tpu)
150                 tf.tpu.experimental.initialize_tpu_system(tpu)
151 
152                 strategy = tf.distribute.experimental.TPUStrategy(tpu)
153 
154             elif len(gpus) == 0:
155                 strategy = tf.distribute.OneDeviceStrategy(device="/cpu:0")
156             elif len(gpus) == 1:
157                 strategy = tf.distribute.OneDeviceStrategy(device="/gpu:0")
158             elif len(gpus) > 1:
159                 # If you only want to use a specific subset of GPUs use `CUDA_VISIBLE_DEVICES=0`
160                 strategy = tf.distribute.MirroredStrategy()
161             else:
162                 raise ValueError("Cannot find the proper strategy please check your environment properties.")
163 
164         return strategy
165 
166     @property
167     @tf_required
168     def strategy(self) -> "tf.distribute.Strategy":
169         """
170         The strategy used for distributed training.
171         """
172         return self._setup_strategy
173 
174     @property
175     @tf_required
176     def n_replicas(self) -> int:
177         """
178         The number of replicas (CPUs, GPUs or TPU cores) used in this training.
179         """
180         return self._setup_strategy.num_replicas_in_sync
181 
182     @property
183     def train_batch_size(self) -> int:
184         """
185         The actual batch size for training (may differ from :obj:`per_gpu_train_batch_size` in distributed training).
186         """
187         if self.per_gpu_train_batch_size:
188             logger.warning(
189                 "Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future "
190                 "version. Using `--per_device_train_batch_size` is preferred."
191             )
192         per_device_batch_size = self.per_gpu_train_batch_size or self.per_device_train_batch_size
193         return per_device_batch_size * self.n_replicas
194 
195     @property
196     def eval_batch_size(self) -> int:
197         """
198         The actual batch size for evaluation (may differ from :obj:`per_gpu_eval_batch_size` in distributed training).
199         """
200         if self.per_gpu_eval_batch_size:
201             logger.warning(
202                 "Using deprecated `--per_gpu_eval_batch_size` argument which will be removed in a future "
203                 "version. Using `--per_device_eval_batch_size` is preferred."
204             )
205         per_device_batch_size = self.per_gpu_eval_batch_size or self.per_device_eval_batch_size
206         return per_device_batch_size * self.n_replicas
207 
208     @property
209     @tf_required
210     def n_gpu(self) -> int:
211         """
212         The number of replicas (CPUs, GPUs or TPU cores) used in this training.
213         """
214         warnings.warn(
215             "The n_gpu argument is deprecated and will be removed in a future version, use n_replicas instead.",
216             FutureWarning,
217         )
218         return self._setup_strategy.num_replicas_in_sync
219
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/src/transformers/training_args_tf.py b/src/transformers/training_args_tf.py
--- a/src/transformers/training_args_tf.py
+++ b/src/transformers/training_args_tf.py
@@ -119,7 +119,7 @@
     def _setup_strategy(self) -> Tuple["tf.distribute.Strategy", int]:
         logger.info("Tensorflow: setting up strategy")
 
-        if self.args.xla:
+        if self.xla:
             tf.config.optimizer.set_jit(True)
 
         gpus = tf.config.list_physical_devices("GPU")

{"golden_diff": "diff --git a/src/transformers/training_args_tf.py b/src/transformers/training_args_tf.py\n--- a/src/transformers/training_args_tf.py\n+++ b/src/transformers/training_args_tf.py\n@@ -119,7 +119,7 @@\n     def _setup_strategy(self) -> Tuple[\"tf.distribute.Strategy\", int]:\n         logger.info(\"Tensorflow: setting up strategy\")\n \n-        if self.args.xla:\n+        if self.xla:\n             tf.config.optimizer.set_jit(True)\n \n         gpus = tf.config.list_physical_devices(\"GPU\")\n", "issue": "AttributeError: 'TFTrainingArguments' object has no attribute 'args'\n## Environment info\r\n<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.\r\n     Don't forget to fill out the missing fields in that output! -->\r\n     \r\n- `transformers` version: 3.2.0\r\n- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic\r\n- Python version: 3.6.9\r\n- PyTorch version (GPU?): 1.6.0+cu101 (True)\r\n- Tensorflow version (GPU?): 2.3.0 (True)\r\n- Using GPU in script?: True\r\n- Using distributed or parallel set-up in script?: False\r\n\r\n### Who can help\r\n<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @\r\n If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.\r\n Please tag fewer than 3 people.\r\n -->\r\n Trainer: @sgugger \r\n tensorflow: @jplu \r\n\r\n\r\n## Information\r\n\r\nModel I am using (Bert, XLNet ...): `distilbert-base-uncased`\r\n\r\nThe problem arises when using:\r\n* [x] the official example scripts: (give details below)\r\n* [ ] my own modified scripts: (give details below)\r\n\r\nThe tasks I am working on is:\r\n* [ ] an official GLUE/SQUaD task: (give the name)\r\n* [x] my own task or dataset: (give details below)\r\n    IMDB Sequence Classification\r\n\r\n## To reproduce\r\n\r\nSteps to reproduce the behavior:\r\n\r\n1. Follow the [fine-tuning tutorial here and use TensorFlow](https://huggingface.co/transformers/master/custom_datasets.html#fine-tuning-with-trainer)\r\n\r\n<!-- If you have code snippets, error messages, stack traces please provide them here as well.\r\n     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting\r\n     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->\r\n\r\n```python\r\nAttributeError                            Traceback (most recent call last)\r\n\r\n<ipython-input-10-c5306faf2c2f> in <module>()\r\n     12 )\r\n     13 \r\n---> 14 with training_args.strategy.scope():\r\n     15     model = TFDistilBertForSequenceClassification.from_pretrained(\"distilbert-base-uncased\")\r\n     16 \r\n\r\n4 frames\r\n\r\n/usr/local/lib/python3.6/dist-packages/transformers/training_args_tf.py in _setup_strategy(self)\r\n    120         logger.info(\"Tensorflow: setting up strategy\")\r\n    121 \r\n--> 122         if self.args.xla:\r\n    123             tf.config.optimizer.set_jit(True)\r\n    124 \r\n\r\nAttributeError: 'TFTrainingArguments' object has no attribute 'args'\r\n```\r\n\r\n## Expected behavior\r\n\r\n<!-- A clear and concise description of what you would expect to happen. -->\r\n\n", "before_files": [{"content": "import warnings\nfrom dataclasses import dataclass, field\nfrom typing import Tuple\n\nfrom .file_utils import cached_property, is_tf_available, tf_required\nfrom .training_args import TrainingArguments\nfrom .utils import logging\n\n\nlogger = logging.get_logger(__name__)\n\nif is_tf_available():\n    import tensorflow as tf\n\n\n@dataclass\nclass TFTrainingArguments(TrainingArguments):\n    \"\"\"\n    TrainingArguments is the subset of the arguments we use in our example scripts\n    **which relate to the training loop itself**.\n\n    Using :class:`~transformers.HfArgumentParser` we can turn this class\n    into argparse arguments to be able to specify them on the command line.\n\n    Parameters:\n        output_dir (:obj:`str`):\n            The output directory where the model predictions and checkpoints will be written.\n        overwrite_output_dir (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            If :obj:`True`, overwrite the content of the output directory. Use this to continue training if\n            :obj:`output_dir` points to a checkpoint directory.\n        do_train (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to run training or not.\n        do_eval (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to run evaluation on the dev set or not.\n        do_predict (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to run predictions on the test set or not.\n        evaluate_during_training (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to run evaluation during training at each logging step or not.\n        per_device_train_batch_size (:obj:`int`, `optional`, defaults to 8):\n            The batch size per GPU/TPU core/CPU for training.\n        per_device_eval_batch_size (:obj:`int`, `optional`, defaults to 8):\n            The batch size per GPU/TPU core/CPU for evaluation.\n        gradient_accumulation_steps: (:obj:`int`, `optional`, defaults to 1):\n            Number of updates steps to accumulate the gradients for, before performing a backward/update pass.\n\n            .. warning::\n\n                When using gradient accumulation, one step is counted as one step with backward pass. Therefore,\n                logging, evaluation, save will be conducted every ``gradient_accumulation_steps * xxx_step`` training\n                examples.\n        learning_rate (:obj:`float`, `optional`, defaults to 5e-5):\n            The initial learning rate for Adam.\n        weight_decay (:obj:`float`, `optional`, defaults to 0):\n            The weight decay to apply (if not zero).\n        adam_epsilon (:obj:`float`, `optional`, defaults to 1e-8):\n            Epsilon for the Adam optimizer.\n        max_grad_norm (:obj:`float`, `optional`, defaults to 1.0):\n            Maximum gradient norm (for gradient clipping).\n        num_train_epochs(:obj:`float`, `optional`, defaults to 3.0):\n            Total number of training epochs to perform.\n        max_steps (:obj:`int`, `optional`, defaults to -1):\n            If set to a positive number, the total number of training steps to perform. Overrides\n            :obj:`num_train_epochs`.\n        warmup_steps (:obj:`int`, `optional`, defaults to 0):\n            Number of steps used for a linear warmup from 0 to :obj:`learning_rate`.\n        logging_dir (:obj:`str`, `optional`):\n            Tensorboard log directory. Will default to `runs/**CURRENT_DATETIME_HOSTNAME**`.\n        logging_first_step (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Wheter to log and evalulate the first :obj:`global_step` or not.\n        logging_steps (:obj:`int`, `optional`, defaults to 500):\n            Number of update steps between two logs.\n        save_steps (:obj:`int`, `optional`, defaults to 500):\n            Number of updates steps before two checkpoint saves.\n        save_total_limit (:obj:`int`, `optional`):\n            If a value is passed, will limit the total amount of checkpoints. Deletes the older checkpoints in\n            :obj:`output_dir`.\n        no_cuda (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to not use CUDA even when it is available or not.\n        seed (:obj:`int`, `optional`, defaults to 42):\n            Random seed for initialization.\n        fp16 (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to use 16-bit (mixed) precision training (through NVIDIA apex) instead of 32-bit training.\n        fp16_opt_level (:obj:`str`, `optional`, defaults to 'O1'):\n            For :obj:`fp16` training, apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. See details\n            on the `apex documentation <https://nvidia.github.io/apex/amp.html>`__.\n        local_rank (:obj:`int`, `optional`, defaults to -1):\n            During distributed training, the rank of the process.\n        tpu_num_cores (:obj:`int`, `optional`):\n            When training on TPU, the mumber of TPU cores (automatically passed by launcher script).\n        debug (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to activate the trace to record computation graphs and profiling information or not.\n        dataloader_drop_last (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to drop the last incomplete batch (if the length of the dataset is not divisible by the batch size)\n            or not.\n        eval_steps (:obj:`int`, `optional`, defaults to 1000):\n            Number of update steps before two evaluations.\n        past_index (:obj:`int`, `optional`, defaults to -1):\n            Some models like :doc:`TransformerXL <../model_doc/transformerxl>` or :doc`XLNet <../model_doc/xlnet>` can\n            make use of the past hidden states for their predictions. If this argument is set to a positive int, the\n            ``Trainer`` will use the corresponding output (usually index 2) as the past state and feed it to the model\n            at the next training step under the keyword argument ``mems``.\n        tpu_name (:obj:`str`, `optional`):\n            The name of the TPU the process is running on.\n        run_name (:obj:`str`, `optional`):\n            A descriptor for the run. Notably used for wandb logging.\n        xla (:obj:`bool`, `optional`):\n            Whether to activate the XLA compilation or not.\n    \"\"\"\n\n    tpu_name: str = field(\n        default=None,\n        metadata={\"help\": \"Name of TPU\"},\n    )\n\n    xla: bool = field(default=False, metadata={\"help\": \"Whether to activate the XLA compilation or not\"})\n\n    @cached_property\n    @tf_required\n    def _setup_strategy(self) -> Tuple[\"tf.distribute.Strategy\", int]:\n        logger.info(\"Tensorflow: setting up strategy\")\n\n        if self.args.xla:\n            tf.config.optimizer.set_jit(True)\n\n        gpus = tf.config.list_physical_devices(\"GPU\")\n\n        # Set to float16 at first\n        if self.fp16:\n            policy = tf.keras.mixed_precision.experimental.Policy(\"mixed_float16\")\n            tf.keras.mixed_precision.experimental.set_policy(policy)\n\n        if self.no_cuda:\n            strategy = tf.distribute.OneDeviceStrategy(device=\"/cpu:0\")\n        else:\n            try:\n                if self.tpu_name:\n                    tpu = tf.distribute.cluster_resolver.TPUClusterResolver(self.tpu_name)\n                else:\n                    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n            except ValueError:\n                tpu = None\n\n            if tpu:\n                # Set to bfloat16 in case of TPU\n                if self.fp16:\n                    policy = tf.keras.mixed_precision.experimental.Policy(\"mixed_bfloat16\")\n                    tf.keras.mixed_precision.experimental.set_policy(policy)\n\n                tf.config.experimental_connect_to_cluster(tpu)\n                tf.tpu.experimental.initialize_tpu_system(tpu)\n\n                strategy = tf.distribute.experimental.TPUStrategy(tpu)\n\n            elif len(gpus) == 0:\n                strategy = tf.distribute.OneDeviceStrategy(device=\"/cpu:0\")\n            elif len(gpus) == 1:\n                strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n            elif len(gpus) > 1:\n                # If you only want to use a specific subset of GPUs use `CUDA_VISIBLE_DEVICES=0`\n                strategy = tf.distribute.MirroredStrategy()\n            else:\n                raise ValueError(\"Cannot find the proper strategy please check your environment properties.\")\n\n        return strategy\n\n    @property\n    @tf_required\n    def strategy(self) -> \"tf.distribute.Strategy\":\n        \"\"\"\n        The strategy used for distributed training.\n        \"\"\"\n        return self._setup_strategy\n\n    @property\n    @tf_required\n    def n_replicas(self) -> int:\n        \"\"\"\n        The number of replicas (CPUs, GPUs or TPU cores) used in this training.\n        \"\"\"\n        return self._setup_strategy.num_replicas_in_sync\n\n    @property\n    def train_batch_size(self) -> int:\n        \"\"\"\n        The actual batch size for training (may differ from :obj:`per_gpu_train_batch_size` in distributed training).\n        \"\"\"\n        if self.per_gpu_train_batch_size:\n            logger.warning(\n                \"Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future \"\n                \"version. Using `--per_device_train_batch_size` is preferred.\"\n            )\n        per_device_batch_size = self.per_gpu_train_batch_size or self.per_device_train_batch_size\n        return per_device_batch_size * self.n_replicas\n\n    @property\n    def eval_batch_size(self) -> int:\n        \"\"\"\n        The actual batch size for evaluation (may differ from :obj:`per_gpu_eval_batch_size` in distributed training).\n        \"\"\"\n        if self.per_gpu_eval_batch_size:\n            logger.warning(\n                \"Using deprecated `--per_gpu_eval_batch_size` argument which will be removed in a future \"\n                \"version. Using `--per_device_eval_batch_size` is preferred.\"\n            )\n        per_device_batch_size = self.per_gpu_eval_batch_size or self.per_device_eval_batch_size\n        return per_device_batch_size * self.n_replicas\n\n    @property\n    @tf_required\n    def n_gpu(self) -> int:\n        \"\"\"\n        The number of replicas (CPUs, GPUs or TPU cores) used in this training.\n        \"\"\"\n        warnings.warn(\n            \"The n_gpu argument is deprecated and will be removed in a future version, use n_replicas instead.\",\n            FutureWarning,\n        )\n        return self._setup_strategy.num_replicas_in_sync\n", "path": "src/transformers/training_args_tf.py"}], "after_files": [{"content": "import warnings\nfrom dataclasses import dataclass, field\nfrom typing import Tuple\n\nfrom .file_utils import cached_property, is_tf_available, tf_required\nfrom .training_args import TrainingArguments\nfrom .utils import logging\n\n\nlogger = logging.get_logger(__name__)\n\nif is_tf_available():\n    import tensorflow as tf\n\n\n@dataclass\nclass TFTrainingArguments(TrainingArguments):\n    \"\"\"\n    TrainingArguments is the subset of the arguments we use in our example scripts\n    **which relate to the training loop itself**.\n\n    Using :class:`~transformers.HfArgumentParser` we can turn this class\n    into argparse arguments to be able to specify them on the command line.\n\n    Parameters:\n        output_dir (:obj:`str`):\n            The output directory where the model predictions and checkpoints will be written.\n        overwrite_output_dir (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            If :obj:`True`, overwrite the content of the output directory. Use this to continue training if\n            :obj:`output_dir` points to a checkpoint directory.\n        do_train (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to run training or not.\n        do_eval (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to run evaluation on the dev set or not.\n        do_predict (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to run predictions on the test set or not.\n        evaluate_during_training (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to run evaluation during training at each logging step or not.\n        per_device_train_batch_size (:obj:`int`, `optional`, defaults to 8):\n            The batch size per GPU/TPU core/CPU for training.\n        per_device_eval_batch_size (:obj:`int`, `optional`, defaults to 8):\n            The batch size per GPU/TPU core/CPU for evaluation.\n        gradient_accumulation_steps: (:obj:`int`, `optional`, defaults to 1):\n            Number of updates steps to accumulate the gradients for, before performing a backward/update pass.\n\n            .. warning::\n\n                When using gradient accumulation, one step is counted as one step with backward pass. Therefore,\n                logging, evaluation, save will be conducted every ``gradient_accumulation_steps * xxx_step`` training\n                examples.\n        learning_rate (:obj:`float`, `optional`, defaults to 5e-5):\n            The initial learning rate for Adam.\n        weight_decay (:obj:`float`, `optional`, defaults to 0):\n            The weight decay to apply (if not zero).\n        adam_epsilon (:obj:`float`, `optional`, defaults to 1e-8):\n            Epsilon for the Adam optimizer.\n        max_grad_norm (:obj:`float`, `optional`, defaults to 1.0):\n            Maximum gradient norm (for gradient clipping).\n        num_train_epochs(:obj:`float`, `optional`, defaults to 3.0):\n            Total number of training epochs to perform.\n        max_steps (:obj:`int`, `optional`, defaults to -1):\n            If set to a positive number, the total number of training steps to perform. Overrides\n            :obj:`num_train_epochs`.\n        warmup_steps (:obj:`int`, `optional`, defaults to 0):\n            Number of steps used for a linear warmup from 0 to :obj:`learning_rate`.\n        logging_dir (:obj:`str`, `optional`):\n            Tensorboard log directory. Will default to `runs/**CURRENT_DATETIME_HOSTNAME**`.\n        logging_first_step (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Wheter to log and evalulate the first :obj:`global_step` or not.\n        logging_steps (:obj:`int`, `optional`, defaults to 500):\n            Number of update steps between two logs.\n        save_steps (:obj:`int`, `optional`, defaults to 500):\n            Number of updates steps before two checkpoint saves.\n        save_total_limit (:obj:`int`, `optional`):\n            If a value is passed, will limit the total amount of checkpoints. Deletes the older checkpoints in\n            :obj:`output_dir`.\n        no_cuda (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to not use CUDA even when it is available or not.\n        seed (:obj:`int`, `optional`, defaults to 42):\n            Random seed for initialization.\n        fp16 (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to use 16-bit (mixed) precision training (through NVIDIA apex) instead of 32-bit training.\n        fp16_opt_level (:obj:`str`, `optional`, defaults to 'O1'):\n            For :obj:`fp16` training, apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. See details\n            on the `apex documentation <https://nvidia.github.io/apex/amp.html>`__.\n        local_rank (:obj:`int`, `optional`, defaults to -1):\n            During distributed training, the rank of the process.\n        tpu_num_cores (:obj:`int`, `optional`):\n            When training on TPU, the mumber of TPU cores (automatically passed by launcher script).\n        debug (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to activate the trace to record computation graphs and profiling information or not.\n        dataloader_drop_last (:obj:`bool`, `optional`, defaults to :obj:`False`):\n            Whether to drop the last incomplete batch (if the length of the dataset is not divisible by the batch size)\n            or not.\n        eval_steps (:obj:`int`, `optional`, defaults to 1000):\n            Number of update steps before two evaluations.\n        past_index (:obj:`int`, `optional`, defaults to -1):\n            Some models like :doc:`TransformerXL <../model_doc/transformerxl>` or :doc`XLNet <../model_doc/xlnet>` can\n            make use of the past hidden states for their predictions. If this argument is set to a positive int, the\n            ``Trainer`` will use the corresponding output (usually index 2) as the past state and feed it to the model\n            at the next training step under the keyword argument ``mems``.\n        tpu_name (:obj:`str`, `optional`):\n            The name of the TPU the process is running on.\n        run_name (:obj:`str`, `optional`):\n            A descriptor for the run. Notably used for wandb logging.\n        xla (:obj:`bool`, `optional`):\n            Whether to activate the XLA compilation or not.\n    \"\"\"\n\n    tpu_name: str = field(\n        default=None,\n        metadata={\"help\": \"Name of TPU\"},\n    )\n\n    xla: bool = field(default=False, metadata={\"help\": \"Whether to activate the XLA compilation or not\"})\n\n    @cached_property\n    @tf_required\n    def _setup_strategy(self) -> Tuple[\"tf.distribute.Strategy\", int]:\n        logger.info(\"Tensorflow: setting up strategy\")\n\n        if self.xla:\n            tf.config.optimizer.set_jit(True)\n\n        gpus = tf.config.list_physical_devices(\"GPU\")\n\n        # Set to float16 at first\n        if self.fp16:\n            policy = tf.keras.mixed_precision.experimental.Policy(\"mixed_float16\")\n            tf.keras.mixed_precision.experimental.set_policy(policy)\n\n        if self.no_cuda:\n            strategy = tf.distribute.OneDeviceStrategy(device=\"/cpu:0\")\n        else:\n            try:\n                if self.tpu_name:\n                    tpu = tf.distribute.cluster_resolver.TPUClusterResolver(self.tpu_name)\n                else:\n                    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n            except ValueError:\n                tpu = None\n\n            if tpu:\n                # Set to bfloat16 in case of TPU\n                if self.fp16:\n                    policy = tf.keras.mixed_precision.experimental.Policy(\"mixed_bfloat16\")\n                    tf.keras.mixed_precision.experimental.set_policy(policy)\n\n                tf.config.experimental_connect_to_cluster(tpu)\n                tf.tpu.experimental.initialize_tpu_system(tpu)\n\n                strategy = tf.distribute.experimental.TPUStrategy(tpu)\n\n            elif len(gpus) == 0:\n                strategy = tf.distribute.OneDeviceStrategy(device=\"/cpu:0\")\n            elif len(gpus) == 1:\n                strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n            elif len(gpus) > 1:\n                # If you only want to use a specific subset of GPUs use `CUDA_VISIBLE_DEVICES=0`\n                strategy = tf.distribute.MirroredStrategy()\n            else:\n                raise ValueError(\"Cannot find the proper strategy please check your environment properties.\")\n\n        return strategy\n\n    @property\n    @tf_required\n    def strategy(self) -> \"tf.distribute.Strategy\":\n        \"\"\"\n        The strategy used for distributed training.\n        \"\"\"\n        return self._setup_strategy\n\n    @property\n    @tf_required\n    def n_replicas(self) -> int:\n        \"\"\"\n        The number of replicas (CPUs, GPUs or TPU cores) used in this training.\n        \"\"\"\n        return self._setup_strategy.num_replicas_in_sync\n\n    @property\n    def train_batch_size(self) -> int:\n        \"\"\"\n        The actual batch size for training (may differ from :obj:`per_gpu_train_batch_size` in distributed training).\n        \"\"\"\n        if self.per_gpu_train_batch_size:\n            logger.warning(\n                \"Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future \"\n                \"version. Using `--per_device_train_batch_size` is preferred.\"\n            )\n        per_device_batch_size = self.per_gpu_train_batch_size or self.per_device_train_batch_size\n        return per_device_batch_size * self.n_replicas\n\n    @property\n    def eval_batch_size(self) -> int:\n        \"\"\"\n        The actual batch size for evaluation (may differ from :obj:`per_gpu_eval_batch_size` in distributed training).\n        \"\"\"\n        if self.per_gpu_eval_batch_size:\n            logger.warning(\n                \"Using deprecated `--per_gpu_eval_batch_size` argument which will be removed in a future \"\n                \"version. Using `--per_device_eval_batch_size` is preferred.\"\n            )\n        per_device_batch_size = self.per_gpu_eval_batch_size or self.per_device_eval_batch_size\n        return per_device_batch_size * self.n_replicas\n\n    @property\n    @tf_required\n    def n_gpu(self) -> int:\n        \"\"\"\n        The number of replicas (CPUs, GPUs or TPU cores) used in this training.\n        \"\"\"\n        warnings.warn(\n            \"The n_gpu argument is deprecated and will be removed in a future version, use n_replicas instead.\",\n            FutureWarning,\n        )\n        return self._setup_strategy.num_replicas_in_sync\n", "path": "src/transformers/training_args_tf.py"}]}