We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
mismatch for docs in `perplexity`
## 📚 Documentation

I am working on the NLP models and find that the implementation of `perplexity` in https://github.com/Lightning-AI/torchmetrics/blob/e4df07109586d1f605e06aed85fbc6794e8ed833/src/torchmetrics/functional/text/perplexity.py#L114-L126 is probably wrong.

Since `_perplexity_update` in https://github.com/Lightning-AI/torchmetrics/blob/e4df07109586d1f605e06aed85fbc6794e8ed833/src/torchmetrics/functional/text/perplexity.py#L84-L88 execute the `softmax`, it indicates that the `preds` **should be a logit or a unnormalized score** and definitely NOT a log probability.

I wonder if this is intentioned move or is just a simple typo. 

I also find the similar fix in issue #1366, which made the mistake. Should I make a PR to this?



--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `src/torchmetrics/functional/text/perplexity.py`
Content:
```
1 # Copyright The Lightning team.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 #
7 #     http://www.apache.org/licenses/LICENSE-2.0
8 #
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 
15 from typing import Optional, Tuple
16 
17 import torch
18 from torch import Tensor
19 from torch.nn import functional as F  # noqa: N812
20 
21 _TORCH_FLOAT_OR_DOUBLE = (torch.float32, torch.float64)
22 
23 
24 def _check_shape_and_type_consistency(preds: Tensor, target: Tensor) -> None:
25     """Check shape and type consistency of input vectors.
26 
27     Args:
28         preds:
29             Probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].
30         target:
31             Ground truth values with a shape [batch_size, seq_len].
32 
33     Raises:
34         ValueError:
35             If ``preds`` tensor has no 3 dimensions.
36         ValueError:
37             If ``target`` tensor has no 2 dimensions.
38         ValueError:
39             If the first two dimensions of ``preds`` and ``target`` do not equal.
40         TypeError:
41             If ``preds`` dtype is not one of ``(torch.float16, torch.float32, torch.float64)``
42         TypeError:
43             If ``target`` is not of a type LongTensor (torch.int64)
44     """
45     if len(preds.shape) != 3:
46         raise ValueError(
47             "Input tensor `preds` is expected to have 3 dimensions, [batch_size, seq_len, vocab_size],"
48             f" but got {len(preds.shape)}."
49         )
50     if len(target.shape) != 2:
51         raise ValueError(
52             "Input tensor `target` is expected to have 2 dimensions, [batch_size, seq_len],"
53             f" but got {len(target.shape)}."
54         )
55     if preds.shape[:2] != target.shape:
56         raise ValueError(
57             "Input tensors `preds` and `target` are expected to have equaling first two dimensions,"
58             f" [batch_size, seq_len], but got {preds.shape[:2]} and {target.shape}."
59         )
60     if preds.dtype not in _TORCH_FLOAT_OR_DOUBLE:
61         raise TypeError(
62             f"Input tensor `preds` is expected to be of a type one of {_TORCH_FLOAT_OR_DOUBLE} but got {preds.dtype}."
63         )
64     if target.dtype != torch.int64:
65         raise TypeError(f"Input tensor `target` is expected to be of a type {torch.int64} but got {target.dtype}.")
66 
67 
68 def _perplexity_update(preds: Tensor, target: Tensor, ignore_index: Optional[int] = None) -> Tuple[Tensor, Tensor]:
69     """Compute intermediate statistics for Perplexity.
70 
71     Args:
72         preds:
73             Probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].
74         target:
75             Ground truth values with a shape [batch_size, seq_len].
76         ignore_index:
77             Integer specifying a target class to ignore. If given, this class index does not contribute
78             to the returned score.
79 
80     Returns:
81         Log probabilities, summed over all samples
82         Number of samples
83     """
84     _check_shape_and_type_consistency(preds, target)
85 
86     probs = F.softmax(preds.reshape(-1, preds.shape[-1]), dim=1)
87     target = target.reshape(-1)
88 
89     if ignore_index is not None:
90         mask = target.ne(ignore_index)
91         target = target.where(target != ignore_index, torch.tensor(0, device=target.device))
92     else:
93         mask = torch.ones_like(target, dtype=torch.bool)
94 
95     probs = probs[:, target].diagonal()[mask]
96     total_log_probs = -probs.log().sum()
97     count = mask.sum()
98 
99     return total_log_probs, count
100 
101 
102 def _perplexity_compute(total: Tensor, count: Tensor) -> Tensor:
103     """Compute the Perplexity.
104 
105     Args:
106         total: Log probabilities, summed over all samples
107         count: Number of samples
108     Returns:
109         Perplexity
110     """
111     return torch.exp(total / count)
112 
113 
114 def perplexity(preds: Tensor, target: Tensor, ignore_index: Optional[int] = None) -> Tensor:
115     """Perplexity measures how well a language model predicts a text sample.
116 
117     This metric is calculated as the average number of bits per word a model needs to represent the sample.
118 
119     Args:
120         preds:
121             Log probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].
122         target:
123             Ground truth values with a shape [batch_size, seq_len].
124         ignore_index:
125             Integer specifying a target class to ignore. If given, this class index does not contribute
126             to the returned score.
127 
128     Returns:
129         Perplexity value
130 
131     Examples:
132         >>> import torch
133         >>> preds = torch.rand(2, 8, 5, generator=torch.manual_seed(22))
134         >>> target = torch.randint(5, (2, 8), generator=torch.manual_seed(22))
135         >>> target[0, 6:] = -100
136         >>> perplexity(preds, target, ignore_index=-100)
137         tensor(5.2545)
138     """
139     total, count = _perplexity_update(preds, target, ignore_index)
140     return _perplexity_compute(total, count)
141
```

Path: `src/torchmetrics/text/perplexity.py`
Content:
```
1 # Copyright The Lightning team.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 #
7 #     http://www.apache.org/licenses/LICENSE-2.0
8 #
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 
15 from typing import Any, Dict, Optional, Sequence, Union
16 
17 from torch import Tensor, tensor
18 
19 from torchmetrics.functional.text.perplexity import _perplexity_compute, _perplexity_update
20 from torchmetrics.metric import Metric
21 from torchmetrics.utilities.imports import _MATPLOTLIB_AVAILABLE
22 from torchmetrics.utilities.plot import _AX_TYPE, _PLOT_OUT_TYPE
23 
24 if not _MATPLOTLIB_AVAILABLE:
25     __doctest_skip__ = ["Perplexity.plot"]
26 
27 
28 class Perplexity(Metric):
29     r"""Perplexity measures how well a language model predicts a text sample.
30 
31     It's calculated as the average number of bits per word a model needs to represent the sample.
32 
33     As input to ``forward`` and ``update`` the metric accepts the following input:
34 
35     - ``preds`` (:class:`~torch.Tensor`): Probabilities assigned to each token in a sequence with shape
36         [batch_size, seq_len, vocab_size]
37     - ``target`` (:class:`~torch.Tensor`): Ground truth values with a shape [batch_size, seq_len]
38 
39     As output of ``forward`` and ``compute`` the metric returns the following output:
40 
41     - ``perp`` (:class:`~torch.Tensor`): A tensor with the perplexity score
42 
43     Args:
44         ignore_index: Integer specifying a target class to ignore.
45             If given, this class index does not contribute to the returned score.
46         kwargs:
47             Additional keyword arguments, see :ref:`Metric kwargs` for more info.
48 
49     Examples:
50         >>> from torchmetrics.text import Perplexity
51         >>> import torch
52         >>> preds = torch.rand(2, 8, 5, generator=torch.manual_seed(22))
53         >>> target = torch.randint(5, (2, 8), generator=torch.manual_seed(22))
54         >>> target[0, 6:] = -100
55         >>> perp = Perplexity(ignore_index=-100)
56         >>> perp(preds, target)
57         tensor(5.2545)
58     """
59     is_differentiable = True
60     higher_is_better = False
61     full_state_update = False
62     total_log_probs: Tensor
63     count: Tensor
64 
65     def __init__(
66         self,
67         ignore_index: Optional[int] = None,
68         **kwargs: Dict[str, Any],
69     ) -> None:
70         super().__init__(**kwargs)
71         if ignore_index is not None and not isinstance(ignore_index, int):
72             raise ValueError(f"Argument `ignore_index` expected to either be `None` or an `int` but got {ignore_index}")
73         self.ignore_index = ignore_index
74         self.add_state("total_log_probs", default=tensor(0.0), dist_reduce_fx="sum")
75         self.add_state("count", default=tensor(0.0), dist_reduce_fx="sum")
76 
77     def update(self, preds: Tensor, target: Tensor) -> None:
78         """Update state with predictions and targets."""
79         total_log_probs, count = _perplexity_update(preds, target, self.ignore_index)
80         self.total_log_probs += total_log_probs
81         self.count += count
82 
83     def compute(self) -> Tensor:
84         """Compute the Perplexity."""
85         return _perplexity_compute(self.total_log_probs, self.count)
86 
87     def plot(
88         self, val: Optional[Union[Tensor, Sequence[Tensor]]] = None, ax: Optional[_AX_TYPE] = None
89     ) -> _PLOT_OUT_TYPE:
90         """Plot a single or multiple values from the metric.
91 
92         Args:
93             val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results.
94                 If no value is provided, will automatically call `metric.compute` and plot that result.
95             ax: An matplotlib axis object. If provided will add plot to that axis
96 
97         Returns:
98             Figure and Axes object
99 
100         Raises:
101             ModuleNotFoundError:
102                 If `matplotlib` is not installed
103 
104         .. plot::
105             :scale: 75
106 
107             >>> # Example plotting a single value
108             >>> import torch
109             >>> from torchmetrics.text import Perplexity
110             >>> metric = Perplexity()
111             >>> metric.update(torch.rand(2, 8, 5), torch.randint(5, (2, 8)))
112             >>> fig_, ax_ = metric.plot()
113 
114         .. plot::
115             :scale: 75
116 
117             >>> # Example plotting multiple values
118             >>> import torch
119             >>> from torchmetrics.text import Perplexity
120             >>> metric = Perplexity()
121             >>> values = [ ]
122             >>> for _ in range(10):
123             ...     values.append(metric(torch.rand(2, 8, 5), torch.randint(5, (2, 8))))
124             >>> fig_, ax_ = metric.plot(values)
125         """
126         return self._plot(val, ax)
127
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/src/torchmetrics/functional/text/perplexity.py b/src/torchmetrics/functional/text/perplexity.py
--- a/src/torchmetrics/functional/text/perplexity.py
+++ b/src/torchmetrics/functional/text/perplexity.py
@@ -26,7 +26,8 @@
 
     Args:
         preds:
-            Probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].
+            Logits or a unnormalized score assigned to each token in a sequence with shape [batch_size, seq_len,
+            vocab_size]. Scores will be normalized internally using softmax.
         target:
             Ground truth values with a shape [batch_size, seq_len].
 
@@ -70,7 +71,8 @@
 
     Args:
         preds:
-            Probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].
+            Logits or a unnormalized score assigned to each token in a sequence with shape [batch_size, seq_len,
+            vocab_size]. Scores will be normalized internally using softmax.
         target:
             Ground truth values with a shape [batch_size, seq_len].
         ignore_index:
@@ -118,7 +120,8 @@
 
     Args:
         preds:
-            Log probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].
+            Logits or a unnormalized score assigned to each token in a sequence with shape [batch_size, seq_len,
+            vocab_size], which is the output of a language model. Scores will be normalized internally using softmax.
         target:
             Ground truth values with a shape [batch_size, seq_len].
         ignore_index:
diff --git a/src/torchmetrics/text/perplexity.py b/src/torchmetrics/text/perplexity.py
--- a/src/torchmetrics/text/perplexity.py
+++ b/src/torchmetrics/text/perplexity.py
@@ -32,8 +32,9 @@
 
     As input to ``forward`` and ``update`` the metric accepts the following input:
 
-    - ``preds`` (:class:`~torch.Tensor`): Probabilities assigned to each token in a sequence with shape
-        [batch_size, seq_len, vocab_size]
+    - ``preds`` (:class:`~torch.Tensor`): Logits or a unnormalized score assigned to each token in a sequence with shape
+        [batch_size, seq_len, vocab_size], which is the output of a language model. Scores will be normalized internally
+        using softmax.
     - ``target`` (:class:`~torch.Tensor`): Ground truth values with a shape [batch_size, seq_len]
 
     As output of ``forward`` and ``compute`` the metric returns the following output:

{"golden_diff": "diff --git a/src/torchmetrics/functional/text/perplexity.py b/src/torchmetrics/functional/text/perplexity.py\n--- a/src/torchmetrics/functional/text/perplexity.py\n+++ b/src/torchmetrics/functional/text/perplexity.py\n@@ -26,7 +26,8 @@\n \n     Args:\n         preds:\n-            Probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].\n+            Logits or a unnormalized score assigned to each token in a sequence with shape [batch_size, seq_len,\n+            vocab_size]. Scores will be normalized internally using softmax.\n         target:\n             Ground truth values with a shape [batch_size, seq_len].\n \n@@ -70,7 +71,8 @@\n \n     Args:\n         preds:\n-            Probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].\n+            Logits or a unnormalized score assigned to each token in a sequence with shape [batch_size, seq_len,\n+            vocab_size]. Scores will be normalized internally using softmax.\n         target:\n             Ground truth values with a shape [batch_size, seq_len].\n         ignore_index:\n@@ -118,7 +120,8 @@\n \n     Args:\n         preds:\n-            Log probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].\n+            Logits or a unnormalized score assigned to each token in a sequence with shape [batch_size, seq_len,\n+            vocab_size], which is the output of a language model. Scores will be normalized internally using softmax.\n         target:\n             Ground truth values with a shape [batch_size, seq_len].\n         ignore_index:\ndiff --git a/src/torchmetrics/text/perplexity.py b/src/torchmetrics/text/perplexity.py\n--- a/src/torchmetrics/text/perplexity.py\n+++ b/src/torchmetrics/text/perplexity.py\n@@ -32,8 +32,9 @@\n \n     As input to ``forward`` and ``update`` the metric accepts the following input:\n \n-    - ``preds`` (:class:`~torch.Tensor`): Probabilities assigned to each token in a sequence with shape\n-        [batch_size, seq_len, vocab_size]\n+    - ``preds`` (:class:`~torch.Tensor`): Logits or a unnormalized score assigned to each token in a sequence with shape\n+        [batch_size, seq_len, vocab_size], which is the output of a language model. Scores will be normalized internally\n+        using softmax.\n     - ``target`` (:class:`~torch.Tensor`): Ground truth values with a shape [batch_size, seq_len]\n \n     As output of ``forward`` and ``compute`` the metric returns the following output:\n", "issue": "mismatch for docs in `perplexity`\n## \ud83d\udcda Documentation\r\n\r\nI am working on the NLP models and find that the implementation of `perplexity` in https://github.com/Lightning-AI/torchmetrics/blob/e4df07109586d1f605e06aed85fbc6794e8ed833/src/torchmetrics/functional/text/perplexity.py#L114-L126 is probably wrong.\r\n\r\nSince `_perplexity_update` in https://github.com/Lightning-AI/torchmetrics/blob/e4df07109586d1f605e06aed85fbc6794e8ed833/src/torchmetrics/functional/text/perplexity.py#L84-L88 execute the `softmax`, it indicates that the `preds` **should be a logit or a unnormalized score** and definitely NOT a log probability.\r\n\r\nI wonder if this is intentioned move or is just a simple typo. \r\n\r\nI also find the similar fix in issue #1366, which made the mistake. Should I make a PR to this?\r\n\r\n\n", "before_files": [{"content": "# Copyright The Lightning team.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom typing import Optional, Tuple\n\nimport torch\nfrom torch import Tensor\nfrom torch.nn import functional as F  # noqa: N812\n\n_TORCH_FLOAT_OR_DOUBLE = (torch.float32, torch.float64)\n\n\ndef _check_shape_and_type_consistency(preds: Tensor, target: Tensor) -> None:\n    \"\"\"Check shape and type consistency of input vectors.\n\n    Args:\n        preds:\n            Probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].\n        target:\n            Ground truth values with a shape [batch_size, seq_len].\n\n    Raises:\n        ValueError:\n            If ``preds`` tensor has no 3 dimensions.\n        ValueError:\n            If ``target`` tensor has no 2 dimensions.\n        ValueError:\n            If the first two dimensions of ``preds`` and ``target`` do not equal.\n        TypeError:\n            If ``preds`` dtype is not one of ``(torch.float16, torch.float32, torch.float64)``\n        TypeError:\n            If ``target`` is not of a type LongTensor (torch.int64)\n    \"\"\"\n    if len(preds.shape) != 3:\n        raise ValueError(\n            \"Input tensor `preds` is expected to have 3 dimensions, [batch_size, seq_len, vocab_size],\"\n            f\" but got {len(preds.shape)}.\"\n        )\n    if len(target.shape) != 2:\n        raise ValueError(\n            \"Input tensor `target` is expected to have 2 dimensions, [batch_size, seq_len],\"\n            f\" but got {len(target.shape)}.\"\n        )\n    if preds.shape[:2] != target.shape:\n        raise ValueError(\n            \"Input tensors `preds` and `target` are expected to have equaling first two dimensions,\"\n            f\" [batch_size, seq_len], but got {preds.shape[:2]} and {target.shape}.\"\n        )\n    if preds.dtype not in _TORCH_FLOAT_OR_DOUBLE:\n        raise TypeError(\n            f\"Input tensor `preds` is expected to be of a type one of {_TORCH_FLOAT_OR_DOUBLE} but got {preds.dtype}.\"\n        )\n    if target.dtype != torch.int64:\n        raise TypeError(f\"Input tensor `target` is expected to be of a type {torch.int64} but got {target.dtype}.\")\n\n\ndef _perplexity_update(preds: Tensor, target: Tensor, ignore_index: Optional[int] = None) -> Tuple[Tensor, Tensor]:\n    \"\"\"Compute intermediate statistics for Perplexity.\n\n    Args:\n        preds:\n            Probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].\n        target:\n            Ground truth values with a shape [batch_size, seq_len].\n        ignore_index:\n            Integer specifying a target class to ignore. If given, this class index does not contribute\n            to the returned score.\n\n    Returns:\n        Log probabilities, summed over all samples\n        Number of samples\n    \"\"\"\n    _check_shape_and_type_consistency(preds, target)\n\n    probs = F.softmax(preds.reshape(-1, preds.shape[-1]), dim=1)\n    target = target.reshape(-1)\n\n    if ignore_index is not None:\n        mask = target.ne(ignore_index)\n        target = target.where(target != ignore_index, torch.tensor(0, device=target.device))\n    else:\n        mask = torch.ones_like(target, dtype=torch.bool)\n\n    probs = probs[:, target].diagonal()[mask]\n    total_log_probs = -probs.log().sum()\n    count = mask.sum()\n\n    return total_log_probs, count\n\n\ndef _perplexity_compute(total: Tensor, count: Tensor) -> Tensor:\n    \"\"\"Compute the Perplexity.\n\n    Args:\n        total: Log probabilities, summed over all samples\n        count: Number of samples\n    Returns:\n        Perplexity\n    \"\"\"\n    return torch.exp(total / count)\n\n\ndef perplexity(preds: Tensor, target: Tensor, ignore_index: Optional[int] = None) -> Tensor:\n    \"\"\"Perplexity measures how well a language model predicts a text sample.\n\n    This metric is calculated as the average number of bits per word a model needs to represent the sample.\n\n    Args:\n        preds:\n            Log probabilities assigned to each token in a sequence with shape [batch_size, seq_len, vocab_size].\n        target:\n            Ground truth values with a shape [batch_size, seq_len].\n        ignore_index:\n            Integer specifying a target class to ignore. If given, this class index does not contribute\n            to the returned score.\n\n    Returns:\n        Perplexity value\n\n    Examples:\n        >>> import torch\n        >>> preds = torch.rand(2, 8, 5, generator=torch.manual_seed(22))\n        >>> target = torch.randint(5, (2, 8), generator=torch.manual_seed(22))\n        >>> target[0, 6:] = -100\n        >>> perplexity(preds, target, ignore_index=-100)\n        tensor(5.2545)\n    \"\"\"\n    total, count = _perplexity_update(preds, target, ignore_index)\n    return _perplexity_compute(total, count)\n", "path": "src/torchmetrics/functional/text/perplexity.py"}, {"content": "# Copyright The Lightning team.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom typing import Any, Dict, Optional, Sequence, Union\n\nfrom torch import Tensor, tensor\n\nfrom torchmetrics.functional.text.perplexity import _perplexity_compute, _perplexity_update\nfrom torchmetrics.metric import Metric\nfrom torchmetrics.utilities.imports import _MATPLOTLIB_AVAILABLE\nfrom torchmetrics.utilities.plot import _AX_TYPE, _PLOT_OUT_TYPE\n\nif not _MATPLOTLIB_AVAILABLE:\n    __doctest_skip__ = [\"Perplexity.plot\"]\n\n\nclass Perplexity(Metric):\n    r\"\"\"Perplexity measures how well a language model predicts a text sample.\n\n    It's calculated as the average number of bits per word a model needs to represent the sample.\n\n    As input to ``forward`` and ``update`` the metric accepts the following input:\n\n    - ``preds`` (:class:`~torch.Tensor`): Probabilities assigned to each token in a sequence with shape\n        [batch_size, seq_len, vocab_size]\n    - ``target`` (:class:`~torch.Tensor`): Ground truth values with a shape [batch_size, seq_len]\n\n    As output of ``forward`` and ``compute`` the metric returns the following output:\n\n    - ``perp`` (:class:`~torch.Tensor`): A tensor with the perplexity score\n\n    Args:\n        ignore_index: Integer specifying a target class to ignore.\n            If given, this class index does not contribute to the returned score.\n        kwargs:\n            Additional keyword arguments, see :ref:`Metric kwargs` for more info.\n\n    Examples:\n        >>> from torchmetrics.text import Perplexity\n        >>> import torch\n        >>> preds = torch.rand(2, 8, 5, generator=torch.manual_seed(22))\n        >>> target = torch.randint(5, (2, 8), generator=torch.manual_seed(22))\n        >>> target[0, 6:] = -100\n        >>> perp = Perplexity(ignore_index=-100)\n        >>> perp(preds, target)\n        tensor(5.2545)\n    \"\"\"\n    is_differentiable = True\n    higher_is_better = False\n    full_state_update = False\n    total_log_probs: Tensor\n    count: Tensor\n\n    def __init__(\n        self,\n        ignore_index: Optional[int] = None,\n        **kwargs: Dict[str, Any],\n    ) -> None:\n        super().__init__(**kwargs)\n        if ignore_index is not None and not isinstance(ignore_index, int):\n            raise ValueError(f\"Argument `ignore_index` expected to either be `None` or an `int` but got {ignore_index}\")\n        self.ignore_index = ignore_index\n        self.add_state(\"total_log_probs\", default=tensor(0.0), dist_reduce_fx=\"sum\")\n        self.add_state(\"count\", default=tensor(0.0), dist_reduce_fx=\"sum\")\n\n    def update(self, preds: Tensor, target: Tensor) -> None:\n        \"\"\"Update state with predictions and targets.\"\"\"\n        total_log_probs, count = _perplexity_update(preds, target, self.ignore_index)\n        self.total_log_probs += total_log_probs\n        self.count += count\n\n    def compute(self) -> Tensor:\n        \"\"\"Compute the Perplexity.\"\"\"\n        return _perplexity_compute(self.total_log_probs, self.count)\n\n    def plot(\n        self, val: Optional[Union[Tensor, Sequence[Tensor]]] = None, ax: Optional[_AX_TYPE] = None\n    ) -> _PLOT_OUT_TYPE:\n        \"\"\"Plot a single or multiple values from the metric.\n\n        Args:\n            val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results.\n                If no value is provided, will automatically call `metric.compute` and plot that result.\n            ax: An matplotlib axis object. If provided will add plot to that axis\n\n        Returns:\n            Figure and Axes object\n\n        Raises:\n            ModuleNotFoundError:\n                If `matplotlib` is not installed\n\n        .. plot::\n            :scale: 75\n\n            >>> # Example plotting a single value\n            >>> import torch\n            >>> from torchmetrics.text import Perplexity\n            >>> metric = Perplexity()\n            >>> metric.update(torch.rand(2, 8, 5), torch.randint(5, (2, 8)))\n            >>> fig_, ax_ = metric.plot()\n\n        .. plot::\n            :scale: 75\n\n            >>> # Example plotting multiple values\n            >>> import torch\n            >>> from torchmetrics.text import Perplexity\n            >>> metric = Perplexity()\n            >>> values = [ ]\n            >>> for _ in range(10):\n            ...     values.append(metric(torch.rand(2, 8, 5), torch.randint(5, (2, 8))))\n            >>> fig_, ax_ = metric.plot(values)\n        \"\"\"\n        return self._plot(val, ax)\n", "path": "src/torchmetrics/text/perplexity.py"}], "after_files": [{"content": "# Copyright The Lightning team.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom typing import Optional, Tuple\n\nimport torch\nfrom torch import Tensor\nfrom torch.nn import functional as F  # noqa: N812\n\n_TORCH_FLOAT_OR_DOUBLE = (torch.float32, torch.float64)\n\n\ndef _check_shape_and_type_consistency(preds: Tensor, target: Tensor) -> None:\n    \"\"\"Check shape and type consistency of input vectors.\n\n    Args:\n        preds:\n            Logits or a unnormalized score assigned to each token in a sequence with shape [batch_size, seq_len,\n            vocab_size]. Scores will be normalized internally using softmax.\n        target:\n            Ground truth values with a shape [batch_size, seq_len].\n\n    Raises:\n        ValueError:\n            If ``preds`` tensor has no 3 dimensions.\n        ValueError:\n            If ``target`` tensor has no 2 dimensions.\n        ValueError:\n            If the first two dimensions of ``preds`` and ``target`` do not equal.\n        TypeError:\n            If ``preds`` dtype is not one of ``(torch.float16, torch.float32, torch.float64)``\n        TypeError:\n            If ``target`` is not of a type LongTensor (torch.int64)\n    \"\"\"\n    if len(preds.shape) != 3:\n        raise ValueError(\n            \"Input tensor `preds` is expected to have 3 dimensions, [batch_size, seq_len, vocab_size],\"\n            f\" but got {len(preds.shape)}.\"\n        )\n    if len(target.shape) != 2:\n        raise ValueError(\n            \"Input tensor `target` is expected to have 2 dimensions, [batch_size, seq_len],\"\n            f\" but got {len(target.shape)}.\"\n        )\n    if preds.shape[:2] != target.shape:\n        raise ValueError(\n            \"Input tensors `preds` and `target` are expected to have equaling first two dimensions,\"\n            f\" [batch_size, seq_len], but got {preds.shape[:2]} and {target.shape}.\"\n        )\n    if preds.dtype not in _TORCH_FLOAT_OR_DOUBLE:\n        raise TypeError(\n            f\"Input tensor `preds` is expected to be of a type one of {_TORCH_FLOAT_OR_DOUBLE} but got {preds.dtype}.\"\n        )\n    if target.dtype != torch.int64:\n        raise TypeError(f\"Input tensor `target` is expected to be of a type {torch.int64} but got {target.dtype}.\")\n\n\ndef _perplexity_update(preds: Tensor, target: Tensor, ignore_index: Optional[int] = None) -> Tuple[Tensor, Tensor]:\n    \"\"\"Compute intermediate statistics for Perplexity.\n\n    Args:\n        preds:\n            Logits or a unnormalized score assigned to each token in a sequence with shape [batch_size, seq_len,\n            vocab_size]. Scores will be normalized internally using softmax.\n        target:\n            Ground truth values with a shape [batch_size, seq_len].\n        ignore_index:\n            Integer specifying a target class to ignore. If given, this class index does not contribute\n            to the returned score.\n\n    Returns:\n        Log probabilities, summed over all samples\n        Number of samples\n    \"\"\"\n    _check_shape_and_type_consistency(preds, target)\n\n    probs = F.softmax(preds.reshape(-1, preds.shape[-1]), dim=1)\n    target = target.reshape(-1)\n\n    if ignore_index is not None:\n        mask = target.ne(ignore_index)\n        target = target.where(target != ignore_index, torch.tensor(0, device=target.device))\n    else:\n        mask = torch.ones_like(target, dtype=torch.bool)\n\n    probs = probs[:, target].diagonal()[mask]\n    total_log_probs = -probs.log().sum()\n    count = mask.sum()\n\n    return total_log_probs, count\n\n\ndef _perplexity_compute(total: Tensor, count: Tensor) -> Tensor:\n    \"\"\"Compute the Perplexity.\n\n    Args:\n        total: Log probabilities, summed over all samples\n        count: Number of samples\n    Returns:\n        Perplexity\n    \"\"\"\n    return torch.exp(total / count)\n\n\ndef perplexity(preds: Tensor, target: Tensor, ignore_index: Optional[int] = None) -> Tensor:\n    \"\"\"Perplexity measures how well a language model predicts a text sample.\n\n    This metric is calculated as the average number of bits per word a model needs to represent the sample.\n\n    Args:\n        preds:\n            Logits or a unnormalized score assigned to each token in a sequence with shape [batch_size, seq_len,\n            vocab_size], which is the output of a language model. Scores will be normalized internally using softmax.\n        target:\n            Ground truth values with a shape [batch_size, seq_len].\n        ignore_index:\n            Integer specifying a target class to ignore. If given, this class index does not contribute\n            to the returned score.\n\n    Returns:\n        Perplexity value\n\n    Examples:\n        >>> import torch\n        >>> preds = torch.rand(2, 8, 5, generator=torch.manual_seed(22))\n        >>> target = torch.randint(5, (2, 8), generator=torch.manual_seed(22))\n        >>> target[0, 6:] = -100\n        >>> perplexity(preds, target, ignore_index=-100)\n        tensor(5.2545)\n    \"\"\"\n    total, count = _perplexity_update(preds, target, ignore_index)\n    return _perplexity_compute(total, count)\n", "path": "src/torchmetrics/functional/text/perplexity.py"}, {"content": "# Copyright The Lightning team.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom typing import Any, Dict, Optional, Sequence, Union\n\nfrom torch import Tensor, tensor\n\nfrom torchmetrics.functional.text.perplexity import _perplexity_compute, _perplexity_update\nfrom torchmetrics.metric import Metric\nfrom torchmetrics.utilities.imports import _MATPLOTLIB_AVAILABLE\nfrom torchmetrics.utilities.plot import _AX_TYPE, _PLOT_OUT_TYPE\n\nif not _MATPLOTLIB_AVAILABLE:\n    __doctest_skip__ = [\"Perplexity.plot\"]\n\n\nclass Perplexity(Metric):\n    r\"\"\"Perplexity measures how well a language model predicts a text sample.\n\n    It's calculated as the average number of bits per word a model needs to represent the sample.\n\n    As input to ``forward`` and ``update`` the metric accepts the following input:\n\n    - ``preds`` (:class:`~torch.Tensor`): Logits or a unnormalized score assigned to each token in a sequence with shape\n        [batch_size, seq_len, vocab_size], which is the output of a language model. Scores will be normalized internally\n        using softmax.\n    - ``target`` (:class:`~torch.Tensor`): Ground truth values with a shape [batch_size, seq_len]\n\n    As output of ``forward`` and ``compute`` the metric returns the following output:\n\n    - ``perp`` (:class:`~torch.Tensor`): A tensor with the perplexity score\n\n    Args:\n        ignore_index: Integer specifying a target class to ignore.\n            If given, this class index does not contribute to the returned score.\n        kwargs:\n            Additional keyword arguments, see :ref:`Metric kwargs` for more info.\n\n    Examples:\n        >>> from torchmetrics.text import Perplexity\n        >>> import torch\n        >>> preds = torch.rand(2, 8, 5, generator=torch.manual_seed(22))\n        >>> target = torch.randint(5, (2, 8), generator=torch.manual_seed(22))\n        >>> target[0, 6:] = -100\n        >>> perp = Perplexity(ignore_index=-100)\n        >>> perp(preds, target)\n        tensor(5.2545)\n    \"\"\"\n    is_differentiable = True\n    higher_is_better = False\n    full_state_update = False\n    total_log_probs: Tensor\n    count: Tensor\n\n    def __init__(\n        self,\n        ignore_index: Optional[int] = None,\n        **kwargs: Dict[str, Any],\n    ) -> None:\n        super().__init__(**kwargs)\n        if ignore_index is not None and not isinstance(ignore_index, int):\n            raise ValueError(f\"Argument `ignore_index` expected to either be `None` or an `int` but got {ignore_index}\")\n        self.ignore_index = ignore_index\n        self.add_state(\"total_log_probs\", default=tensor(0.0), dist_reduce_fx=\"sum\")\n        self.add_state(\"count\", default=tensor(0.0), dist_reduce_fx=\"sum\")\n\n    def update(self, preds: Tensor, target: Tensor) -> None:\n        \"\"\"Update state with predictions and targets.\"\"\"\n        total_log_probs, count = _perplexity_update(preds, target, self.ignore_index)\n        self.total_log_probs += total_log_probs\n        self.count += count\n\n    def compute(self) -> Tensor:\n        \"\"\"Compute the Perplexity.\"\"\"\n        return _perplexity_compute(self.total_log_probs, self.count)\n\n    def plot(\n        self, val: Optional[Union[Tensor, Sequence[Tensor]]] = None, ax: Optional[_AX_TYPE] = None\n    ) -> _PLOT_OUT_TYPE:\n        \"\"\"Plot a single or multiple values from the metric.\n\n        Args:\n            val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results.\n                If no value is provided, will automatically call `metric.compute` and plot that result.\n            ax: An matplotlib axis object. If provided will add plot to that axis\n\n        Returns:\n            Figure and Axes object\n\n        Raises:\n            ModuleNotFoundError:\n                If `matplotlib` is not installed\n\n        .. plot::\n            :scale: 75\n\n            >>> # Example plotting a single value\n            >>> import torch\n            >>> from torchmetrics.text import Perplexity\n            >>> metric = Perplexity()\n            >>> metric.update(torch.rand(2, 8, 5), torch.randint(5, (2, 8)))\n            >>> fig_, ax_ = metric.plot()\n\n        .. plot::\n            :scale: 75\n\n            >>> # Example plotting multiple values\n            >>> import torch\n            >>> from torchmetrics.text import Perplexity\n            >>> metric = Perplexity()\n            >>> values = [ ]\n            >>> for _ in range(10):\n            ...     values.append(metric(torch.rand(2, 8, 5), torch.randint(5, (2, 8))))\n            >>> fig_, ax_ = metric.plot(values)\n        \"\"\"\n        return self._plot(val, ax)\n", "path": "src/torchmetrics/text/perplexity.py"}]}