We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Local Install: No rule to make target 'install-flash-attention-v2'
### System Info

**Release:** v1.3.4, main branch
**Target**: x86_64-unknown-linux-gnu
**Cargo version**: 1.70.0
**Commit sha**: 630800eed37b15c4b0c9eb8e6ab47212026720f7
**Docker label**: N/A
**nvidia-smi**: Wed Jan  3 18:13:50 2024
  
```
Mon Jan  8 15:13:03 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        On  | 00000000:01:00.0 Off |                  Off |
|  0%   40C    P8              26W / 450W |      3MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        On  | 00000000:02:00.0 Off |                  Off |
|  0%   46C    P8              26W / 450W |      3MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
```

### Information

- [ ] Docker
- [X] The CLI directly

### Tasks

- [X] An officially supported command
- [ ] My own modifications

### Reproduction

Followed steps found within the local install instructions (https://github.com/huggingface/text-generation-inference#local-install), in addition to a couple undocumented make steps 

1. Fork & pull from main branch, v1.3.4
2. Install and test Rust: 1.70.0 (90c541806 2023-05-31)
3. Install & test Conda: 23.11.0
5. Install and test libprotoc: 3.21.12
6. Execute `export BUILD_EXTENSIONS=True`
7. Execute `make install`
8. Execute `cd server && make install-vllm-cuda && cd ..`
9. Update source code for paged attention [as described here](https://github.com/huggingface/text-generation-inference/pull/1386/commits/ad7f8396732c12368c7961e15fe2287f1aa42db3)
10. Execute `cd server && make install-flash-attention && cd ..`
11. Execute `make run-falcon-7b-instruct`, which yields the following message:

_Flash Attention V2 is not installed.
Use the official Docker image (ghcr.io/huggingface/text-generation-inference:latest) or install flash attention v2 with `cd server && make install install-flash-attention-v2`_

12. Execute `cd server && make install install-flash-attention-v2`, which yields the following error message:

_make: *** No rule to make target 'install-flash-attention-v2'.  Stop._

### Expected behavior

Make target should be found and executed. Solution below.

**SOLUTION**: 

The file that issues the warning, `server/text_generation_server/utils/flash_attn.py`, is missing the proper make target suffix. It should be: 

`make install install-flash-attention-v2-cuda` or `install-flash-attention-v2-rocm`

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `server/text_generation_server/utils/flash_attn.py`
Content:
```
1 import os
2 import torch
3 
4 from loguru import logger
5 
6 from text_generation_server.utils.import_utils import IS_CUDA_SYSTEM, IS_ROCM_SYSTEM
7 
8 if os.getenv("USE_FLASH_ATTENTION", "").lower() == "false":
9     raise ImportError("`USE_FLASH_ATTENTION` is false.")
10 
11 if not torch.cuda.is_available():
12     raise ImportError("CUDA is not available")
13 
14 major, minor = torch.cuda.get_device_capability()
15 is_sm75 = major == 7 and minor == 5
16 is_sm8x = major == 8 and minor >= 0
17 is_sm90 = major == 9 and minor == 0
18 
19 HAS_FLASH_ATTN = False
20 HAS_FLASH_ATTN_V2_CUDA = False
21 HAS_FLASH_ATTN_V2_ROCM = False
22 try:
23     try:
24         import flash_attn_2_cuda
25     except ImportError:
26         raise ImportError(
27             "Flash Attention V2 is not installed.\n"
28             "Use the official Docker image (ghcr.io/huggingface/text-generation-inference:latest) "
29             "or install flash attention v2 with `cd server && make install install-flash-attention-v2`"
30         )
31     if not (is_sm8x or is_sm90):
32         raise ImportError(
33             f"GPU with CUDA capability {major} {minor} is not supported for "
34             "Flash Attention V2"
35         )
36     HAS_FLASH_ATTN_V2_CUDA = IS_CUDA_SYSTEM
37     HAS_FLASH_ATTN_V2_ROCM = IS_ROCM_SYSTEM
38 except ImportError as e:
39     try:
40         import flash_attn_cuda
41     except ImportError:
42         raise ImportError(
43             "Flash Attention is not installed.\n"
44             "Use the official Docker image (ghcr.io/huggingface/text-generation-inference:latest) "
45             "or install flash attention with `cd server && make install install-flash-attention`"
46         ) from e
47 
48     if IS_CUDA_SYSTEM and not (is_sm75 or is_sm8x or is_sm90):
49         raise ImportError(
50             f"GPU with CUDA capability {major} {minor} is not supported"
51         ) from e
52     elif IS_ROCM_SYSTEM:
53         for idx in range(torch.cuda.device_count()):
54             if "MI210" not in torch.cuda.get_device_name(
55                 idx
56             ) and "MI250" not in torch.cuda.get_device_name(idx):
57                 raise ImportError(
58                     f"AMD GPU {torch.cuda.get_device_name(idx)} does not support flash-attention"
59                 )
60 
61     logger.warning(f"Unable to use Flash Attention V2: {e}")
62     HAS_FLASH_ATTN = True
63 
64 
65 def attention(
66     q,
67     k,
68     v,
69     out,
70     cu_seqlens,
71     max_s,
72     softmax_scale,
73     window_size_left=-1,
74 ):
75     if window_size_left <= 0 and window_size_left != -1:
76         raise ValueError("`window_size_left` must be > 0 or -1")
77 
78     if HAS_FLASH_ATTN_V2_CUDA:
79         return flash_attn_2_cuda.varlen_fwd(
80             q,
81             k,
82             v,
83             out,
84             cu_seqlens,
85             cu_seqlens,
86             max_s,
87             max_s,
88             0.0,
89             softmax_scale,
90             False,
91             True,
92             window_size_left,
93             0,
94             False,
95             None,
96         )
97     elif HAS_FLASH_ATTN_V2_ROCM:
98         if window_size_left != -1:
99             raise ValueError(
100                 f"RoCm version of Flash Attention v2 does not support window attention (window_size_left != -1, got window_size_left={window_size_left})."
101             )
102 
103         # RoCm flash API does not take the window_size_left and window_size_right arguments.
104         return flash_attn_2_cuda.varlen_fwd(
105             q,
106             k,
107             v,
108             out,
109             cu_seqlens,
110             cu_seqlens,
111             max_s,
112             max_s,
113             0.0,
114             softmax_scale,
115             False,
116             True,
117             False,
118             None,
119         )
120     elif HAS_FLASH_ATTN:
121         if window_size_left != -1:
122             raise NotImplementedError(
123                 "window_size_left is only available with flash attn v2"
124             )
125 
126         # Flash attention v1 requires q, k and v to have the same number of heads
127         if k.shape[1] != q.shape[1]:
128             # MQA expand
129             if k.shape[1] == 1:
130                 k = k.expand(-1, q.shape[1], -1)
131             # Grouped attention reshape
132             else:
133                 original_shape = k.shape
134                 k = (
135                     k.unsqueeze(2)
136                     .expand(-1, -1, q.shape[1] // k.shape[1], -1)
137                     .reshape(original_shape[0], -1, original_shape[2])
138                 )
139         if v.shape[1] != q.shape[1]:
140             # MQA expand
141             if v.shape[1] == 1:
142                 v = v.expand(-1, q.shape[1], -1)
143             # Grouped attention reshape
144             else:
145                 original_shape = v.shape
146                 v = (
147                     v.unsqueeze(2)
148                     .expand(-1, -1, q.shape[1] // v.shape[1], -1)
149                     .reshape(original_shape[0], -1, original_shape[2])
150                 )
151 
152         return flash_attn_cuda.fwd(
153             q,
154             k,
155             v,
156             out,
157             cu_seqlens,
158             cu_seqlens,
159             max_s,
160             max_s,
161             0.0,
162             softmax_scale,
163             False,
164             True,
165             False,
166             0,
167             None,
168         )
169 
170     raise NotImplementedError("flash attention is not installed")
171
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/server/text_generation_server/utils/flash_attn.py b/server/text_generation_server/utils/flash_attn.py
--- a/server/text_generation_server/utils/flash_attn.py
+++ b/server/text_generation_server/utils/flash_attn.py
@@ -23,10 +23,15 @@
     try:
         import flash_attn_2_cuda
     except ImportError:
+        architecture_suffix = ""
+        if IS_CUDA_SYSTEM:
+            architecture_suffix = "-cuda"
+        elif IS_ROCM_SYSTEM:
+            architecture_suffix = "-rocm"
         raise ImportError(
             "Flash Attention V2 is not installed.\n"
             "Use the official Docker image (ghcr.io/huggingface/text-generation-inference:latest) "
-            "or install flash attention v2 with `cd server && make install install-flash-attention-v2`"
+            f"or install flash attention v2 with `cd server && make install install-flash-attention-v2{architecture_suffix}`"
         )
     if not (is_sm8x or is_sm90):
         raise ImportError(

{"golden_diff": "diff --git a/server/text_generation_server/utils/flash_attn.py b/server/text_generation_server/utils/flash_attn.py\n--- a/server/text_generation_server/utils/flash_attn.py\n+++ b/server/text_generation_server/utils/flash_attn.py\n@@ -23,10 +23,15 @@\n     try:\n         import flash_attn_2_cuda\n     except ImportError:\n+        architecture_suffix = \"\"\n+        if IS_CUDA_SYSTEM:\n+            architecture_suffix = \"-cuda\"\n+        elif IS_ROCM_SYSTEM:\n+            architecture_suffix = \"-rocm\"\n         raise ImportError(\n             \"Flash Attention V2 is not installed.\\n\"\n             \"Use the official Docker image (ghcr.io/huggingface/text-generation-inference:latest) \"\n-            \"or install flash attention v2 with `cd server && make install install-flash-attention-v2`\"\n+            f\"or install flash attention v2 with `cd server && make install install-flash-attention-v2{architecture_suffix}`\"\n         )\n     if not (is_sm8x or is_sm90):\n         raise ImportError(\n", "issue": "Local Install: No rule to make target 'install-flash-attention-v2'\n### System Info\r\n\r\n**Release:** v1.3.4, main branch\r\n**Target**: x86_64-unknown-linux-gnu\r\n**Cargo version**: 1.70.0\r\n**Commit sha**: 630800eed37b15c4b0c9eb8e6ab47212026720f7\r\n**Docker label**: N/A\r\n**nvidia-smi**: Wed Jan  3 18:13:50 2024\r\n  \r\n```\r\nMon Jan  8 15:13:03 2024\r\n+---------------------------------------------------------------------------------------+\r\n| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |\r\n|-----------------------------------------+----------------------+----------------------+\r\n| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |\r\n|                                         |                      |               MIG M. |\r\n|=========================================+======================+======================|\r\n|   0  NVIDIA GeForce RTX 4090        On  | 00000000:01:00.0 Off |                  Off |\r\n|  0%   40C    P8              26W / 450W |      3MiB / 24564MiB |      0%      Default |\r\n|                                         |                      |                  N/A |\r\n+-----------------------------------------+----------------------+----------------------+\r\n|   1  NVIDIA GeForce RTX 4090        On  | 00000000:02:00.0 Off |                  Off |\r\n|  0%   46C    P8              26W / 450W |      3MiB / 24564MiB |      0%      Default |\r\n|                                         |                      |                  N/A |\r\n+-----------------------------------------+----------------------+----------------------+\r\n\r\n+---------------------------------------------------------------------------------------+\r\n| Processes:                                                                            |\r\n|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |\r\n|        ID   ID                                                             Usage      |\r\n|=======================================================================================|\r\n|  No running processes found                                                           |\r\n+---------------------------------------------------------------------------------------+\r\n```\r\n\r\n### Information\r\n\r\n- [ ] Docker\r\n- [X] The CLI directly\r\n\r\n### Tasks\r\n\r\n- [X] An officially supported command\r\n- [ ] My own modifications\r\n\r\n### Reproduction\r\n\r\nFollowed steps found within the local install instructions (https://github.com/huggingface/text-generation-inference#local-install), in addition to a couple undocumented make steps \r\n\r\n1. Fork & pull from main branch, v1.3.4\r\n2. Install and test Rust: 1.70.0 (90c541806 2023-05-31)\r\n3. Install & test Conda: 23.11.0\r\n5. Install and test libprotoc: 3.21.12\r\n6. Execute `export BUILD_EXTENSIONS=True`\r\n7. Execute `make install`\r\n8. Execute `cd server && make install-vllm-cuda && cd ..`\r\n9. Update source code for paged attention [as described here](https://github.com/huggingface/text-generation-inference/pull/1386/commits/ad7f8396732c12368c7961e15fe2287f1aa42db3)\r\n10. Execute `cd server && make install-flash-attention && cd ..`\r\n11. Execute `make run-falcon-7b-instruct`, which yields the following message:\r\n\r\n_Flash Attention V2 is not installed.\r\nUse the official Docker image (ghcr.io/huggingface/text-generation-inference:latest) or install flash attention v2 with `cd server && make install install-flash-attention-v2`_\r\n\r\n12. Execute `cd server && make install install-flash-attention-v2`, which yields the following error message:\r\n\r\n_make: *** No rule to make target 'install-flash-attention-v2'.  Stop._\r\n\r\n### Expected behavior\r\n\r\nMake target should be found and executed. Solution below.\r\n\r\n**SOLUTION**: \r\n\r\nThe file that issues the warning, `server/text_generation_server/utils/flash_attn.py`, is missing the proper make target suffix. It should be: \r\n\r\n`make install install-flash-attention-v2-cuda` or `install-flash-attention-v2-rocm`\n", "before_files": [{"content": "import os\nimport torch\n\nfrom loguru import logger\n\nfrom text_generation_server.utils.import_utils import IS_CUDA_SYSTEM, IS_ROCM_SYSTEM\n\nif os.getenv(\"USE_FLASH_ATTENTION\", \"\").lower() == \"false\":\n    raise ImportError(\"`USE_FLASH_ATTENTION` is false.\")\n\nif not torch.cuda.is_available():\n    raise ImportError(\"CUDA is not available\")\n\nmajor, minor = torch.cuda.get_device_capability()\nis_sm75 = major == 7 and minor == 5\nis_sm8x = major == 8 and minor >= 0\nis_sm90 = major == 9 and minor == 0\n\nHAS_FLASH_ATTN = False\nHAS_FLASH_ATTN_V2_CUDA = False\nHAS_FLASH_ATTN_V2_ROCM = False\ntry:\n    try:\n        import flash_attn_2_cuda\n    except ImportError:\n        raise ImportError(\n            \"Flash Attention V2 is not installed.\\n\"\n            \"Use the official Docker image (ghcr.io/huggingface/text-generation-inference:latest) \"\n            \"or install flash attention v2 with `cd server && make install install-flash-attention-v2`\"\n        )\n    if not (is_sm8x or is_sm90):\n        raise ImportError(\n            f\"GPU with CUDA capability {major} {minor} is not supported for \"\n            \"Flash Attention V2\"\n        )\n    HAS_FLASH_ATTN_V2_CUDA = IS_CUDA_SYSTEM\n    HAS_FLASH_ATTN_V2_ROCM = IS_ROCM_SYSTEM\nexcept ImportError as e:\n    try:\n        import flash_attn_cuda\n    except ImportError:\n        raise ImportError(\n            \"Flash Attention is not installed.\\n\"\n            \"Use the official Docker image (ghcr.io/huggingface/text-generation-inference:latest) \"\n            \"or install flash attention with `cd server && make install install-flash-attention`\"\n        ) from e\n\n    if IS_CUDA_SYSTEM and not (is_sm75 or is_sm8x or is_sm90):\n        raise ImportError(\n            f\"GPU with CUDA capability {major} {minor} is not supported\"\n        ) from e\n    elif IS_ROCM_SYSTEM:\n        for idx in range(torch.cuda.device_count()):\n            if \"MI210\" not in torch.cuda.get_device_name(\n                idx\n            ) and \"MI250\" not in torch.cuda.get_device_name(idx):\n                raise ImportError(\n                    f\"AMD GPU {torch.cuda.get_device_name(idx)} does not support flash-attention\"\n                )\n\n    logger.warning(f\"Unable to use Flash Attention V2: {e}\")\n    HAS_FLASH_ATTN = True\n\n\ndef attention(\n    q,\n    k,\n    v,\n    out,\n    cu_seqlens,\n    max_s,\n    softmax_scale,\n    window_size_left=-1,\n):\n    if window_size_left <= 0 and window_size_left != -1:\n        raise ValueError(\"`window_size_left` must be > 0 or -1\")\n\n    if HAS_FLASH_ATTN_V2_CUDA:\n        return flash_attn_2_cuda.varlen_fwd(\n            q,\n            k,\n            v,\n            out,\n            cu_seqlens,\n            cu_seqlens,\n            max_s,\n            max_s,\n            0.0,\n            softmax_scale,\n            False,\n            True,\n            window_size_left,\n            0,\n            False,\n            None,\n        )\n    elif HAS_FLASH_ATTN_V2_ROCM:\n        if window_size_left != -1:\n            raise ValueError(\n                f\"RoCm version of Flash Attention v2 does not support window attention (window_size_left != -1, got window_size_left={window_size_left}).\"\n            )\n\n        # RoCm flash API does not take the window_size_left and window_size_right arguments.\n        return flash_attn_2_cuda.varlen_fwd(\n            q,\n            k,\n            v,\n            out,\n            cu_seqlens,\n            cu_seqlens,\n            max_s,\n            max_s,\n            0.0,\n            softmax_scale,\n            False,\n            True,\n            False,\n            None,\n        )\n    elif HAS_FLASH_ATTN:\n        if window_size_left != -1:\n            raise NotImplementedError(\n                \"window_size_left is only available with flash attn v2\"\n            )\n\n        # Flash attention v1 requires q, k and v to have the same number of heads\n        if k.shape[1] != q.shape[1]:\n            # MQA expand\n            if k.shape[1] == 1:\n                k = k.expand(-1, q.shape[1], -1)\n            # Grouped attention reshape\n            else:\n                original_shape = k.shape\n                k = (\n                    k.unsqueeze(2)\n                    .expand(-1, -1, q.shape[1] // k.shape[1], -1)\n                    .reshape(original_shape[0], -1, original_shape[2])\n                )\n        if v.shape[1] != q.shape[1]:\n            # MQA expand\n            if v.shape[1] == 1:\n                v = v.expand(-1, q.shape[1], -1)\n            # Grouped attention reshape\n            else:\n                original_shape = v.shape\n                v = (\n                    v.unsqueeze(2)\n                    .expand(-1, -1, q.shape[1] // v.shape[1], -1)\n                    .reshape(original_shape[0], -1, original_shape[2])\n                )\n\n        return flash_attn_cuda.fwd(\n            q,\n            k,\n            v,\n            out,\n            cu_seqlens,\n            cu_seqlens,\n            max_s,\n            max_s,\n            0.0,\n            softmax_scale,\n            False,\n            True,\n            False,\n            0,\n            None,\n        )\n\n    raise NotImplementedError(\"flash attention is not installed\")\n", "path": "server/text_generation_server/utils/flash_attn.py"}], "after_files": [{"content": "import os\nimport torch\n\nfrom loguru import logger\n\nfrom text_generation_server.utils.import_utils import IS_CUDA_SYSTEM, IS_ROCM_SYSTEM\n\nif os.getenv(\"USE_FLASH_ATTENTION\", \"\").lower() == \"false\":\n    raise ImportError(\"`USE_FLASH_ATTENTION` is false.\")\n\nif not torch.cuda.is_available():\n    raise ImportError(\"CUDA is not available\")\n\nmajor, minor = torch.cuda.get_device_capability()\nis_sm75 = major == 7 and minor == 5\nis_sm8x = major == 8 and minor >= 0\nis_sm90 = major == 9 and minor == 0\n\nHAS_FLASH_ATTN = False\nHAS_FLASH_ATTN_V2_CUDA = False\nHAS_FLASH_ATTN_V2_ROCM = False\ntry:\n    try:\n        import flash_attn_2_cuda\n    except ImportError:\n        architecture_suffix = \"\"\n        if IS_CUDA_SYSTEM:\n            architecture_suffix = \"-cuda\"\n        elif IS_ROCM_SYSTEM:\n            architecture_suffix = \"-rocm\"\n        raise ImportError(\n            \"Flash Attention V2 is not installed.\\n\"\n            \"Use the official Docker image (ghcr.io/huggingface/text-generation-inference:latest) \"\n            f\"or install flash attention v2 with `cd server && make install install-flash-attention-v2{architecture_suffix}`\"\n        )\n    if not (is_sm8x or is_sm90):\n        raise ImportError(\n            f\"GPU with CUDA capability {major} {minor} is not supported for \"\n            \"Flash Attention V2\"\n        )\n    HAS_FLASH_ATTN_V2_CUDA = IS_CUDA_SYSTEM\n    HAS_FLASH_ATTN_V2_ROCM = IS_ROCM_SYSTEM\nexcept ImportError as e:\n    try:\n        import flash_attn_cuda\n    except ImportError:\n        raise ImportError(\n            \"Flash Attention is not installed.\\n\"\n            \"Use the official Docker image (ghcr.io/huggingface/text-generation-inference:latest) \"\n            \"or install flash attention with `cd server && make install install-flash-attention`\"\n        ) from e\n\n    if IS_CUDA_SYSTEM and not (is_sm75 or is_sm8x or is_sm90):\n        raise ImportError(\n            f\"GPU with CUDA capability {major} {minor} is not supported\"\n        ) from e\n    elif IS_ROCM_SYSTEM:\n        for idx in range(torch.cuda.device_count()):\n            if \"MI210\" not in torch.cuda.get_device_name(\n                idx\n            ) and \"MI250\" not in torch.cuda.get_device_name(idx):\n                raise ImportError(\n                    f\"AMD GPU {torch.cuda.get_device_name(idx)} does not support flash-attention\"\n                )\n\n    logger.warning(f\"Unable to use Flash Attention V2: {e}\")\n    HAS_FLASH_ATTN = True\n\n\ndef attention(\n    q,\n    k,\n    v,\n    out,\n    cu_seqlens,\n    max_s,\n    softmax_scale,\n    window_size_left=-1,\n):\n    if window_size_left <= 0 and window_size_left != -1:\n        raise ValueError(\"`window_size_left` must be > 0 or -1\")\n\n    if HAS_FLASH_ATTN_V2_CUDA:\n        return flash_attn_2_cuda.varlen_fwd(\n            q,\n            k,\n            v,\n            out,\n            cu_seqlens,\n            cu_seqlens,\n            max_s,\n            max_s,\n            0.0,\n            softmax_scale,\n            False,\n            True,\n            window_size_left,\n            0,\n            False,\n            None,\n        )\n    elif HAS_FLASH_ATTN_V2_ROCM:\n        if window_size_left != -1:\n            raise ValueError(\n                f\"RoCm version of Flash Attention v2 does not support window attention (window_size_left != -1, got window_size_left={window_size_left}).\"\n            )\n\n        # RoCm flash API does not take the window_size_left and window_size_right arguments.\n        return flash_attn_2_cuda.varlen_fwd(\n            q,\n            k,\n            v,\n            out,\n            cu_seqlens,\n            cu_seqlens,\n            max_s,\n            max_s,\n            0.0,\n            softmax_scale,\n            False,\n            True,\n            False,\n            None,\n        )\n    elif HAS_FLASH_ATTN:\n        if window_size_left != -1:\n            raise NotImplementedError(\n                \"window_size_left is only available with flash attn v2\"\n            )\n\n        # Flash attention v1 requires q, k and v to have the same number of heads\n        if k.shape[1] != q.shape[1]:\n            # MQA expand\n            if k.shape[1] == 1:\n                k = k.expand(-1, q.shape[1], -1)\n            # Grouped attention reshape\n            else:\n                original_shape = k.shape\n                k = (\n                    k.unsqueeze(2)\n                    .expand(-1, -1, q.shape[1] // k.shape[1], -1)\n                    .reshape(original_shape[0], -1, original_shape[2])\n                )\n        if v.shape[1] != q.shape[1]:\n            # MQA expand\n            if v.shape[1] == 1:\n                v = v.expand(-1, q.shape[1], -1)\n            # Grouped attention reshape\n            else:\n                original_shape = v.shape\n                v = (\n                    v.unsqueeze(2)\n                    .expand(-1, -1, q.shape[1] // v.shape[1], -1)\n                    .reshape(original_shape[0], -1, original_shape[2])\n                )\n\n        return flash_attn_cuda.fwd(\n            q,\n            k,\n            v,\n            out,\n            cu_seqlens,\n            cu_seqlens,\n            max_s,\n            max_s,\n            0.0,\n            softmax_scale,\n            False,\n            True,\n            False,\n            0,\n            None,\n        )\n\n    raise NotImplementedError(\"flash attention is not installed\")\n", "path": "server/text_generation_server/utils/flash_attn.py"}]}