We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Mixtral tutorial doesn't work without huggingface access token
The tutorial for running mixtral on VLLM doesn't work since the model cannot be downloaded without a huggingface access token. This is because mixtral is now a gated model: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1

```
Image build for im-1P0Aou6cl9H3BAwictbALw failed with the exception:                                           │
│ GatedRepoError('401 Client Error. (Request ID:                                                                 │
│ Root=1-66213747-475d6ad5261bb9eb4931c4fd;025f8bf1-0bb2-42ac-86a0-743e752004a0)\n\nCannot access gated repo for │
│ url https://huggingface.co/api/models/mistralai/Mixtral-8x7B-Instruct-v0.1/revision/main.\nRepo model          │
│ mistralai/Mixtral-8x7B-Instruct-v0.1 is gated. You must be authenticated to access it.')
```

Affected Tutorial: https://modal.com/docs/examples/vllm_mixtral
Affected Code: https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/llm-serving/vllm_mixtral.py

This can be fixed using a similar approach as done [here]( https://github.com/modal-labs/modal-examples/pull/539) (adding an env var for a `HF_TOKEN`) to the [function call](https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/llm-serving/vllm_mixtral.py#L45-L49) where the model is downloaded.
Also the tutorial needs to be updated to inform the user that a huggingface access token is required.

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `06_gpu_and_ml/llm-serving/vllm_mixtral.py`
Content:
```
1 # # Fast inference with vLLM (Mixtral 8x7B)
2 #
3 # In this example, we show how to run basic inference, using [`vLLM`](https://github.com/vllm-project/vllm)
4 # to take advantage of PagedAttention, which speeds up sequential inferences with optimized key-value caching.
5 #
6 # We are running the [Mixtral 8x7B Instruct](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) model here,
7 # which is a mixture-of-experts model finetuned for conversation.
8 # You can expect ~3 minute cold starts.
9 # For a single request, the throughput is over 50 tokens/second.
10 # The larger the batch of prompts, the higher the throughput (up to hundreds of tokens per second).
11 #
12 # ## Setup
13 #
14 # First we import the components we need from `modal`.
15 
16 import os
17 import time
18 
19 import modal
20 
21 MODEL_DIR = "/model"
22 MODEL_NAME = "mistralai/Mixtral-8x7B-Instruct-v0.1"
23 GPU_CONFIG = modal.gpu.A100(memory=80, count=2)
24 
25 
26 # ## Define a container image
27 #
28 # We want to create a Modal image which has the model weights pre-saved to a directory. The benefit of this
29 # is that the container no longer has to re-download the model from Huggingface - instead, it will take
30 # advantage of Modal's internal filesystem for faster cold starts.
31 #
32 # ### Download the weights
33 #
34 # We can download the model to a particular directory using the HuggingFace utility function `snapshot_download`.
35 #
36 # Mixtral is beefy, at nearly 100 GB in `safetensors` format, so this can take some time -- at least a few minutes.
37 #
38 # Tip: avoid using global variables in this function. Changes to code outside this function will not be detected and the download step will not re-run.
39 def download_model_to_image(model_dir, model_name):
40     from huggingface_hub import snapshot_download
41     from transformers.utils import move_cache
42 
43     os.makedirs(model_dir, exist_ok=True)
44 
45     snapshot_download(
46         model_name,
47         local_dir=model_dir,
48         ignore_patterns=["*.pt", "*.bin"],  # Using safetensors
49     )
50     move_cache()
51 
52 
53 # ### Image definition
54 # We’ll start from a Dockerhub image recommended by `vLLM`, and use
55 # run_function to run the function defined above to ensure the weights of
56 # the model are saved within the container image.
57 
58 vllm_image = (
59     modal.Image.debian_slim()
60     .pip_install(
61         "vllm==0.4.0.post1",
62         "torch==2.1.2",
63         "transformers==4.39.3",
64         "ray==2.10.0",
65         "hf-transfer==0.1.6",
66         "huggingface_hub==0.22.2",
67     )
68     .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})
69     .run_function(
70         download_model_to_image,
71         timeout=60 * 20,
72         kwargs={"model_dir": MODEL_DIR, "model_name": MODEL_NAME},
73     )
74 )
75 
76 stub = modal.Stub("example-vllm-mixtral")
77 
78 
79 # ## The model class
80 #
81 # The inference function is best represented with Modal's [class syntax](/docs/guide/lifecycle-functions) and the `@enter` decorator.
82 # This enables us to load the model into memory just once every time a container starts up, and keep it cached
83 # on the GPU for each subsequent invocation of the function.
84 #
85 # The `vLLM` library allows the code to remain quite clean. We do have to patch the multi-GPU setup due to issues with Ray.
86 @stub.cls(
87     gpu=GPU_CONFIG,
88     timeout=60 * 10,
89     container_idle_timeout=60 * 10,
90     allow_concurrent_inputs=10,
91     image=vllm_image,
92 )
93 class Model:
94     @modal.enter()
95     def start_engine(self):
96         from vllm.engine.arg_utils import AsyncEngineArgs
97         from vllm.engine.async_llm_engine import AsyncLLMEngine
98 
99         print("🥶 cold starting inference")
100         start = time.monotonic_ns()
101 
102         engine_args = AsyncEngineArgs(
103             model=MODEL_DIR,
104             tensor_parallel_size=GPU_CONFIG.count,
105             gpu_memory_utilization=0.90,
106             enforce_eager=False,  # capture the graph for faster inference, but slower cold starts
107             disable_log_stats=True,  # disable logging so we can stream tokens
108             disable_log_requests=True,
109         )
110         self.template = "<s> [INST] {user} [/INST] "
111 
112         # this can take some time!
113         self.engine = AsyncLLMEngine.from_engine_args(engine_args)
114         duration_s = (time.monotonic_ns() - start) / 1e9
115         print(f"🏎️ engine started in {duration_s:.0f}s")
116 
117     @modal.method()
118     async def completion_stream(self, user_question):
119         from vllm import SamplingParams
120         from vllm.utils import random_uuid
121 
122         sampling_params = SamplingParams(
123             temperature=0.75,
124             max_tokens=128,
125             repetition_penalty=1.1,
126         )
127 
128         request_id = random_uuid()
129         result_generator = self.engine.generate(
130             self.template.format(user=user_question),
131             sampling_params,
132             request_id,
133         )
134         index, num_tokens = 0, 0
135         start = time.monotonic_ns()
136         async for output in result_generator:
137             if (
138                 output.outputs[0].text
139                 and "\ufffd" == output.outputs[0].text[-1]
140             ):
141                 continue
142             text_delta = output.outputs[0].text[index:]
143             index = len(output.outputs[0].text)
144             num_tokens = len(output.outputs[0].token_ids)
145 
146             yield text_delta
147         duration_s = (time.monotonic_ns() - start) / 1e9
148 
149         yield (
150             f"\n\tGenerated {num_tokens} tokens from {MODEL_NAME} in {duration_s:.1f}s,"
151             f" throughput = {num_tokens / duration_s:.0f} tokens/second on {GPU_CONFIG}.\n"
152         )
153 
154     @modal.exit()
155     def stop_engine(self):
156         if GPU_CONFIG.count > 1:
157             import ray
158 
159             ray.shutdown()
160 
161 
162 # ## Run the model
163 # We define a [`local_entrypoint`](/docs/guide/apps#entrypoints-for-ephemeral-apps) to call our remote function
164 # sequentially for a list of inputs. You can run this locally with `modal run -q vllm_mixtral.py`. The `q` flag
165 # enables the text to stream in your local terminal.
166 @stub.local_entrypoint()
167 def main():
168     questions = [
169         "Implement a Python function to compute the Fibonacci numbers.",
170         "What is the fable involving a fox and grapes?",
171         "What were the major contributing factors to the fall of the Roman Empire?",
172         "Describe the city of the future, considering advances in technology, environmental changes, and societal shifts.",
173         "What is the product of 9 and 8?",
174         "Who was Emperor Norton I, and what was his significance in San Francisco's history?",
175     ]
176     model = Model()
177     for question in questions:
178         print("Sending new request:", question, "\n\n")
179         for text in model.completion_stream.remote_gen(question):
180             print(text, end="", flush=text.endswith("\n"))
181 
182 
183 # ## Deploy and invoke the model
184 # Once we deploy this model with `modal deploy text_generation_inference.py`,
185 # we can invoke inference from other apps, sharing the same pool
186 # of GPU containers with all other apps we might need.
187 #
188 # ```
189 # $ python
190 # >>> import modal
191 # >>> f = modal.Function.lookup("example-tgi-Mixtral-8x7B-Instruct-v0.1", "Model.generate")
192 # >>> f.remote("What is the story about the fox and grapes?")
193 # 'The story about the fox and grapes ...
194 # ```
195 
196 # ## Coupling a frontend web application
197 #
198 # We can stream inference from a FastAPI backend, also deployed on Modal.
199 #
200 # You can try our deployment [here](https://modal-labs--vllm-mixtral.modal.run).
201 
202 from pathlib import Path
203 
204 from modal import Mount, asgi_app
205 
206 frontend_path = Path(__file__).parent.parent / "llm-frontend"
207 
208 
209 @stub.function(
210     mounts=[Mount.from_local_dir(frontend_path, remote_path="/assets")],
211     keep_warm=1,
212     allow_concurrent_inputs=20,
213     timeout=60 * 10,
214 )
215 @asgi_app()
216 def vllm_mixtral():
217     import json
218 
219     import fastapi
220     import fastapi.staticfiles
221     from fastapi.responses import StreamingResponse
222 
223     web_app = fastapi.FastAPI()
224 
225     @web_app.get("/stats")
226     async def stats():
227         stats = await Model().completion_stream.get_current_stats.aio()
228         return {
229             "backlog": stats.backlog,
230             "num_total_runners": stats.num_total_runners,
231             "model": MODEL_NAME + " (vLLM)",
232         }
233 
234     @web_app.get("/completion/{question}")
235     async def completion(question: str):
236         from urllib.parse import unquote
237 
238         async def generate():
239             async for text in Model().completion_stream.remote_gen.aio(
240                 unquote(question)
241             ):
242                 yield f"data: {json.dumps(dict(text=text), ensure_ascii=False)}\n\n"
243 
244         return StreamingResponse(generate(), media_type="text/event-stream")
245 
246     web_app.mount(
247         "/", fastapi.staticfiles.StaticFiles(directory="/assets", html=True)
248     )
249     return web_app
250
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/06_gpu_and_ml/llm-serving/vllm_mixtral.py b/06_gpu_and_ml/llm-serving/vllm_mixtral.py
--- a/06_gpu_and_ml/llm-serving/vllm_mixtral.py
+++ b/06_gpu_and_ml/llm-serving/vllm_mixtral.py
@@ -20,6 +20,7 @@
 
 MODEL_DIR = "/model"
 MODEL_NAME = "mistralai/Mixtral-8x7B-Instruct-v0.1"
+MODEL_REVISION = "1e637f2d7cb0a9d6fb1922f305cb784995190a83"
 GPU_CONFIG = modal.gpu.A100(memory=80, count=2)
 
 
@@ -33,10 +34,15 @@
 #
 # We can download the model to a particular directory using the HuggingFace utility function `snapshot_download`.
 #
-# Mixtral is beefy, at nearly 100 GB in `safetensors` format, so this can take some time -- at least a few minutes.
+# For this step to work on a [gated model](https://huggingface.co/docs/hub/en/models-gated)
+# like Mixtral 8x7B, the `HF_TOKEN` environment variable must be set.
+#
+# After [creating a HuggingFace access token](https://huggingface.co/settings/tokens)
+# and accepting the [terms of use](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1),
+# head to the [secrets page](https://modal.com/secrets) to share it with Modal as `huggingface-secret`.
 #
-# Tip: avoid using global variables in this function. Changes to code outside this function will not be detected and the download step will not re-run.
-def download_model_to_image(model_dir, model_name):
+# Mixtral is beefy, at nearly 100 GB in `safetensors` format, so this can take some time -- at least a few minutes.
+def download_model_to_image(model_dir, model_name, model_revision):
     from huggingface_hub import snapshot_download
     from transformers.utils import move_cache
 
@@ -44,6 +50,7 @@
 
     snapshot_download(
         model_name,
+        revision=model_revision,
         local_dir=model_dir,
         ignore_patterns=["*.pt", "*.bin"],  # Using safetensors
     )
@@ -69,7 +76,12 @@
     .run_function(
         download_model_to_image,
         timeout=60 * 20,
-        kwargs={"model_dir": MODEL_DIR, "model_name": MODEL_NAME},
+        kwargs={
+            "model_dir": MODEL_DIR,
+            "model_name": MODEL_NAME,
+            "model_revision": MODEL_REVISION,
+        },
+        secrets=[modal.Secret.from_name("huggingface-secret")],
     )
 )

{"golden_diff": "diff --git a/06_gpu_and_ml/llm-serving/vllm_mixtral.py b/06_gpu_and_ml/llm-serving/vllm_mixtral.py\n--- a/06_gpu_and_ml/llm-serving/vllm_mixtral.py\n+++ b/06_gpu_and_ml/llm-serving/vllm_mixtral.py\n@@ -20,6 +20,7 @@\n \n MODEL_DIR = \"/model\"\n MODEL_NAME = \"mistralai/Mixtral-8x7B-Instruct-v0.1\"\n+MODEL_REVISION = \"1e637f2d7cb0a9d6fb1922f305cb784995190a83\"\n GPU_CONFIG = modal.gpu.A100(memory=80, count=2)\n \n \n@@ -33,10 +34,15 @@\n #\n # We can download the model to a particular directory using the HuggingFace utility function `snapshot_download`.\n #\n-# Mixtral is beefy, at nearly 100 GB in `safetensors` format, so this can take some time -- at least a few minutes.\n+# For this step to work on a [gated model](https://huggingface.co/docs/hub/en/models-gated)\n+# like Mixtral 8x7B, the `HF_TOKEN` environment variable must be set.\n+#\n+# After [creating a HuggingFace access token](https://huggingface.co/settings/tokens)\n+# and accepting the [terms of use](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1),\n+# head to the [secrets page](https://modal.com/secrets) to share it with Modal as `huggingface-secret`.\n #\n-# Tip: avoid using global variables in this function. Changes to code outside this function will not be detected and the download step will not re-run.\n-def download_model_to_image(model_dir, model_name):\n+# Mixtral is beefy, at nearly 100 GB in `safetensors` format, so this can take some time -- at least a few minutes.\n+def download_model_to_image(model_dir, model_name, model_revision):\n     from huggingface_hub import snapshot_download\n     from transformers.utils import move_cache\n \n@@ -44,6 +50,7 @@\n \n     snapshot_download(\n         model_name,\n+        revision=model_revision,\n         local_dir=model_dir,\n         ignore_patterns=[\"*.pt\", \"*.bin\"],  # Using safetensors\n     )\n@@ -69,7 +76,12 @@\n     .run_function(\n         download_model_to_image,\n         timeout=60 * 20,\n-        kwargs={\"model_dir\": MODEL_DIR, \"model_name\": MODEL_NAME},\n+        kwargs={\n+            \"model_dir\": MODEL_DIR,\n+            \"model_name\": MODEL_NAME,\n+            \"model_revision\": MODEL_REVISION,\n+        },\n+        secrets=[modal.Secret.from_name(\"huggingface-secret\")],\n     )\n )\n", "issue": "Mixtral tutorial doesn't work without huggingface access token\nThe tutorial for running mixtral on VLLM doesn't work since the model cannot be downloaded without a huggingface access token. This is because mixtral is now a gated model: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1\r\n\r\n```\r\nImage build for im-1P0Aou6cl9H3BAwictbALw failed with the exception:                                           \u2502\r\n\u2502 GatedRepoError('401 Client Error. (Request ID:                                                                 \u2502\r\n\u2502 Root=1-66213747-475d6ad5261bb9eb4931c4fd;025f8bf1-0bb2-42ac-86a0-743e752004a0)\\n\\nCannot access gated repo for \u2502\r\n\u2502 url https://huggingface.co/api/models/mistralai/Mixtral-8x7B-Instruct-v0.1/revision/main.\\nRepo model          \u2502\r\n\u2502 mistralai/Mixtral-8x7B-Instruct-v0.1 is gated. You must be authenticated to access it.')\r\n```\r\n\r\nAffected Tutorial: https://modal.com/docs/examples/vllm_mixtral\r\nAffected Code: https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/llm-serving/vllm_mixtral.py\r\n\r\nThis can be fixed using a similar approach as done [here]( https://github.com/modal-labs/modal-examples/pull/539) (adding an env var for a `HF_TOKEN`) to the [function call](https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/llm-serving/vllm_mixtral.py#L45-L49) where the model is downloaded.\r\nAlso the tutorial needs to be updated to inform the user that a huggingface access token is required.\n", "before_files": [{"content": "# # Fast inference with vLLM (Mixtral 8x7B)\n#\n# In this example, we show how to run basic inference, using [`vLLM`](https://github.com/vllm-project/vllm)\n# to take advantage of PagedAttention, which speeds up sequential inferences with optimized key-value caching.\n#\n# We are running the [Mixtral 8x7B Instruct](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) model here,\n# which is a mixture-of-experts model finetuned for conversation.\n# You can expect ~3 minute cold starts.\n# For a single request, the throughput is over 50 tokens/second.\n# The larger the batch of prompts, the higher the throughput (up to hundreds of tokens per second).\n#\n# ## Setup\n#\n# First we import the components we need from `modal`.\n\nimport os\nimport time\n\nimport modal\n\nMODEL_DIR = \"/model\"\nMODEL_NAME = \"mistralai/Mixtral-8x7B-Instruct-v0.1\"\nGPU_CONFIG = modal.gpu.A100(memory=80, count=2)\n\n\n# ## Define a container image\n#\n# We want to create a Modal image which has the model weights pre-saved to a directory. The benefit of this\n# is that the container no longer has to re-download the model from Huggingface - instead, it will take\n# advantage of Modal's internal filesystem for faster cold starts.\n#\n# ### Download the weights\n#\n# We can download the model to a particular directory using the HuggingFace utility function `snapshot_download`.\n#\n# Mixtral is beefy, at nearly 100 GB in `safetensors` format, so this can take some time -- at least a few minutes.\n#\n# Tip: avoid using global variables in this function. Changes to code outside this function will not be detected and the download step will not re-run.\ndef download_model_to_image(model_dir, model_name):\n    from huggingface_hub import snapshot_download\n    from transformers.utils import move_cache\n\n    os.makedirs(model_dir, exist_ok=True)\n\n    snapshot_download(\n        model_name,\n        local_dir=model_dir,\n        ignore_patterns=[\"*.pt\", \"*.bin\"],  # Using safetensors\n    )\n    move_cache()\n\n\n# ### Image definition\n# We\u2019ll start from a Dockerhub image recommended by `vLLM`, and use\n# run_function to run the function defined above to ensure the weights of\n# the model are saved within the container image.\n\nvllm_image = (\n    modal.Image.debian_slim()\n    .pip_install(\n        \"vllm==0.4.0.post1\",\n        \"torch==2.1.2\",\n        \"transformers==4.39.3\",\n        \"ray==2.10.0\",\n        \"hf-transfer==0.1.6\",\n        \"huggingface_hub==0.22.2\",\n    )\n    .env({\"HF_HUB_ENABLE_HF_TRANSFER\": \"1\"})\n    .run_function(\n        download_model_to_image,\n        timeout=60 * 20,\n        kwargs={\"model_dir\": MODEL_DIR, \"model_name\": MODEL_NAME},\n    )\n)\n\nstub = modal.Stub(\"example-vllm-mixtral\")\n\n\n# ## The model class\n#\n# The inference function is best represented with Modal's [class syntax](/docs/guide/lifecycle-functions) and the `@enter` decorator.\n# This enables us to load the model into memory just once every time a container starts up, and keep it cached\n# on the GPU for each subsequent invocation of the function.\n#\n# The `vLLM` library allows the code to remain quite clean. We do have to patch the multi-GPU setup due to issues with Ray.\n@stub.cls(\n    gpu=GPU_CONFIG,\n    timeout=60 * 10,\n    container_idle_timeout=60 * 10,\n    allow_concurrent_inputs=10,\n    image=vllm_image,\n)\nclass Model:\n    @modal.enter()\n    def start_engine(self):\n        from vllm.engine.arg_utils import AsyncEngineArgs\n        from vllm.engine.async_llm_engine import AsyncLLMEngine\n\n        print(\"\ud83e\udd76 cold starting inference\")\n        start = time.monotonic_ns()\n\n        engine_args = AsyncEngineArgs(\n            model=MODEL_DIR,\n            tensor_parallel_size=GPU_CONFIG.count,\n            gpu_memory_utilization=0.90,\n            enforce_eager=False,  # capture the graph for faster inference, but slower cold starts\n            disable_log_stats=True,  # disable logging so we can stream tokens\n            disable_log_requests=True,\n        )\n        self.template = \"<s> [INST] {user} [/INST] \"\n\n        # this can take some time!\n        self.engine = AsyncLLMEngine.from_engine_args(engine_args)\n        duration_s = (time.monotonic_ns() - start) / 1e9\n        print(f\"\ud83c\udfce\ufe0f engine started in {duration_s:.0f}s\")\n\n    @modal.method()\n    async def completion_stream(self, user_question):\n        from vllm import SamplingParams\n        from vllm.utils import random_uuid\n\n        sampling_params = SamplingParams(\n            temperature=0.75,\n            max_tokens=128,\n            repetition_penalty=1.1,\n        )\n\n        request_id = random_uuid()\n        result_generator = self.engine.generate(\n            self.template.format(user=user_question),\n            sampling_params,\n            request_id,\n        )\n        index, num_tokens = 0, 0\n        start = time.monotonic_ns()\n        async for output in result_generator:\n            if (\n                output.outputs[0].text\n                and \"\\ufffd\" == output.outputs[0].text[-1]\n            ):\n                continue\n            text_delta = output.outputs[0].text[index:]\n            index = len(output.outputs[0].text)\n            num_tokens = len(output.outputs[0].token_ids)\n\n            yield text_delta\n        duration_s = (time.monotonic_ns() - start) / 1e9\n\n        yield (\n            f\"\\n\\tGenerated {num_tokens} tokens from {MODEL_NAME} in {duration_s:.1f}s,\"\n            f\" throughput = {num_tokens / duration_s:.0f} tokens/second on {GPU_CONFIG}.\\n\"\n        )\n\n    @modal.exit()\n    def stop_engine(self):\n        if GPU_CONFIG.count > 1:\n            import ray\n\n            ray.shutdown()\n\n\n# ## Run the model\n# We define a [`local_entrypoint`](/docs/guide/apps#entrypoints-for-ephemeral-apps) to call our remote function\n# sequentially for a list of inputs. You can run this locally with `modal run -q vllm_mixtral.py`. The `q` flag\n# enables the text to stream in your local terminal.\n@stub.local_entrypoint()\ndef main():\n    questions = [\n        \"Implement a Python function to compute the Fibonacci numbers.\",\n        \"What is the fable involving a fox and grapes?\",\n        \"What were the major contributing factors to the fall of the Roman Empire?\",\n        \"Describe the city of the future, considering advances in technology, environmental changes, and societal shifts.\",\n        \"What is the product of 9 and 8?\",\n        \"Who was Emperor Norton I, and what was his significance in San Francisco's history?\",\n    ]\n    model = Model()\n    for question in questions:\n        print(\"Sending new request:\", question, \"\\n\\n\")\n        for text in model.completion_stream.remote_gen(question):\n            print(text, end=\"\", flush=text.endswith(\"\\n\"))\n\n\n# ## Deploy and invoke the model\n# Once we deploy this model with `modal deploy text_generation_inference.py`,\n# we can invoke inference from other apps, sharing the same pool\n# of GPU containers with all other apps we might need.\n#\n# ```\n# $ python\n# >>> import modal\n# >>> f = modal.Function.lookup(\"example-tgi-Mixtral-8x7B-Instruct-v0.1\", \"Model.generate\")\n# >>> f.remote(\"What is the story about the fox and grapes?\")\n# 'The story about the fox and grapes ...\n# ```\n\n# ## Coupling a frontend web application\n#\n# We can stream inference from a FastAPI backend, also deployed on Modal.\n#\n# You can try our deployment [here](https://modal-labs--vllm-mixtral.modal.run).\n\nfrom pathlib import Path\n\nfrom modal import Mount, asgi_app\n\nfrontend_path = Path(__file__).parent.parent / \"llm-frontend\"\n\n\n@stub.function(\n    mounts=[Mount.from_local_dir(frontend_path, remote_path=\"/assets\")],\n    keep_warm=1,\n    allow_concurrent_inputs=20,\n    timeout=60 * 10,\n)\n@asgi_app()\ndef vllm_mixtral():\n    import json\n\n    import fastapi\n    import fastapi.staticfiles\n    from fastapi.responses import StreamingResponse\n\n    web_app = fastapi.FastAPI()\n\n    @web_app.get(\"/stats\")\n    async def stats():\n        stats = await Model().completion_stream.get_current_stats.aio()\n        return {\n            \"backlog\": stats.backlog,\n            \"num_total_runners\": stats.num_total_runners,\n            \"model\": MODEL_NAME + \" (vLLM)\",\n        }\n\n    @web_app.get(\"/completion/{question}\")\n    async def completion(question: str):\n        from urllib.parse import unquote\n\n        async def generate():\n            async for text in Model().completion_stream.remote_gen.aio(\n                unquote(question)\n            ):\n                yield f\"data: {json.dumps(dict(text=text), ensure_ascii=False)}\\n\\n\"\n\n        return StreamingResponse(generate(), media_type=\"text/event-stream\")\n\n    web_app.mount(\n        \"/\", fastapi.staticfiles.StaticFiles(directory=\"/assets\", html=True)\n    )\n    return web_app\n", "path": "06_gpu_and_ml/llm-serving/vllm_mixtral.py"}], "after_files": [{"content": "# # Fast inference with vLLM (Mixtral 8x7B)\n#\n# In this example, we show how to run basic inference, using [`vLLM`](https://github.com/vllm-project/vllm)\n# to take advantage of PagedAttention, which speeds up sequential inferences with optimized key-value caching.\n#\n# We are running the [Mixtral 8x7B Instruct](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) model here,\n# which is a mixture-of-experts model finetuned for conversation.\n# You can expect ~3 minute cold starts.\n# For a single request, the throughput is over 50 tokens/second.\n# The larger the batch of prompts, the higher the throughput (up to hundreds of tokens per second).\n#\n# ## Setup\n#\n# First we import the components we need from `modal`.\n\nimport os\nimport time\n\nimport modal\n\nMODEL_DIR = \"/model\"\nMODEL_NAME = \"mistralai/Mixtral-8x7B-Instruct-v0.1\"\nMODEL_REVISION = \"1e637f2d7cb0a9d6fb1922f305cb784995190a83\"\nGPU_CONFIG = modal.gpu.A100(memory=80, count=2)\n\n\n# ## Define a container image\n#\n# We want to create a Modal image which has the model weights pre-saved to a directory. The benefit of this\n# is that the container no longer has to re-download the model from Huggingface - instead, it will take\n# advantage of Modal's internal filesystem for faster cold starts.\n#\n# ### Download the weights\n#\n# We can download the model to a particular directory using the HuggingFace utility function `snapshot_download`.\n#\n# For this step to work on a [gated model](https://huggingface.co/docs/hub/en/models-gated)\n# like Mixtral 8x7B, the `HF_TOKEN` environment variable must be set.\n#\n# After [creating a HuggingFace access token](https://huggingface.co/settings/tokens)\n# and accepting the [terms of use](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1),\n# head to the [secrets page](https://modal.com/secrets) to share it with Modal as `huggingface-secret`.\n#\n# Mixtral is beefy, at nearly 100 GB in `safetensors` format, so this can take some time -- at least a few minutes.\ndef download_model_to_image(model_dir, model_name, model_revision):\n    from huggingface_hub import snapshot_download\n    from transformers.utils import move_cache\n\n    os.makedirs(model_dir, exist_ok=True)\n\n    snapshot_download(\n        model_name,\n        revision=model_revision,\n        local_dir=model_dir,\n        ignore_patterns=[\"*.pt\", \"*.bin\"],  # Using safetensors\n    )\n    move_cache()\n\n\n# ### Image definition\n# We\u2019ll start from a Dockerhub image recommended by `vLLM`, and use\n# run_function to run the function defined above to ensure the weights of\n# the model are saved within the container image.\n\nvllm_image = (\n    modal.Image.debian_slim()\n    .pip_install(\n        \"vllm==0.4.0.post1\",\n        \"torch==2.1.2\",\n        \"transformers==4.39.3\",\n        \"ray==2.10.0\",\n        \"hf-transfer==0.1.6\",\n        \"huggingface_hub==0.22.2\",\n    )\n    .env({\"HF_HUB_ENABLE_HF_TRANSFER\": \"1\"})\n    .run_function(\n        download_model_to_image,\n        timeout=60 * 20,\n        kwargs={\n            \"model_dir\": MODEL_DIR,\n            \"model_name\": MODEL_NAME,\n            \"model_revision\": MODEL_REVISION,\n        },\n        secrets=[modal.Secret.from_name(\"huggingface-secret\")],\n    )\n)\n\nstub = modal.Stub(\"example-vllm-mixtral\")\n\n\n# ## The model class\n#\n# The inference function is best represented with Modal's [class syntax](/docs/guide/lifecycle-functions) and the `@enter` decorator.\n# This enables us to load the model into memory just once every time a container starts up, and keep it cached\n# on the GPU for each subsequent invocation of the function.\n#\n# The `vLLM` library allows the code to remain quite clean. We do have to patch the multi-GPU setup due to issues with Ray.\n@stub.cls(\n    gpu=GPU_CONFIG,\n    timeout=60 * 10,\n    container_idle_timeout=60 * 10,\n    allow_concurrent_inputs=10,\n    image=vllm_image,\n)\nclass Model:\n    @modal.enter()\n    def start_engine(self):\n        from vllm.engine.arg_utils import AsyncEngineArgs\n        from vllm.engine.async_llm_engine import AsyncLLMEngine\n\n        print(\"\ud83e\udd76 cold starting inference\")\n        start = time.monotonic_ns()\n\n        engine_args = AsyncEngineArgs(\n            model=MODEL_DIR,\n            tensor_parallel_size=GPU_CONFIG.count,\n            gpu_memory_utilization=0.90,\n            enforce_eager=False,  # capture the graph for faster inference, but slower cold starts\n            disable_log_stats=True,  # disable logging so we can stream tokens\n            disable_log_requests=True,\n        )\n        self.template = \"<s> [INST] {user} [/INST] \"\n\n        # this can take some time!\n        self.engine = AsyncLLMEngine.from_engine_args(engine_args)\n        duration_s = (time.monotonic_ns() - start) / 1e9\n        print(f\"\ud83c\udfce\ufe0f engine started in {duration_s:.0f}s\")\n\n    @modal.method()\n    async def completion_stream(self, user_question):\n        from vllm import SamplingParams\n        from vllm.utils import random_uuid\n\n        sampling_params = SamplingParams(\n            temperature=0.75,\n            max_tokens=128,\n            repetition_penalty=1.1,\n        )\n\n        request_id = random_uuid()\n        result_generator = self.engine.generate(\n            self.template.format(user=user_question),\n            sampling_params,\n            request_id,\n        )\n        index, num_tokens = 0, 0\n        start = time.monotonic_ns()\n        async for output in result_generator:\n            if (\n                output.outputs[0].text\n                and \"\\ufffd\" == output.outputs[0].text[-1]\n            ):\n                continue\n            text_delta = output.outputs[0].text[index:]\n            index = len(output.outputs[0].text)\n            num_tokens = len(output.outputs[0].token_ids)\n\n            yield text_delta\n        duration_s = (time.monotonic_ns() - start) / 1e9\n\n        yield (\n            f\"\\n\\tGenerated {num_tokens} tokens from {MODEL_NAME} in {duration_s:.1f}s,\"\n            f\" throughput = {num_tokens / duration_s:.0f} tokens/second on {GPU_CONFIG}.\\n\"\n        )\n\n    @modal.exit()\n    def stop_engine(self):\n        if GPU_CONFIG.count > 1:\n            import ray\n\n            ray.shutdown()\n\n\n# ## Run the model\n# We define a [`local_entrypoint`](/docs/guide/apps#entrypoints-for-ephemeral-apps) to call our remote function\n# sequentially for a list of inputs. You can run this locally with `modal run -q vllm_mixtral.py`. The `q` flag\n# enables the text to stream in your local terminal.\n@stub.local_entrypoint()\ndef main():\n    questions = [\n        \"Implement a Python function to compute the Fibonacci numbers.\",\n        \"What is the fable involving a fox and grapes?\",\n        \"What were the major contributing factors to the fall of the Roman Empire?\",\n        \"Describe the city of the future, considering advances in technology, environmental changes, and societal shifts.\",\n        \"What is the product of 9 and 8?\",\n        \"Who was Emperor Norton I, and what was his significance in San Francisco's history?\",\n    ]\n    model = Model()\n    for question in questions:\n        print(\"Sending new request:\", question, \"\\n\\n\")\n        for text in model.completion_stream.remote_gen(question):\n            print(text, end=\"\", flush=text.endswith(\"\\n\"))\n\n\n# ## Deploy and invoke the model\n# Once we deploy this model with `modal deploy text_generation_inference.py`,\n# we can invoke inference from other apps, sharing the same pool\n# of GPU containers with all other apps we might need.\n#\n# ```\n# $ python\n# >>> import modal\n# >>> f = modal.Function.lookup(\"example-tgi-Mixtral-8x7B-Instruct-v0.1\", \"Model.generate\")\n# >>> f.remote(\"What is the story about the fox and grapes?\")\n# 'The story about the fox and grapes ...\n# ```\n\n# ## Coupling a frontend web application\n#\n# We can stream inference from a FastAPI backend, also deployed on Modal.\n#\n# You can try our deployment [here](https://modal-labs--vllm-mixtral.modal.run).\n\nfrom pathlib import Path\n\nfrom modal import Mount, asgi_app\n\nfrontend_path = Path(__file__).parent.parent / \"llm-frontend\"\n\n\n@stub.function(\n    mounts=[Mount.from_local_dir(frontend_path, remote_path=\"/assets\")],\n    keep_warm=1,\n    allow_concurrent_inputs=20,\n    timeout=60 * 10,\n)\n@asgi_app()\ndef vllm_mixtral():\n    import json\n\n    import fastapi\n    import fastapi.staticfiles\n    from fastapi.responses import StreamingResponse\n\n    web_app = fastapi.FastAPI()\n\n    @web_app.get(\"/stats\")\n    async def stats():\n        stats = await Model().completion_stream.get_current_stats.aio()\n        return {\n            \"backlog\": stats.backlog,\n            \"num_total_runners\": stats.num_total_runners,\n            \"model\": MODEL_NAME + \" (vLLM)\",\n        }\n\n    @web_app.get(\"/completion/{question}\")\n    async def completion(question: str):\n        from urllib.parse import unquote\n\n        async def generate():\n            async for text in Model().completion_stream.remote_gen.aio(\n                unquote(question)\n            ):\n                yield f\"data: {json.dumps(dict(text=text), ensure_ascii=False)}\\n\\n\"\n\n        return StreamingResponse(generate(), media_type=\"text/event-stream\")\n\n    web_app.mount(\n        \"/\", fastapi.staticfiles.StaticFiles(directory=\"/assets\", html=True)\n    )\n    return web_app\n", "path": "06_gpu_and_ml/llm-serving/vllm_mixtral.py"}]}