We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Stop Error When disabled_grpc=True
/kind bug

**What steps did you take and what happened:**
In our use case we don't use the grpc_server inside `ModelServer` thus we set `enable_grpc=False`. But when server stops it will still call stop from this unstarted server thus got the error below:
```
>       await self._server.stop(grace=10)
E       AttributeError: 'NoneType' object has no attribute 'stop'
```

The bug is because that this [if](https://github.com/kserve/kserve/blob/master/python/kserve/kserve/model_server.py#L174) will always be evaluated as `True`. Can we please add a one line change to make it support `enable_grpc=False`? Maybe change it to `if self._grpc_server._server` or `if self.enable_grpc`? Should be a very quick fix and that helps a lot! Thank you!

**What did you expect to happen:**
A one line change from [code](https://github.com/kserve/kserve/blob/master/python/kserve/kserve/model_server.py#L174) to `if self._grpc_server._server` or `if self.enable_grpc`

**What's the InferenceService yaml:**
[To help us debug please run `kubectl get isvc $name -n $namespace -oyaml` and paste the output]

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]


**Environment:**

- Istio Version:
- Knative Version:
- KServe Version:
- Kubeflow version:
- Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
- Minikube/Kind version:
- Kubernetes version: (use `kubectl version`):
- OS (e.g. from `/etc/os-release`):

Stop Error When disabled_grpc=True
/kind bug

**What steps did you take and what happened:**
In our use case we don't use the grpc_server inside `ModelServer` thus we set `enable_grpc=False`. But when server stops it will still call stop from this unstarted server thus got the error below:
```
>       await self._server.stop(grace=10)
E       AttributeError: 'NoneType' object has no attribute 'stop'
```

The bug is because that this [if](https://github.com/kserve/kserve/blob/master/python/kserve/kserve/model_server.py#L174) will always be evaluated as `True`. Can we please add a one line change to make it support `enable_grpc=False`? Maybe change it to `if self._grpc_server._server` or `if self.enable_grpc`? Should be a very quick fix and that helps a lot! Thank you!

**What did you expect to happen:**
A one line change from [code](https://github.com/kserve/kserve/blob/master/python/kserve/kserve/model_server.py#L174) to `if self._grpc_server._server` or `if self.enable_grpc`

**What's the InferenceService yaml:**
[To help us debug please run `kubectl get isvc $name -n $namespace -oyaml` and paste the output]

**Anything else you would like to add:**
[Miscellaneous information that will assist in solving the issue.]


**Environment:**

- Istio Version:
- Knative Version:
- KServe Version:
- Kubeflow version:
- Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]
- Minikube/Kind version:
- Kubernetes version: (use `kubectl version`):
- OS (e.g. from `/etc/os-release`):


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `python/kserve/kserve/model_server.py`
Content:
```
1 # Copyright 2022 The KServe Authors.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 #
7 #    http://www.apache.org/licenses/LICENSE-2.0
8 #
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 
15 import argparse
16 import asyncio
17 import concurrent.futures
18 import logging
19 import signal
20 import socket
21 from distutils.util import strtobool
22 from typing import List, Dict, Union, Optional
23 
24 from ray import serve as rayserve
25 from ray.serve.api import Deployment, RayServeHandle
26 
27 from .protocol.grpc.server import GRPCServer
28 from .protocol.rest.server import UvicornServer
29 from .utils import utils
30 import multiprocessing
31 from multiprocessing import Process
32 
33 from .model import Model
34 from .protocol.dataplane import DataPlane
35 from .protocol.model_repository_extension import ModelRepositoryExtension
36 from .model_repository import ModelRepository
37 
38 DEFAULT_HTTP_PORT = 8080
39 DEFAULT_GRPC_PORT = 8081
40 
41 parser = argparse.ArgumentParser(add_help=False)
42 parser.add_argument("--http_port", default=DEFAULT_HTTP_PORT, type=int,
43                     help="The HTTP Port listened to by the model server.")
44 parser.add_argument("--grpc_port", default=DEFAULT_GRPC_PORT, type=int,
45                     help="The GRPC Port listened to by the model server.")
46 parser.add_argument("--workers", default=1, type=int,
47                     help="The number of workers for multi-processing.")
48 parser.add_argument("--max_threads", default=4, type=int,
49                     help="The number of max processing threads in each worker.")
50 parser.add_argument('--max_asyncio_workers', default=None, type=int,
51                     help='Max number of asyncio workers to spawn')
52 parser.add_argument("--enable_grpc", default=True, type=lambda x: bool(strtobool(x)),
53                     help="Enable gRPC for the model server")
54 parser.add_argument("--enable_docs_url", default=False, type=lambda x: bool(strtobool(x)),
55                     help="Enable docs url '/docs' to display Swagger UI.")
56 parser.add_argument("--enable_latency_logging", default=True, type=lambda x: bool(strtobool(x)),
57                     help="Output a log per request with latency metrics.")
58 parser.add_argument("--log_config_file", default=None, type=str,
59                     help="File path containing UvicornServer's log config. Needs to be a yaml or json file.")
60 parser.add_argument("--access_log_format", default=None, type=str,
61                     help="Format to set for the access log (provided by asgi-logger).")
62 
63 args, _ = parser.parse_known_args()
64 
65 FORMAT = '%(asctime)s.%(msecs)03d %(process)s %(name)s %(levelname)s [%(funcName)s():%(lineno)s] %(message)s'
66 DATE_FORMAT = "%Y-%m-%d %H:%M:%S"
67 logging.basicConfig(level=logging.INFO, format=FORMAT, datefmt=DATE_FORMAT)
68 
69 
70 class ModelServer:
71     """KServe ModelServer
72 
73     Args:
74         http_port (int): HTTP port. Default: ``8080``.
75         grpc_port (int): GRPC port. Default: ``8081``.
76         workers (int): Number of workers for uvicorn. Default: ``1``.
77         max_threads (int): Max number of processing threads. Default: ``4``
78         max_asyncio_workers (int): Max number of AsyncIO threads. Default: ``None``
79         registered_models (ModelRepository): Model repository with registered models.
80         enable_grpc (bool): Whether to turn on grpc server. Default: ``True``
81         enable_docs_url (bool): Whether to turn on ``/docs`` Swagger UI. Default: ``False``.
82         enable_latency_logging (bool): Whether to log latency metric. Default: ``True``.
83         log_config_file (dict): File path containing UvicornServer's log config. Default: ``None``.
84         access_log_format (string): Format to set for the access log (provided by asgi-logger). Default: ``None``
85     """
86 
87     def __init__(self, http_port: int = args.http_port,
88                  grpc_port: int = args.grpc_port,
89                  workers: int = args.workers,
90                  max_threads: int = args.max_threads,
91                  max_asyncio_workers: int = args.max_asyncio_workers,
92                  registered_models: ModelRepository = ModelRepository(),
93                  enable_grpc: bool = args.enable_grpc,
94                  enable_docs_url: bool = args.enable_docs_url,
95                  enable_latency_logging: bool = args.enable_latency_logging,
96                  log_config_file: str = args.log_config_file,
97                  access_log_format: str = args.access_log_format):
98         self.registered_models = registered_models
99         self.http_port = http_port
100         self.grpc_port = grpc_port
101         self.workers = workers
102         self.max_threads = max_threads
103         self.max_asyncio_workers = max_asyncio_workers
104         self.enable_grpc = enable_grpc
105         self.enable_docs_url = enable_docs_url
106         self.enable_latency_logging = enable_latency_logging
107         self.dataplane = DataPlane(model_registry=registered_models)
108         self.model_repository_extension = ModelRepositoryExtension(
109             model_registry=self.registered_models)
110         self._grpc_server = GRPCServer(grpc_port, self.dataplane, self.model_repository_extension)
111         self.log_config_file = log_config_file
112         self.access_log_format = access_log_format
113 
114     def start(self, models: Union[List[Model], Dict[str, Deployment]]) -> None:
115         if isinstance(models, list):
116             for model in models:
117                 if isinstance(model, Model):
118                     self.register_model(model)
119                     # pass whether to log request latency into the model
120                     model.enable_latency_logging = self.enable_latency_logging
121                 else:
122                     raise RuntimeError("Model type should be 'Model'")
123         elif isinstance(models, dict):
124             if all([isinstance(v, Deployment) for v in models.values()]):
125                 # TODO: make this port number a variable
126                 rayserve.start(detached=True, http_options={"host": "0.0.0.0", "port": 9071})
127                 for key in models:
128                     models[key].deploy()
129                     handle = models[key].get_handle()
130                     self.register_model_handle(key, handle)
131             else:
132                 raise RuntimeError("Model type should be RayServe Deployment")
133         else:
134             raise RuntimeError("Unknown model collection types")
135 
136         if self.max_asyncio_workers is None:
137             # formula as suggest in https://bugs.python.org/issue35279
138             self.max_asyncio_workers = min(32, utils.cpu_count() + 4)
139         logging.info(f"Setting max asyncio worker threads as {self.max_asyncio_workers}")
140         asyncio.get_event_loop().set_default_executor(
141             concurrent.futures.ThreadPoolExecutor(max_workers=self.max_asyncio_workers))
142 
143         async def serve():
144             serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
145             serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
146             serversocket.bind(('0.0.0.0', self.http_port))
147             serversocket.listen(5)
148 
149             logging.info(f"Starting uvicorn with {self.workers} workers")
150             loop = asyncio.get_event_loop()
151             for sig in [signal.SIGINT, signal.SIGTERM, signal.SIGQUIT]:
152                 loop.add_signal_handler(
153                     sig, lambda s=sig: asyncio.create_task(self.stop(sig=s))
154                 )
155             self._rest_server = UvicornServer(self.http_port, [serversocket],
156                                               self.dataplane, self.model_repository_extension,
157                                               self.enable_docs_url, log_config_file=self.log_config_file,
158                                               access_log_format=self.access_log_format)
159             if self.workers == 1:
160                 await self._rest_server.run()
161             else:
162                 # Since py38 MacOS/Windows defaults to use spawn for starting multiprocessing.
163                 # https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
164                 # Spawn does not work with FastAPI/uvicorn in multiprocessing mode, use fork for multiprocessing
165                 # https://github.com/tiangolo/fastapi/issues/1586
166                 multiprocessing.set_start_method('fork')
167                 server = UvicornServer(self.http_port, [serversocket],
168                                        self.dataplane, self.model_repository_extension,
169                                        self.enable_docs_url, custom_log_config=self.log_config)
170                 for _ in range(self.workers):
171                     p = Process(target=server.run_sync)
172                     p.start()
173 
174         async def servers_task():
175             servers = [serve()]
176             if self.enable_grpc:
177                 servers.append(self._grpc_server.start(self.max_threads))
178             await asyncio.gather(*servers)
179 
180         asyncio.run(servers_task())
181 
182     async def stop(self, sig: Optional[int] = None):
183         logging.info("Stopping the model server")
184         if self._rest_server:
185             logging.info("Stopping the rest server")
186             await self._rest_server.stop()
187         if self._grpc_server:
188             logging.info("Stopping the grpc server")
189             await self._grpc_server.stop(sig)
190 
191     def register_model_handle(self, name: str, model_handle: RayServeHandle):
192         self.registered_models.update_handle(name, model_handle)
193         logging.info("Registering model handle: %s", name)
194 
195     def register_model(self, model: Model):
196         if not model.name:
197             raise Exception(
198                 "Failed to register model, model.name must be provided.")
199         self.registered_models.update(model)
200         logging.info("Registering model: %s", model.name)
201
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/python/kserve/kserve/model_server.py b/python/kserve/kserve/model_server.py
--- a/python/kserve/kserve/model_server.py
+++ b/python/kserve/kserve/model_server.py
@@ -107,7 +107,10 @@
         self.dataplane = DataPlane(model_registry=registered_models)
         self.model_repository_extension = ModelRepositoryExtension(
             model_registry=self.registered_models)
-        self._grpc_server = GRPCServer(grpc_port, self.dataplane, self.model_repository_extension)
+        self._grpc_server = None
+        if self.enable_grpc:
+            self._grpc_server = GRPCServer(grpc_port, self.dataplane,
+                                           self.model_repository_extension)
         self.log_config_file = log_config_file
         self.access_log_format = access_log_format

{"golden_diff": "diff --git a/python/kserve/kserve/model_server.py b/python/kserve/kserve/model_server.py\n--- a/python/kserve/kserve/model_server.py\n+++ b/python/kserve/kserve/model_server.py\n@@ -107,7 +107,10 @@\n         self.dataplane = DataPlane(model_registry=registered_models)\n         self.model_repository_extension = ModelRepositoryExtension(\n             model_registry=self.registered_models)\n-        self._grpc_server = GRPCServer(grpc_port, self.dataplane, self.model_repository_extension)\n+        self._grpc_server = None\n+        if self.enable_grpc:\n+            self._grpc_server = GRPCServer(grpc_port, self.dataplane,\n+                                           self.model_repository_extension)\n         self.log_config_file = log_config_file\n         self.access_log_format = access_log_format\n", "issue": "Stop Error When disabled_grpc=True\n/kind bug\r\n\r\n**What steps did you take and what happened:**\r\nIn our use case we don't use the grpc_server inside `ModelServer` thus we set `enable_grpc=False`. But when server stops it will still call stop from this unstarted server thus got the error below:\r\n```\r\n>       await self._server.stop(grace=10)\r\nE       AttributeError: 'NoneType' object has no attribute 'stop'\r\n```\r\n\r\nThe bug is because that this [if](https://github.com/kserve/kserve/blob/master/python/kserve/kserve/model_server.py#L174) will always be evaluated as `True`. Can we please add a one line change to make it support `enable_grpc=False`? Maybe change it to `if self._grpc_server._server` or `if self.enable_grpc`? Should be a very quick fix and that helps a lot! Thank you!\r\n\r\n**What did you expect to happen:**\r\nA one line change from [code](https://github.com/kserve/kserve/blob/master/python/kserve/kserve/model_server.py#L174) to `if self._grpc_server._server` or `if self.enable_grpc`\r\n\r\n**What's the InferenceService yaml:**\r\n[To help us debug please run `kubectl get isvc $name -n $namespace -oyaml` and paste the output]\r\n\r\n**Anything else you would like to add:**\r\n[Miscellaneous information that will assist in solving the issue.]\r\n\r\n\r\n**Environment:**\r\n\r\n- Istio Version:\r\n- Knative Version:\r\n- KServe Version:\r\n- Kubeflow version:\r\n- Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]\r\n- Minikube/Kind version:\r\n- Kubernetes version: (use `kubectl version`):\r\n- OS (e.g. from `/etc/os-release`):\r\n\nStop Error When disabled_grpc=True\n/kind bug\r\n\r\n**What steps did you take and what happened:**\r\nIn our use case we don't use the grpc_server inside `ModelServer` thus we set `enable_grpc=False`. But when server stops it will still call stop from this unstarted server thus got the error below:\r\n```\r\n>       await self._server.stop(grace=10)\r\nE       AttributeError: 'NoneType' object has no attribute 'stop'\r\n```\r\n\r\nThe bug is because that this [if](https://github.com/kserve/kserve/blob/master/python/kserve/kserve/model_server.py#L174) will always be evaluated as `True`. Can we please add a one line change to make it support `enable_grpc=False`? Maybe change it to `if self._grpc_server._server` or `if self.enable_grpc`? Should be a very quick fix and that helps a lot! Thank you!\r\n\r\n**What did you expect to happen:**\r\nA one line change from [code](https://github.com/kserve/kserve/blob/master/python/kserve/kserve/model_server.py#L174) to `if self._grpc_server._server` or `if self.enable_grpc`\r\n\r\n**What's the InferenceService yaml:**\r\n[To help us debug please run `kubectl get isvc $name -n $namespace -oyaml` and paste the output]\r\n\r\n**Anything else you would like to add:**\r\n[Miscellaneous information that will assist in solving the issue.]\r\n\r\n\r\n**Environment:**\r\n\r\n- Istio Version:\r\n- Knative Version:\r\n- KServe Version:\r\n- Kubeflow version:\r\n- Cloud Environment:[k8s_istio/istio_dex/gcp_basic_auth/gcp_iap/aws/aws_cognito/ibm]\r\n- Minikube/Kind version:\r\n- Kubernetes version: (use `kubectl version`):\r\n- OS (e.g. from `/etc/os-release`):\r\n\n", "before_files": [{"content": "# Copyright 2022 The KServe Authors.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#    http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport argparse\nimport asyncio\nimport concurrent.futures\nimport logging\nimport signal\nimport socket\nfrom distutils.util import strtobool\nfrom typing import List, Dict, Union, Optional\n\nfrom ray import serve as rayserve\nfrom ray.serve.api import Deployment, RayServeHandle\n\nfrom .protocol.grpc.server import GRPCServer\nfrom .protocol.rest.server import UvicornServer\nfrom .utils import utils\nimport multiprocessing\nfrom multiprocessing import Process\n\nfrom .model import Model\nfrom .protocol.dataplane import DataPlane\nfrom .protocol.model_repository_extension import ModelRepositoryExtension\nfrom .model_repository import ModelRepository\n\nDEFAULT_HTTP_PORT = 8080\nDEFAULT_GRPC_PORT = 8081\n\nparser = argparse.ArgumentParser(add_help=False)\nparser.add_argument(\"--http_port\", default=DEFAULT_HTTP_PORT, type=int,\n                    help=\"The HTTP Port listened to by the model server.\")\nparser.add_argument(\"--grpc_port\", default=DEFAULT_GRPC_PORT, type=int,\n                    help=\"The GRPC Port listened to by the model server.\")\nparser.add_argument(\"--workers\", default=1, type=int,\n                    help=\"The number of workers for multi-processing.\")\nparser.add_argument(\"--max_threads\", default=4, type=int,\n                    help=\"The number of max processing threads in each worker.\")\nparser.add_argument('--max_asyncio_workers', default=None, type=int,\n                    help='Max number of asyncio workers to spawn')\nparser.add_argument(\"--enable_grpc\", default=True, type=lambda x: bool(strtobool(x)),\n                    help=\"Enable gRPC for the model server\")\nparser.add_argument(\"--enable_docs_url\", default=False, type=lambda x: bool(strtobool(x)),\n                    help=\"Enable docs url '/docs' to display Swagger UI.\")\nparser.add_argument(\"--enable_latency_logging\", default=True, type=lambda x: bool(strtobool(x)),\n                    help=\"Output a log per request with latency metrics.\")\nparser.add_argument(\"--log_config_file\", default=None, type=str,\n                    help=\"File path containing UvicornServer's log config. Needs to be a yaml or json file.\")\nparser.add_argument(\"--access_log_format\", default=None, type=str,\n                    help=\"Format to set for the access log (provided by asgi-logger).\")\n\nargs, _ = parser.parse_known_args()\n\nFORMAT = '%(asctime)s.%(msecs)03d %(process)s %(name)s %(levelname)s [%(funcName)s():%(lineno)s] %(message)s'\nDATE_FORMAT = \"%Y-%m-%d %H:%M:%S\"\nlogging.basicConfig(level=logging.INFO, format=FORMAT, datefmt=DATE_FORMAT)\n\n\nclass ModelServer:\n    \"\"\"KServe ModelServer\n\n    Args:\n        http_port (int): HTTP port. Default: ``8080``.\n        grpc_port (int): GRPC port. Default: ``8081``.\n        workers (int): Number of workers for uvicorn. Default: ``1``.\n        max_threads (int): Max number of processing threads. Default: ``4``\n        max_asyncio_workers (int): Max number of AsyncIO threads. Default: ``None``\n        registered_models (ModelRepository): Model repository with registered models.\n        enable_grpc (bool): Whether to turn on grpc server. Default: ``True``\n        enable_docs_url (bool): Whether to turn on ``/docs`` Swagger UI. Default: ``False``.\n        enable_latency_logging (bool): Whether to log latency metric. Default: ``True``.\n        log_config_file (dict): File path containing UvicornServer's log config. Default: ``None``.\n        access_log_format (string): Format to set for the access log (provided by asgi-logger). Default: ``None``\n    \"\"\"\n\n    def __init__(self, http_port: int = args.http_port,\n                 grpc_port: int = args.grpc_port,\n                 workers: int = args.workers,\n                 max_threads: int = args.max_threads,\n                 max_asyncio_workers: int = args.max_asyncio_workers,\n                 registered_models: ModelRepository = ModelRepository(),\n                 enable_grpc: bool = args.enable_grpc,\n                 enable_docs_url: bool = args.enable_docs_url,\n                 enable_latency_logging: bool = args.enable_latency_logging,\n                 log_config_file: str = args.log_config_file,\n                 access_log_format: str = args.access_log_format):\n        self.registered_models = registered_models\n        self.http_port = http_port\n        self.grpc_port = grpc_port\n        self.workers = workers\n        self.max_threads = max_threads\n        self.max_asyncio_workers = max_asyncio_workers\n        self.enable_grpc = enable_grpc\n        self.enable_docs_url = enable_docs_url\n        self.enable_latency_logging = enable_latency_logging\n        self.dataplane = DataPlane(model_registry=registered_models)\n        self.model_repository_extension = ModelRepositoryExtension(\n            model_registry=self.registered_models)\n        self._grpc_server = GRPCServer(grpc_port, self.dataplane, self.model_repository_extension)\n        self.log_config_file = log_config_file\n        self.access_log_format = access_log_format\n\n    def start(self, models: Union[List[Model], Dict[str, Deployment]]) -> None:\n        if isinstance(models, list):\n            for model in models:\n                if isinstance(model, Model):\n                    self.register_model(model)\n                    # pass whether to log request latency into the model\n                    model.enable_latency_logging = self.enable_latency_logging\n                else:\n                    raise RuntimeError(\"Model type should be 'Model'\")\n        elif isinstance(models, dict):\n            if all([isinstance(v, Deployment) for v in models.values()]):\n                # TODO: make this port number a variable\n                rayserve.start(detached=True, http_options={\"host\": \"0.0.0.0\", \"port\": 9071})\n                for key in models:\n                    models[key].deploy()\n                    handle = models[key].get_handle()\n                    self.register_model_handle(key, handle)\n            else:\n                raise RuntimeError(\"Model type should be RayServe Deployment\")\n        else:\n            raise RuntimeError(\"Unknown model collection types\")\n\n        if self.max_asyncio_workers is None:\n            # formula as suggest in https://bugs.python.org/issue35279\n            self.max_asyncio_workers = min(32, utils.cpu_count() + 4)\n        logging.info(f\"Setting max asyncio worker threads as {self.max_asyncio_workers}\")\n        asyncio.get_event_loop().set_default_executor(\n            concurrent.futures.ThreadPoolExecutor(max_workers=self.max_asyncio_workers))\n\n        async def serve():\n            serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n            serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n            serversocket.bind(('0.0.0.0', self.http_port))\n            serversocket.listen(5)\n\n            logging.info(f\"Starting uvicorn with {self.workers} workers\")\n            loop = asyncio.get_event_loop()\n            for sig in [signal.SIGINT, signal.SIGTERM, signal.SIGQUIT]:\n                loop.add_signal_handler(\n                    sig, lambda s=sig: asyncio.create_task(self.stop(sig=s))\n                )\n            self._rest_server = UvicornServer(self.http_port, [serversocket],\n                                              self.dataplane, self.model_repository_extension,\n                                              self.enable_docs_url, log_config_file=self.log_config_file,\n                                              access_log_format=self.access_log_format)\n            if self.workers == 1:\n                await self._rest_server.run()\n            else:\n                # Since py38 MacOS/Windows defaults to use spawn for starting multiprocessing.\n                # https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods\n                # Spawn does not work with FastAPI/uvicorn in multiprocessing mode, use fork for multiprocessing\n                # https://github.com/tiangolo/fastapi/issues/1586\n                multiprocessing.set_start_method('fork')\n                server = UvicornServer(self.http_port, [serversocket],\n                                       self.dataplane, self.model_repository_extension,\n                                       self.enable_docs_url, custom_log_config=self.log_config)\n                for _ in range(self.workers):\n                    p = Process(target=server.run_sync)\n                    p.start()\n\n        async def servers_task():\n            servers = [serve()]\n            if self.enable_grpc:\n                servers.append(self._grpc_server.start(self.max_threads))\n            await asyncio.gather(*servers)\n\n        asyncio.run(servers_task())\n\n    async def stop(self, sig: Optional[int] = None):\n        logging.info(\"Stopping the model server\")\n        if self._rest_server:\n            logging.info(\"Stopping the rest server\")\n            await self._rest_server.stop()\n        if self._grpc_server:\n            logging.info(\"Stopping the grpc server\")\n            await self._grpc_server.stop(sig)\n\n    def register_model_handle(self, name: str, model_handle: RayServeHandle):\n        self.registered_models.update_handle(name, model_handle)\n        logging.info(\"Registering model handle: %s\", name)\n\n    def register_model(self, model: Model):\n        if not model.name:\n            raise Exception(\n                \"Failed to register model, model.name must be provided.\")\n        self.registered_models.update(model)\n        logging.info(\"Registering model: %s\", model.name)\n", "path": "python/kserve/kserve/model_server.py"}], "after_files": [{"content": "# Copyright 2022 The KServe Authors.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#    http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport argparse\nimport asyncio\nimport concurrent.futures\nimport logging\nimport signal\nimport socket\nfrom distutils.util import strtobool\nfrom typing import List, Dict, Union, Optional\n\nfrom ray import serve as rayserve\nfrom ray.serve.api import Deployment, RayServeHandle\n\nfrom .protocol.grpc.server import GRPCServer\nfrom .protocol.rest.server import UvicornServer\nfrom .utils import utils\nimport multiprocessing\nfrom multiprocessing import Process\n\nfrom .model import Model\nfrom .protocol.dataplane import DataPlane\nfrom .protocol.model_repository_extension import ModelRepositoryExtension\nfrom .model_repository import ModelRepository\n\nDEFAULT_HTTP_PORT = 8080\nDEFAULT_GRPC_PORT = 8081\n\nparser = argparse.ArgumentParser(add_help=False)\nparser.add_argument(\"--http_port\", default=DEFAULT_HTTP_PORT, type=int,\n                    help=\"The HTTP Port listened to by the model server.\")\nparser.add_argument(\"--grpc_port\", default=DEFAULT_GRPC_PORT, type=int,\n                    help=\"The GRPC Port listened to by the model server.\")\nparser.add_argument(\"--workers\", default=1, type=int,\n                    help=\"The number of workers for multi-processing.\")\nparser.add_argument(\"--max_threads\", default=4, type=int,\n                    help=\"The number of max processing threads in each worker.\")\nparser.add_argument('--max_asyncio_workers', default=None, type=int,\n                    help='Max number of asyncio workers to spawn')\nparser.add_argument(\"--enable_grpc\", default=True, type=lambda x: bool(strtobool(x)),\n                    help=\"Enable gRPC for the model server\")\nparser.add_argument(\"--enable_docs_url\", default=False, type=lambda x: bool(strtobool(x)),\n                    help=\"Enable docs url '/docs' to display Swagger UI.\")\nparser.add_argument(\"--enable_latency_logging\", default=True, type=lambda x: bool(strtobool(x)),\n                    help=\"Output a log per request with latency metrics.\")\nparser.add_argument(\"--log_config_file\", default=None, type=str,\n                    help=\"File path containing UvicornServer's log config. Needs to be a yaml or json file.\")\nparser.add_argument(\"--access_log_format\", default=None, type=str,\n                    help=\"Format to set for the access log (provided by asgi-logger).\")\n\nargs, _ = parser.parse_known_args()\n\nFORMAT = '%(asctime)s.%(msecs)03d %(process)s %(name)s %(levelname)s [%(funcName)s():%(lineno)s] %(message)s'\nDATE_FORMAT = \"%Y-%m-%d %H:%M:%S\"\nlogging.basicConfig(level=logging.INFO, format=FORMAT, datefmt=DATE_FORMAT)\n\n\nclass ModelServer:\n    \"\"\"KServe ModelServer\n\n    Args:\n        http_port (int): HTTP port. Default: ``8080``.\n        grpc_port (int): GRPC port. Default: ``8081``.\n        workers (int): Number of workers for uvicorn. Default: ``1``.\n        max_threads (int): Max number of processing threads. Default: ``4``\n        max_asyncio_workers (int): Max number of AsyncIO threads. Default: ``None``\n        registered_models (ModelRepository): Model repository with registered models.\n        enable_grpc (bool): Whether to turn on grpc server. Default: ``True``\n        enable_docs_url (bool): Whether to turn on ``/docs`` Swagger UI. Default: ``False``.\n        enable_latency_logging (bool): Whether to log latency metric. Default: ``True``.\n        log_config_file (dict): File path containing UvicornServer's log config. Default: ``None``.\n        access_log_format (string): Format to set for the access log (provided by asgi-logger). Default: ``None``\n    \"\"\"\n\n    def __init__(self, http_port: int = args.http_port,\n                 grpc_port: int = args.grpc_port,\n                 workers: int = args.workers,\n                 max_threads: int = args.max_threads,\n                 max_asyncio_workers: int = args.max_asyncio_workers,\n                 registered_models: ModelRepository = ModelRepository(),\n                 enable_grpc: bool = args.enable_grpc,\n                 enable_docs_url: bool = args.enable_docs_url,\n                 enable_latency_logging: bool = args.enable_latency_logging,\n                 log_config_file: str = args.log_config_file,\n                 access_log_format: str = args.access_log_format):\n        self.registered_models = registered_models\n        self.http_port = http_port\n        self.grpc_port = grpc_port\n        self.workers = workers\n        self.max_threads = max_threads\n        self.max_asyncio_workers = max_asyncio_workers\n        self.enable_grpc = enable_grpc\n        self.enable_docs_url = enable_docs_url\n        self.enable_latency_logging = enable_latency_logging\n        self.dataplane = DataPlane(model_registry=registered_models)\n        self.model_repository_extension = ModelRepositoryExtension(\n            model_registry=self.registered_models)\n        self._grpc_server = None\n        if self.enable_grpc:\n            self._grpc_server = GRPCServer(grpc_port, self.dataplane,\n                                           self.model_repository_extension)\n        self.log_config_file = log_config_file\n        self.access_log_format = access_log_format\n\n    def start(self, models: Union[List[Model], Dict[str, Deployment]]) -> None:\n        if isinstance(models, list):\n            for model in models:\n                if isinstance(model, Model):\n                    self.register_model(model)\n                    # pass whether to log request latency into the model\n                    model.enable_latency_logging = self.enable_latency_logging\n                else:\n                    raise RuntimeError(\"Model type should be 'Model'\")\n        elif isinstance(models, dict):\n            if all([isinstance(v, Deployment) for v in models.values()]):\n                # TODO: make this port number a variable\n                rayserve.start(detached=True, http_options={\"host\": \"0.0.0.0\", \"port\": 9071})\n                for key in models:\n                    models[key].deploy()\n                    handle = models[key].get_handle()\n                    self.register_model_handle(key, handle)\n            else:\n                raise RuntimeError(\"Model type should be RayServe Deployment\")\n        else:\n            raise RuntimeError(\"Unknown model collection types\")\n\n        if self.max_asyncio_workers is None:\n            # formula as suggest in https://bugs.python.org/issue35279\n            self.max_asyncio_workers = min(32, utils.cpu_count() + 4)\n        logging.info(f\"Setting max asyncio worker threads as {self.max_asyncio_workers}\")\n        asyncio.get_event_loop().set_default_executor(\n            concurrent.futures.ThreadPoolExecutor(max_workers=self.max_asyncio_workers))\n\n        async def serve():\n            serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n            serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)\n            serversocket.bind(('0.0.0.0', self.http_port))\n            serversocket.listen(5)\n\n            logging.info(f\"Starting uvicorn with {self.workers} workers\")\n            loop = asyncio.get_event_loop()\n            for sig in [signal.SIGINT, signal.SIGTERM, signal.SIGQUIT]:\n                loop.add_signal_handler(\n                    sig, lambda s=sig: asyncio.create_task(self.stop(sig=s))\n                )\n            self._rest_server = UvicornServer(self.http_port, [serversocket],\n                                              self.dataplane, self.model_repository_extension,\n                                              self.enable_docs_url, log_config_file=self.log_config_file,\n                                              access_log_format=self.access_log_format)\n            if self.workers == 1:\n                await self._rest_server.run()\n            else:\n                # Since py38 MacOS/Windows defaults to use spawn for starting multiprocessing.\n                # https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods\n                # Spawn does not work with FastAPI/uvicorn in multiprocessing mode, use fork for multiprocessing\n                # https://github.com/tiangolo/fastapi/issues/1586\n                multiprocessing.set_start_method('fork')\n                server = UvicornServer(self.http_port, [serversocket],\n                                       self.dataplane, self.model_repository_extension,\n                                       self.enable_docs_url, custom_log_config=self.log_config)\n                for _ in range(self.workers):\n                    p = Process(target=server.run_sync)\n                    p.start()\n\n        async def servers_task():\n            servers = [serve()]\n            if self.enable_grpc:\n                servers.append(self._grpc_server.start(self.max_threads))\n            await asyncio.gather(*servers)\n\n        asyncio.run(servers_task())\n\n    async def stop(self, sig: Optional[int] = None):\n        logging.info(\"Stopping the model server\")\n        if self._rest_server:\n            logging.info(\"Stopping the rest server\")\n            await self._rest_server.stop()\n        if self._grpc_server:\n            logging.info(\"Stopping the grpc server\")\n            await self._grpc_server.stop(sig)\n\n    def register_model_handle(self, name: str, model_handle: RayServeHandle):\n        self.registered_models.update_handle(name, model_handle)\n        logging.info(\"Registering model handle: %s\", name)\n\n    def register_model(self, model: Model):\n        if not model.name:\n            raise Exception(\n                \"Failed to register model, model.name must be provided.\")\n        self.registered_models.update(model)\n        logging.info(\"Registering model: %s\", model.name)\n", "path": "python/kserve/kserve/model_server.py"}]}