We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[huggingface] Merge predictive unit parameters from env with model settings
# Background
## What are PREDICTIVE_UNIT_PARAMETERS
This is a collection of parameters passed though the environment to a HuggingFace model. This is opposed to declaring them in a `model-settings.json` file as it is the case in Seldon Core v2. As of this moment, those parameters are injected via Seldon Core v1 only, where as Seldon Core v2 uses `model-settings.json` to provide the metadata for the model.

PREDICTIVE_UNIT_PARAMETERS are used only  by the HuggingFace runtime and injected only by SCv1.

You can find the code for creating `HuggingFaceSettings` in `./runtimes/huggingface/ml-server-huggingface/settings.py` fle along with functions for parsing those params from env vars or from `model-settings.json`  

# What is the problem
Currently, `HuggingFaceSettings` are created either from parsing the PREDICTIVE_UNIT_PARAMETERS from the environment OR from the `model-settings.json`. Meaning that if there is at least one parameter set in the env var, the `model-settings.json` extra parameters will be ignored. This makes it cumbersome when a deployment is created from the UI because additional params such as `task`, `pretrained_model`, `pretrained_tokenizer`, `framework`, etc. will have to be added one by one in the Wizard. Why they have to be added from the wizard and not just specified in `model-settings.json` - because currently SCv1 always injects `model_uri` param so the PREDICTIVE_UNIT_PARAMETERS env var so it's not empty. Because this var is not empty, the HF settings are initialised from it and the `model-settings.json` is ignored.

# What needs to be done
When creating HuggingFace settings, env vars needs to be merged with params from `model-settings.json`, giving priority to env vars. For example:
If such env var exists:
```
PREDICTIVE_UNIT_PARAMETERS = [{"name":"model_uri","value":"/mnt/models","type":"STRING"}, {"name":"task_suffix","value":"else","type":"STRING"}]
```
and such `model-settings.json` file exists:
```
{
    "name": "transformer",
    "implementation": "mlserver_huggingface.HuggingFaceRuntime",
    "parameters": {
        "extra": {
            "task": "text-generation",
            "task_suffix": "something",
            "framework": "pt"
        }
    }
}
```
The outcome should be that the `task` parameter doesn't need to be specified in the wizard and
The HuggingFace settings should contain values: task = text-generation, task_suffix = else, framework = pt

# Scope
This only relating to `HuggingFace` runtime and when it's used from SCv1 and only valid as long as SCv1 is still operational and related code is present in MLServer.

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `runtimes/huggingface/mlserver_huggingface/settings.py`
Content:
```
1 import os
2 import orjson
3 
4 from typing import Optional, Dict
5 from pydantic import BaseSettings
6 from distutils.util import strtobool
7 from transformers.pipelines import SUPPORTED_TASKS
8 
9 try:
10     # Optimum 1.7 changed the import name from `SUPPORTED_TASKS` to
11     # `ORT_SUPPORTED_TASKS`.
12     # We'll try to import the more recent one, falling back to the previous
13     # import name if not present.
14     # https://github.com/huggingface/optimum/blob/987b02e4f6e2a1c9325b364ff764da2e57e89902/optimum/pipelines/__init__.py#L18
15     from optimum.pipelines import ORT_SUPPORTED_TASKS as SUPPORTED_OPTIMUM_TASKS
16 except ImportError:
17     from optimum.pipelines import SUPPORTED_TASKS as SUPPORTED_OPTIMUM_TASKS
18 
19 from mlserver.settings import ModelSettings
20 
21 from .errors import (
22     MissingHuggingFaceSettings,
23     InvalidTransformersTask,
24     InvalidOptimumTask,
25     InvalidModelParameter,
26     InvalidModelParameterType,
27 )
28 
29 ENV_PREFIX_HUGGINGFACE_SETTINGS = "MLSERVER_MODEL_HUGGINGFACE_"
30 PARAMETERS_ENV_NAME = "PREDICTIVE_UNIT_PARAMETERS"
31 
32 
33 class HuggingFaceSettings(BaseSettings):
34     """
35     Parameters that apply only to HuggingFace models
36     """
37 
38     class Config:
39         env_prefix = ENV_PREFIX_HUGGINGFACE_SETTINGS
40 
41     # TODO: Document fields
42     task: str = ""
43     """
44     Pipeline task to load.
45     You can see the available Optimum and Transformers tasks available in the
46     links below:
47 
48     - `Optimum Tasks <https://huggingface.co/docs/optimum/onnxruntime/usage_guides/pipelines#inference-pipelines-with-the-onnx-runtime-accelerator>`_
49     - `Transformer Tasks <https://huggingface.co/docs/transformers/task_summary>`_
50     """  # noqa: E501
51 
52     task_suffix: str = ""
53     """
54     Suffix to append to the base task name.
55     Useful for, e.g. translation tasks which require a suffix on the task name
56     to specify source and target.
57     """
58 
59     pretrained_model: Optional[str] = None
60     """
61     Name of the model that should be loaded in the pipeline.
62     """
63 
64     pretrained_tokenizer: Optional[str] = None
65     """
66     Name of the tokenizer that should be loaded in the pipeline.
67     """
68 
69     framework: Optional[str] = None
70     """
71     The framework to use, either "pt" for PyTorch or "tf" for TensorFlow.
72     """
73 
74     optimum_model: bool = False
75     """
76     Flag to decide whether the pipeline should use a Optimum-optimised model or
77     the standard Transformers model.
78     Under the hood, this will enable the model to use the optimised ONNX
79     runtime.
80     """
81 
82     device: int = -1
83     """
84     Device in which this pipeline will be loaded (e.g., "cpu", "cuda:1", "mps",
85     or a GPU ordinal rank like 1).
86     """
87 
88     inter_op_threads: Optional[int] = None
89     """
90     Threads used for parallelism between independent operations.
91     PyTorch:
92     https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html
93     Tensorflow:
94     https://www.tensorflow.org/api_docs/python/tf/config/threading/set_inter_op_parallelism_threads
95     """
96 
97     intra_op_threads: Optional[int] = None
98     """
99     Threads used within an individual op for parallelism.
100     PyTorch:
101     https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html
102     Tensorflow:
103     https://www.tensorflow.org/api_docs/python/tf/config/threading/set_intra_op_parallelism_threads
104     """
105 
106     @property
107     def task_name(self):
108         if self.task == "translation":
109             return f"{self.task}{self.task_suffix}"
110         return self.task
111 
112 
113 def parse_parameters_from_env() -> Dict:
114     """
115     This method parses the environment variables injected via SCv1.
116     """
117     # TODO: Once support for SCv1 is deprecated, we should remove this method and rely
118     # purely on settings coming via the `model-settings.json` file.
119     parameters = orjson.loads(os.environ.get(PARAMETERS_ENV_NAME, "[]"))
120 
121     type_dict = {
122         "INT": int,
123         "FLOAT": float,
124         "DOUBLE": float,
125         "STRING": str,
126         "BOOL": bool,
127     }
128 
129     parsed_parameters = {}
130     for param in parameters:
131         name = param.get("name")
132         value = param.get("value")
133         type_ = param.get("type")
134         if type_ == "BOOL":
135             parsed_parameters[name] = bool(strtobool(value))
136         else:
137             try:
138                 parsed_parameters[name] = type_dict[type_](value)
139             except ValueError:
140                 raise InvalidModelParameter(name, value, type_)
141             except KeyError:
142                 raise InvalidModelParameterType(type_)
143     return parsed_parameters
144 
145 
146 def get_huggingface_settings(model_settings: ModelSettings) -> HuggingFaceSettings:
147     env_params = parse_parameters_from_env()
148     if not env_params and (
149         not model_settings.parameters or not model_settings.parameters.extra
150     ):
151         raise MissingHuggingFaceSettings()
152 
153     extra = env_params or model_settings.parameters.extra  # type: ignore
154     hf_settings = HuggingFaceSettings(**extra)  # type: ignore
155 
156     if hf_settings.task not in SUPPORTED_TASKS:
157         raise InvalidTransformersTask(hf_settings.task, SUPPORTED_TASKS.keys())
158 
159     if hf_settings.optimum_model:
160         if hf_settings.task not in SUPPORTED_OPTIMUM_TASKS:
161             raise InvalidOptimumTask(hf_settings.task, SUPPORTED_OPTIMUM_TASKS.keys())
162 
163     return hf_settings
164
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/runtimes/huggingface/mlserver_huggingface/settings.py b/runtimes/huggingface/mlserver_huggingface/settings.py
--- a/runtimes/huggingface/mlserver_huggingface/settings.py
+++ b/runtimes/huggingface/mlserver_huggingface/settings.py
@@ -1,7 +1,7 @@
 import os
 import orjson
 
-from typing import Optional, Dict
+from typing import Optional, Dict, Union, NewType
 from pydantic import BaseSettings
 from distutils.util import strtobool
 from transformers.pipelines import SUPPORTED_TASKS
@@ -110,23 +110,33 @@
         return self.task
 
 
-def parse_parameters_from_env() -> Dict:
+EXTRA_TYPE_DICT = {
+    "INT": int,
+    "FLOAT": float,
+    "DOUBLE": float,
+    "STRING": str,
+    "BOOL": bool,
+}
+
+ExtraDict = NewType("ExtraDict", Dict[str, Union[str, bool, float, int]])
+
+
+def parse_parameters_from_env() -> ExtraDict:
     """
     This method parses the environment variables injected via SCv1.
+
+    At least an empty dict is always returned.
     """
     # TODO: Once support for SCv1 is deprecated, we should remove this method and rely
     # purely on settings coming via the `model-settings.json` file.
     parameters = orjson.loads(os.environ.get(PARAMETERS_ENV_NAME, "[]"))
 
-    type_dict = {
-        "INT": int,
-        "FLOAT": float,
-        "DOUBLE": float,
-        "STRING": str,
-        "BOOL": bool,
-    }
+    parsed_parameters: ExtraDict = ExtraDict({})
+
+    # Guard: Exit early if there's no parameters
+    if len(parameters) == 0:
+        return parsed_parameters
 
-    parsed_parameters = {}
     for param in parameters:
         name = param.get("name")
         value = param.get("value")
@@ -135,22 +145,20 @@
             parsed_parameters[name] = bool(strtobool(value))
         else:
             try:
-                parsed_parameters[name] = type_dict[type_](value)
+                parsed_parameters[name] = EXTRA_TYPE_DICT[type_](value)
             except ValueError:
                 raise InvalidModelParameter(name, value, type_)
             except KeyError:
                 raise InvalidModelParameterType(type_)
+
     return parsed_parameters
 
 
 def get_huggingface_settings(model_settings: ModelSettings) -> HuggingFaceSettings:
-    env_params = parse_parameters_from_env()
-    if not env_params and (
-        not model_settings.parameters or not model_settings.parameters.extra
-    ):
-        raise MissingHuggingFaceSettings()
+    """Get the HuggingFace settings provided to the runtime"""
 
-    extra = env_params or model_settings.parameters.extra  # type: ignore
+    env_params = parse_parameters_from_env()
+    extra = merge_huggingface_settings_extra(model_settings, env_params)
     hf_settings = HuggingFaceSettings(**extra)  # type: ignore
 
     if hf_settings.task not in SUPPORTED_TASKS:
@@ -161,3 +169,35 @@
             raise InvalidOptimumTask(hf_settings.task, SUPPORTED_OPTIMUM_TASKS.keys())
 
     return hf_settings
+
+
+def merge_huggingface_settings_extra(
+    model_settings: ModelSettings, env_params: ExtraDict
+) -> ExtraDict:
+    """
+    This function returns the Extra field of the Settings.
+
+    It merges them, iff they're both present, from the
+    environment AND model settings file. Precedence is
+    giving to the environment.
+    """
+
+    # Both `parameters` and `extra` are Optional, so we
+    # need to get the value, or nothing.
+    settings_params = (
+        model_settings.parameters.extra
+        if model_settings.parameters is not None
+        else None
+    )
+
+    if settings_params is None and env_params == {}:
+        # There must be settings provided by at least the environment OR model settings
+        raise MissingHuggingFaceSettings()
+
+    # Set the default value
+    settings_params = settings_params or {}
+
+    # Overwrite any conflicting keys, giving precedence to the environment
+    settings_params.update(env_params)
+
+    return ExtraDict(settings_params)

{"golden_diff": "diff --git a/runtimes/huggingface/mlserver_huggingface/settings.py b/runtimes/huggingface/mlserver_huggingface/settings.py\n--- a/runtimes/huggingface/mlserver_huggingface/settings.py\n+++ b/runtimes/huggingface/mlserver_huggingface/settings.py\n@@ -1,7 +1,7 @@\n import os\n import orjson\n \n-from typing import Optional, Dict\n+from typing import Optional, Dict, Union, NewType\n from pydantic import BaseSettings\n from distutils.util import strtobool\n from transformers.pipelines import SUPPORTED_TASKS\n@@ -110,23 +110,33 @@\n         return self.task\n \n \n-def parse_parameters_from_env() -> Dict:\n+EXTRA_TYPE_DICT = {\n+    \"INT\": int,\n+    \"FLOAT\": float,\n+    \"DOUBLE\": float,\n+    \"STRING\": str,\n+    \"BOOL\": bool,\n+}\n+\n+ExtraDict = NewType(\"ExtraDict\", Dict[str, Union[str, bool, float, int]])\n+\n+\n+def parse_parameters_from_env() -> ExtraDict:\n     \"\"\"\n     This method parses the environment variables injected via SCv1.\n+\n+    At least an empty dict is always returned.\n     \"\"\"\n     # TODO: Once support for SCv1 is deprecated, we should remove this method and rely\n     # purely on settings coming via the `model-settings.json` file.\n     parameters = orjson.loads(os.environ.get(PARAMETERS_ENV_NAME, \"[]\"))\n \n-    type_dict = {\n-        \"INT\": int,\n-        \"FLOAT\": float,\n-        \"DOUBLE\": float,\n-        \"STRING\": str,\n-        \"BOOL\": bool,\n-    }\n+    parsed_parameters: ExtraDict = ExtraDict({})\n+\n+    # Guard: Exit early if there's no parameters\n+    if len(parameters) == 0:\n+        return parsed_parameters\n \n-    parsed_parameters = {}\n     for param in parameters:\n         name = param.get(\"name\")\n         value = param.get(\"value\")\n@@ -135,22 +145,20 @@\n             parsed_parameters[name] = bool(strtobool(value))\n         else:\n             try:\n-                parsed_parameters[name] = type_dict[type_](value)\n+                parsed_parameters[name] = EXTRA_TYPE_DICT[type_](value)\n             except ValueError:\n                 raise InvalidModelParameter(name, value, type_)\n             except KeyError:\n                 raise InvalidModelParameterType(type_)\n+\n     return parsed_parameters\n \n \n def get_huggingface_settings(model_settings: ModelSettings) -> HuggingFaceSettings:\n-    env_params = parse_parameters_from_env()\n-    if not env_params and (\n-        not model_settings.parameters or not model_settings.parameters.extra\n-    ):\n-        raise MissingHuggingFaceSettings()\n+    \"\"\"Get the HuggingFace settings provided to the runtime\"\"\"\n \n-    extra = env_params or model_settings.parameters.extra  # type: ignore\n+    env_params = parse_parameters_from_env()\n+    extra = merge_huggingface_settings_extra(model_settings, env_params)\n     hf_settings = HuggingFaceSettings(**extra)  # type: ignore\n \n     if hf_settings.task not in SUPPORTED_TASKS:\n@@ -161,3 +169,35 @@\n             raise InvalidOptimumTask(hf_settings.task, SUPPORTED_OPTIMUM_TASKS.keys())\n \n     return hf_settings\n+\n+\n+def merge_huggingface_settings_extra(\n+    model_settings: ModelSettings, env_params: ExtraDict\n+) -> ExtraDict:\n+    \"\"\"\n+    This function returns the Extra field of the Settings.\n+\n+    It merges them, iff they're both present, from the\n+    environment AND model settings file. Precedence is\n+    giving to the environment.\n+    \"\"\"\n+\n+    # Both `parameters` and `extra` are Optional, so we\n+    # need to get the value, or nothing.\n+    settings_params = (\n+        model_settings.parameters.extra\n+        if model_settings.parameters is not None\n+        else None\n+    )\n+\n+    if settings_params is None and env_params == {}:\n+        # There must be settings provided by at least the environment OR model settings\n+        raise MissingHuggingFaceSettings()\n+\n+    # Set the default value\n+    settings_params = settings_params or {}\n+\n+    # Overwrite any conflicting keys, giving precedence to the environment\n+    settings_params.update(env_params)\n+\n+    return ExtraDict(settings_params)\n", "issue": "[huggingface] Merge predictive unit parameters from env with model settings\n# Background\r\n## What are PREDICTIVE_UNIT_PARAMETERS\r\nThis is a collection of parameters passed though the environment to a HuggingFace model. This is opposed to declaring them in a `model-settings.json` file as it is the case in Seldon Core v2. As of this moment, those parameters are injected via Seldon Core v1 only, where as Seldon Core v2 uses `model-settings.json` to provide the metadata for the model.\r\n\r\nPREDICTIVE_UNIT_PARAMETERS are used only  by the HuggingFace runtime and injected only by SCv1.\r\n\r\nYou can find the code for creating `HuggingFaceSettings` in `./runtimes/huggingface/ml-server-huggingface/settings.py` fle along with functions for parsing those params from env vars or from `model-settings.json`  \r\n\r\n# What is the problem\r\nCurrently, `HuggingFaceSettings` are created either from parsing the PREDICTIVE_UNIT_PARAMETERS from the environment OR from the `model-settings.json`. Meaning that if there is at least one parameter set in the env var, the `model-settings.json` extra parameters will be ignored. This makes it cumbersome when a deployment is created from the UI because additional params such as `task`, `pretrained_model`, `pretrained_tokenizer`, `framework`, etc. will have to be added one by one in the Wizard. Why they have to be added from the wizard and not just specified in `model-settings.json` - because currently SCv1 always injects `model_uri` param so the PREDICTIVE_UNIT_PARAMETERS env var so it's not empty. Because this var is not empty, the HF settings are initialised from it and the `model-settings.json` is ignored.\r\n\r\n# What needs to be done\r\nWhen creating HuggingFace settings, env vars needs to be merged with params from `model-settings.json`, giving priority to env vars. For example:\r\nIf such env var exists:\r\n```\r\nPREDICTIVE_UNIT_PARAMETERS = [{\"name\":\"model_uri\",\"value\":\"/mnt/models\",\"type\":\"STRING\"}, {\"name\":\"task_suffix\",\"value\":\"else\",\"type\":\"STRING\"}]\r\n```\r\nand such `model-settings.json` file exists:\r\n```\r\n{\r\n    \"name\": \"transformer\",\r\n    \"implementation\": \"mlserver_huggingface.HuggingFaceRuntime\",\r\n    \"parameters\": {\r\n        \"extra\": {\r\n            \"task\": \"text-generation\",\r\n            \"task_suffix\": \"something\",\r\n            \"framework\": \"pt\"\r\n        }\r\n    }\r\n}\r\n```\r\nThe outcome should be that the `task` parameter doesn't need to be specified in the wizard and\r\nThe HuggingFace settings should contain values: task = text-generation, task_suffix = else, framework = pt\r\n\r\n# Scope\r\nThis only relating to `HuggingFace` runtime and when it's used from SCv1 and only valid as long as SCv1 is still operational and related code is present in MLServer.\n", "before_files": [{"content": "import os\nimport orjson\n\nfrom typing import Optional, Dict\nfrom pydantic import BaseSettings\nfrom distutils.util import strtobool\nfrom transformers.pipelines import SUPPORTED_TASKS\n\ntry:\n    # Optimum 1.7 changed the import name from `SUPPORTED_TASKS` to\n    # `ORT_SUPPORTED_TASKS`.\n    # We'll try to import the more recent one, falling back to the previous\n    # import name if not present.\n    # https://github.com/huggingface/optimum/blob/987b02e4f6e2a1c9325b364ff764da2e57e89902/optimum/pipelines/__init__.py#L18\n    from optimum.pipelines import ORT_SUPPORTED_TASKS as SUPPORTED_OPTIMUM_TASKS\nexcept ImportError:\n    from optimum.pipelines import SUPPORTED_TASKS as SUPPORTED_OPTIMUM_TASKS\n\nfrom mlserver.settings import ModelSettings\n\nfrom .errors import (\n    MissingHuggingFaceSettings,\n    InvalidTransformersTask,\n    InvalidOptimumTask,\n    InvalidModelParameter,\n    InvalidModelParameterType,\n)\n\nENV_PREFIX_HUGGINGFACE_SETTINGS = \"MLSERVER_MODEL_HUGGINGFACE_\"\nPARAMETERS_ENV_NAME = \"PREDICTIVE_UNIT_PARAMETERS\"\n\n\nclass HuggingFaceSettings(BaseSettings):\n    \"\"\"\n    Parameters that apply only to HuggingFace models\n    \"\"\"\n\n    class Config:\n        env_prefix = ENV_PREFIX_HUGGINGFACE_SETTINGS\n\n    # TODO: Document fields\n    task: str = \"\"\n    \"\"\"\n    Pipeline task to load.\n    You can see the available Optimum and Transformers tasks available in the\n    links below:\n\n    - `Optimum Tasks <https://huggingface.co/docs/optimum/onnxruntime/usage_guides/pipelines#inference-pipelines-with-the-onnx-runtime-accelerator>`_\n    - `Transformer Tasks <https://huggingface.co/docs/transformers/task_summary>`_\n    \"\"\"  # noqa: E501\n\n    task_suffix: str = \"\"\n    \"\"\"\n    Suffix to append to the base task name.\n    Useful for, e.g. translation tasks which require a suffix on the task name\n    to specify source and target.\n    \"\"\"\n\n    pretrained_model: Optional[str] = None\n    \"\"\"\n    Name of the model that should be loaded in the pipeline.\n    \"\"\"\n\n    pretrained_tokenizer: Optional[str] = None\n    \"\"\"\n    Name of the tokenizer that should be loaded in the pipeline.\n    \"\"\"\n\n    framework: Optional[str] = None\n    \"\"\"\n    The framework to use, either \"pt\" for PyTorch or \"tf\" for TensorFlow.\n    \"\"\"\n\n    optimum_model: bool = False\n    \"\"\"\n    Flag to decide whether the pipeline should use a Optimum-optimised model or\n    the standard Transformers model.\n    Under the hood, this will enable the model to use the optimised ONNX\n    runtime.\n    \"\"\"\n\n    device: int = -1\n    \"\"\"\n    Device in which this pipeline will be loaded (e.g., \"cpu\", \"cuda:1\", \"mps\",\n    or a GPU ordinal rank like 1).\n    \"\"\"\n\n    inter_op_threads: Optional[int] = None\n    \"\"\"\n    Threads used for parallelism between independent operations.\n    PyTorch:\n    https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html\n    Tensorflow:\n    https://www.tensorflow.org/api_docs/python/tf/config/threading/set_inter_op_parallelism_threads\n    \"\"\"\n\n    intra_op_threads: Optional[int] = None\n    \"\"\"\n    Threads used within an individual op for parallelism.\n    PyTorch:\n    https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html\n    Tensorflow:\n    https://www.tensorflow.org/api_docs/python/tf/config/threading/set_intra_op_parallelism_threads\n    \"\"\"\n\n    @property\n    def task_name(self):\n        if self.task == \"translation\":\n            return f\"{self.task}{self.task_suffix}\"\n        return self.task\n\n\ndef parse_parameters_from_env() -> Dict:\n    \"\"\"\n    This method parses the environment variables injected via SCv1.\n    \"\"\"\n    # TODO: Once support for SCv1 is deprecated, we should remove this method and rely\n    # purely on settings coming via the `model-settings.json` file.\n    parameters = orjson.loads(os.environ.get(PARAMETERS_ENV_NAME, \"[]\"))\n\n    type_dict = {\n        \"INT\": int,\n        \"FLOAT\": float,\n        \"DOUBLE\": float,\n        \"STRING\": str,\n        \"BOOL\": bool,\n    }\n\n    parsed_parameters = {}\n    for param in parameters:\n        name = param.get(\"name\")\n        value = param.get(\"value\")\n        type_ = param.get(\"type\")\n        if type_ == \"BOOL\":\n            parsed_parameters[name] = bool(strtobool(value))\n        else:\n            try:\n                parsed_parameters[name] = type_dict[type_](value)\n            except ValueError:\n                raise InvalidModelParameter(name, value, type_)\n            except KeyError:\n                raise InvalidModelParameterType(type_)\n    return parsed_parameters\n\n\ndef get_huggingface_settings(model_settings: ModelSettings) -> HuggingFaceSettings:\n    env_params = parse_parameters_from_env()\n    if not env_params and (\n        not model_settings.parameters or not model_settings.parameters.extra\n    ):\n        raise MissingHuggingFaceSettings()\n\n    extra = env_params or model_settings.parameters.extra  # type: ignore\n    hf_settings = HuggingFaceSettings(**extra)  # type: ignore\n\n    if hf_settings.task not in SUPPORTED_TASKS:\n        raise InvalidTransformersTask(hf_settings.task, SUPPORTED_TASKS.keys())\n\n    if hf_settings.optimum_model:\n        if hf_settings.task not in SUPPORTED_OPTIMUM_TASKS:\n            raise InvalidOptimumTask(hf_settings.task, SUPPORTED_OPTIMUM_TASKS.keys())\n\n    return hf_settings\n", "path": "runtimes/huggingface/mlserver_huggingface/settings.py"}], "after_files": [{"content": "import os\nimport orjson\n\nfrom typing import Optional, Dict, Union, NewType\nfrom pydantic import BaseSettings\nfrom distutils.util import strtobool\nfrom transformers.pipelines import SUPPORTED_TASKS\n\ntry:\n    # Optimum 1.7 changed the import name from `SUPPORTED_TASKS` to\n    # `ORT_SUPPORTED_TASKS`.\n    # We'll try to import the more recent one, falling back to the previous\n    # import name if not present.\n    # https://github.com/huggingface/optimum/blob/987b02e4f6e2a1c9325b364ff764da2e57e89902/optimum/pipelines/__init__.py#L18\n    from optimum.pipelines import ORT_SUPPORTED_TASKS as SUPPORTED_OPTIMUM_TASKS\nexcept ImportError:\n    from optimum.pipelines import SUPPORTED_TASKS as SUPPORTED_OPTIMUM_TASKS\n\nfrom mlserver.settings import ModelSettings\n\nfrom .errors import (\n    MissingHuggingFaceSettings,\n    InvalidTransformersTask,\n    InvalidOptimumTask,\n    InvalidModelParameter,\n    InvalidModelParameterType,\n)\n\nENV_PREFIX_HUGGINGFACE_SETTINGS = \"MLSERVER_MODEL_HUGGINGFACE_\"\nPARAMETERS_ENV_NAME = \"PREDICTIVE_UNIT_PARAMETERS\"\n\n\nclass HuggingFaceSettings(BaseSettings):\n    \"\"\"\n    Parameters that apply only to HuggingFace models\n    \"\"\"\n\n    class Config:\n        env_prefix = ENV_PREFIX_HUGGINGFACE_SETTINGS\n\n    # TODO: Document fields\n    task: str = \"\"\n    \"\"\"\n    Pipeline task to load.\n    You can see the available Optimum and Transformers tasks available in the\n    links below:\n\n    - `Optimum Tasks <https://huggingface.co/docs/optimum/onnxruntime/usage_guides/pipelines#inference-pipelines-with-the-onnx-runtime-accelerator>`_\n    - `Transformer Tasks <https://huggingface.co/docs/transformers/task_summary>`_\n    \"\"\"  # noqa: E501\n\n    task_suffix: str = \"\"\n    \"\"\"\n    Suffix to append to the base task name.\n    Useful for, e.g. translation tasks which require a suffix on the task name\n    to specify source and target.\n    \"\"\"\n\n    pretrained_model: Optional[str] = None\n    \"\"\"\n    Name of the model that should be loaded in the pipeline.\n    \"\"\"\n\n    pretrained_tokenizer: Optional[str] = None\n    \"\"\"\n    Name of the tokenizer that should be loaded in the pipeline.\n    \"\"\"\n\n    framework: Optional[str] = None\n    \"\"\"\n    The framework to use, either \"pt\" for PyTorch or \"tf\" for TensorFlow.\n    \"\"\"\n\n    optimum_model: bool = False\n    \"\"\"\n    Flag to decide whether the pipeline should use a Optimum-optimised model or\n    the standard Transformers model.\n    Under the hood, this will enable the model to use the optimised ONNX\n    runtime.\n    \"\"\"\n\n    device: int = -1\n    \"\"\"\n    Device in which this pipeline will be loaded (e.g., \"cpu\", \"cuda:1\", \"mps\",\n    or a GPU ordinal rank like 1).\n    \"\"\"\n\n    inter_op_threads: Optional[int] = None\n    \"\"\"\n    Threads used for parallelism between independent operations.\n    PyTorch:\n    https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html\n    Tensorflow:\n    https://www.tensorflow.org/api_docs/python/tf/config/threading/set_inter_op_parallelism_threads\n    \"\"\"\n\n    intra_op_threads: Optional[int] = None\n    \"\"\"\n    Threads used within an individual op for parallelism.\n    PyTorch:\n    https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html\n    Tensorflow:\n    https://www.tensorflow.org/api_docs/python/tf/config/threading/set_intra_op_parallelism_threads\n    \"\"\"\n\n    @property\n    def task_name(self):\n        if self.task == \"translation\":\n            return f\"{self.task}{self.task_suffix}\"\n        return self.task\n\n\nEXTRA_TYPE_DICT = {\n    \"INT\": int,\n    \"FLOAT\": float,\n    \"DOUBLE\": float,\n    \"STRING\": str,\n    \"BOOL\": bool,\n}\n\nExtraDict = NewType(\"ExtraDict\", Dict[str, Union[str, bool, float, int]])\n\n\ndef parse_parameters_from_env() -> ExtraDict:\n    \"\"\"\n    This method parses the environment variables injected via SCv1.\n\n    At least an empty dict is always returned.\n    \"\"\"\n    # TODO: Once support for SCv1 is deprecated, we should remove this method and rely\n    # purely on settings coming via the `model-settings.json` file.\n    parameters = orjson.loads(os.environ.get(PARAMETERS_ENV_NAME, \"[]\"))\n\n    parsed_parameters: ExtraDict = ExtraDict({})\n\n    # Guard: Exit early if there's no parameters\n    if len(parameters) == 0:\n        return parsed_parameters\n\n    for param in parameters:\n        name = param.get(\"name\")\n        value = param.get(\"value\")\n        type_ = param.get(\"type\")\n        if type_ == \"BOOL\":\n            parsed_parameters[name] = bool(strtobool(value))\n        else:\n            try:\n                parsed_parameters[name] = EXTRA_TYPE_DICT[type_](value)\n            except ValueError:\n                raise InvalidModelParameter(name, value, type_)\n            except KeyError:\n                raise InvalidModelParameterType(type_)\n\n    return parsed_parameters\n\n\ndef get_huggingface_settings(model_settings: ModelSettings) -> HuggingFaceSettings:\n    \"\"\"Get the HuggingFace settings provided to the runtime\"\"\"\n\n    env_params = parse_parameters_from_env()\n    extra = merge_huggingface_settings_extra(model_settings, env_params)\n    hf_settings = HuggingFaceSettings(**extra)  # type: ignore\n\n    if hf_settings.task not in SUPPORTED_TASKS:\n        raise InvalidTransformersTask(hf_settings.task, SUPPORTED_TASKS.keys())\n\n    if hf_settings.optimum_model:\n        if hf_settings.task not in SUPPORTED_OPTIMUM_TASKS:\n            raise InvalidOptimumTask(hf_settings.task, SUPPORTED_OPTIMUM_TASKS.keys())\n\n    return hf_settings\n\n\ndef merge_huggingface_settings_extra(\n    model_settings: ModelSettings, env_params: ExtraDict\n) -> ExtraDict:\n    \"\"\"\n    This function returns the Extra field of the Settings.\n\n    It merges them, iff they're both present, from the\n    environment AND model settings file. Precedence is\n    giving to the environment.\n    \"\"\"\n\n    # Both `parameters` and `extra` are Optional, so we\n    # need to get the value, or nothing.\n    settings_params = (\n        model_settings.parameters.extra\n        if model_settings.parameters is not None\n        else None\n    )\n\n    if settings_params is None and env_params == {}:\n        # There must be settings provided by at least the environment OR model settings\n        raise MissingHuggingFaceSettings()\n\n    # Set the default value\n    settings_params = settings_params or {}\n\n    # Overwrite any conflicting keys, giving precedence to the environment\n    settings_params.update(env_params)\n\n    return ExtraDict(settings_params)\n", "path": "runtimes/huggingface/mlserver_huggingface/settings.py"}]}