We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
SASEvaluator returns score even when `predicted_answers` contains `None`
In a RAG pipeline, there are cases where the prompt instructs the LLM to return a placeholder answer if there is missing information in the context, such as "No answer". See an [example](https://github.com/deepset-ai/haystack-evaluation/blob/cbfcde5afcaf7574aa82b1ad6b5ce307c8a7c97e/evaluations/architectures/basic_rag.py#L10) in our evaluation repository.

In that case, it makes sense to replace the predicted answer in the RAG pipeline with `None`, instead of computing any metric on the placeholder answer. However, the SASEvaluator returns a score even if `None` is passed.

**Example**:
```
from haystack.components.evaluators import SASEvaluator

sas = SASEvaluator(model="sentence-transformers/all-mpnet-base-v2")
sas.warm_up()
sas.run(predicted_answers=[None, "Whatever"], ground_truth_answers=["Whatever", "Another whatever"])
```

the output is:

```
{'score': 0.46286168694496155,
 'individual_scores': [0.26153379678726196, 0.6641895771026611]}
```

**Expected**

```
{'score': 0.6641895771026611,
 'individual_scores': [None, 0.6641895771026611]}
```

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `haystack/components/evaluators/sas_evaluator.py`
Content:
```
1 # SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>
2 #
3 # SPDX-License-Identifier: Apache-2.0
4 
5 from typing import Any, Dict, List, Optional
6 
7 from numpy import mean as np_mean
8 
9 from haystack import component, default_from_dict, default_to_dict
10 from haystack.lazy_imports import LazyImport
11 from haystack.utils import ComponentDevice, expit
12 from haystack.utils.auth import Secret, deserialize_secrets_inplace
13 
14 with LazyImport(message="Run 'pip install \"sentence-transformers>=2.3.0\"'") as sas_import:
15     from sentence_transformers import CrossEncoder, SentenceTransformer, util
16     from transformers import AutoConfig
17 
18 
19 @component
20 class SASEvaluator:
21     """
22     SASEvaluator computes the Semantic Answer Similarity (SAS) between a list of predictions and a one of ground truths.
23 
24     It's usually used in Retrieval Augmented Generation (RAG) pipelines to evaluate the quality of the generated
25     answers. The SAS is computed using a pre-trained model from the Hugging Face model hub. The model can be either a
26     Bi-Encoder or a Cross-Encoder. The choice of the model is based on the `model` parameter.
27 
28     Usage example:
29     ```python
30     from haystack.components.evaluators.sas_evaluator import SASEvaluator
31 
32     evaluator = SASEvaluator(model="cross-encoder/ms-marco-MiniLM-L-6-v2")
33     evaluator.warm_up()
34     ground_truths = [
35         "A construction budget of US $2.3 billion",
36         "The Eiffel Tower, completed in 1889, symbolizes Paris's cultural magnificence.",
37         "The Meiji Restoration in 1868 transformed Japan into a modernized world power.",
38     ]
39     predictions = [
40         "A construction budget of US $2.3 billion",
41         "The Eiffel Tower, completed in 1889, symbolizes Paris's cultural magnificence.",
42         "The Meiji Restoration in 1868 transformed Japan into a modernized world power.",
43     ]
44     result = evaluator.run(
45         ground_truths_answers=ground_truths, predicted_answers=predictions
46     )
47 
48     print(result["score"])
49     # 0.9999673763910929
50 
51     print(result["individual_scores"])
52     # [0.9999765157699585, 0.999968409538269, 0.9999572038650513]
53     ```
54     """
55 
56     def __init__(
57         self,
58         model: str = "sentence-transformers/paraphrase-multilingual-mpnet-base-v2",
59         batch_size: int = 32,
60         device: Optional[ComponentDevice] = None,
61         token: Secret = Secret.from_env_var("HF_API_TOKEN", strict=False),
62     ):
63         """
64         Creates a new instance of SASEvaluator.
65 
66         :param model:
67             SentenceTransformers semantic textual similarity model, should be path or string pointing to a downloadable
68             model.
69         :param batch_size:
70             Number of prediction-label pairs to encode at once.
71         :param device:
72             The device on which the model is loaded. If `None`, the default device is automatically selected.
73         :param token:
74             The Hugging Face token for HTTP bearer authorization.
75             You can find your HF token in your [account settings](https://huggingface.co/settings/tokens)
76         """
77         sas_import.check()
78 
79         self._model = model
80         self._batch_size = batch_size
81         self._device = device
82         self._token = token
83         self._similarity_model = None
84 
85     def to_dict(self) -> Dict[str, Any]:
86         """
87         Serialize this component to a dictionary.
88 
89         :returns:
90             The serialized component as a dictionary.
91         """
92         return default_to_dict(
93             self,
94             model=self._model,
95             batch_size=self._batch_size,
96             device=self._device.to_dict() if self._device else None,
97             token=self._token.to_dict() if self._token else None,
98         )
99 
100     @classmethod
101     def from_dict(cls, data: Dict[str, Any]) -> "SASEvaluator":
102         """
103         Deserialize this component from a dictionary.
104 
105         :param data:
106             The dictionary representation of this component.
107         :returns:
108             The deserialized component instance.
109         """
110         deserialize_secrets_inplace(data["init_parameters"], keys=["token"])
111         if device := data.get("init_parameters", {}).get("device"):
112             data["init_parameters"]["device"] = ComponentDevice.from_dict(device)
113         return default_from_dict(cls, data)
114 
115     def warm_up(self):
116         """
117         Initializes the component.
118         """
119         if self._similarity_model:
120             return
121 
122         token = self._token.resolve_value() if self._token else None
123         config = AutoConfig.from_pretrained(self._model, use_auth_token=token)
124         cross_encoder_used = False
125         if config.architectures:
126             cross_encoder_used = any(arch.endswith("ForSequenceClassification") for arch in config.architectures)
127         device = ComponentDevice.resolve_device(self._device).to_torch_str()
128         # Based on the Model string we can load either Bi-Encoders or Cross Encoders.
129         # Similarity computation changes for both approaches
130         if cross_encoder_used:
131             self._similarity_model = CrossEncoder(
132                 self._model,
133                 device=device,
134                 tokenizer_args={"use_auth_token": token},
135                 automodel_args={"use_auth_token": token},
136             )
137         else:
138             self._similarity_model = SentenceTransformer(self._model, device=device, use_auth_token=token)
139 
140     @component.output_types(score=float, individual_scores=List[float])
141     def run(self, ground_truth_answers: List[str], predicted_answers: List[str]) -> Dict[str, Any]:
142         """
143         SASEvaluator component run method.
144 
145         Run the SASEvaluator to compute the Semantic Answer Similarity (SAS) between a list of predicted answers
146         and a list of ground truth answers. Both must be list of strings of same length.
147 
148         :param ground_truth_answers:
149             A list of expected answers for each question.
150         :param predicted_answers:
151             A list of generated answers for each question.
152         :returns:
153             A dictionary with the following outputs:
154                 - `score`: Mean SAS score over all the predictions/ground-truth pairs.
155                 - `individual_scores`: A list of similarity scores for each prediction/ground-truth pair.
156         """
157         if len(ground_truth_answers) != len(predicted_answers):
158             raise ValueError("The number of predictions and labels must be the same.")
159 
160         if len(predicted_answers) == 0:
161             return {"score": 0.0, "individual_scores": [0.0]}
162 
163         if not self._similarity_model:
164             msg = "The model has not been initialized. Call warm_up() before running the evaluator."
165             raise RuntimeError(msg)
166 
167         if isinstance(self._similarity_model, CrossEncoder):
168             # For Cross Encoders we create a list of pairs of predictions and labels
169             sentence_pairs = list(zip(predicted_answers, ground_truth_answers))
170             similarity_scores = self._similarity_model.predict(
171                 sentence_pairs, batch_size=self._batch_size, convert_to_numpy=True
172             )
173 
174             # All Cross Encoders do not return a set of logits scores that are normalized
175             # We normalize scores if they are larger than 1
176             if (similarity_scores > 1).any():
177                 similarity_scores = expit(similarity_scores)
178 
179             # Convert scores to list of floats from numpy array
180             similarity_scores = similarity_scores.tolist()
181 
182         else:
183             # For Bi-encoders we create embeddings separately for predictions and labels
184             predictions_embeddings = self._similarity_model.encode(
185                 predicted_answers, batch_size=self._batch_size, convert_to_tensor=True
186             )
187             label_embeddings = self._similarity_model.encode(
188                 ground_truth_answers, batch_size=self._batch_size, convert_to_tensor=True
189             )
190 
191             # Compute cosine-similarities
192             similarity_scores = [
193                 float(util.cos_sim(p, l).cpu().numpy()) for p, l in zip(predictions_embeddings, label_embeddings)
194             ]
195 
196         sas_score = np_mean(similarity_scores)
197 
198         return {"score": sas_score, "individual_scores": similarity_scores}
199
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/haystack/components/evaluators/sas_evaluator.py b/haystack/components/evaluators/sas_evaluator.py
--- a/haystack/components/evaluators/sas_evaluator.py
+++ b/haystack/components/evaluators/sas_evaluator.py
@@ -157,6 +157,9 @@
         if len(ground_truth_answers) != len(predicted_answers):
             raise ValueError("The number of predictions and labels must be the same.")
 
+        if any(answer is None for answer in predicted_answers):
+            raise ValueError("Predicted answers must not contain None values.")
+
         if len(predicted_answers) == 0:
             return {"score": 0.0, "individual_scores": [0.0]}

{"golden_diff": "diff --git a/haystack/components/evaluators/sas_evaluator.py b/haystack/components/evaluators/sas_evaluator.py\n--- a/haystack/components/evaluators/sas_evaluator.py\n+++ b/haystack/components/evaluators/sas_evaluator.py\n@@ -157,6 +157,9 @@\n         if len(ground_truth_answers) != len(predicted_answers):\n             raise ValueError(\"The number of predictions and labels must be the same.\")\n \n+        if any(answer is None for answer in predicted_answers):\n+            raise ValueError(\"Predicted answers must not contain None values.\")\n+\n         if len(predicted_answers) == 0:\n             return {\"score\": 0.0, \"individual_scores\": [0.0]}\n", "issue": "SASEvaluator returns score even when `predicted_answers` contains `None`\nIn a RAG pipeline, there are cases where the prompt instructs the LLM to return a placeholder answer if there is missing information in the context, such as \"No answer\". See an [example](https://github.com/deepset-ai/haystack-evaluation/blob/cbfcde5afcaf7574aa82b1ad6b5ce307c8a7c97e/evaluations/architectures/basic_rag.py#L10) in our evaluation repository.\r\n\r\nIn that case, it makes sense to replace the predicted answer in the RAG pipeline with `None`, instead of computing any metric on the placeholder answer. However, the SASEvaluator returns a score even if `None` is passed.\r\n\r\n**Example**:\r\n```\r\nfrom haystack.components.evaluators import SASEvaluator\r\n\r\nsas = SASEvaluator(model=\"sentence-transformers/all-mpnet-base-v2\")\r\nsas.warm_up()\r\nsas.run(predicted_answers=[None, \"Whatever\"], ground_truth_answers=[\"Whatever\", \"Another whatever\"])\r\n```\r\n\r\nthe output is:\r\n\r\n```\r\n{'score': 0.46286168694496155,\r\n 'individual_scores': [0.26153379678726196, 0.6641895771026611]}\r\n```\r\n\r\n**Expected**\r\n\r\n```\r\n{'score': 0.6641895771026611,\r\n 'individual_scores': [None, 0.6641895771026611]}\r\n```\n", "before_files": [{"content": "# SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>\n#\n# SPDX-License-Identifier: Apache-2.0\n\nfrom typing import Any, Dict, List, Optional\n\nfrom numpy import mean as np_mean\n\nfrom haystack import component, default_from_dict, default_to_dict\nfrom haystack.lazy_imports import LazyImport\nfrom haystack.utils import ComponentDevice, expit\nfrom haystack.utils.auth import Secret, deserialize_secrets_inplace\n\nwith LazyImport(message=\"Run 'pip install \\\"sentence-transformers>=2.3.0\\\"'\") as sas_import:\n    from sentence_transformers import CrossEncoder, SentenceTransformer, util\n    from transformers import AutoConfig\n\n\n@component\nclass SASEvaluator:\n    \"\"\"\n    SASEvaluator computes the Semantic Answer Similarity (SAS) between a list of predictions and a one of ground truths.\n\n    It's usually used in Retrieval Augmented Generation (RAG) pipelines to evaluate the quality of the generated\n    answers. The SAS is computed using a pre-trained model from the Hugging Face model hub. The model can be either a\n    Bi-Encoder or a Cross-Encoder. The choice of the model is based on the `model` parameter.\n\n    Usage example:\n    ```python\n    from haystack.components.evaluators.sas_evaluator import SASEvaluator\n\n    evaluator = SASEvaluator(model=\"cross-encoder/ms-marco-MiniLM-L-6-v2\")\n    evaluator.warm_up()\n    ground_truths = [\n        \"A construction budget of US $2.3 billion\",\n        \"The Eiffel Tower, completed in 1889, symbolizes Paris's cultural magnificence.\",\n        \"The Meiji Restoration in 1868 transformed Japan into a modernized world power.\",\n    ]\n    predictions = [\n        \"A construction budget of US $2.3 billion\",\n        \"The Eiffel Tower, completed in 1889, symbolizes Paris's cultural magnificence.\",\n        \"The Meiji Restoration in 1868 transformed Japan into a modernized world power.\",\n    ]\n    result = evaluator.run(\n        ground_truths_answers=ground_truths, predicted_answers=predictions\n    )\n\n    print(result[\"score\"])\n    # 0.9999673763910929\n\n    print(result[\"individual_scores\"])\n    # [0.9999765157699585, 0.999968409538269, 0.9999572038650513]\n    ```\n    \"\"\"\n\n    def __init__(\n        self,\n        model: str = \"sentence-transformers/paraphrase-multilingual-mpnet-base-v2\",\n        batch_size: int = 32,\n        device: Optional[ComponentDevice] = None,\n        token: Secret = Secret.from_env_var(\"HF_API_TOKEN\", strict=False),\n    ):\n        \"\"\"\n        Creates a new instance of SASEvaluator.\n\n        :param model:\n            SentenceTransformers semantic textual similarity model, should be path or string pointing to a downloadable\n            model.\n        :param batch_size:\n            Number of prediction-label pairs to encode at once.\n        :param device:\n            The device on which the model is loaded. If `None`, the default device is automatically selected.\n        :param token:\n            The Hugging Face token for HTTP bearer authorization.\n            You can find your HF token in your [account settings](https://huggingface.co/settings/tokens)\n        \"\"\"\n        sas_import.check()\n\n        self._model = model\n        self._batch_size = batch_size\n        self._device = device\n        self._token = token\n        self._similarity_model = None\n\n    def to_dict(self) -> Dict[str, Any]:\n        \"\"\"\n        Serialize this component to a dictionary.\n\n        :returns:\n            The serialized component as a dictionary.\n        \"\"\"\n        return default_to_dict(\n            self,\n            model=self._model,\n            batch_size=self._batch_size,\n            device=self._device.to_dict() if self._device else None,\n            token=self._token.to_dict() if self._token else None,\n        )\n\n    @classmethod\n    def from_dict(cls, data: Dict[str, Any]) -> \"SASEvaluator\":\n        \"\"\"\n        Deserialize this component from a dictionary.\n\n        :param data:\n            The dictionary representation of this component.\n        :returns:\n            The deserialized component instance.\n        \"\"\"\n        deserialize_secrets_inplace(data[\"init_parameters\"], keys=[\"token\"])\n        if device := data.get(\"init_parameters\", {}).get(\"device\"):\n            data[\"init_parameters\"][\"device\"] = ComponentDevice.from_dict(device)\n        return default_from_dict(cls, data)\n\n    def warm_up(self):\n        \"\"\"\n        Initializes the component.\n        \"\"\"\n        if self._similarity_model:\n            return\n\n        token = self._token.resolve_value() if self._token else None\n        config = AutoConfig.from_pretrained(self._model, use_auth_token=token)\n        cross_encoder_used = False\n        if config.architectures:\n            cross_encoder_used = any(arch.endswith(\"ForSequenceClassification\") for arch in config.architectures)\n        device = ComponentDevice.resolve_device(self._device).to_torch_str()\n        # Based on the Model string we can load either Bi-Encoders or Cross Encoders.\n        # Similarity computation changes for both approaches\n        if cross_encoder_used:\n            self._similarity_model = CrossEncoder(\n                self._model,\n                device=device,\n                tokenizer_args={\"use_auth_token\": token},\n                automodel_args={\"use_auth_token\": token},\n            )\n        else:\n            self._similarity_model = SentenceTransformer(self._model, device=device, use_auth_token=token)\n\n    @component.output_types(score=float, individual_scores=List[float])\n    def run(self, ground_truth_answers: List[str], predicted_answers: List[str]) -> Dict[str, Any]:\n        \"\"\"\n        SASEvaluator component run method.\n\n        Run the SASEvaluator to compute the Semantic Answer Similarity (SAS) between a list of predicted answers\n        and a list of ground truth answers. Both must be list of strings of same length.\n\n        :param ground_truth_answers:\n            A list of expected answers for each question.\n        :param predicted_answers:\n            A list of generated answers for each question.\n        :returns:\n            A dictionary with the following outputs:\n                - `score`: Mean SAS score over all the predictions/ground-truth pairs.\n                - `individual_scores`: A list of similarity scores for each prediction/ground-truth pair.\n        \"\"\"\n        if len(ground_truth_answers) != len(predicted_answers):\n            raise ValueError(\"The number of predictions and labels must be the same.\")\n\n        if len(predicted_answers) == 0:\n            return {\"score\": 0.0, \"individual_scores\": [0.0]}\n\n        if not self._similarity_model:\n            msg = \"The model has not been initialized. Call warm_up() before running the evaluator.\"\n            raise RuntimeError(msg)\n\n        if isinstance(self._similarity_model, CrossEncoder):\n            # For Cross Encoders we create a list of pairs of predictions and labels\n            sentence_pairs = list(zip(predicted_answers, ground_truth_answers))\n            similarity_scores = self._similarity_model.predict(\n                sentence_pairs, batch_size=self._batch_size, convert_to_numpy=True\n            )\n\n            # All Cross Encoders do not return a set of logits scores that are normalized\n            # We normalize scores if they are larger than 1\n            if (similarity_scores > 1).any():\n                similarity_scores = expit(similarity_scores)\n\n            # Convert scores to list of floats from numpy array\n            similarity_scores = similarity_scores.tolist()\n\n        else:\n            # For Bi-encoders we create embeddings separately for predictions and labels\n            predictions_embeddings = self._similarity_model.encode(\n                predicted_answers, batch_size=self._batch_size, convert_to_tensor=True\n            )\n            label_embeddings = self._similarity_model.encode(\n                ground_truth_answers, batch_size=self._batch_size, convert_to_tensor=True\n            )\n\n            # Compute cosine-similarities\n            similarity_scores = [\n                float(util.cos_sim(p, l).cpu().numpy()) for p, l in zip(predictions_embeddings, label_embeddings)\n            ]\n\n        sas_score = np_mean(similarity_scores)\n\n        return {\"score\": sas_score, \"individual_scores\": similarity_scores}\n", "path": "haystack/components/evaluators/sas_evaluator.py"}], "after_files": [{"content": "# SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>\n#\n# SPDX-License-Identifier: Apache-2.0\n\nfrom typing import Any, Dict, List, Optional\n\nfrom numpy import mean as np_mean\n\nfrom haystack import component, default_from_dict, default_to_dict\nfrom haystack.lazy_imports import LazyImport\nfrom haystack.utils import ComponentDevice, expit\nfrom haystack.utils.auth import Secret, deserialize_secrets_inplace\n\nwith LazyImport(message=\"Run 'pip install \\\"sentence-transformers>=2.3.0\\\"'\") as sas_import:\n    from sentence_transformers import CrossEncoder, SentenceTransformer, util\n    from transformers import AutoConfig\n\n\n@component\nclass SASEvaluator:\n    \"\"\"\n    SASEvaluator computes the Semantic Answer Similarity (SAS) between a list of predictions and a one of ground truths.\n\n    It's usually used in Retrieval Augmented Generation (RAG) pipelines to evaluate the quality of the generated\n    answers. The SAS is computed using a pre-trained model from the Hugging Face model hub. The model can be either a\n    Bi-Encoder or a Cross-Encoder. The choice of the model is based on the `model` parameter.\n\n    Usage example:\n    ```python\n    from haystack.components.evaluators.sas_evaluator import SASEvaluator\n\n    evaluator = SASEvaluator(model=\"cross-encoder/ms-marco-MiniLM-L-6-v2\")\n    evaluator.warm_up()\n    ground_truths = [\n        \"A construction budget of US $2.3 billion\",\n        \"The Eiffel Tower, completed in 1889, symbolizes Paris's cultural magnificence.\",\n        \"The Meiji Restoration in 1868 transformed Japan into a modernized world power.\",\n    ]\n    predictions = [\n        \"A construction budget of US $2.3 billion\",\n        \"The Eiffel Tower, completed in 1889, symbolizes Paris's cultural magnificence.\",\n        \"The Meiji Restoration in 1868 transformed Japan into a modernized world power.\",\n    ]\n    result = evaluator.run(\n        ground_truths_answers=ground_truths, predicted_answers=predictions\n    )\n\n    print(result[\"score\"])\n    # 0.9999673763910929\n\n    print(result[\"individual_scores\"])\n    # [0.9999765157699585, 0.999968409538269, 0.9999572038650513]\n    ```\n    \"\"\"\n\n    def __init__(\n        self,\n        model: str = \"sentence-transformers/paraphrase-multilingual-mpnet-base-v2\",\n        batch_size: int = 32,\n        device: Optional[ComponentDevice] = None,\n        token: Secret = Secret.from_env_var(\"HF_API_TOKEN\", strict=False),\n    ):\n        \"\"\"\n        Creates a new instance of SASEvaluator.\n\n        :param model:\n            SentenceTransformers semantic textual similarity model, should be path or string pointing to a downloadable\n            model.\n        :param batch_size:\n            Number of prediction-label pairs to encode at once.\n        :param device:\n            The device on which the model is loaded. If `None`, the default device is automatically selected.\n        :param token:\n            The Hugging Face token for HTTP bearer authorization.\n            You can find your HF token in your [account settings](https://huggingface.co/settings/tokens)\n        \"\"\"\n        sas_import.check()\n\n        self._model = model\n        self._batch_size = batch_size\n        self._device = device\n        self._token = token\n        self._similarity_model = None\n\n    def to_dict(self) -> Dict[str, Any]:\n        \"\"\"\n        Serialize this component to a dictionary.\n\n        :returns:\n            The serialized component as a dictionary.\n        \"\"\"\n        return default_to_dict(\n            self,\n            model=self._model,\n            batch_size=self._batch_size,\n            device=self._device.to_dict() if self._device else None,\n            token=self._token.to_dict() if self._token else None,\n        )\n\n    @classmethod\n    def from_dict(cls, data: Dict[str, Any]) -> \"SASEvaluator\":\n        \"\"\"\n        Deserialize this component from a dictionary.\n\n        :param data:\n            The dictionary representation of this component.\n        :returns:\n            The deserialized component instance.\n        \"\"\"\n        deserialize_secrets_inplace(data[\"init_parameters\"], keys=[\"token\"])\n        if device := data.get(\"init_parameters\", {}).get(\"device\"):\n            data[\"init_parameters\"][\"device\"] = ComponentDevice.from_dict(device)\n        return default_from_dict(cls, data)\n\n    def warm_up(self):\n        \"\"\"\n        Initializes the component.\n        \"\"\"\n        if self._similarity_model:\n            return\n\n        token = self._token.resolve_value() if self._token else None\n        config = AutoConfig.from_pretrained(self._model, use_auth_token=token)\n        cross_encoder_used = False\n        if config.architectures:\n            cross_encoder_used = any(arch.endswith(\"ForSequenceClassification\") for arch in config.architectures)\n        device = ComponentDevice.resolve_device(self._device).to_torch_str()\n        # Based on the Model string we can load either Bi-Encoders or Cross Encoders.\n        # Similarity computation changes for both approaches\n        if cross_encoder_used:\n            self._similarity_model = CrossEncoder(\n                self._model,\n                device=device,\n                tokenizer_args={\"use_auth_token\": token},\n                automodel_args={\"use_auth_token\": token},\n            )\n        else:\n            self._similarity_model = SentenceTransformer(self._model, device=device, use_auth_token=token)\n\n    @component.output_types(score=float, individual_scores=List[float])\n    def run(self, ground_truth_answers: List[str], predicted_answers: List[str]) -> Dict[str, Any]:\n        \"\"\"\n        SASEvaluator component run method.\n\n        Run the SASEvaluator to compute the Semantic Answer Similarity (SAS) between a list of predicted answers\n        and a list of ground truth answers. Both must be list of strings of same length.\n\n        :param ground_truth_answers:\n            A list of expected answers for each question.\n        :param predicted_answers:\n            A list of generated answers for each question.\n        :returns:\n            A dictionary with the following outputs:\n                - `score`: Mean SAS score over all the predictions/ground-truth pairs.\n                - `individual_scores`: A list of similarity scores for each prediction/ground-truth pair.\n        \"\"\"\n        if len(ground_truth_answers) != len(predicted_answers):\n            raise ValueError(\"The number of predictions and labels must be the same.\")\n\n        if any(answer is None for answer in predicted_answers):\n            raise ValueError(\"Predicted answers must not contain None values.\")\n\n        if len(predicted_answers) == 0:\n            return {\"score\": 0.0, \"individual_scores\": [0.0]}\n\n        if not self._similarity_model:\n            msg = \"The model has not been initialized. Call warm_up() before running the evaluator.\"\n            raise RuntimeError(msg)\n\n        if isinstance(self._similarity_model, CrossEncoder):\n            # For Cross Encoders we create a list of pairs of predictions and labels\n            sentence_pairs = list(zip(predicted_answers, ground_truth_answers))\n            similarity_scores = self._similarity_model.predict(\n                sentence_pairs, batch_size=self._batch_size, convert_to_numpy=True\n            )\n\n            # All Cross Encoders do not return a set of logits scores that are normalized\n            # We normalize scores if they are larger than 1\n            if (similarity_scores > 1).any():\n                similarity_scores = expit(similarity_scores)\n\n            # Convert scores to list of floats from numpy array\n            similarity_scores = similarity_scores.tolist()\n\n        else:\n            # For Bi-encoders we create embeddings separately for predictions and labels\n            predictions_embeddings = self._similarity_model.encode(\n                predicted_answers, batch_size=self._batch_size, convert_to_tensor=True\n            )\n            label_embeddings = self._similarity_model.encode(\n                ground_truth_answers, batch_size=self._batch_size, convert_to_tensor=True\n            )\n\n            # Compute cosine-similarities\n            similarity_scores = [\n                float(util.cos_sim(p, l).cpu().numpy()) for p, l in zip(predictions_embeddings, label_embeddings)\n            ]\n\n        sas_score = np_mean(similarity_scores)\n\n        return {\"score\": sas_score, \"individual_scores\": similarity_scores}\n", "path": "haystack/components/evaluators/sas_evaluator.py"}]}