We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
`ContextRelevanceEvaluator`: statements extraction can be made more robust
Slightly different inputs produce very different outputs in the `ContextRelevanceEvaluator`.

**Example in the docs**:
```python
from haystack.components.evaluators import ContextRelevanceEvaluator

questions = ["Who created the Python language?"]
contexts = [
    [
        "Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects."
    ],
]

evaluator = ContextRelevanceEvaluator()
result = evaluator.run(questions=questions, contexts=contexts)
print(result["score"])
# 1.0
print(result["individual_scores"])
# [1.0]
print(result["results"])
# [{'statements': ['Python, created by Guido van Rossum in the late 1980s.'], 'statement_scores': [1], 'score': 1.0}]
```

**A different example**:
```python
from haystack.components.evaluators import ContextRelevanceEvaluator
questions = ["Who created the Python language?"]
contexts = [
    [
        "Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.",
        "Java is a high-level, class-based, object-oriented programming language. It allows you to write once, run anywhere, meaning that compiled Java code can run on all platforms that support Java without the need for recompilation.",
        "Scala is a high-level, statically typed programming language.",
    ],
]
evaluator = ContextRelevanceEvaluator()
result = evaluator.run(questions=questions, contexts=contexts)

print(result)
# {'results': [{'statements': ['Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language.', 'Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.'], 'statement_scores': [1, 0], 'score': 0.5}], 'score': 0.5, 'individual_scores': [0.5]}
```

In this case, the first context is broken into 2 statements, while the 2nd and 3rd statements are not considered.

**Expected behavior**
The extraction of statements should be more robust/consistent.

**Additional context**
From @julian-risch:
> I checked that the prompt sent to the LLM correctly contains all three contexts. The problem is that the LLM extract statements only for the first context at least in this example. It seems that the prompt instructions aren't good enough. You could try extending the _DEFAULT_EXAMPLES with an example that has multiple contexts for one question.


**System:**
 - Haystack version (commit or version number): 2.2.0

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `haystack/components/evaluators/context_relevance.py`
Content:
```
1 # SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>
2 #
3 # SPDX-License-Identifier: Apache-2.0
4 
5 from typing import Any, Dict, List, Optional
6 
7 from numpy import mean as np_mean
8 
9 from haystack import component, default_from_dict, default_to_dict
10 from haystack.components.evaluators.llm_evaluator import LLMEvaluator
11 from haystack.utils import Secret, deserialize_secrets_inplace
12 
13 # Private global variable for default examples to include in the prompt if the user does not provide any examples
14 _DEFAULT_EXAMPLES = [
15     {
16         "inputs": {
17             "questions": "What is the capital of Germany?",
18             "contexts": ["Berlin is the capital of Germany and was founded in 1244."],
19         },
20         "outputs": {
21             "statements": ["Berlin is the capital of Germany.", "Berlin was founded in 1244."],
22             "statement_scores": [1, 0],
23         },
24     },
25     {
26         "inputs": {"questions": "What is the capital of France?", "contexts": ["Berlin is the capital of Germany."]},
27         "outputs": {"statements": ["Berlin is the capital of Germany."], "statement_scores": [0]},
28     },
29     {
30         "inputs": {"questions": "What is the capital of Italy?", "contexts": ["Rome is the capital of Italy."]},
31         "outputs": {"statements": ["Rome is the capital of Italy."], "statement_scores": [1]},
32     },
33 ]
34 
35 
36 @component
37 class ContextRelevanceEvaluator(LLMEvaluator):
38     """
39     Evaluator that checks if a provided context is relevant to the question.
40 
41     An LLM breaks up the context into multiple statements and checks whether each statement
42     is relevant for answering a question.
43     The final score for the context relevance is a number from 0.0 to 1.0. It represents the proportion of
44     statements that can be inferred from the provided contexts.
45 
46     Usage example:
47     ```python
48     from haystack.components.evaluators import ContextRelevanceEvaluator
49 
50     questions = ["Who created the Python language?"]
51     contexts = [
52         [(
53             "Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming "
54             "language. Its design philosophy emphasizes code readability, and its language constructs aim to help "
55             "programmers write clear, logical code for both small and large-scale software projects."
56         )],
57     ]
58 
59     evaluator = ContextRelevanceEvaluator()
60     result = evaluator.run(questions=questions, contexts=contexts)
61     print(result["score"])
62     # 1.0
63     print(result["individual_scores"])
64     # [1.0]
65     print(result["results"])
66     # [{
67     #   'statements': ['Python, created by Guido van Rossum in the late 1980s.'],
68     #   'statement_scores': [1],
69     #   'score': 1.0
70     # }]
71     ```
72     """
73 
74     def __init__(
75         self,
76         examples: Optional[List[Dict[str, Any]]] = None,
77         progress_bar: bool = True,
78         api: str = "openai",
79         api_key: Secret = Secret.from_env_var("OPENAI_API_KEY"),
80         raise_on_failure: bool = True,
81     ):
82         """
83         Creates an instance of ContextRelevanceEvaluator.
84 
85         :param examples:
86             Optional few-shot examples conforming to the expected input and output format of ContextRelevanceEvaluator.
87             Default examples will be used if none are provided.
88             Each example must be a dictionary with keys "inputs" and "outputs".
89             "inputs" must be a dictionary with keys "questions" and "contexts".
90             "outputs" must be a dictionary with "statements" and "statement_scores".
91             Expected format:
92             [{
93                 "inputs": {
94                     "questions": "What is the capital of Italy?", "contexts": ["Rome is the capital of Italy."],
95                 },
96                 "outputs": {
97                     "statements": ["Rome is the capital of Italy."],
98                     "statement_scores": [1],
99                 },
100             }]
101         :param progress_bar:
102             Whether to show a progress bar during the evaluation.
103         :param api:
104             The API to use for calling an LLM through a Generator.
105             Supported APIs: "openai".
106         :param api_key:
107             The API key.
108         :param raise_on_failure:
109             Whether to raise an exception if the API call fails.
110 
111         """
112         self.instructions = (
113             "Your task is to judge how relevant the provided context is for answering a question. "
114             "First, please extract statements from the provided context. "
115             "Second, calculate a relevance score for each statement in the context. "
116             "The score is 1 if the statement is relevant to answer the question or 0 if it is not relevant."
117         )
118         self.inputs = [("questions", List[str]), ("contexts", List[List[str]])]
119         self.outputs = ["statements", "statement_scores"]
120         self.examples = examples or _DEFAULT_EXAMPLES
121         self.api = api
122         self.api_key = api_key
123 
124         super(ContextRelevanceEvaluator, self).__init__(
125             instructions=self.instructions,
126             inputs=self.inputs,
127             outputs=self.outputs,
128             examples=self.examples,
129             api=self.api,
130             api_key=self.api_key,
131             raise_on_failure=raise_on_failure,
132             progress_bar=progress_bar,
133         )
134 
135     @component.output_types(individual_scores=List[int], score=float, results=List[Dict[str, Any]])
136     def run(self, questions: List[str], contexts: List[List[str]]) -> Dict[str, Any]:
137         """
138         Run the LLM evaluator.
139 
140         :param questions:
141             A list of questions.
142         :param contexts:
143             A list of lists of contexts. Each list of contexts corresponds to one question.
144         :returns:
145             A dictionary with the following outputs:
146                 - `score`: Mean context relevance score over all the provided input questions.
147                 - `individual_scores`: A list of context relevance scores for each input question.
148                 - `results`: A list of dictionaries with `statements` and `statement_scores` for each input context.
149         """
150         result = super(ContextRelevanceEvaluator, self).run(questions=questions, contexts=contexts)
151 
152         # calculate average statement relevance score per query
153         for idx, res in enumerate(result["results"]):
154             if res is None:
155                 result["results"][idx] = {"statements": [], "statement_scores": [], "score": float("nan")}
156                 continue
157             if not res["statements"]:
158                 res["score"] = 0
159             else:
160                 res["score"] = np_mean(res["statement_scores"])
161 
162         # calculate average context relevance score over all queries
163         result["score"] = np_mean([res["score"] for res in result["results"]])
164         result["individual_scores"] = [res["score"] for res in result["results"]]
165 
166         return result
167 
168     def to_dict(self) -> Dict[str, Any]:
169         """
170         Serialize this component to a dictionary.
171 
172         :returns:
173             A dictionary with serialized data.
174         """
175         return default_to_dict(
176             self,
177             api=self.api,
178             api_key=self.api_key.to_dict() if self.api_key else None,
179             examples=self.examples,
180             progress_bar=self.progress_bar,
181             raise_on_failure=self.raise_on_failure,
182         )
183 
184     @classmethod
185     def from_dict(cls, data: Dict[str, Any]) -> "ContextRelevanceEvaluator":
186         """
187         Deserialize this component from a dictionary.
188 
189         :param data:
190             The dictionary representation of this component.
191         :returns:
192             The deserialized component instance.
193         """
194         deserialize_secrets_inplace(data["init_parameters"], keys=["api_key"])
195         return default_from_dict(cls, data)
196
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/haystack/components/evaluators/context_relevance.py b/haystack/components/evaluators/context_relevance.py
--- a/haystack/components/evaluators/context_relevance.py
+++ b/haystack/components/evaluators/context_relevance.py
@@ -23,8 +23,23 @@
         },
     },
     {
-        "inputs": {"questions": "What is the capital of France?", "contexts": ["Berlin is the capital of Germany."]},
-        "outputs": {"statements": ["Berlin is the capital of Germany."], "statement_scores": [0]},
+        "inputs": {
+            "questions": "What is the capital of France?",
+            "contexts": [
+                "Berlin is the capital of Germany and was founded in 1244.",
+                "Europe is a continent with 44 countries.",
+                "Madrid is the capital of Spain.",
+            ],
+        },
+        "outputs": {
+            "statements": [
+                "Berlin is the capital of Germany.",
+                "Berlin was founded in 1244.",
+                "Europe is a continent with 44 countries.",
+                "Madrid is the capital of Spain.",
+            ],
+            "statement_scores": [0, 0, 0, 0],
+        },
     },
     {
         "inputs": {"questions": "What is the capital of Italy?", "contexts": ["Rome is the capital of Italy."]},
@@ -113,7 +128,8 @@
             "Your task is to judge how relevant the provided context is for answering a question. "
             "First, please extract statements from the provided context. "
             "Second, calculate a relevance score for each statement in the context. "
-            "The score is 1 if the statement is relevant to answer the question or 0 if it is not relevant."
+            "The score is 1 if the statement is relevant to answer the question or 0 if it is not relevant. "
+            "Each statement should be scored individually."
         )
         self.inputs = [("questions", List[str]), ("contexts", List[List[str]])]
         self.outputs = ["statements", "statement_scores"]

{"golden_diff": "diff --git a/haystack/components/evaluators/context_relevance.py b/haystack/components/evaluators/context_relevance.py\n--- a/haystack/components/evaluators/context_relevance.py\n+++ b/haystack/components/evaluators/context_relevance.py\n@@ -23,8 +23,23 @@\n         },\n     },\n     {\n-        \"inputs\": {\"questions\": \"What is the capital of France?\", \"contexts\": [\"Berlin is the capital of Germany.\"]},\n-        \"outputs\": {\"statements\": [\"Berlin is the capital of Germany.\"], \"statement_scores\": [0]},\n+        \"inputs\": {\n+            \"questions\": \"What is the capital of France?\",\n+            \"contexts\": [\n+                \"Berlin is the capital of Germany and was founded in 1244.\",\n+                \"Europe is a continent with 44 countries.\",\n+                \"Madrid is the capital of Spain.\",\n+            ],\n+        },\n+        \"outputs\": {\n+            \"statements\": [\n+                \"Berlin is the capital of Germany.\",\n+                \"Berlin was founded in 1244.\",\n+                \"Europe is a continent with 44 countries.\",\n+                \"Madrid is the capital of Spain.\",\n+            ],\n+            \"statement_scores\": [0, 0, 0, 0],\n+        },\n     },\n     {\n         \"inputs\": {\"questions\": \"What is the capital of Italy?\", \"contexts\": [\"Rome is the capital of Italy.\"]},\n@@ -113,7 +128,8 @@\n             \"Your task is to judge how relevant the provided context is for answering a question. \"\n             \"First, please extract statements from the provided context. \"\n             \"Second, calculate a relevance score for each statement in the context. \"\n-            \"The score is 1 if the statement is relevant to answer the question or 0 if it is not relevant.\"\n+            \"The score is 1 if the statement is relevant to answer the question or 0 if it is not relevant. \"\n+            \"Each statement should be scored individually.\"\n         )\n         self.inputs = [(\"questions\", List[str]), (\"contexts\", List[List[str]])]\n         self.outputs = [\"statements\", \"statement_scores\"]\n", "issue": "`ContextRelevanceEvaluator`: statements extraction can be made more robust\nSlightly different inputs produce very different outputs in the `ContextRelevanceEvaluator`.\r\n\r\n**Example in the docs**:\r\n```python\r\nfrom haystack.components.evaluators import ContextRelevanceEvaluator\r\n\r\nquestions = [\"Who created the Python language?\"]\r\ncontexts = [\r\n    [\r\n        \"Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.\"\r\n    ],\r\n]\r\n\r\nevaluator = ContextRelevanceEvaluator()\r\nresult = evaluator.run(questions=questions, contexts=contexts)\r\nprint(result[\"score\"])\r\n# 1.0\r\nprint(result[\"individual_scores\"])\r\n# [1.0]\r\nprint(result[\"results\"])\r\n# [{'statements': ['Python, created by Guido van Rossum in the late 1980s.'], 'statement_scores': [1], 'score': 1.0}]\r\n```\r\n\r\n**A different example**:\r\n```python\r\nfrom haystack.components.evaluators import ContextRelevanceEvaluator\r\nquestions = [\"Who created the Python language?\"]\r\ncontexts = [\r\n    [\r\n        \"Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language. Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.\",\r\n        \"Java is a high-level, class-based, object-oriented programming language. It allows you to write once, run anywhere, meaning that compiled Java code can run on all platforms that support Java without the need for recompilation.\",\r\n        \"Scala is a high-level, statically typed programming language.\",\r\n    ],\r\n]\r\nevaluator = ContextRelevanceEvaluator()\r\nresult = evaluator.run(questions=questions, contexts=contexts)\r\n\r\nprint(result)\r\n# {'results': [{'statements': ['Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming language.', 'Its design philosophy emphasizes code readability, and its language constructs aim to help programmers write clear, logical code for both small and large-scale software projects.'], 'statement_scores': [1, 0], 'score': 0.5}], 'score': 0.5, 'individual_scores': [0.5]}\r\n```\r\n\r\nIn this case, the first context is broken into 2 statements, while the 2nd and 3rd statements are not considered.\r\n\r\n**Expected behavior**\r\nThe extraction of statements should be more robust/consistent.\r\n\r\n**Additional context**\r\nFrom @julian-risch:\r\n> I checked that the prompt sent to the LLM correctly contains all three contexts. The problem is that the LLM extract statements only for the first context at least in this example. It seems that the prompt instructions aren't good enough. You could try extending the _DEFAULT_EXAMPLES with an example that has multiple contexts for one question.\r\n\r\n\r\n**System:**\r\n - Haystack version (commit or version number): 2.2.0\n", "before_files": [{"content": "# SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>\n#\n# SPDX-License-Identifier: Apache-2.0\n\nfrom typing import Any, Dict, List, Optional\n\nfrom numpy import mean as np_mean\n\nfrom haystack import component, default_from_dict, default_to_dict\nfrom haystack.components.evaluators.llm_evaluator import LLMEvaluator\nfrom haystack.utils import Secret, deserialize_secrets_inplace\n\n# Private global variable for default examples to include in the prompt if the user does not provide any examples\n_DEFAULT_EXAMPLES = [\n    {\n        \"inputs\": {\n            \"questions\": \"What is the capital of Germany?\",\n            \"contexts\": [\"Berlin is the capital of Germany and was founded in 1244.\"],\n        },\n        \"outputs\": {\n            \"statements\": [\"Berlin is the capital of Germany.\", \"Berlin was founded in 1244.\"],\n            \"statement_scores\": [1, 0],\n        },\n    },\n    {\n        \"inputs\": {\"questions\": \"What is the capital of France?\", \"contexts\": [\"Berlin is the capital of Germany.\"]},\n        \"outputs\": {\"statements\": [\"Berlin is the capital of Germany.\"], \"statement_scores\": [0]},\n    },\n    {\n        \"inputs\": {\"questions\": \"What is the capital of Italy?\", \"contexts\": [\"Rome is the capital of Italy.\"]},\n        \"outputs\": {\"statements\": [\"Rome is the capital of Italy.\"], \"statement_scores\": [1]},\n    },\n]\n\n\n@component\nclass ContextRelevanceEvaluator(LLMEvaluator):\n    \"\"\"\n    Evaluator that checks if a provided context is relevant to the question.\n\n    An LLM breaks up the context into multiple statements and checks whether each statement\n    is relevant for answering a question.\n    The final score for the context relevance is a number from 0.0 to 1.0. It represents the proportion of\n    statements that can be inferred from the provided contexts.\n\n    Usage example:\n    ```python\n    from haystack.components.evaluators import ContextRelevanceEvaluator\n\n    questions = [\"Who created the Python language?\"]\n    contexts = [\n        [(\n            \"Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming \"\n            \"language. Its design philosophy emphasizes code readability, and its language constructs aim to help \"\n            \"programmers write clear, logical code for both small and large-scale software projects.\"\n        )],\n    ]\n\n    evaluator = ContextRelevanceEvaluator()\n    result = evaluator.run(questions=questions, contexts=contexts)\n    print(result[\"score\"])\n    # 1.0\n    print(result[\"individual_scores\"])\n    # [1.0]\n    print(result[\"results\"])\n    # [{\n    #   'statements': ['Python, created by Guido van Rossum in the late 1980s.'],\n    #   'statement_scores': [1],\n    #   'score': 1.0\n    # }]\n    ```\n    \"\"\"\n\n    def __init__(\n        self,\n        examples: Optional[List[Dict[str, Any]]] = None,\n        progress_bar: bool = True,\n        api: str = \"openai\",\n        api_key: Secret = Secret.from_env_var(\"OPENAI_API_KEY\"),\n        raise_on_failure: bool = True,\n    ):\n        \"\"\"\n        Creates an instance of ContextRelevanceEvaluator.\n\n        :param examples:\n            Optional few-shot examples conforming to the expected input and output format of ContextRelevanceEvaluator.\n            Default examples will be used if none are provided.\n            Each example must be a dictionary with keys \"inputs\" and \"outputs\".\n            \"inputs\" must be a dictionary with keys \"questions\" and \"contexts\".\n            \"outputs\" must be a dictionary with \"statements\" and \"statement_scores\".\n            Expected format:\n            [{\n                \"inputs\": {\n                    \"questions\": \"What is the capital of Italy?\", \"contexts\": [\"Rome is the capital of Italy.\"],\n                },\n                \"outputs\": {\n                    \"statements\": [\"Rome is the capital of Italy.\"],\n                    \"statement_scores\": [1],\n                },\n            }]\n        :param progress_bar:\n            Whether to show a progress bar during the evaluation.\n        :param api:\n            The API to use for calling an LLM through a Generator.\n            Supported APIs: \"openai\".\n        :param api_key:\n            The API key.\n        :param raise_on_failure:\n            Whether to raise an exception if the API call fails.\n\n        \"\"\"\n        self.instructions = (\n            \"Your task is to judge how relevant the provided context is for answering a question. \"\n            \"First, please extract statements from the provided context. \"\n            \"Second, calculate a relevance score for each statement in the context. \"\n            \"The score is 1 if the statement is relevant to answer the question or 0 if it is not relevant.\"\n        )\n        self.inputs = [(\"questions\", List[str]), (\"contexts\", List[List[str]])]\n        self.outputs = [\"statements\", \"statement_scores\"]\n        self.examples = examples or _DEFAULT_EXAMPLES\n        self.api = api\n        self.api_key = api_key\n\n        super(ContextRelevanceEvaluator, self).__init__(\n            instructions=self.instructions,\n            inputs=self.inputs,\n            outputs=self.outputs,\n            examples=self.examples,\n            api=self.api,\n            api_key=self.api_key,\n            raise_on_failure=raise_on_failure,\n            progress_bar=progress_bar,\n        )\n\n    @component.output_types(individual_scores=List[int], score=float, results=List[Dict[str, Any]])\n    def run(self, questions: List[str], contexts: List[List[str]]) -> Dict[str, Any]:\n        \"\"\"\n        Run the LLM evaluator.\n\n        :param questions:\n            A list of questions.\n        :param contexts:\n            A list of lists of contexts. Each list of contexts corresponds to one question.\n        :returns:\n            A dictionary with the following outputs:\n                - `score`: Mean context relevance score over all the provided input questions.\n                - `individual_scores`: A list of context relevance scores for each input question.\n                - `results`: A list of dictionaries with `statements` and `statement_scores` for each input context.\n        \"\"\"\n        result = super(ContextRelevanceEvaluator, self).run(questions=questions, contexts=contexts)\n\n        # calculate average statement relevance score per query\n        for idx, res in enumerate(result[\"results\"]):\n            if res is None:\n                result[\"results\"][idx] = {\"statements\": [], \"statement_scores\": [], \"score\": float(\"nan\")}\n                continue\n            if not res[\"statements\"]:\n                res[\"score\"] = 0\n            else:\n                res[\"score\"] = np_mean(res[\"statement_scores\"])\n\n        # calculate average context relevance score over all queries\n        result[\"score\"] = np_mean([res[\"score\"] for res in result[\"results\"]])\n        result[\"individual_scores\"] = [res[\"score\"] for res in result[\"results\"]]\n\n        return result\n\n    def to_dict(self) -> Dict[str, Any]:\n        \"\"\"\n        Serialize this component to a dictionary.\n\n        :returns:\n            A dictionary with serialized data.\n        \"\"\"\n        return default_to_dict(\n            self,\n            api=self.api,\n            api_key=self.api_key.to_dict() if self.api_key else None,\n            examples=self.examples,\n            progress_bar=self.progress_bar,\n            raise_on_failure=self.raise_on_failure,\n        )\n\n    @classmethod\n    def from_dict(cls, data: Dict[str, Any]) -> \"ContextRelevanceEvaluator\":\n        \"\"\"\n        Deserialize this component from a dictionary.\n\n        :param data:\n            The dictionary representation of this component.\n        :returns:\n            The deserialized component instance.\n        \"\"\"\n        deserialize_secrets_inplace(data[\"init_parameters\"], keys=[\"api_key\"])\n        return default_from_dict(cls, data)\n", "path": "haystack/components/evaluators/context_relevance.py"}], "after_files": [{"content": "# SPDX-FileCopyrightText: 2022-present deepset GmbH <info@deepset.ai>\n#\n# SPDX-License-Identifier: Apache-2.0\n\nfrom typing import Any, Dict, List, Optional\n\nfrom numpy import mean as np_mean\n\nfrom haystack import component, default_from_dict, default_to_dict\nfrom haystack.components.evaluators.llm_evaluator import LLMEvaluator\nfrom haystack.utils import Secret, deserialize_secrets_inplace\n\n# Private global variable for default examples to include in the prompt if the user does not provide any examples\n_DEFAULT_EXAMPLES = [\n    {\n        \"inputs\": {\n            \"questions\": \"What is the capital of Germany?\",\n            \"contexts\": [\"Berlin is the capital of Germany and was founded in 1244.\"],\n        },\n        \"outputs\": {\n            \"statements\": [\"Berlin is the capital of Germany.\", \"Berlin was founded in 1244.\"],\n            \"statement_scores\": [1, 0],\n        },\n    },\n    {\n        \"inputs\": {\n            \"questions\": \"What is the capital of France?\",\n            \"contexts\": [\n                \"Berlin is the capital of Germany and was founded in 1244.\",\n                \"Europe is a continent with 44 countries.\",\n                \"Madrid is the capital of Spain.\",\n            ],\n        },\n        \"outputs\": {\n            \"statements\": [\n                \"Berlin is the capital of Germany.\",\n                \"Berlin was founded in 1244.\",\n                \"Europe is a continent with 44 countries.\",\n                \"Madrid is the capital of Spain.\",\n            ],\n            \"statement_scores\": [0, 0, 0, 0],\n        },\n    },\n    {\n        \"inputs\": {\"questions\": \"What is the capital of Italy?\", \"contexts\": [\"Rome is the capital of Italy.\"]},\n        \"outputs\": {\"statements\": [\"Rome is the capital of Italy.\"], \"statement_scores\": [1]},\n    },\n]\n\n\n@component\nclass ContextRelevanceEvaluator(LLMEvaluator):\n    \"\"\"\n    Evaluator that checks if a provided context is relevant to the question.\n\n    An LLM breaks up the context into multiple statements and checks whether each statement\n    is relevant for answering a question.\n    The final score for the context relevance is a number from 0.0 to 1.0. It represents the proportion of\n    statements that can be inferred from the provided contexts.\n\n    Usage example:\n    ```python\n    from haystack.components.evaluators import ContextRelevanceEvaluator\n\n    questions = [\"Who created the Python language?\"]\n    contexts = [\n        [(\n            \"Python, created by Guido van Rossum in the late 1980s, is a high-level general-purpose programming \"\n            \"language. Its design philosophy emphasizes code readability, and its language constructs aim to help \"\n            \"programmers write clear, logical code for both small and large-scale software projects.\"\n        )],\n    ]\n\n    evaluator = ContextRelevanceEvaluator()\n    result = evaluator.run(questions=questions, contexts=contexts)\n    print(result[\"score\"])\n    # 1.0\n    print(result[\"individual_scores\"])\n    # [1.0]\n    print(result[\"results\"])\n    # [{\n    #   'statements': ['Python, created by Guido van Rossum in the late 1980s.'],\n    #   'statement_scores': [1],\n    #   'score': 1.0\n    # }]\n    ```\n    \"\"\"\n\n    def __init__(\n        self,\n        examples: Optional[List[Dict[str, Any]]] = None,\n        progress_bar: bool = True,\n        api: str = \"openai\",\n        api_key: Secret = Secret.from_env_var(\"OPENAI_API_KEY\"),\n        raise_on_failure: bool = True,\n    ):\n        \"\"\"\n        Creates an instance of ContextRelevanceEvaluator.\n\n        :param examples:\n            Optional few-shot examples conforming to the expected input and output format of ContextRelevanceEvaluator.\n            Default examples will be used if none are provided.\n            Each example must be a dictionary with keys \"inputs\" and \"outputs\".\n            \"inputs\" must be a dictionary with keys \"questions\" and \"contexts\".\n            \"outputs\" must be a dictionary with \"statements\" and \"statement_scores\".\n            Expected format:\n            [{\n                \"inputs\": {\n                    \"questions\": \"What is the capital of Italy?\", \"contexts\": [\"Rome is the capital of Italy.\"],\n                },\n                \"outputs\": {\n                    \"statements\": [\"Rome is the capital of Italy.\"],\n                    \"statement_scores\": [1],\n                },\n            }]\n        :param progress_bar:\n            Whether to show a progress bar during the evaluation.\n        :param api:\n            The API to use for calling an LLM through a Generator.\n            Supported APIs: \"openai\".\n        :param api_key:\n            The API key.\n        :param raise_on_failure:\n            Whether to raise an exception if the API call fails.\n\n        \"\"\"\n        self.instructions = (\n            \"Your task is to judge how relevant the provided context is for answering a question. \"\n            \"First, please extract statements from the provided context. \"\n            \"Second, calculate a relevance score for each statement in the context. \"\n            \"The score is 1 if the statement is relevant to answer the question or 0 if it is not relevant. \"\n            \"Each statement should be scored individually.\"\n        )\n        self.inputs = [(\"questions\", List[str]), (\"contexts\", List[List[str]])]\n        self.outputs = [\"statements\", \"statement_scores\"]\n        self.examples = examples or _DEFAULT_EXAMPLES\n        self.api = api\n        self.api_key = api_key\n\n        super(ContextRelevanceEvaluator, self).__init__(\n            instructions=self.instructions,\n            inputs=self.inputs,\n            outputs=self.outputs,\n            examples=self.examples,\n            api=self.api,\n            api_key=self.api_key,\n            raise_on_failure=raise_on_failure,\n            progress_bar=progress_bar,\n        )\n\n    @component.output_types(individual_scores=List[int], score=float, results=List[Dict[str, Any]])\n    def run(self, questions: List[str], contexts: List[List[str]]) -> Dict[str, Any]:\n        \"\"\"\n        Run the LLM evaluator.\n\n        :param questions:\n            A list of questions.\n        :param contexts:\n            A list of lists of contexts. Each list of contexts corresponds to one question.\n        :returns:\n            A dictionary with the following outputs:\n                - `score`: Mean context relevance score over all the provided input questions.\n                - `individual_scores`: A list of context relevance scores for each input question.\n                - `results`: A list of dictionaries with `statements` and `statement_scores` for each input context.\n        \"\"\"\n        result = super(ContextRelevanceEvaluator, self).run(questions=questions, contexts=contexts)\n\n        # calculate average statement relevance score per query\n        for idx, res in enumerate(result[\"results\"]):\n            if res is None:\n                result[\"results\"][idx] = {\"statements\": [], \"statement_scores\": [], \"score\": float(\"nan\")}\n                continue\n            if not res[\"statements\"]:\n                res[\"score\"] = 0\n            else:\n                res[\"score\"] = np_mean(res[\"statement_scores\"])\n\n        # calculate average context relevance score over all queries\n        result[\"score\"] = np_mean([res[\"score\"] for res in result[\"results\"]])\n        result[\"individual_scores\"] = [res[\"score\"] for res in result[\"results\"]]\n\n        return result\n\n    def to_dict(self) -> Dict[str, Any]:\n        \"\"\"\n        Serialize this component to a dictionary.\n\n        :returns:\n            A dictionary with serialized data.\n        \"\"\"\n        return default_to_dict(\n            self,\n            api=self.api,\n            api_key=self.api_key.to_dict() if self.api_key else None,\n            examples=self.examples,\n            progress_bar=self.progress_bar,\n            raise_on_failure=self.raise_on_failure,\n        )\n\n    @classmethod\n    def from_dict(cls, data: Dict[str, Any]) -> \"ContextRelevanceEvaluator\":\n        \"\"\"\n        Deserialize this component from a dictionary.\n\n        :param data:\n            The dictionary representation of this component.\n        :returns:\n            The deserialized component instance.\n        \"\"\"\n        deserialize_secrets_inplace(data[\"init_parameters\"], keys=[\"api_key\"])\n        return default_from_dict(cls, data)\n", "path": "haystack/components/evaluators/context_relevance.py"}]}