We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
QuestionGenerator generates wrong document questions for non-default `num_queries_per_doc`
**Describe the bug**
QuestionGenerator, for the non-default values of `num_queries_per_doc` parameter, generates questions that don't match the order of given documents. For example:

```
question_generator = QuestionGenerator(model_name_or_path="valhalla/t5-small-e2e-qg", num_queries_per_doc=2)
questions = question_generator.generate_batch(texts=["Some document text..", "Another document text..."])

assert isinstance(questions, list)
assert len(questions) == 2
assert isinstance(questions[0], list)
assert isinstance(questions[1], list)
assert len(questions[0]) > 0
assert len(questions[1]) > 0
```

However, not all the questions[0] and questions[1] are guaranteed to come from the first and second documents, respectively, but are likely to be mixed up. For example, questions[0] and questions[1] are likely to contain the questions from the first documents. This mixup then spills over into the next item in the questions list, and so on.


**Error message**
Unfortunately, no error is thrown; the errors are silent.

**Expected behavior**
Generated question buckets should match the originating documents.

**Additional context**
None

**To Reproduce**
See `test_questions_generator.py` and `test_qa_generator_no_default_params` method 

**FAQ Check**
- [ ] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: 
 - GPU/CPU:
 - Haystack version (commit or version number): 
 - DocumentStore:
 - Reader:
 - Retriever:


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `haystack/nodes/question_generator/question_generator.py`
Content:
```
1 import logging
2 from typing import List, Union, Optional, Iterator
3 import itertools
4 import torch
5 
6 from tqdm.auto import tqdm
7 from transformers import AutoModelForSeq2SeqLM
8 from transformers import AutoTokenizer
9 
10 from haystack.errors import HaystackError
11 from haystack.schema import Document
12 from haystack.nodes.base import BaseComponent
13 from haystack.nodes.preprocessor import PreProcessor
14 from haystack.modeling.utils import initialize_device_settings
15 
16 logger = logging.getLogger(__name__)
17 
18 
19 class QuestionGenerator(BaseComponent):
20     """
21     The Question Generator takes only a document as input and outputs questions that it thinks can be
22     answered by this document. In our current implementation, input texts are split into chunks of 50 words
23     with a 10 word overlap. This is because the default model `valhalla/t5-base-e2e-qg` seems to generate only
24     about 3 questions per passage regardless of length. Our approach prioritizes the creation of more questions
25     over processing efficiency (T5 is able to digest much more than 50 words at once). The returned questions
26     generally come in an order dictated by the order of their answers i.e. early questions in the list generally
27     come from earlier in the document.
28     """
29 
30     outgoing_edges = 1
31 
32     def __init__(
33         self,
34         model_name_or_path: str = "valhalla/t5-base-e2e-qg",
35         model_version: Optional[str] = None,
36         num_beams: int = 4,
37         max_length: int = 256,
38         no_repeat_ngram_size: int = 3,
39         length_penalty: float = 1.5,
40         early_stopping: bool = True,
41         split_length: int = 50,
42         split_overlap: int = 10,
43         use_gpu: bool = True,
44         prompt: str = "generate questions:",
45         num_queries_per_doc: int = 1,
46         sep_token: str = "<sep>",
47         batch_size: int = 16,
48         progress_bar: bool = True,
49         use_auth_token: Optional[Union[str, bool]] = None,
50         devices: Optional[List[Union[str, torch.device]]] = None,
51     ):
52         """
53         Uses the valhalla/t5-base-e2e-qg model by default. This class supports any question generation model that is
54         implemented as a Seq2SeqLM in HuggingFace Transformers. Note that this style of question generation (where the only input
55         is a document) is sometimes referred to as end-to-end question generation. Answer-supervised question
56         generation is not currently supported.
57 
58         :param model_name_or_path: Directory of a saved model or the name of a public model e.g. "valhalla/t5-base-e2e-qg".
59                                    See https://huggingface.co/models for full list of available models.
60         :param model_version: The version of model to use from the HuggingFace model hub. Can be tag name, branch name, or commit hash.
61         :param use_gpu: Whether to use GPU or the CPU. Falls back on CPU if no GPU is available.
62         :param batch_size: Number of documents to process at a time.
63         :param progress_bar: Whether to show a tqdm progress bar or not.
64         :param use_auth_token: The API token used to download private models from Huggingface.
65                                If this parameter is set to `True`, then the token generated when running
66                                `transformers-cli login` (stored in ~/.huggingface) will be used.
67                                Additional information can be found here
68                                https://huggingface.co/transformers/main_classes/model.html#transformers.PreTrainedModel.from_pretrained
69         :param devices: List of torch devices (e.g. cuda, cpu, mps) to limit inference to specific devices.
70                         A list containing torch device objects and/or strings is supported (For example
71                         [torch.device('cuda:0'), "mps", "cuda:1"]). When specifying `use_gpu=False` the devices
72                         parameter is not used and a single cpu device is used for inference.
73 
74         """
75         super().__init__()
76         self.devices, _ = initialize_device_settings(devices=devices, use_cuda=use_gpu, multi_gpu=False)
77         if len(self.devices) > 1:
78             logger.warning(
79                 f"Multiple devices are not supported in {self.__class__.__name__} inference, "
80                 f"using the first device {self.devices[0]}."
81             )
82         self.model = AutoModelForSeq2SeqLM.from_pretrained(
83             model_name_or_path, revision=model_version, use_auth_token=use_auth_token
84         )
85         self.model.to(str(self.devices[0]))
86         self.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_auth_token=use_auth_token)
87         self.num_beams = num_beams
88         self.max_length = max_length
89         self.no_repeat_ngram_size = no_repeat_ngram_size
90         self.length_penalty = length_penalty
91         self.early_stopping = early_stopping
92         self.split_length = split_length
93         self.split_overlap = split_overlap
94         self.preprocessor = PreProcessor()
95         self.prompt = prompt
96         self.num_queries_per_doc = num_queries_per_doc
97         self.batch_size = batch_size
98         self.sep_token = self.tokenizer.sep_token or sep_token
99         self.progress_bar = progress_bar
100 
101     def run(self, documents: List[Document]):  # type: ignore
102         generated_questions = []
103         for d in documents:
104             questions = self.generate(d.content)
105             curr_dict = {"document_id": d.id, "document_sample": d.content[:200], "questions": questions}
106             generated_questions.append(curr_dict)
107         output = {"generated_questions": generated_questions, "documents": documents}
108         return output, "output_1"
109 
110     def run_batch(self, documents: Union[List[Document], List[List[Document]]], batch_size: Optional[int] = None):  # type: ignore
111         generated_questions = []
112         if isinstance(documents[0], Document):
113             questions = self.generate_batch(
114                 texts=[d.content for d in documents if isinstance(d, Document)], batch_size=batch_size
115             )
116             questions_iterator = questions  # type: ignore
117             documents_iterator = documents
118         else:
119             questions = self.generate_batch(
120                 texts=[[d.content for d in doc_list] for doc_list in documents if isinstance(doc_list, list)],
121                 batch_size=batch_size,
122             )
123             questions_iterator = itertools.chain.from_iterable(questions)  # type: ignore
124             documents_iterator = itertools.chain.from_iterable(documents)  # type: ignore
125         for cur_questions, doc in zip(questions_iterator, documents_iterator):
126             if not isinstance(doc, Document):
127                 raise HaystackError(f"doc was of type {type(doc)}, but expected a Document.")
128             curr_dict = {"document_id": doc.id, "document_sample": doc.content[:200], "questions": cur_questions}
129             generated_questions.append(curr_dict)
130         output = {"generated_questions": generated_questions, "documents": documents}
131         return output, "output_1"
132 
133     def generate(self, text: str) -> List[str]:
134         # Performing splitting because T5 has a max input length
135         # Also currently, it seems that it only generates about 3 questions for the beginning section of text
136         split_texts_docs = self.preprocessor.split(
137             document={"content": text},
138             split_by="word",
139             split_respect_sentence_boundary=False,
140             split_overlap=self.split_overlap,
141             split_length=self.split_length,
142         )
143         split_texts = [
144             f"{self.prompt} {text.content}" if self.prompt not in text.content else text.content
145             for text in split_texts_docs
146         ]
147         tokenized = self.tokenizer(split_texts, return_tensors="pt", padding=True)
148         input_ids = tokenized["input_ids"].to(self.devices[0])
149         # Necessary if padding is enabled so the model won't attend pad tokens
150         attention_mask = tokenized["attention_mask"].to(self.devices[0])
151         tokens_output = self.model.generate(
152             input_ids=input_ids,
153             attention_mask=attention_mask,
154             num_beams=self.num_beams,
155             max_length=self.max_length,
156             no_repeat_ngram_size=self.no_repeat_ngram_size,
157             length_penalty=self.length_penalty,
158             early_stopping=self.early_stopping,
159             num_return_sequences=self.num_queries_per_doc,
160         )
161 
162         string_output = self.tokenizer.batch_decode(tokens_output, skip_special_tokens=True)
163 
164         ret = []
165         for split in string_output:
166             for question in split.split(self.sep_token):
167                 question = question.strip()
168                 if question and question not in ret:
169                     ret.append(question)
170 
171         return ret
172 
173     def generate_batch(
174         self, texts: Union[List[str], List[List[str]]], batch_size: Optional[int] = None
175     ) -> Union[List[List[str]], List[List[List[str]]]]:
176         """
177         Generates questions for a list of strings or a list of lists of strings.
178 
179         :param texts: List of str or list of list of str.
180         :param batch_size: Number of texts to process at a time.
181         """
182 
183         if batch_size is None:
184             batch_size = self.batch_size
185 
186         if isinstance(texts[0], str):
187             single_doc_list = True
188             number_of_docs = [1 for text_list in texts]
189             text_iterator = texts
190         else:
191             single_doc_list = False
192             number_of_docs = [len(text_list) for text_list in texts]
193             text_iterator = itertools.chain.from_iterable(texts)  # type: ignore
194 
195         split_texts_docs = [
196             self.preprocessor.split(
197                 document={"content": text},
198                 split_by="word",
199                 split_respect_sentence_boundary=False,
200                 split_overlap=self.split_overlap,
201                 split_length=self.split_length,
202             )
203             for text in text_iterator
204         ]
205         split_texts = [[doc.content for doc in split if isinstance(doc.content, str)] for split in split_texts_docs]
206         number_of_splits = [len(split) for split in split_texts]
207         flat_split_texts = [
208             f"{self.prompt} {text}" if self.prompt not in text else text
209             for text in itertools.chain.from_iterable(split_texts)
210         ]
211 
212         batches = self._get_batches(flat_split_texts, batch_size=batch_size)
213         all_string_outputs = []
214         pb = tqdm(total=len(flat_split_texts), disable=not self.progress_bar, desc="Generating questions")
215         for batch in batches:
216             tokenized = self.tokenizer(batch, return_tensors="pt", padding=True)
217             input_ids = tokenized["input_ids"].to(self.devices[0])
218             # Necessary if padding is enabled so the model won't attend pad tokens
219             attention_mask = tokenized["attention_mask"].to(self.devices[0])
220             tokens_output = self.model.generate(
221                 input_ids=input_ids,
222                 attention_mask=attention_mask,
223                 num_beams=self.num_beams,
224                 max_length=self.max_length,
225                 no_repeat_ngram_size=self.no_repeat_ngram_size,
226                 length_penalty=self.length_penalty,
227                 early_stopping=self.early_stopping,
228                 num_return_sequences=self.num_queries_per_doc,
229             )
230 
231             string_output = self.tokenizer.batch_decode(tokens_output, skip_special_tokens=True)
232             all_string_outputs.extend(string_output)
233             pb.update(len(batch))
234         pb.close()
235         # Group predictions together by split
236         grouped_predictions_split = []
237         left_idx = 0
238         right_idx = 0
239         for number in number_of_splits:
240             right_idx = left_idx + number
241             grouped_predictions_split.append(all_string_outputs[left_idx:right_idx])
242             left_idx = right_idx
243         # Group predictions together by doc list
244         grouped_predictions_doc_list = []
245         left_idx = 0
246         right_idx = 0
247         for number in number_of_docs:
248             right_idx = left_idx + number
249             grouped_predictions_doc_list.append(grouped_predictions_split[left_idx:right_idx])
250             left_idx = right_idx
251 
252         results = []
253         for group in grouped_predictions_doc_list:
254             group_preds = []
255             for doc in group:
256                 doc_preds = []
257                 for split in doc:
258                     for question in split.split(self.sep_token):
259                         question = question.strip()
260                         if question and question not in doc_preds:
261                             doc_preds.append(question)
262                 group_preds.append(doc_preds)
263             if single_doc_list:
264                 results.append(group_preds[0])
265             else:
266                 results.append(group_preds)
267 
268         return results
269 
270     @staticmethod
271     def _get_batches(texts: List[str], batch_size: Optional[int]) -> Iterator[List[str]]:
272         if batch_size is None:
273             yield texts
274             return
275         else:
276             for index in range(0, len(texts), batch_size):
277                 yield texts[index : index + batch_size]
278
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/haystack/nodes/question_generator/question_generator.py b/haystack/nodes/question_generator/question_generator.py
--- a/haystack/nodes/question_generator/question_generator.py
+++ b/haystack/nodes/question_generator/question_generator.py
@@ -60,6 +60,11 @@
         :param model_version: The version of model to use from the HuggingFace model hub. Can be tag name, branch name, or commit hash.
         :param use_gpu: Whether to use GPU or the CPU. Falls back on CPU if no GPU is available.
         :param batch_size: Number of documents to process at a time.
+        :param num_queries_per_doc: Number of questions to generate per document. However, this is actually a number
+                                    of question to generate per split in the document where the `split_length` determines
+                                    the length of the split and the `split_overlap` determines the overlap between splits.
+                                    Therefore, this parameter is multiplied by the resulting number of splits to get the
+                                    total number of questions generated per document. This value is capped at 3.
         :param progress_bar: Whether to show a tqdm progress bar or not.
         :param use_auth_token: The API token used to download private models from Huggingface.
                                If this parameter is set to `True`, then the token generated when running
@@ -93,7 +98,7 @@
         self.split_overlap = split_overlap
         self.preprocessor = PreProcessor()
         self.prompt = prompt
-        self.num_queries_per_doc = num_queries_per_doc
+        self.num_queries_per_doc = min(num_queries_per_doc, 3)
         self.batch_size = batch_size
         self.sep_token = self.tokenizer.sep_token or sep_token
         self.progress_bar = progress_bar
@@ -237,7 +242,7 @@
         left_idx = 0
         right_idx = 0
         for number in number_of_splits:
-            right_idx = left_idx + number
+            right_idx = left_idx + number * self.num_queries_per_doc
             grouped_predictions_split.append(all_string_outputs[left_idx:right_idx])
             left_idx = right_idx
         # Group predictions together by doc list

{"golden_diff": "diff --git a/haystack/nodes/question_generator/question_generator.py b/haystack/nodes/question_generator/question_generator.py\n--- a/haystack/nodes/question_generator/question_generator.py\n+++ b/haystack/nodes/question_generator/question_generator.py\n@@ -60,6 +60,11 @@\n         :param model_version: The version of model to use from the HuggingFace model hub. Can be tag name, branch name, or commit hash.\n         :param use_gpu: Whether to use GPU or the CPU. Falls back on CPU if no GPU is available.\n         :param batch_size: Number of documents to process at a time.\n+        :param num_queries_per_doc: Number of questions to generate per document. However, this is actually a number\n+                                    of question to generate per split in the document where the `split_length` determines\n+                                    the length of the split and the `split_overlap` determines the overlap between splits.\n+                                    Therefore, this parameter is multiplied by the resulting number of splits to get the\n+                                    total number of questions generated per document. This value is capped at 3.\n         :param progress_bar: Whether to show a tqdm progress bar or not.\n         :param use_auth_token: The API token used to download private models from Huggingface.\n                                If this parameter is set to `True`, then the token generated when running\n@@ -93,7 +98,7 @@\n         self.split_overlap = split_overlap\n         self.preprocessor = PreProcessor()\n         self.prompt = prompt\n-        self.num_queries_per_doc = num_queries_per_doc\n+        self.num_queries_per_doc = min(num_queries_per_doc, 3)\n         self.batch_size = batch_size\n         self.sep_token = self.tokenizer.sep_token or sep_token\n         self.progress_bar = progress_bar\n@@ -237,7 +242,7 @@\n         left_idx = 0\n         right_idx = 0\n         for number in number_of_splits:\n-            right_idx = left_idx + number\n+            right_idx = left_idx + number * self.num_queries_per_doc\n             grouped_predictions_split.append(all_string_outputs[left_idx:right_idx])\n             left_idx = right_idx\n         # Group predictions together by doc list\n", "issue": "QuestionGenerator generates wrong document questions for non-default `num_queries_per_doc`\n**Describe the bug**\r\nQuestionGenerator, for the non-default values of `num_queries_per_doc` parameter, generates questions that don't match the order of given documents. For example:\r\n\r\n```\r\nquestion_generator = QuestionGenerator(model_name_or_path=\"valhalla/t5-small-e2e-qg\", num_queries_per_doc=2)\r\nquestions = question_generator.generate_batch(texts=[\"Some document text..\", \"Another document text...\"])\r\n\r\nassert isinstance(questions, list)\r\nassert len(questions) == 2\r\nassert isinstance(questions[0], list)\r\nassert isinstance(questions[1], list)\r\nassert len(questions[0]) > 0\r\nassert len(questions[1]) > 0\r\n```\r\n\r\nHowever, not all the questions[0] and questions[1] are guaranteed to come from the first and second documents, respectively, but are likely to be mixed up. For example, questions[0] and questions[1] are likely to contain the questions from the first documents. This mixup then spills over into the next item in the questions list, and so on.\r\n\r\n\r\n**Error message**\r\nUnfortunately, no error is thrown; the errors are silent.\r\n\r\n**Expected behavior**\r\nGenerated question buckets should match the originating documents.\r\n\r\n**Additional context**\r\nNone\r\n\r\n**To Reproduce**\r\nSee `test_questions_generator.py` and `test_qa_generator_no_default_params` method \r\n\r\n**FAQ Check**\r\n- [ ] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?\r\n\r\n**System:**\r\n - OS: \r\n - GPU/CPU:\r\n - Haystack version (commit or version number): \r\n - DocumentStore:\r\n - Reader:\r\n - Retriever:\r\n\n", "before_files": [{"content": "import logging\nfrom typing import List, Union, Optional, Iterator\nimport itertools\nimport torch\n\nfrom tqdm.auto import tqdm\nfrom transformers import AutoModelForSeq2SeqLM\nfrom transformers import AutoTokenizer\n\nfrom haystack.errors import HaystackError\nfrom haystack.schema import Document\nfrom haystack.nodes.base import BaseComponent\nfrom haystack.nodes.preprocessor import PreProcessor\nfrom haystack.modeling.utils import initialize_device_settings\n\nlogger = logging.getLogger(__name__)\n\n\nclass QuestionGenerator(BaseComponent):\n    \"\"\"\n    The Question Generator takes only a document as input and outputs questions that it thinks can be\n    answered by this document. In our current implementation, input texts are split into chunks of 50 words\n    with a 10 word overlap. This is because the default model `valhalla/t5-base-e2e-qg` seems to generate only\n    about 3 questions per passage regardless of length. Our approach prioritizes the creation of more questions\n    over processing efficiency (T5 is able to digest much more than 50 words at once). The returned questions\n    generally come in an order dictated by the order of their answers i.e. early questions in the list generally\n    come from earlier in the document.\n    \"\"\"\n\n    outgoing_edges = 1\n\n    def __init__(\n        self,\n        model_name_or_path: str = \"valhalla/t5-base-e2e-qg\",\n        model_version: Optional[str] = None,\n        num_beams: int = 4,\n        max_length: int = 256,\n        no_repeat_ngram_size: int = 3,\n        length_penalty: float = 1.5,\n        early_stopping: bool = True,\n        split_length: int = 50,\n        split_overlap: int = 10,\n        use_gpu: bool = True,\n        prompt: str = \"generate questions:\",\n        num_queries_per_doc: int = 1,\n        sep_token: str = \"<sep>\",\n        batch_size: int = 16,\n        progress_bar: bool = True,\n        use_auth_token: Optional[Union[str, bool]] = None,\n        devices: Optional[List[Union[str, torch.device]]] = None,\n    ):\n        \"\"\"\n        Uses the valhalla/t5-base-e2e-qg model by default. This class supports any question generation model that is\n        implemented as a Seq2SeqLM in HuggingFace Transformers. Note that this style of question generation (where the only input\n        is a document) is sometimes referred to as end-to-end question generation. Answer-supervised question\n        generation is not currently supported.\n\n        :param model_name_or_path: Directory of a saved model or the name of a public model e.g. \"valhalla/t5-base-e2e-qg\".\n                                   See https://huggingface.co/models for full list of available models.\n        :param model_version: The version of model to use from the HuggingFace model hub. Can be tag name, branch name, or commit hash.\n        :param use_gpu: Whether to use GPU or the CPU. Falls back on CPU if no GPU is available.\n        :param batch_size: Number of documents to process at a time.\n        :param progress_bar: Whether to show a tqdm progress bar or not.\n        :param use_auth_token: The API token used to download private models from Huggingface.\n                               If this parameter is set to `True`, then the token generated when running\n                               `transformers-cli login` (stored in ~/.huggingface) will be used.\n                               Additional information can be found here\n                               https://huggingface.co/transformers/main_classes/model.html#transformers.PreTrainedModel.from_pretrained\n        :param devices: List of torch devices (e.g. cuda, cpu, mps) to limit inference to specific devices.\n                        A list containing torch device objects and/or strings is supported (For example\n                        [torch.device('cuda:0'), \"mps\", \"cuda:1\"]). When specifying `use_gpu=False` the devices\n                        parameter is not used and a single cpu device is used for inference.\n\n        \"\"\"\n        super().__init__()\n        self.devices, _ = initialize_device_settings(devices=devices, use_cuda=use_gpu, multi_gpu=False)\n        if len(self.devices) > 1:\n            logger.warning(\n                f\"Multiple devices are not supported in {self.__class__.__name__} inference, \"\n                f\"using the first device {self.devices[0]}.\"\n            )\n        self.model = AutoModelForSeq2SeqLM.from_pretrained(\n            model_name_or_path, revision=model_version, use_auth_token=use_auth_token\n        )\n        self.model.to(str(self.devices[0]))\n        self.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_auth_token=use_auth_token)\n        self.num_beams = num_beams\n        self.max_length = max_length\n        self.no_repeat_ngram_size = no_repeat_ngram_size\n        self.length_penalty = length_penalty\n        self.early_stopping = early_stopping\n        self.split_length = split_length\n        self.split_overlap = split_overlap\n        self.preprocessor = PreProcessor()\n        self.prompt = prompt\n        self.num_queries_per_doc = num_queries_per_doc\n        self.batch_size = batch_size\n        self.sep_token = self.tokenizer.sep_token or sep_token\n        self.progress_bar = progress_bar\n\n    def run(self, documents: List[Document]):  # type: ignore\n        generated_questions = []\n        for d in documents:\n            questions = self.generate(d.content)\n            curr_dict = {\"document_id\": d.id, \"document_sample\": d.content[:200], \"questions\": questions}\n            generated_questions.append(curr_dict)\n        output = {\"generated_questions\": generated_questions, \"documents\": documents}\n        return output, \"output_1\"\n\n    def run_batch(self, documents: Union[List[Document], List[List[Document]]], batch_size: Optional[int] = None):  # type: ignore\n        generated_questions = []\n        if isinstance(documents[0], Document):\n            questions = self.generate_batch(\n                texts=[d.content for d in documents if isinstance(d, Document)], batch_size=batch_size\n            )\n            questions_iterator = questions  # type: ignore\n            documents_iterator = documents\n        else:\n            questions = self.generate_batch(\n                texts=[[d.content for d in doc_list] for doc_list in documents if isinstance(doc_list, list)],\n                batch_size=batch_size,\n            )\n            questions_iterator = itertools.chain.from_iterable(questions)  # type: ignore\n            documents_iterator = itertools.chain.from_iterable(documents)  # type: ignore\n        for cur_questions, doc in zip(questions_iterator, documents_iterator):\n            if not isinstance(doc, Document):\n                raise HaystackError(f\"doc was of type {type(doc)}, but expected a Document.\")\n            curr_dict = {\"document_id\": doc.id, \"document_sample\": doc.content[:200], \"questions\": cur_questions}\n            generated_questions.append(curr_dict)\n        output = {\"generated_questions\": generated_questions, \"documents\": documents}\n        return output, \"output_1\"\n\n    def generate(self, text: str) -> List[str]:\n        # Performing splitting because T5 has a max input length\n        # Also currently, it seems that it only generates about 3 questions for the beginning section of text\n        split_texts_docs = self.preprocessor.split(\n            document={\"content\": text},\n            split_by=\"word\",\n            split_respect_sentence_boundary=False,\n            split_overlap=self.split_overlap,\n            split_length=self.split_length,\n        )\n        split_texts = [\n            f\"{self.prompt} {text.content}\" if self.prompt not in text.content else text.content\n            for text in split_texts_docs\n        ]\n        tokenized = self.tokenizer(split_texts, return_tensors=\"pt\", padding=True)\n        input_ids = tokenized[\"input_ids\"].to(self.devices[0])\n        # Necessary if padding is enabled so the model won't attend pad tokens\n        attention_mask = tokenized[\"attention_mask\"].to(self.devices[0])\n        tokens_output = self.model.generate(\n            input_ids=input_ids,\n            attention_mask=attention_mask,\n            num_beams=self.num_beams,\n            max_length=self.max_length,\n            no_repeat_ngram_size=self.no_repeat_ngram_size,\n            length_penalty=self.length_penalty,\n            early_stopping=self.early_stopping,\n            num_return_sequences=self.num_queries_per_doc,\n        )\n\n        string_output = self.tokenizer.batch_decode(tokens_output, skip_special_tokens=True)\n\n        ret = []\n        for split in string_output:\n            for question in split.split(self.sep_token):\n                question = question.strip()\n                if question and question not in ret:\n                    ret.append(question)\n\n        return ret\n\n    def generate_batch(\n        self, texts: Union[List[str], List[List[str]]], batch_size: Optional[int] = None\n    ) -> Union[List[List[str]], List[List[List[str]]]]:\n        \"\"\"\n        Generates questions for a list of strings or a list of lists of strings.\n\n        :param texts: List of str or list of list of str.\n        :param batch_size: Number of texts to process at a time.\n        \"\"\"\n\n        if batch_size is None:\n            batch_size = self.batch_size\n\n        if isinstance(texts[0], str):\n            single_doc_list = True\n            number_of_docs = [1 for text_list in texts]\n            text_iterator = texts\n        else:\n            single_doc_list = False\n            number_of_docs = [len(text_list) for text_list in texts]\n            text_iterator = itertools.chain.from_iterable(texts)  # type: ignore\n\n        split_texts_docs = [\n            self.preprocessor.split(\n                document={\"content\": text},\n                split_by=\"word\",\n                split_respect_sentence_boundary=False,\n                split_overlap=self.split_overlap,\n                split_length=self.split_length,\n            )\n            for text in text_iterator\n        ]\n        split_texts = [[doc.content for doc in split if isinstance(doc.content, str)] for split in split_texts_docs]\n        number_of_splits = [len(split) for split in split_texts]\n        flat_split_texts = [\n            f\"{self.prompt} {text}\" if self.prompt not in text else text\n            for text in itertools.chain.from_iterable(split_texts)\n        ]\n\n        batches = self._get_batches(flat_split_texts, batch_size=batch_size)\n        all_string_outputs = []\n        pb = tqdm(total=len(flat_split_texts), disable=not self.progress_bar, desc=\"Generating questions\")\n        for batch in batches:\n            tokenized = self.tokenizer(batch, return_tensors=\"pt\", padding=True)\n            input_ids = tokenized[\"input_ids\"].to(self.devices[0])\n            # Necessary if padding is enabled so the model won't attend pad tokens\n            attention_mask = tokenized[\"attention_mask\"].to(self.devices[0])\n            tokens_output = self.model.generate(\n                input_ids=input_ids,\n                attention_mask=attention_mask,\n                num_beams=self.num_beams,\n                max_length=self.max_length,\n                no_repeat_ngram_size=self.no_repeat_ngram_size,\n                length_penalty=self.length_penalty,\n                early_stopping=self.early_stopping,\n                num_return_sequences=self.num_queries_per_doc,\n            )\n\n            string_output = self.tokenizer.batch_decode(tokens_output, skip_special_tokens=True)\n            all_string_outputs.extend(string_output)\n            pb.update(len(batch))\n        pb.close()\n        # Group predictions together by split\n        grouped_predictions_split = []\n        left_idx = 0\n        right_idx = 0\n        for number in number_of_splits:\n            right_idx = left_idx + number\n            grouped_predictions_split.append(all_string_outputs[left_idx:right_idx])\n            left_idx = right_idx\n        # Group predictions together by doc list\n        grouped_predictions_doc_list = []\n        left_idx = 0\n        right_idx = 0\n        for number in number_of_docs:\n            right_idx = left_idx + number\n            grouped_predictions_doc_list.append(grouped_predictions_split[left_idx:right_idx])\n            left_idx = right_idx\n\n        results = []\n        for group in grouped_predictions_doc_list:\n            group_preds = []\n            for doc in group:\n                doc_preds = []\n                for split in doc:\n                    for question in split.split(self.sep_token):\n                        question = question.strip()\n                        if question and question not in doc_preds:\n                            doc_preds.append(question)\n                group_preds.append(doc_preds)\n            if single_doc_list:\n                results.append(group_preds[0])\n            else:\n                results.append(group_preds)\n\n        return results\n\n    @staticmethod\n    def _get_batches(texts: List[str], batch_size: Optional[int]) -> Iterator[List[str]]:\n        if batch_size is None:\n            yield texts\n            return\n        else:\n            for index in range(0, len(texts), batch_size):\n                yield texts[index : index + batch_size]\n", "path": "haystack/nodes/question_generator/question_generator.py"}], "after_files": [{"content": "import logging\nfrom typing import List, Union, Optional, Iterator\nimport itertools\nimport torch\n\nfrom tqdm.auto import tqdm\nfrom transformers import AutoModelForSeq2SeqLM\nfrom transformers import AutoTokenizer\n\nfrom haystack.errors import HaystackError\nfrom haystack.schema import Document\nfrom haystack.nodes.base import BaseComponent\nfrom haystack.nodes.preprocessor import PreProcessor\nfrom haystack.modeling.utils import initialize_device_settings\n\nlogger = logging.getLogger(__name__)\n\n\nclass QuestionGenerator(BaseComponent):\n    \"\"\"\n    The Question Generator takes only a document as input and outputs questions that it thinks can be\n    answered by this document. In our current implementation, input texts are split into chunks of 50 words\n    with a 10 word overlap. This is because the default model `valhalla/t5-base-e2e-qg` seems to generate only\n    about 3 questions per passage regardless of length. Our approach prioritizes the creation of more questions\n    over processing efficiency (T5 is able to digest much more than 50 words at once). The returned questions\n    generally come in an order dictated by the order of their answers i.e. early questions in the list generally\n    come from earlier in the document.\n    \"\"\"\n\n    outgoing_edges = 1\n\n    def __init__(\n        self,\n        model_name_or_path: str = \"valhalla/t5-base-e2e-qg\",\n        model_version: Optional[str] = None,\n        num_beams: int = 4,\n        max_length: int = 256,\n        no_repeat_ngram_size: int = 3,\n        length_penalty: float = 1.5,\n        early_stopping: bool = True,\n        split_length: int = 50,\n        split_overlap: int = 10,\n        use_gpu: bool = True,\n        prompt: str = \"generate questions:\",\n        num_queries_per_doc: int = 1,\n        sep_token: str = \"<sep>\",\n        batch_size: int = 16,\n        progress_bar: bool = True,\n        use_auth_token: Optional[Union[str, bool]] = None,\n        devices: Optional[List[Union[str, torch.device]]] = None,\n    ):\n        \"\"\"\n        Uses the valhalla/t5-base-e2e-qg model by default. This class supports any question generation model that is\n        implemented as a Seq2SeqLM in HuggingFace Transformers. Note that this style of question generation (where the only input\n        is a document) is sometimes referred to as end-to-end question generation. Answer-supervised question\n        generation is not currently supported.\n\n        :param model_name_or_path: Directory of a saved model or the name of a public model e.g. \"valhalla/t5-base-e2e-qg\".\n                                   See https://huggingface.co/models for full list of available models.\n        :param model_version: The version of model to use from the HuggingFace model hub. Can be tag name, branch name, or commit hash.\n        :param use_gpu: Whether to use GPU or the CPU. Falls back on CPU if no GPU is available.\n        :param batch_size: Number of documents to process at a time.\n        :param num_queries_per_doc: Number of questions to generate per document. However, this is actually a number\n                                    of question to generate per split in the document where the `split_length` determines\n                                    the length of the split and the `split_overlap` determines the overlap between splits.\n                                    Therefore, this parameter is multiplied by the resulting number of splits to get the\n                                    total number of questions generated per document. This value is capped at 3.\n        :param progress_bar: Whether to show a tqdm progress bar or not.\n        :param use_auth_token: The API token used to download private models from Huggingface.\n                               If this parameter is set to `True`, then the token generated when running\n                               `transformers-cli login` (stored in ~/.huggingface) will be used.\n                               Additional information can be found here\n                               https://huggingface.co/transformers/main_classes/model.html#transformers.PreTrainedModel.from_pretrained\n        :param devices: List of torch devices (e.g. cuda, cpu, mps) to limit inference to specific devices.\n                        A list containing torch device objects and/or strings is supported (For example\n                        [torch.device('cuda:0'), \"mps\", \"cuda:1\"]). When specifying `use_gpu=False` the devices\n                        parameter is not used and a single cpu device is used for inference.\n\n        \"\"\"\n        super().__init__()\n        self.devices, _ = initialize_device_settings(devices=devices, use_cuda=use_gpu, multi_gpu=False)\n        if len(self.devices) > 1:\n            logger.warning(\n                f\"Multiple devices are not supported in {self.__class__.__name__} inference, \"\n                f\"using the first device {self.devices[0]}.\"\n            )\n        self.model = AutoModelForSeq2SeqLM.from_pretrained(\n            model_name_or_path, revision=model_version, use_auth_token=use_auth_token\n        )\n        self.model.to(str(self.devices[0]))\n        self.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_auth_token=use_auth_token)\n        self.num_beams = num_beams\n        self.max_length = max_length\n        self.no_repeat_ngram_size = no_repeat_ngram_size\n        self.length_penalty = length_penalty\n        self.early_stopping = early_stopping\n        self.split_length = split_length\n        self.split_overlap = split_overlap\n        self.preprocessor = PreProcessor()\n        self.prompt = prompt\n        self.num_queries_per_doc = min(num_queries_per_doc, 3)\n        self.batch_size = batch_size\n        self.sep_token = self.tokenizer.sep_token or sep_token\n        self.progress_bar = progress_bar\n\n    def run(self, documents: List[Document]):  # type: ignore\n        generated_questions = []\n        for d in documents:\n            questions = self.generate(d.content)\n            curr_dict = {\"document_id\": d.id, \"document_sample\": d.content[:200], \"questions\": questions}\n            generated_questions.append(curr_dict)\n        output = {\"generated_questions\": generated_questions, \"documents\": documents}\n        return output, \"output_1\"\n\n    def run_batch(self, documents: Union[List[Document], List[List[Document]]], batch_size: Optional[int] = None):  # type: ignore\n        generated_questions = []\n        if isinstance(documents[0], Document):\n            questions = self.generate_batch(\n                texts=[d.content for d in documents if isinstance(d, Document)], batch_size=batch_size\n            )\n            questions_iterator = questions  # type: ignore\n            documents_iterator = documents\n        else:\n            questions = self.generate_batch(\n                texts=[[d.content for d in doc_list] for doc_list in documents if isinstance(doc_list, list)],\n                batch_size=batch_size,\n            )\n            questions_iterator = itertools.chain.from_iterable(questions)  # type: ignore\n            documents_iterator = itertools.chain.from_iterable(documents)  # type: ignore\n        for cur_questions, doc in zip(questions_iterator, documents_iterator):\n            if not isinstance(doc, Document):\n                raise HaystackError(f\"doc was of type {type(doc)}, but expected a Document.\")\n            curr_dict = {\"document_id\": doc.id, \"document_sample\": doc.content[:200], \"questions\": cur_questions}\n            generated_questions.append(curr_dict)\n        output = {\"generated_questions\": generated_questions, \"documents\": documents}\n        return output, \"output_1\"\n\n    def generate(self, text: str) -> List[str]:\n        # Performing splitting because T5 has a max input length\n        # Also currently, it seems that it only generates about 3 questions for the beginning section of text\n        split_texts_docs = self.preprocessor.split(\n            document={\"content\": text},\n            split_by=\"word\",\n            split_respect_sentence_boundary=False,\n            split_overlap=self.split_overlap,\n            split_length=self.split_length,\n        )\n        split_texts = [\n            f\"{self.prompt} {text.content}\" if self.prompt not in text.content else text.content\n            for text in split_texts_docs\n        ]\n        tokenized = self.tokenizer(split_texts, return_tensors=\"pt\", padding=True)\n        input_ids = tokenized[\"input_ids\"].to(self.devices[0])\n        # Necessary if padding is enabled so the model won't attend pad tokens\n        attention_mask = tokenized[\"attention_mask\"].to(self.devices[0])\n        tokens_output = self.model.generate(\n            input_ids=input_ids,\n            attention_mask=attention_mask,\n            num_beams=self.num_beams,\n            max_length=self.max_length,\n            no_repeat_ngram_size=self.no_repeat_ngram_size,\n            length_penalty=self.length_penalty,\n            early_stopping=self.early_stopping,\n            num_return_sequences=self.num_queries_per_doc,\n        )\n\n        string_output = self.tokenizer.batch_decode(tokens_output, skip_special_tokens=True)\n\n        ret = []\n        for split in string_output:\n            for question in split.split(self.sep_token):\n                question = question.strip()\n                if question and question not in ret:\n                    ret.append(question)\n\n        return ret\n\n    def generate_batch(\n        self, texts: Union[List[str], List[List[str]]], batch_size: Optional[int] = None\n    ) -> Union[List[List[str]], List[List[List[str]]]]:\n        \"\"\"\n        Generates questions for a list of strings or a list of lists of strings.\n\n        :param texts: List of str or list of list of str.\n        :param batch_size: Number of texts to process at a time.\n        \"\"\"\n\n        if batch_size is None:\n            batch_size = self.batch_size\n\n        if isinstance(texts[0], str):\n            single_doc_list = True\n            number_of_docs = [1 for text_list in texts]\n            text_iterator = texts\n        else:\n            single_doc_list = False\n            number_of_docs = [len(text_list) for text_list in texts]\n            text_iterator = itertools.chain.from_iterable(texts)  # type: ignore\n\n        split_texts_docs = [\n            self.preprocessor.split(\n                document={\"content\": text},\n                split_by=\"word\",\n                split_respect_sentence_boundary=False,\n                split_overlap=self.split_overlap,\n                split_length=self.split_length,\n            )\n            for text in text_iterator\n        ]\n        split_texts = [[doc.content for doc in split if isinstance(doc.content, str)] for split in split_texts_docs]\n        number_of_splits = [len(split) for split in split_texts]\n        flat_split_texts = [\n            f\"{self.prompt} {text}\" if self.prompt not in text else text\n            for text in itertools.chain.from_iterable(split_texts)\n        ]\n\n        batches = self._get_batches(flat_split_texts, batch_size=batch_size)\n        all_string_outputs = []\n        pb = tqdm(total=len(flat_split_texts), disable=not self.progress_bar, desc=\"Generating questions\")\n        for batch in batches:\n            tokenized = self.tokenizer(batch, return_tensors=\"pt\", padding=True)\n            input_ids = tokenized[\"input_ids\"].to(self.devices[0])\n            # Necessary if padding is enabled so the model won't attend pad tokens\n            attention_mask = tokenized[\"attention_mask\"].to(self.devices[0])\n            tokens_output = self.model.generate(\n                input_ids=input_ids,\n                attention_mask=attention_mask,\n                num_beams=self.num_beams,\n                max_length=self.max_length,\n                no_repeat_ngram_size=self.no_repeat_ngram_size,\n                length_penalty=self.length_penalty,\n                early_stopping=self.early_stopping,\n                num_return_sequences=self.num_queries_per_doc,\n            )\n\n            string_output = self.tokenizer.batch_decode(tokens_output, skip_special_tokens=True)\n            all_string_outputs.extend(string_output)\n            pb.update(len(batch))\n        pb.close()\n        # Group predictions together by split\n        grouped_predictions_split = []\n        left_idx = 0\n        right_idx = 0\n        for number in number_of_splits:\n            right_idx = left_idx + number * self.num_queries_per_doc\n            grouped_predictions_split.append(all_string_outputs[left_idx:right_idx])\n            left_idx = right_idx\n        # Group predictions together by doc list\n        grouped_predictions_doc_list = []\n        left_idx = 0\n        right_idx = 0\n        for number in number_of_docs:\n            right_idx = left_idx + number\n            grouped_predictions_doc_list.append(grouped_predictions_split[left_idx:right_idx])\n            left_idx = right_idx\n\n        results = []\n        for group in grouped_predictions_doc_list:\n            group_preds = []\n            for doc in group:\n                doc_preds = []\n                for split in doc:\n                    for question in split.split(self.sep_token):\n                        question = question.strip()\n                        if question and question not in doc_preds:\n                            doc_preds.append(question)\n                group_preds.append(doc_preds)\n            if single_doc_list:\n                results.append(group_preds[0])\n            else:\n                results.append(group_preds)\n\n        return results\n\n    @staticmethod\n    def _get_batches(texts: List[str], batch_size: Optional[int]) -> Iterator[List[str]]:\n        if batch_size is None:\n            yield texts\n            return\n        else:\n            for index in range(0, len(texts), batch_size):\n                yield texts[index : index + batch_size]\n", "path": "haystack/nodes/question_generator/question_generator.py"}]}