We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Unable to install latest haystack version on Windows
**Describe the bug:**

I can't install the latest haystack version on Windows and therefore can't use haystack properly. Tried using pip install farm-haystack==0.4.0 and pip install git+https://github.com/deepset-ai/haystack.git. I suspect it has something to do with faiss-cpu not being compatible with Windows. Is there a way to use haystack anyways? Thanks :)

**Error message:**

 AttributeError: 'MSVCCompiler' object has no attribute 'compiler'
    ----------------------------------------
  ERROR: Failed building wheel for faiss-cpu
  Running setup.py clean for faiss-cpu
Failed to build faiss-cpu
Installing collected packages: faiss-cpu, farm-haystack
    Running setup.py install for faiss-cpu ... error
    ERROR: Command errored out with exit status 1

**System:**
 - OS: Windows
 - Haystack version (commit or version number): 0.4.0


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `haystack/document_store/faiss.py`
Content:
```
1 import logging
2 from pathlib import Path
3 from typing import Union, List, Optional, Dict
4 from tqdm import tqdm
5 import faiss
6 import numpy as np
7 import random
8 
9 from haystack import Document
10 from haystack.document_store.sql import SQLDocumentStore
11 from haystack.retriever.base import BaseRetriever
12 
13 logger = logging.getLogger(__name__)
14 
15 class FAISSDocumentStore(SQLDocumentStore):
16     """
17     Document store for very large scale embedding based dense retrievers like the DPR.
18 
19     It implements the FAISS library(https://github.com/facebookresearch/faiss)
20     to perform similarity search on vectors.
21 
22     The document text and meta-data (for filtering) are stored using the SQLDocumentStore, while
23     the vector embeddings are indexed in a FAISS Index.
24 
25     """
26 
27     def __init__(
28         self,
29         sql_url: str = "sqlite:///",
30         index_buffer_size: int = 10_000,
31         vector_dim: int = 768,
32         faiss_index_factory_str: str = "Flat",
33         faiss_index: Optional[faiss.swigfaiss.Index] = None,
34         **kwargs,
35     ):
36         """
37         :param sql_url: SQL connection URL for database. It defaults to local file based SQLite DB. For large scale
38                         deployment, Postgres is recommended.
39         :param index_buffer_size: When working with large datasets, the ingestion process(FAISS + SQL) can be buffered in
40                                   smaller chunks to reduce memory footprint.
41         :param vector_dim: the embedding vector size.
42         :param faiss_index_factory_str: Create a new FAISS index of the specified type.
43                                         The type is determined from the given string following the conventions
44                                         of the original FAISS index factory.
45                                         Recommended options:
46                                         - "Flat" (default): Best accuracy (= exact). Becomes slow and RAM intense for > 1 Mio docs.
47                                         - "HNSW": Graph-based heuristic. If not further specified,
48                                                   we use a RAM intense, but more accurate config:
49                                                   HNSW256, efConstruction=256 and efSearch=256
50                                         - "IVFx,Flat": Inverted Index. Replace x with the number of centroids aka nlist.
51                                                           Rule of thumb: nlist = 10 * sqrt (num_docs) is a good starting point.
52                                         For more details see:
53                                         - Overview of indices https://github.com/facebookresearch/faiss/wiki/Faiss-indexes
54                                         - Guideline for choosing an index https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index
55                                         - FAISS Index factory https://github.com/facebookresearch/faiss/wiki/The-index-factory
56                                         Benchmarks: XXX
57         :param faiss_index: Pass an existing FAISS Index, i.e. an empty one that you configured manually
58                             or one with docs that you used in Haystack before and want to load again.
59         """
60         self.vector_dim = vector_dim
61 
62         if faiss_index:
63             self.faiss_index = faiss_index
64         else:
65             self.faiss_index = self._create_new_index(vector_dim=self.vector_dim, index_factory=faiss_index_factory_str, **kwargs)
66 
67         self.index_buffer_size = index_buffer_size
68         super().__init__(url=sql_url)
69 
70     def _create_new_index(self, vector_dim: int, index_factory: str = "Flat", metric_type=faiss.METRIC_INNER_PRODUCT, **kwargs):
71         if index_factory == "HNSW" and metric_type == faiss.METRIC_INNER_PRODUCT:
72             # faiss index factory doesn't give the same results for HNSW IP, therefore direct init.
73             # defaults here are similar to DPR codebase (good accuracy, but very high RAM consumption)
74             n_links = kwargs.get("n_links", 128)
75             index = faiss.IndexHNSWFlat(vector_dim, n_links, metric_type)
76             index.hnsw.efSearch = kwargs.get("efSearch", 20)#20
77             index.hnsw.efConstruction = kwargs.get("efConstruction", 80)#80
78             logger.info(f"HNSW params: n_links: {n_links}, efSearch: {index.hnsw.efSearch}, efConstruction: {index.hnsw.efConstruction}")
79         else:
80             index = faiss.index_factory(vector_dim, index_factory, metric_type)
81         return index
82 
83     def write_documents(self, documents: Union[List[dict], List[Document]], index: Optional[str] = None):
84         """
85         Add new documents to the DocumentStore.
86         :param documents: List of `Dicts` or List of `Documents`. If they already contain the embeddings, we'll index
87                           them right away in FAISS. If not, you can later call update_embeddings() to create & index them.
88         :param index: (SQL) index name for storing the docs and metadata
89         :return:
90         """
91         # vector index
92         if not self.faiss_index:
93             raise ValueError("Couldn't find a FAISS index. Try to init the FAISSDocumentStore() again ...")
94         # doc + metadata index
95         index = index or self.index
96         document_objects = [Document.from_dict(d) if isinstance(d, dict) else d for d in documents]
97 
98         add_vectors = False if document_objects[0].embedding is None else True
99 
100         for i in range(0, len(document_objects), self.index_buffer_size):
101             vector_id = self.faiss_index.ntotal
102             if add_vectors:
103                 embeddings = [doc.embedding for doc in document_objects[i: i + self.index_buffer_size]]
104                 embeddings = np.array(embeddings, dtype="float32")
105                 self.faiss_index.add(embeddings)
106 
107             docs_to_write_in_sql = []
108             for doc in document_objects[i : i + self.index_buffer_size]:
109                 meta = doc.meta
110                 if add_vectors:
111                     meta["vector_id"] = vector_id
112                     vector_id += 1
113                 docs_to_write_in_sql.append(doc)
114 
115             super(FAISSDocumentStore, self).write_documents(docs_to_write_in_sql, index=index)
116 
117     def update_embeddings(self, retriever: BaseRetriever, index: Optional[str] = None):
118         """
119         Updates the embeddings in the the document store using the encoding model specified in the retriever.
120         This can be useful if want to add or change the embeddings for your documents (e.g. after changing the retriever config).
121 
122         :param retriever: Retriever to use to get embeddings for text
123         :param index: (SQL) index name for storing the docs and metadata
124         :return: None
125         """
126         # To clear out the FAISS index contents and frees all memory immediately that is in use by the index
127         self.faiss_index.reset()
128 
129         index = index or self.index
130         documents = self.get_all_documents(index=index)
131 
132         if len(documents) == 0:
133             logger.warning("Calling DocumentStore.update_embeddings() on an empty index")
134             self.faiss_index = None
135             return
136 
137         logger.info(f"Updating embeddings for {len(documents)} docs...")
138         embeddings = retriever.embed_passages(documents)  # type: ignore
139         assert len(documents) == len(embeddings)
140         for i, doc in enumerate(documents):
141             doc.embedding = embeddings[i]
142 
143         logger.info("Indexing embeddings and updating vectors_ids...")
144         for i in tqdm(range(0, len(documents), self.index_buffer_size)):
145             vector_id_map = {}
146             vector_id = self.faiss_index.ntotal
147             embeddings = [doc.embedding for doc in documents[i: i + self.index_buffer_size]]
148             embeddings = np.array(embeddings, dtype="float32")
149             self.faiss_index.add(embeddings)
150 
151             for doc in documents[i: i + self.index_buffer_size]:
152                 vector_id_map[doc.id] = vector_id
153                 vector_id += 1
154             self.update_vector_ids(vector_id_map, index=index)
155 
156     def train_index(self, documents: Optional[Union[List[dict], List[Document]]], embeddings: Optional[np.array] = None):
157         """
158         Some FAISS indices (e.g. IVF) require initial "training" on a sample of vectors before you can add your final vectors.
159         The train vectors should come from the same distribution as your final ones.
160         You can pass either documents (incl. embeddings) or just the plain embeddings that the index shall be trained on.
161 
162         :param documents: Documents (incl. the embeddings)
163         :param embeddings: Plain embeddings
164         :return: None
165         """
166 
167         if embeddings and documents:
168             raise ValueError("Either pass `documents` or `embeddings`. You passed both.")
169         if documents:
170             document_objects = [Document.from_dict(d) if isinstance(d, dict) else d for d in documents]
171             embeddings = [doc.embedding for doc in document_objects]
172             embeddings = np.array(embeddings, dtype="float32")
173         self.faiss_index.train(embeddings)
174 
175     def delete_all_documents(self, index=None):
176         index = index or self.index
177         self.faiss_index.reset()
178         super().delete_all_documents(index=index)
179 
180     def query_by_embedding(
181         self, query_emb: np.array, filters: Optional[dict] = None, top_k: int = 10, index: Optional[str] = None
182     ) -> List[Document]:
183         """
184         Find the document that is most similar to the provided `query_emb` by using a vector similarity metric.
185 
186         :param query_emb: Embedding of the query (e.g. gathered from DPR)
187         :param filters: Optional filters to narrow down the search space.
188                         Example: {"name": ["some", "more"], "category": ["only_one"]}
189         :param top_k: How many documents to return
190         :param index: (SQL) index name for storing the docs and metadata
191         :return:
192         """
193         if filters:
194             raise Exception("Query filters are not implemented for the FAISSDocumentStore.")
195         if not self.faiss_index:
196             raise Exception("No index exists. Use 'update_embeddings()` to create an index.")
197 
198         query_emb = query_emb.reshape(1, -1).astype(np.float32)
199         score_matrix, vector_id_matrix = self.faiss_index.search(query_emb, top_k)
200         vector_ids_for_query = [str(vector_id) for vector_id in vector_id_matrix[0] if vector_id != -1]
201 
202         documents = self.get_documents_by_vector_ids(vector_ids_for_query, index=index)
203 
204         #assign query score to each document
205         scores_for_vector_ids: Dict[str, float] = {str(v_id): s for v_id, s in zip(vector_id_matrix[0], score_matrix[0])}
206         for doc in documents:
207             doc.score = scores_for_vector_ids[doc.meta["vector_id"]]  # type: ignore
208             doc.probability = (doc.score + 1) / 2
209         return documents
210 
211     def save(self, file_path: Union[str, Path]):
212         """
213         Save FAISS Index to the specified file.
214 
215         :param file_path: Path to save to.
216         :return: None
217         """
218         faiss.write_index(self.faiss_index, str(file_path))
219 
220     @classmethod
221     def load(
222             cls,
223             faiss_file_path: Union[str, Path],
224             sql_url: str,
225             index_buffer_size: int = 10_000,
226     ):
227         """
228         Load a saved FAISS index from a file and connect to the SQL database.
229         Note: In order to have a correct mapping from FAISS to SQL,
230               make sure to use the same SQL DB that you used when calling `save()`.
231 
232         :param faiss_file_path: Stored FAISS index file. Can be created via calling `save()`
233         :param sql_url: Connection string to the SQL database that contains your docs and metadata.
234         :param index_buffer_size: When working with large datasets, the ingestion process(FAISS + SQL) can be buffered in
235                                   smaller chunks to reduce memory footprint.
236         :return:
237         """
238         """
239         """
240         faiss_index = faiss.read_index(str(faiss_file_path))
241         return cls(
242             faiss_index=faiss_index,
243             sql_url=sql_url,
244             index_buffer_size=index_buffer_size,
245             vector_dim=faiss_index.d
246         )
247 
248
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/haystack/document_store/faiss.py b/haystack/document_store/faiss.py
--- a/haystack/document_store/faiss.py
+++ b/haystack/document_store/faiss.py
@@ -1,17 +1,22 @@
 import logging
+from sys import platform
 from pathlib import Path
 from typing import Union, List, Optional, Dict
 from tqdm import tqdm
-import faiss
 import numpy as np
-import random
 
 from haystack import Document
 from haystack.document_store.sql import SQLDocumentStore
 from haystack.retriever.base import BaseRetriever
 
+if platform != 'win32' and platform != 'cygwin':
+    import faiss
+else:
+    raise ModuleNotFoundError("FAISSDocumentStore on windows platform is not supported")
+
 logger = logging.getLogger(__name__)
 
+
 class FAISSDocumentStore(SQLDocumentStore):
     """
     Document store for very large scale embedding based dense retrievers like the DPR.

{"golden_diff": "diff --git a/haystack/document_store/faiss.py b/haystack/document_store/faiss.py\n--- a/haystack/document_store/faiss.py\n+++ b/haystack/document_store/faiss.py\n@@ -1,17 +1,22 @@\n import logging\n+from sys import platform\n from pathlib import Path\n from typing import Union, List, Optional, Dict\n from tqdm import tqdm\n-import faiss\n import numpy as np\n-import random\n \n from haystack import Document\n from haystack.document_store.sql import SQLDocumentStore\n from haystack.retriever.base import BaseRetriever\n \n+if platform != 'win32' and platform != 'cygwin':\n+    import faiss\n+else:\n+    raise ModuleNotFoundError(\"FAISSDocumentStore on windows platform is not supported\")\n+\n logger = logging.getLogger(__name__)\n \n+\n class FAISSDocumentStore(SQLDocumentStore):\n     \"\"\"\n     Document store for very large scale embedding based dense retrievers like the DPR.\n", "issue": "Unable to install latest haystack version on Windows\n**Describe the bug:**\r\n\r\nI can't install the latest haystack version on Windows and therefore can't use haystack properly. Tried using pip install farm-haystack==0.4.0 and pip install git+https://github.com/deepset-ai/haystack.git. I suspect it has something to do with faiss-cpu not being compatible with Windows. Is there a way to use haystack anyways? Thanks :)\r\n\r\n**Error message:**\r\n\r\n AttributeError: 'MSVCCompiler' object has no attribute 'compiler'\r\n    ----------------------------------------\r\n  ERROR: Failed building wheel for faiss-cpu\r\n  Running setup.py clean for faiss-cpu\r\nFailed to build faiss-cpu\r\nInstalling collected packages: faiss-cpu, farm-haystack\r\n    Running setup.py install for faiss-cpu ... error\r\n    ERROR: Command errored out with exit status 1\r\n\r\n**System:**\r\n - OS: Windows\r\n - Haystack version (commit or version number): 0.4.0\r\n\n", "before_files": [{"content": "import logging\nfrom pathlib import Path\nfrom typing import Union, List, Optional, Dict\nfrom tqdm import tqdm\nimport faiss\nimport numpy as np\nimport random\n\nfrom haystack import Document\nfrom haystack.document_store.sql import SQLDocumentStore\nfrom haystack.retriever.base import BaseRetriever\n\nlogger = logging.getLogger(__name__)\n\nclass FAISSDocumentStore(SQLDocumentStore):\n    \"\"\"\n    Document store for very large scale embedding based dense retrievers like the DPR.\n\n    It implements the FAISS library(https://github.com/facebookresearch/faiss)\n    to perform similarity search on vectors.\n\n    The document text and meta-data (for filtering) are stored using the SQLDocumentStore, while\n    the vector embeddings are indexed in a FAISS Index.\n\n    \"\"\"\n\n    def __init__(\n        self,\n        sql_url: str = \"sqlite:///\",\n        index_buffer_size: int = 10_000,\n        vector_dim: int = 768,\n        faiss_index_factory_str: str = \"Flat\",\n        faiss_index: Optional[faiss.swigfaiss.Index] = None,\n        **kwargs,\n    ):\n        \"\"\"\n        :param sql_url: SQL connection URL for database. It defaults to local file based SQLite DB. For large scale\n                        deployment, Postgres is recommended.\n        :param index_buffer_size: When working with large datasets, the ingestion process(FAISS + SQL) can be buffered in\n                                  smaller chunks to reduce memory footprint.\n        :param vector_dim: the embedding vector size.\n        :param faiss_index_factory_str: Create a new FAISS index of the specified type.\n                                        The type is determined from the given string following the conventions\n                                        of the original FAISS index factory.\n                                        Recommended options:\n                                        - \"Flat\" (default): Best accuracy (= exact). Becomes slow and RAM intense for > 1 Mio docs.\n                                        - \"HNSW\": Graph-based heuristic. If not further specified,\n                                                  we use a RAM intense, but more accurate config:\n                                                  HNSW256, efConstruction=256 and efSearch=256\n                                        - \"IVFx,Flat\": Inverted Index. Replace x with the number of centroids aka nlist.\n                                                          Rule of thumb: nlist = 10 * sqrt (num_docs) is a good starting point.\n                                        For more details see:\n                                        - Overview of indices https://github.com/facebookresearch/faiss/wiki/Faiss-indexes\n                                        - Guideline for choosing an index https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index\n                                        - FAISS Index factory https://github.com/facebookresearch/faiss/wiki/The-index-factory\n                                        Benchmarks: XXX\n        :param faiss_index: Pass an existing FAISS Index, i.e. an empty one that you configured manually\n                            or one with docs that you used in Haystack before and want to load again.\n        \"\"\"\n        self.vector_dim = vector_dim\n\n        if faiss_index:\n            self.faiss_index = faiss_index\n        else:\n            self.faiss_index = self._create_new_index(vector_dim=self.vector_dim, index_factory=faiss_index_factory_str, **kwargs)\n\n        self.index_buffer_size = index_buffer_size\n        super().__init__(url=sql_url)\n\n    def _create_new_index(self, vector_dim: int, index_factory: str = \"Flat\", metric_type=faiss.METRIC_INNER_PRODUCT, **kwargs):\n        if index_factory == \"HNSW\" and metric_type == faiss.METRIC_INNER_PRODUCT:\n            # faiss index factory doesn't give the same results for HNSW IP, therefore direct init.\n            # defaults here are similar to DPR codebase (good accuracy, but very high RAM consumption)\n            n_links = kwargs.get(\"n_links\", 128)\n            index = faiss.IndexHNSWFlat(vector_dim, n_links, metric_type)\n            index.hnsw.efSearch = kwargs.get(\"efSearch\", 20)#20\n            index.hnsw.efConstruction = kwargs.get(\"efConstruction\", 80)#80\n            logger.info(f\"HNSW params: n_links: {n_links}, efSearch: {index.hnsw.efSearch}, efConstruction: {index.hnsw.efConstruction}\")\n        else:\n            index = faiss.index_factory(vector_dim, index_factory, metric_type)\n        return index\n\n    def write_documents(self, documents: Union[List[dict], List[Document]], index: Optional[str] = None):\n        \"\"\"\n        Add new documents to the DocumentStore.\n        :param documents: List of `Dicts` or List of `Documents`. If they already contain the embeddings, we'll index\n                          them right away in FAISS. If not, you can later call update_embeddings() to create & index them.\n        :param index: (SQL) index name for storing the docs and metadata\n        :return:\n        \"\"\"\n        # vector index\n        if not self.faiss_index:\n            raise ValueError(\"Couldn't find a FAISS index. Try to init the FAISSDocumentStore() again ...\")\n        # doc + metadata index\n        index = index or self.index\n        document_objects = [Document.from_dict(d) if isinstance(d, dict) else d for d in documents]\n\n        add_vectors = False if document_objects[0].embedding is None else True\n\n        for i in range(0, len(document_objects), self.index_buffer_size):\n            vector_id = self.faiss_index.ntotal\n            if add_vectors:\n                embeddings = [doc.embedding for doc in document_objects[i: i + self.index_buffer_size]]\n                embeddings = np.array(embeddings, dtype=\"float32\")\n                self.faiss_index.add(embeddings)\n\n            docs_to_write_in_sql = []\n            for doc in document_objects[i : i + self.index_buffer_size]:\n                meta = doc.meta\n                if add_vectors:\n                    meta[\"vector_id\"] = vector_id\n                    vector_id += 1\n                docs_to_write_in_sql.append(doc)\n\n            super(FAISSDocumentStore, self).write_documents(docs_to_write_in_sql, index=index)\n\n    def update_embeddings(self, retriever: BaseRetriever, index: Optional[str] = None):\n        \"\"\"\n        Updates the embeddings in the the document store using the encoding model specified in the retriever.\n        This can be useful if want to add or change the embeddings for your documents (e.g. after changing the retriever config).\n\n        :param retriever: Retriever to use to get embeddings for text\n        :param index: (SQL) index name for storing the docs and metadata\n        :return: None\n        \"\"\"\n        # To clear out the FAISS index contents and frees all memory immediately that is in use by the index\n        self.faiss_index.reset()\n\n        index = index or self.index\n        documents = self.get_all_documents(index=index)\n\n        if len(documents) == 0:\n            logger.warning(\"Calling DocumentStore.update_embeddings() on an empty index\")\n            self.faiss_index = None\n            return\n\n        logger.info(f\"Updating embeddings for {len(documents)} docs...\")\n        embeddings = retriever.embed_passages(documents)  # type: ignore\n        assert len(documents) == len(embeddings)\n        for i, doc in enumerate(documents):\n            doc.embedding = embeddings[i]\n\n        logger.info(\"Indexing embeddings and updating vectors_ids...\")\n        for i in tqdm(range(0, len(documents), self.index_buffer_size)):\n            vector_id_map = {}\n            vector_id = self.faiss_index.ntotal\n            embeddings = [doc.embedding for doc in documents[i: i + self.index_buffer_size]]\n            embeddings = np.array(embeddings, dtype=\"float32\")\n            self.faiss_index.add(embeddings)\n\n            for doc in documents[i: i + self.index_buffer_size]:\n                vector_id_map[doc.id] = vector_id\n                vector_id += 1\n            self.update_vector_ids(vector_id_map, index=index)\n\n    def train_index(self, documents: Optional[Union[List[dict], List[Document]]], embeddings: Optional[np.array] = None):\n        \"\"\"\n        Some FAISS indices (e.g. IVF) require initial \"training\" on a sample of vectors before you can add your final vectors.\n        The train vectors should come from the same distribution as your final ones.\n        You can pass either documents (incl. embeddings) or just the plain embeddings that the index shall be trained on.\n\n        :param documents: Documents (incl. the embeddings)\n        :param embeddings: Plain embeddings\n        :return: None\n        \"\"\"\n\n        if embeddings and documents:\n            raise ValueError(\"Either pass `documents` or `embeddings`. You passed both.\")\n        if documents:\n            document_objects = [Document.from_dict(d) if isinstance(d, dict) else d for d in documents]\n            embeddings = [doc.embedding for doc in document_objects]\n            embeddings = np.array(embeddings, dtype=\"float32\")\n        self.faiss_index.train(embeddings)\n\n    def delete_all_documents(self, index=None):\n        index = index or self.index\n        self.faiss_index.reset()\n        super().delete_all_documents(index=index)\n\n    def query_by_embedding(\n        self, query_emb: np.array, filters: Optional[dict] = None, top_k: int = 10, index: Optional[str] = None\n    ) -> List[Document]:\n        \"\"\"\n        Find the document that is most similar to the provided `query_emb` by using a vector similarity metric.\n\n        :param query_emb: Embedding of the query (e.g. gathered from DPR)\n        :param filters: Optional filters to narrow down the search space.\n                        Example: {\"name\": [\"some\", \"more\"], \"category\": [\"only_one\"]}\n        :param top_k: How many documents to return\n        :param index: (SQL) index name for storing the docs and metadata\n        :return:\n        \"\"\"\n        if filters:\n            raise Exception(\"Query filters are not implemented for the FAISSDocumentStore.\")\n        if not self.faiss_index:\n            raise Exception(\"No index exists. Use 'update_embeddings()` to create an index.\")\n\n        query_emb = query_emb.reshape(1, -1).astype(np.float32)\n        score_matrix, vector_id_matrix = self.faiss_index.search(query_emb, top_k)\n        vector_ids_for_query = [str(vector_id) for vector_id in vector_id_matrix[0] if vector_id != -1]\n\n        documents = self.get_documents_by_vector_ids(vector_ids_for_query, index=index)\n\n        #assign query score to each document\n        scores_for_vector_ids: Dict[str, float] = {str(v_id): s for v_id, s in zip(vector_id_matrix[0], score_matrix[0])}\n        for doc in documents:\n            doc.score = scores_for_vector_ids[doc.meta[\"vector_id\"]]  # type: ignore\n            doc.probability = (doc.score + 1) / 2\n        return documents\n\n    def save(self, file_path: Union[str, Path]):\n        \"\"\"\n        Save FAISS Index to the specified file.\n\n        :param file_path: Path to save to.\n        :return: None\n        \"\"\"\n        faiss.write_index(self.faiss_index, str(file_path))\n\n    @classmethod\n    def load(\n            cls,\n            faiss_file_path: Union[str, Path],\n            sql_url: str,\n            index_buffer_size: int = 10_000,\n    ):\n        \"\"\"\n        Load a saved FAISS index from a file and connect to the SQL database.\n        Note: In order to have a correct mapping from FAISS to SQL,\n              make sure to use the same SQL DB that you used when calling `save()`.\n\n        :param faiss_file_path: Stored FAISS index file. Can be created via calling `save()`\n        :param sql_url: Connection string to the SQL database that contains your docs and metadata.\n        :param index_buffer_size: When working with large datasets, the ingestion process(FAISS + SQL) can be buffered in\n                                  smaller chunks to reduce memory footprint.\n        :return:\n        \"\"\"\n        \"\"\"\n        \"\"\"\n        faiss_index = faiss.read_index(str(faiss_file_path))\n        return cls(\n            faiss_index=faiss_index,\n            sql_url=sql_url,\n            index_buffer_size=index_buffer_size,\n            vector_dim=faiss_index.d\n        )\n\n", "path": "haystack/document_store/faiss.py"}], "after_files": [{"content": "import logging\nfrom sys import platform\nfrom pathlib import Path\nfrom typing import Union, List, Optional, Dict\nfrom tqdm import tqdm\nimport numpy as np\n\nfrom haystack import Document\nfrom haystack.document_store.sql import SQLDocumentStore\nfrom haystack.retriever.base import BaseRetriever\n\nif platform != 'win32' and platform != 'cygwin':\n    import faiss\nelse:\n    raise ModuleNotFoundError(\"FAISSDocumentStore on windows platform is not supported\")\n\nlogger = logging.getLogger(__name__)\n\n\nclass FAISSDocumentStore(SQLDocumentStore):\n    \"\"\"\n    Document store for very large scale embedding based dense retrievers like the DPR.\n\n    It implements the FAISS library(https://github.com/facebookresearch/faiss)\n    to perform similarity search on vectors.\n\n    The document text and meta-data (for filtering) are stored using the SQLDocumentStore, while\n    the vector embeddings are indexed in a FAISS Index.\n\n    \"\"\"\n\n    def __init__(\n        self,\n        sql_url: str = \"sqlite:///\",\n        index_buffer_size: int = 10_000,\n        vector_dim: int = 768,\n        faiss_index_factory_str: str = \"Flat\",\n        faiss_index: Optional[faiss.swigfaiss.Index] = None,\n        **kwargs,\n    ):\n        \"\"\"\n        :param sql_url: SQL connection URL for database. It defaults to local file based SQLite DB. For large scale\n                        deployment, Postgres is recommended.\n        :param index_buffer_size: When working with large datasets, the ingestion process(FAISS + SQL) can be buffered in\n                                  smaller chunks to reduce memory footprint.\n        :param vector_dim: the embedding vector size.\n        :param faiss_index_factory_str: Create a new FAISS index of the specified type.\n                                        The type is determined from the given string following the conventions\n                                        of the original FAISS index factory.\n                                        Recommended options:\n                                        - \"Flat\" (default): Best accuracy (= exact). Becomes slow and RAM intense for > 1 Mio docs.\n                                        - \"HNSW\": Graph-based heuristic. If not further specified,\n                                                  we use a RAM intense, but more accurate config:\n                                                  HNSW256, efConstruction=256 and efSearch=256\n                                        - \"IVFx,Flat\": Inverted Index. Replace x with the number of centroids aka nlist.\n                                                          Rule of thumb: nlist = 10 * sqrt (num_docs) is a good starting point.\n                                        For more details see:\n                                        - Overview of indices https://github.com/facebookresearch/faiss/wiki/Faiss-indexes\n                                        - Guideline for choosing an index https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index\n                                        - FAISS Index factory https://github.com/facebookresearch/faiss/wiki/The-index-factory\n                                        Benchmarks: XXX\n        :param faiss_index: Pass an existing FAISS Index, i.e. an empty one that you configured manually\n                            or one with docs that you used in Haystack before and want to load again.\n        \"\"\"\n        self.vector_dim = vector_dim\n\n        if faiss_index:\n            self.faiss_index = faiss_index\n        else:\n            self.faiss_index = self._create_new_index(vector_dim=self.vector_dim, index_factory=faiss_index_factory_str, **kwargs)\n\n        self.index_buffer_size = index_buffer_size\n        super().__init__(url=sql_url)\n\n    def _create_new_index(self, vector_dim: int, index_factory: str = \"Flat\", metric_type=faiss.METRIC_INNER_PRODUCT, **kwargs):\n        if index_factory == \"HNSW\" and metric_type == faiss.METRIC_INNER_PRODUCT:\n            # faiss index factory doesn't give the same results for HNSW IP, therefore direct init.\n            # defaults here are similar to DPR codebase (good accuracy, but very high RAM consumption)\n            n_links = kwargs.get(\"n_links\", 128)\n            index = faiss.IndexHNSWFlat(vector_dim, n_links, metric_type)\n            index.hnsw.efSearch = kwargs.get(\"efSearch\", 20)#20\n            index.hnsw.efConstruction = kwargs.get(\"efConstruction\", 80)#80\n            logger.info(f\"HNSW params: n_links: {n_links}, efSearch: {index.hnsw.efSearch}, efConstruction: {index.hnsw.efConstruction}\")\n        else:\n            index = faiss.index_factory(vector_dim, index_factory, metric_type)\n        return index\n\n    def write_documents(self, documents: Union[List[dict], List[Document]], index: Optional[str] = None):\n        \"\"\"\n        Add new documents to the DocumentStore.\n        :param documents: List of `Dicts` or List of `Documents`. If they already contain the embeddings, we'll index\n                          them right away in FAISS. If not, you can later call update_embeddings() to create & index them.\n        :param index: (SQL) index name for storing the docs and metadata\n        :return:\n        \"\"\"\n        # vector index\n        if not self.faiss_index:\n            raise ValueError(\"Couldn't find a FAISS index. Try to init the FAISSDocumentStore() again ...\")\n        # doc + metadata index\n        index = index or self.index\n        document_objects = [Document.from_dict(d) if isinstance(d, dict) else d for d in documents]\n\n        add_vectors = False if document_objects[0].embedding is None else True\n\n        for i in range(0, len(document_objects), self.index_buffer_size):\n            vector_id = self.faiss_index.ntotal\n            if add_vectors:\n                embeddings = [doc.embedding for doc in document_objects[i: i + self.index_buffer_size]]\n                embeddings = np.array(embeddings, dtype=\"float32\")\n                self.faiss_index.add(embeddings)\n\n            docs_to_write_in_sql = []\n            for doc in document_objects[i : i + self.index_buffer_size]:\n                meta = doc.meta\n                if add_vectors:\n                    meta[\"vector_id\"] = vector_id\n                    vector_id += 1\n                docs_to_write_in_sql.append(doc)\n\n            super(FAISSDocumentStore, self).write_documents(docs_to_write_in_sql, index=index)\n\n    def update_embeddings(self, retriever: BaseRetriever, index: Optional[str] = None):\n        \"\"\"\n        Updates the embeddings in the the document store using the encoding model specified in the retriever.\n        This can be useful if want to add or change the embeddings for your documents (e.g. after changing the retriever config).\n\n        :param retriever: Retriever to use to get embeddings for text\n        :param index: (SQL) index name for storing the docs and metadata\n        :return: None\n        \"\"\"\n        # To clear out the FAISS index contents and frees all memory immediately that is in use by the index\n        self.faiss_index.reset()\n\n        index = index or self.index\n        documents = self.get_all_documents(index=index)\n\n        if len(documents) == 0:\n            logger.warning(\"Calling DocumentStore.update_embeddings() on an empty index\")\n            self.faiss_index = None\n            return\n\n        logger.info(f\"Updating embeddings for {len(documents)} docs...\")\n        embeddings = retriever.embed_passages(documents)  # type: ignore\n        assert len(documents) == len(embeddings)\n        for i, doc in enumerate(documents):\n            doc.embedding = embeddings[i]\n\n        logger.info(\"Indexing embeddings and updating vectors_ids...\")\n        for i in tqdm(range(0, len(documents), self.index_buffer_size)):\n            vector_id_map = {}\n            vector_id = self.faiss_index.ntotal\n            embeddings = [doc.embedding for doc in documents[i: i + self.index_buffer_size]]\n            embeddings = np.array(embeddings, dtype=\"float32\")\n            self.faiss_index.add(embeddings)\n\n            for doc in documents[i: i + self.index_buffer_size]:\n                vector_id_map[doc.id] = vector_id\n                vector_id += 1\n            self.update_vector_ids(vector_id_map, index=index)\n\n    def train_index(self, documents: Optional[Union[List[dict], List[Document]]], embeddings: Optional[np.array] = None):\n        \"\"\"\n        Some FAISS indices (e.g. IVF) require initial \"training\" on a sample of vectors before you can add your final vectors.\n        The train vectors should come from the same distribution as your final ones.\n        You can pass either documents (incl. embeddings) or just the plain embeddings that the index shall be trained on.\n\n        :param documents: Documents (incl. the embeddings)\n        :param embeddings: Plain embeddings\n        :return: None\n        \"\"\"\n\n        if embeddings and documents:\n            raise ValueError(\"Either pass `documents` or `embeddings`. You passed both.\")\n        if documents:\n            document_objects = [Document.from_dict(d) if isinstance(d, dict) else d for d in documents]\n            embeddings = [doc.embedding for doc in document_objects]\n            embeddings = np.array(embeddings, dtype=\"float32\")\n        self.faiss_index.train(embeddings)\n\n    def delete_all_documents(self, index=None):\n        index = index or self.index\n        self.faiss_index.reset()\n        super().delete_all_documents(index=index)\n\n    def query_by_embedding(\n        self, query_emb: np.array, filters: Optional[dict] = None, top_k: int = 10, index: Optional[str] = None\n    ) -> List[Document]:\n        \"\"\"\n        Find the document that is most similar to the provided `query_emb` by using a vector similarity metric.\n\n        :param query_emb: Embedding of the query (e.g. gathered from DPR)\n        :param filters: Optional filters to narrow down the search space.\n                        Example: {\"name\": [\"some\", \"more\"], \"category\": [\"only_one\"]}\n        :param top_k: How many documents to return\n        :param index: (SQL) index name for storing the docs and metadata\n        :return:\n        \"\"\"\n        if filters:\n            raise Exception(\"Query filters are not implemented for the FAISSDocumentStore.\")\n        if not self.faiss_index:\n            raise Exception(\"No index exists. Use 'update_embeddings()` to create an index.\")\n\n        query_emb = query_emb.reshape(1, -1).astype(np.float32)\n        score_matrix, vector_id_matrix = self.faiss_index.search(query_emb, top_k)\n        vector_ids_for_query = [str(vector_id) for vector_id in vector_id_matrix[0] if vector_id != -1]\n\n        documents = self.get_documents_by_vector_ids(vector_ids_for_query, index=index)\n\n        #assign query score to each document\n        scores_for_vector_ids: Dict[str, float] = {str(v_id): s for v_id, s in zip(vector_id_matrix[0], score_matrix[0])}\n        for doc in documents:\n            doc.score = scores_for_vector_ids[doc.meta[\"vector_id\"]]  # type: ignore\n            doc.probability = (doc.score + 1) / 2\n        return documents\n\n    def save(self, file_path: Union[str, Path]):\n        \"\"\"\n        Save FAISS Index to the specified file.\n\n        :param file_path: Path to save to.\n        :return: None\n        \"\"\"\n        faiss.write_index(self.faiss_index, str(file_path))\n\n    @classmethod\n    def load(\n            cls,\n            faiss_file_path: Union[str, Path],\n            sql_url: str,\n            index_buffer_size: int = 10_000,\n    ):\n        \"\"\"\n        Load a saved FAISS index from a file and connect to the SQL database.\n        Note: In order to have a correct mapping from FAISS to SQL,\n              make sure to use the same SQL DB that you used when calling `save()`.\n\n        :param faiss_file_path: Stored FAISS index file. Can be created via calling `save()`\n        :param sql_url: Connection string to the SQL database that contains your docs and metadata.\n        :param index_buffer_size: When working with large datasets, the ingestion process(FAISS + SQL) can be buffered in\n                                  smaller chunks to reduce memory footprint.\n        :return:\n        \"\"\"\n        \"\"\"\n        \"\"\"\n        faiss_index = faiss.read_index(str(faiss_file_path))\n        return cls(\n            faiss_index=faiss_index,\n            sql_url=sql_url,\n            index_buffer_size=index_buffer_size,\n            vector_dim=faiss_index.d\n        )\n\n", "path": "haystack/document_store/faiss.py"}]}