We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Crawler does not write JSON, but serializes the result dict to a string written in a .json file
First of all, great work on Haystack! It’s an incredible library and I really enjoy playing around with it!

I noticed an odd behavior of the Crawler as in this example:

```python
from haystack.connector import Crawler

crawler = Crawler()
# crawl Haystack docs, i.e. all pages that include haystack.deepset.ai/docs/
docs = crawler.crawl(urls=["https://haystack.deepset.ai/docs/latest/get_startedmd"],
    output_dir="crawled_files",
    filter_urls= ["haystack\.deepset\.ai\/docs\/"])
```

The resulting file looks like this:

```
{'meta': {'url': 'https://haystack.deepset.ai/docs/latest/get_startedmd'}, 'text': 'Knowledge...
```

which is not valid JSON, as described in the docs. The Crawler rather simply serializes the data dict to string and writes it into a JSON file.

A working import to load the result in a next step looks like this:

```python
import ast

# docs[0] being the first result from the crawl run
with open(docs[0], 'r', encoding='utf-8') as f:
    result = ast.literal_eval(f.read())
```

instead of `json.read( ... )`.

As I don’t have the overview of the entire lib and how the created text files are used across the different pipelines, I am hesitant to use propose a solution. So I am raising this as a slightly odd behavior for now. Happy to provide a fix though given guidance from other developers.

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `haystack/connector/crawler.py`
Content:
```
1 import logging
2 import re
3 from pathlib import Path
4 from urllib.parse import urlparse
5 from typing import List, Any, Optional, Dict, Tuple, Union
6 from haystack.schema import Document, BaseComponent
7 logger = logging.getLogger(__name__)
8 
9 
10 class Crawler(BaseComponent):
11     """
12     Crawl texts from a website so that we can use them later in Haystack as a corpus for search / question answering etc.
13 
14     **Example:**
15     ```python
16     |    from haystack.connector import Crawler
17     |
18     |    crawler = Crawler()
19     |    # crawl Haystack docs, i.e. all pages that include haystack.deepset.ai/docs/
20     |    docs = crawler.crawl(urls=["https://haystack.deepset.ai/docs/latest/get_startedmd"],
21     |                         output_dir="crawled_files",
22     |                         filter_urls= ["haystack\.deepset\.ai\/docs\/"])
23     ```
24     """
25 
26     outgoing_edges = 1
27 
28     def __init__(self, output_dir: str, urls: Optional[List[str]] = None, crawler_depth: int = 1,
29                  filter_urls: Optional[List] = None, overwrite_existing_files=True):
30         """
31         Init object with basic params for crawling (can be overwritten later).
32 
33         :param output_dir: Path for the directory to store files
34         :param urls: List of http(s) address(es) (can also be supplied later when calling crawl())
35         :param crawler_depth: How many sublinks to follow from the initial list of URLs. Current options:
36                               0: Only initial list of urls
37                               1: Follow links found on the initial URLs (but no further)
38         :param filter_urls: Optional list of regular expressions that the crawled URLs must comply with.
39                            All URLs not matching at least one of the regular expressions will be dropped.
40         :param overwrite_existing_files: Whether to overwrite existing files in output_dir with new content
41         """
42         try:
43             from webdriver_manager.chrome import ChromeDriverManager
44         except ImportError:
45             raise ImportError("Can't find package `webdriver-manager` \n"
46                               "You can install it via `pip install webdriver-manager`")
47 
48         try:
49             from selenium import webdriver
50         except ImportError:
51             raise ImportError("Can't find package `selenium` \n"
52                               "You can install it via `pip install selenium`")
53 
54         options = webdriver.chrome.options.Options()
55         options.add_argument('--headless')
56         self.driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)
57         self.urls = urls
58         self.output_dir = output_dir
59         self.crawler_depth = crawler_depth
60         self.filter_urls = filter_urls
61         self.overwrite_existing_files = overwrite_existing_files
62 
63     def crawl(self,  output_dir: Union[str, Path, None] = None,
64               urls: Optional[List[str]] = None,
65               crawler_depth: Optional[int] = None,
66               filter_urls: Optional[List] = None,
67               overwrite_existing_files: Optional[bool] = None) -> List[Path]:
68         """
69         Craw URL(s), extract the text from the HTML, create a Haystack Document object out of it and save it (one JSON
70         file per URL, including text and basic meta data).
71         You can optionally specify via `filter_urls` to only crawl URLs that match a certain pattern.
72         All parameters are optional here and only meant to overwrite instance attributes at runtime.
73         If no parameters are provided to this method, the instance attributes that were passed during __init__ will be used.
74 
75         :param output_dir: Path for the directory to store files
76         :param urls: List of http addresses or single http address
77         :param crawler_depth: How many sublinks to follow from the initial list of URLs. Current options:
78                               0: Only initial list of urls
79                               1: Follow links found on the initial URLs (but no further)
80         :param filter_urls: Optional list of regular expressions that the crawled URLs must comply with.
81                            All URLs not matching at least one of the regular expressions will be dropped.
82         :param overwrite_existing_files: Whether to overwrite existing files in output_dir with new content
83 
84         :return: List of paths where the crawled webpages got stored
85         """
86         # use passed params or fallback to instance attributes
87         urls = urls or self.urls
88         if urls is None:
89             raise ValueError("Got no urls to crawl. Set `urls` to a list of URLs in __init__(), crawl() or run(). `")
90         output_dir = output_dir or self.output_dir
91         filter_urls = filter_urls or self.filter_urls
92         if overwrite_existing_files is None:
93             overwrite_existing_files = self.overwrite_existing_files
94         if crawler_depth is None:
95             crawler_depth = self.crawler_depth
96 
97         output_dir = Path(output_dir)
98         if not output_dir.exists():
99             output_dir.mkdir(parents=True)
100 
101         is_not_empty = len(list(output_dir.rglob("*"))) > 0
102         if is_not_empty and not overwrite_existing_files:
103             logger.info(
104                 f"Found data stored in `{output_dir}`. Delete this first if you really want to fetch new data."
105             )
106             return []
107         else:
108             logger.info(f"Fetching from {urls} to `{output_dir}`")
109 
110             filepaths = []
111 
112             sub_links: Dict[str, List] = {}
113 
114             # don't go beyond the initial list of urls
115             if crawler_depth == 0:
116                 filepaths += self._write_to_files(urls, output_dir=output_dir)
117             # follow one level of sublinks
118             elif crawler_depth == 1:
119                 for url_ in urls:
120                     existed_links: List = list(sum(list(sub_links.values()), []))
121                     sub_links[url_] = list(self._extract_sublinks_from_url(base_url=url_, filter_urls=filter_urls,
122                                                                      existed_links=existed_links))
123                 for url in sub_links:
124                     filepaths += self._write_to_files(sub_links[url], output_dir=output_dir, base_url=url)
125 
126             return filepaths
127 
128     def _write_to_files(self, urls: List[str], output_dir: Path, base_url: str = None) -> List[Path]:
129         paths = []
130         for link in urls:
131             logger.info(f"writing contents from `{link}`")
132             self.driver.get(link)
133             el = self.driver.find_element_by_tag_name('body')
134             text = el.text
135 
136             link_split_values = link.replace('https://', '').split('/')
137             file_name = f"{'_'.join(link_split_values)}.json"
138             file_path = output_dir / file_name
139 
140             data = {}
141             data['meta'] = {'url': link}
142             if base_url:
143                 data['meta']['base_url'] = base_url
144             data['text'] = text
145             with open(file_path, 'w', encoding='utf-8') as f:
146                 f.write(str(data))
147             paths.append(file_path)
148 
149         return paths
150 
151     def run(self, output_dir: Union[str, Path, None] = None, urls: Optional[List[str]] = None, # type: ignore
152             crawler_depth: Optional[int] = None, filter_urls: Optional[List] = None, # type: ignore
153             overwrite_existing_files: Optional[bool] = None, **kwargs) -> Tuple[Dict, str]: # type: ignore
154         """
155         Method to be executed when the Crawler is used as a Node within a Haystack pipeline.
156 
157         :param output_dir: Path for the directory to store files
158         :param urls: List of http addresses or single http address
159         :param crawler_depth: How many sublinks to follow from the initial list of URLs. Current options:
160                               0: Only initial list of urls
161                               1: Follow links found on the initial URLs (but no further)
162         :param filter_urls: Optional list of regular expressions that the crawled URLs must comply with.
163                            All URLs not matching at least one of the regular expressions will be dropped.
164         :param overwrite_existing_files: Whether to overwrite existing files in output_dir with new content
165 
166         :return: Tuple({"paths": List of filepaths, ...}, Name of output edge)
167         """
168 
169         filepaths = self.crawl(urls=urls, output_dir=output_dir, crawler_depth=crawler_depth, filter_urls=filter_urls,
170                                overwrite_existing_files=overwrite_existing_files)
171         results = {"paths": filepaths}
172         results.update(**kwargs)
173         return results, "output_1"
174 
175     @staticmethod
176     def _is_internal_url(base_url: str, sub_link: str) -> bool:
177         base_url_ = urlparse(base_url)
178         sub_link_ = urlparse(sub_link)
179         return base_url_.scheme == sub_link_.scheme and base_url_.netloc == sub_link_.netloc
180 
181     @staticmethod
182     def _is_inpage_navigation(base_url: str, sub_link: str) -> bool:
183         base_url_ = urlparse(base_url)
184         sub_link_ = urlparse(sub_link)
185         return base_url_.path == sub_link_.path and base_url_.netloc == sub_link_.netloc
186 
187     def _extract_sublinks_from_url(self, base_url: str,
188                                    filter_urls: Optional[List] = None,
189                                   existed_links: List = None) -> set:
190         self.driver.get(base_url)
191         a_elements = self.driver.find_elements_by_tag_name('a')
192         sub_links = set()
193         if not (existed_links and base_url in existed_links):
194             if filter_urls:
195                 if re.compile('|'.join(filter_urls)).search(base_url):
196                     sub_links.add(base_url)
197 
198         for i in a_elements:
199             sub_link = i.get_attribute('href')
200             if not (existed_links and sub_link in existed_links):
201                 if self._is_internal_url(base_url=base_url, sub_link=sub_link) \
202                         and (not self._is_inpage_navigation(base_url=base_url, sub_link=sub_link)):
203                     if filter_urls:
204                         if re.compile('|'.join(filter_urls)).search(sub_link):
205                             sub_links.add(sub_link)
206                     else:
207                         sub_links.add(sub_link)
208 
209         return sub_links
210
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/haystack/connector/crawler.py b/haystack/connector/crawler.py
--- a/haystack/connector/crawler.py
+++ b/haystack/connector/crawler.py
@@ -1,3 +1,4 @@
+import json
 import logging
 import re
 from pathlib import Path
@@ -15,10 +16,9 @@
     ```python
     |    from haystack.connector import Crawler
     |
-    |    crawler = Crawler()
+    |    crawler = Crawler(output_dir="crawled_files")
     |    # crawl Haystack docs, i.e. all pages that include haystack.deepset.ai/docs/
     |    docs = crawler.crawl(urls=["https://haystack.deepset.ai/docs/latest/get_startedmd"],
-    |                         output_dir="crawled_files",
     |                         filter_urls= ["haystack\.deepset\.ai\/docs\/"])
     ```
     """
@@ -143,7 +143,7 @@
                 data['meta']['base_url'] = base_url
             data['text'] = text
             with open(file_path, 'w', encoding='utf-8') as f:
-                f.write(str(data))
+                json.dump(data, f)
             paths.append(file_path)
 
         return paths

{"golden_diff": "diff --git a/haystack/connector/crawler.py b/haystack/connector/crawler.py\n--- a/haystack/connector/crawler.py\n+++ b/haystack/connector/crawler.py\n@@ -1,3 +1,4 @@\n+import json\n import logging\n import re\n from pathlib import Path\n@@ -15,10 +16,9 @@\n     ```python\n     |    from haystack.connector import Crawler\n     |\n-    |    crawler = Crawler()\n+    |    crawler = Crawler(output_dir=\"crawled_files\")\n     |    # crawl Haystack docs, i.e. all pages that include haystack.deepset.ai/docs/\n     |    docs = crawler.crawl(urls=[\"https://haystack.deepset.ai/docs/latest/get_startedmd\"],\n-    |                         output_dir=\"crawled_files\",\n     |                         filter_urls= [\"haystack\\.deepset\\.ai\\/docs\\/\"])\n     ```\n     \"\"\"\n@@ -143,7 +143,7 @@\n                 data['meta']['base_url'] = base_url\n             data['text'] = text\n             with open(file_path, 'w', encoding='utf-8') as f:\n-                f.write(str(data))\n+                json.dump(data, f)\n             paths.append(file_path)\n \n         return paths\n", "issue": "Crawler does not write JSON, but serializes the result dict to a string written in a .json file\nFirst of all, great work on Haystack! It\u2019s an incredible library and I really enjoy playing around with it!\r\n\r\nI noticed an odd behavior of the Crawler as in this example:\r\n\r\n```python\r\nfrom haystack.connector import Crawler\r\n\r\ncrawler = Crawler()\r\n# crawl Haystack docs, i.e. all pages that include haystack.deepset.ai/docs/\r\ndocs = crawler.crawl(urls=[\"https://haystack.deepset.ai/docs/latest/get_startedmd\"],\r\n    output_dir=\"crawled_files\",\r\n    filter_urls= [\"haystack\\.deepset\\.ai\\/docs\\/\"])\r\n```\r\n\r\nThe resulting file looks like this:\r\n\r\n```\r\n{'meta': {'url': 'https://haystack.deepset.ai/docs/latest/get_startedmd'}, 'text': 'Knowledge...\r\n```\r\n\r\nwhich is not valid JSON, as described in the docs. The Crawler rather simply serializes the data dict to string and writes it into a JSON file.\r\n\r\nA working import to load the result in a next step looks like this:\r\n\r\n```python\r\nimport ast\r\n\r\n# docs[0] being the first result from the crawl run\r\nwith open(docs[0], 'r', encoding='utf-8') as f:\r\n    result = ast.literal_eval(f.read())\r\n```\r\n\r\ninstead of `json.read( ... )`.\r\n\r\nAs I don\u2019t have the overview of the entire lib and how the created text files are used across the different pipelines, I am hesitant to use propose a solution. So I am raising this as a slightly odd behavior for now. Happy to provide a fix though given guidance from other developers.\n", "before_files": [{"content": "import logging\nimport re\nfrom pathlib import Path\nfrom urllib.parse import urlparse\nfrom typing import List, Any, Optional, Dict, Tuple, Union\nfrom haystack.schema import Document, BaseComponent\nlogger = logging.getLogger(__name__)\n\n\nclass Crawler(BaseComponent):\n    \"\"\"\n    Crawl texts from a website so that we can use them later in Haystack as a corpus for search / question answering etc.\n\n    **Example:**\n    ```python\n    |    from haystack.connector import Crawler\n    |\n    |    crawler = Crawler()\n    |    # crawl Haystack docs, i.e. all pages that include haystack.deepset.ai/docs/\n    |    docs = crawler.crawl(urls=[\"https://haystack.deepset.ai/docs/latest/get_startedmd\"],\n    |                         output_dir=\"crawled_files\",\n    |                         filter_urls= [\"haystack\\.deepset\\.ai\\/docs\\/\"])\n    ```\n    \"\"\"\n\n    outgoing_edges = 1\n\n    def __init__(self, output_dir: str, urls: Optional[List[str]] = None, crawler_depth: int = 1,\n                 filter_urls: Optional[List] = None, overwrite_existing_files=True):\n        \"\"\"\n        Init object with basic params for crawling (can be overwritten later).\n\n        :param output_dir: Path for the directory to store files\n        :param urls: List of http(s) address(es) (can also be supplied later when calling crawl())\n        :param crawler_depth: How many sublinks to follow from the initial list of URLs. Current options:\n                              0: Only initial list of urls\n                              1: Follow links found on the initial URLs (but no further)\n        :param filter_urls: Optional list of regular expressions that the crawled URLs must comply with.\n                           All URLs not matching at least one of the regular expressions will be dropped.\n        :param overwrite_existing_files: Whether to overwrite existing files in output_dir with new content\n        \"\"\"\n        try:\n            from webdriver_manager.chrome import ChromeDriverManager\n        except ImportError:\n            raise ImportError(\"Can't find package `webdriver-manager` \\n\"\n                              \"You can install it via `pip install webdriver-manager`\")\n\n        try:\n            from selenium import webdriver\n        except ImportError:\n            raise ImportError(\"Can't find package `selenium` \\n\"\n                              \"You can install it via `pip install selenium`\")\n\n        options = webdriver.chrome.options.Options()\n        options.add_argument('--headless')\n        self.driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)\n        self.urls = urls\n        self.output_dir = output_dir\n        self.crawler_depth = crawler_depth\n        self.filter_urls = filter_urls\n        self.overwrite_existing_files = overwrite_existing_files\n\n    def crawl(self,  output_dir: Union[str, Path, None] = None,\n              urls: Optional[List[str]] = None,\n              crawler_depth: Optional[int] = None,\n              filter_urls: Optional[List] = None,\n              overwrite_existing_files: Optional[bool] = None) -> List[Path]:\n        \"\"\"\n        Craw URL(s), extract the text from the HTML, create a Haystack Document object out of it and save it (one JSON\n        file per URL, including text and basic meta data).\n        You can optionally specify via `filter_urls` to only crawl URLs that match a certain pattern.\n        All parameters are optional here and only meant to overwrite instance attributes at runtime.\n        If no parameters are provided to this method, the instance attributes that were passed during __init__ will be used.\n\n        :param output_dir: Path for the directory to store files\n        :param urls: List of http addresses or single http address\n        :param crawler_depth: How many sublinks to follow from the initial list of URLs. Current options:\n                              0: Only initial list of urls\n                              1: Follow links found on the initial URLs (but no further)\n        :param filter_urls: Optional list of regular expressions that the crawled URLs must comply with.\n                           All URLs not matching at least one of the regular expressions will be dropped.\n        :param overwrite_existing_files: Whether to overwrite existing files in output_dir with new content\n\n        :return: List of paths where the crawled webpages got stored\n        \"\"\"\n        # use passed params or fallback to instance attributes\n        urls = urls or self.urls\n        if urls is None:\n            raise ValueError(\"Got no urls to crawl. Set `urls` to a list of URLs in __init__(), crawl() or run(). `\")\n        output_dir = output_dir or self.output_dir\n        filter_urls = filter_urls or self.filter_urls\n        if overwrite_existing_files is None:\n            overwrite_existing_files = self.overwrite_existing_files\n        if crawler_depth is None:\n            crawler_depth = self.crawler_depth\n\n        output_dir = Path(output_dir)\n        if not output_dir.exists():\n            output_dir.mkdir(parents=True)\n\n        is_not_empty = len(list(output_dir.rglob(\"*\"))) > 0\n        if is_not_empty and not overwrite_existing_files:\n            logger.info(\n                f\"Found data stored in `{output_dir}`. Delete this first if you really want to fetch new data.\"\n            )\n            return []\n        else:\n            logger.info(f\"Fetching from {urls} to `{output_dir}`\")\n\n            filepaths = []\n\n            sub_links: Dict[str, List] = {}\n\n            # don't go beyond the initial list of urls\n            if crawler_depth == 0:\n                filepaths += self._write_to_files(urls, output_dir=output_dir)\n            # follow one level of sublinks\n            elif crawler_depth == 1:\n                for url_ in urls:\n                    existed_links: List = list(sum(list(sub_links.values()), []))\n                    sub_links[url_] = list(self._extract_sublinks_from_url(base_url=url_, filter_urls=filter_urls,\n                                                                     existed_links=existed_links))\n                for url in sub_links:\n                    filepaths += self._write_to_files(sub_links[url], output_dir=output_dir, base_url=url)\n\n            return filepaths\n\n    def _write_to_files(self, urls: List[str], output_dir: Path, base_url: str = None) -> List[Path]:\n        paths = []\n        for link in urls:\n            logger.info(f\"writing contents from `{link}`\")\n            self.driver.get(link)\n            el = self.driver.find_element_by_tag_name('body')\n            text = el.text\n\n            link_split_values = link.replace('https://', '').split('/')\n            file_name = f\"{'_'.join(link_split_values)}.json\"\n            file_path = output_dir / file_name\n\n            data = {}\n            data['meta'] = {'url': link}\n            if base_url:\n                data['meta']['base_url'] = base_url\n            data['text'] = text\n            with open(file_path, 'w', encoding='utf-8') as f:\n                f.write(str(data))\n            paths.append(file_path)\n\n        return paths\n\n    def run(self, output_dir: Union[str, Path, None] = None, urls: Optional[List[str]] = None, # type: ignore\n            crawler_depth: Optional[int] = None, filter_urls: Optional[List] = None, # type: ignore\n            overwrite_existing_files: Optional[bool] = None, **kwargs) -> Tuple[Dict, str]: # type: ignore\n        \"\"\"\n        Method to be executed when the Crawler is used as a Node within a Haystack pipeline.\n\n        :param output_dir: Path for the directory to store files\n        :param urls: List of http addresses or single http address\n        :param crawler_depth: How many sublinks to follow from the initial list of URLs. Current options:\n                              0: Only initial list of urls\n                              1: Follow links found on the initial URLs (but no further)\n        :param filter_urls: Optional list of regular expressions that the crawled URLs must comply with.\n                           All URLs not matching at least one of the regular expressions will be dropped.\n        :param overwrite_existing_files: Whether to overwrite existing files in output_dir with new content\n\n        :return: Tuple({\"paths\": List of filepaths, ...}, Name of output edge)\n        \"\"\"\n\n        filepaths = self.crawl(urls=urls, output_dir=output_dir, crawler_depth=crawler_depth, filter_urls=filter_urls,\n                               overwrite_existing_files=overwrite_existing_files)\n        results = {\"paths\": filepaths}\n        results.update(**kwargs)\n        return results, \"output_1\"\n\n    @staticmethod\n    def _is_internal_url(base_url: str, sub_link: str) -> bool:\n        base_url_ = urlparse(base_url)\n        sub_link_ = urlparse(sub_link)\n        return base_url_.scheme == sub_link_.scheme and base_url_.netloc == sub_link_.netloc\n\n    @staticmethod\n    def _is_inpage_navigation(base_url: str, sub_link: str) -> bool:\n        base_url_ = urlparse(base_url)\n        sub_link_ = urlparse(sub_link)\n        return base_url_.path == sub_link_.path and base_url_.netloc == sub_link_.netloc\n\n    def _extract_sublinks_from_url(self, base_url: str,\n                                   filter_urls: Optional[List] = None,\n                                  existed_links: List = None) -> set:\n        self.driver.get(base_url)\n        a_elements = self.driver.find_elements_by_tag_name('a')\n        sub_links = set()\n        if not (existed_links and base_url in existed_links):\n            if filter_urls:\n                if re.compile('|'.join(filter_urls)).search(base_url):\n                    sub_links.add(base_url)\n\n        for i in a_elements:\n            sub_link = i.get_attribute('href')\n            if not (existed_links and sub_link in existed_links):\n                if self._is_internal_url(base_url=base_url, sub_link=sub_link) \\\n                        and (not self._is_inpage_navigation(base_url=base_url, sub_link=sub_link)):\n                    if filter_urls:\n                        if re.compile('|'.join(filter_urls)).search(sub_link):\n                            sub_links.add(sub_link)\n                    else:\n                        sub_links.add(sub_link)\n\n        return sub_links\n", "path": "haystack/connector/crawler.py"}], "after_files": [{"content": "import json\nimport logging\nimport re\nfrom pathlib import Path\nfrom urllib.parse import urlparse\nfrom typing import List, Any, Optional, Dict, Tuple, Union\nfrom haystack.schema import Document, BaseComponent\nlogger = logging.getLogger(__name__)\n\n\nclass Crawler(BaseComponent):\n    \"\"\"\n    Crawl texts from a website so that we can use them later in Haystack as a corpus for search / question answering etc.\n\n    **Example:**\n    ```python\n    |    from haystack.connector import Crawler\n    |\n    |    crawler = Crawler(output_dir=\"crawled_files\")\n    |    # crawl Haystack docs, i.e. all pages that include haystack.deepset.ai/docs/\n    |    docs = crawler.crawl(urls=[\"https://haystack.deepset.ai/docs/latest/get_startedmd\"],\n    |                         filter_urls= [\"haystack\\.deepset\\.ai\\/docs\\/\"])\n    ```\n    \"\"\"\n\n    outgoing_edges = 1\n\n    def __init__(self, output_dir: str, urls: Optional[List[str]] = None, crawler_depth: int = 1,\n                 filter_urls: Optional[List] = None, overwrite_existing_files=True):\n        \"\"\"\n        Init object with basic params for crawling (can be overwritten later).\n\n        :param output_dir: Path for the directory to store files\n        :param urls: List of http(s) address(es) (can also be supplied later when calling crawl())\n        :param crawler_depth: How many sublinks to follow from the initial list of URLs. Current options:\n                              0: Only initial list of urls\n                              1: Follow links found on the initial URLs (but no further)\n        :param filter_urls: Optional list of regular expressions that the crawled URLs must comply with.\n                           All URLs not matching at least one of the regular expressions will be dropped.\n        :param overwrite_existing_files: Whether to overwrite existing files in output_dir with new content\n        \"\"\"\n        try:\n            from webdriver_manager.chrome import ChromeDriverManager\n        except ImportError:\n            raise ImportError(\"Can't find package `webdriver-manager` \\n\"\n                              \"You can install it via `pip install webdriver-manager`\")\n\n        try:\n            from selenium import webdriver\n        except ImportError:\n            raise ImportError(\"Can't find package `selenium` \\n\"\n                              \"You can install it via `pip install selenium`\")\n\n        options = webdriver.chrome.options.Options()\n        options.add_argument('--headless')\n        self.driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)\n        self.urls = urls\n        self.output_dir = output_dir\n        self.crawler_depth = crawler_depth\n        self.filter_urls = filter_urls\n        self.overwrite_existing_files = overwrite_existing_files\n\n    def crawl(self,  output_dir: Union[str, Path, None] = None,\n              urls: Optional[List[str]] = None,\n              crawler_depth: Optional[int] = None,\n              filter_urls: Optional[List] = None,\n              overwrite_existing_files: Optional[bool] = None) -> List[Path]:\n        \"\"\"\n        Craw URL(s), extract the text from the HTML, create a Haystack Document object out of it and save it (one JSON\n        file per URL, including text and basic meta data).\n        You can optionally specify via `filter_urls` to only crawl URLs that match a certain pattern.\n        All parameters are optional here and only meant to overwrite instance attributes at runtime.\n        If no parameters are provided to this method, the instance attributes that were passed during __init__ will be used.\n\n        :param output_dir: Path for the directory to store files\n        :param urls: List of http addresses or single http address\n        :param crawler_depth: How many sublinks to follow from the initial list of URLs. Current options:\n                              0: Only initial list of urls\n                              1: Follow links found on the initial URLs (but no further)\n        :param filter_urls: Optional list of regular expressions that the crawled URLs must comply with.\n                           All URLs not matching at least one of the regular expressions will be dropped.\n        :param overwrite_existing_files: Whether to overwrite existing files in output_dir with new content\n\n        :return: List of paths where the crawled webpages got stored\n        \"\"\"\n        # use passed params or fallback to instance attributes\n        urls = urls or self.urls\n        if urls is None:\n            raise ValueError(\"Got no urls to crawl. Set `urls` to a list of URLs in __init__(), crawl() or run(). `\")\n        output_dir = output_dir or self.output_dir\n        filter_urls = filter_urls or self.filter_urls\n        if overwrite_existing_files is None:\n            overwrite_existing_files = self.overwrite_existing_files\n        if crawler_depth is None:\n            crawler_depth = self.crawler_depth\n\n        output_dir = Path(output_dir)\n        if not output_dir.exists():\n            output_dir.mkdir(parents=True)\n\n        is_not_empty = len(list(output_dir.rglob(\"*\"))) > 0\n        if is_not_empty and not overwrite_existing_files:\n            logger.info(\n                f\"Found data stored in `{output_dir}`. Delete this first if you really want to fetch new data.\"\n            )\n            return []\n        else:\n            logger.info(f\"Fetching from {urls} to `{output_dir}`\")\n\n            filepaths = []\n\n            sub_links: Dict[str, List] = {}\n\n            # don't go beyond the initial list of urls\n            if crawler_depth == 0:\n                filepaths += self._write_to_files(urls, output_dir=output_dir)\n            # follow one level of sublinks\n            elif crawler_depth == 1:\n                for url_ in urls:\n                    existed_links: List = list(sum(list(sub_links.values()), []))\n                    sub_links[url_] = list(self._extract_sublinks_from_url(base_url=url_, filter_urls=filter_urls,\n                                                                     existed_links=existed_links))\n                for url in sub_links:\n                    filepaths += self._write_to_files(sub_links[url], output_dir=output_dir, base_url=url)\n\n            return filepaths\n\n    def _write_to_files(self, urls: List[str], output_dir: Path, base_url: str = None) -> List[Path]:\n        paths = []\n        for link in urls:\n            logger.info(f\"writing contents from `{link}`\")\n            self.driver.get(link)\n            el = self.driver.find_element_by_tag_name('body')\n            text = el.text\n\n            link_split_values = link.replace('https://', '').split('/')\n            file_name = f\"{'_'.join(link_split_values)}.json\"\n            file_path = output_dir / file_name\n\n            data = {}\n            data['meta'] = {'url': link}\n            if base_url:\n                data['meta']['base_url'] = base_url\n            data['text'] = text\n            with open(file_path, 'w', encoding='utf-8') as f:\n                json.dump(data, f)\n            paths.append(file_path)\n\n        return paths\n\n    def run(self, output_dir: Union[str, Path, None] = None, urls: Optional[List[str]] = None, # type: ignore\n            crawler_depth: Optional[int] = None, filter_urls: Optional[List] = None, # type: ignore\n            overwrite_existing_files: Optional[bool] = None, **kwargs) -> Tuple[Dict, str]: # type: ignore\n        \"\"\"\n        Method to be executed when the Crawler is used as a Node within a Haystack pipeline.\n\n        :param output_dir: Path for the directory to store files\n        :param urls: List of http addresses or single http address\n        :param crawler_depth: How many sublinks to follow from the initial list of URLs. Current options:\n                              0: Only initial list of urls\n                              1: Follow links found on the initial URLs (but no further)\n        :param filter_urls: Optional list of regular expressions that the crawled URLs must comply with.\n                           All URLs not matching at least one of the regular expressions will be dropped.\n        :param overwrite_existing_files: Whether to overwrite existing files in output_dir with new content\n\n        :return: Tuple({\"paths\": List of filepaths, ...}, Name of output edge)\n        \"\"\"\n\n        filepaths = self.crawl(urls=urls, output_dir=output_dir, crawler_depth=crawler_depth, filter_urls=filter_urls,\n                               overwrite_existing_files=overwrite_existing_files)\n        results = {\"paths\": filepaths}\n        results.update(**kwargs)\n        return results, \"output_1\"\n\n    @staticmethod\n    def _is_internal_url(base_url: str, sub_link: str) -> bool:\n        base_url_ = urlparse(base_url)\n        sub_link_ = urlparse(sub_link)\n        return base_url_.scheme == sub_link_.scheme and base_url_.netloc == sub_link_.netloc\n\n    @staticmethod\n    def _is_inpage_navigation(base_url: str, sub_link: str) -> bool:\n        base_url_ = urlparse(base_url)\n        sub_link_ = urlparse(sub_link)\n        return base_url_.path == sub_link_.path and base_url_.netloc == sub_link_.netloc\n\n    def _extract_sublinks_from_url(self, base_url: str,\n                                   filter_urls: Optional[List] = None,\n                                  existed_links: List = None) -> set:\n        self.driver.get(base_url)\n        a_elements = self.driver.find_elements_by_tag_name('a')\n        sub_links = set()\n        if not (existed_links and base_url in existed_links):\n            if filter_urls:\n                if re.compile('|'.join(filter_urls)).search(base_url):\n                    sub_links.add(base_url)\n\n        for i in a_elements:\n            sub_link = i.get_attribute('href')\n            if not (existed_links and sub_link in existed_links):\n                if self._is_internal_url(base_url=base_url, sub_link=sub_link) \\\n                        and (not self._is_inpage_navigation(base_url=base_url, sub_link=sub_link)):\n                    if filter_urls:\n                        if re.compile('|'.join(filter_urls)).search(sub_link):\n                            sub_links.add(sub_link)\n                    else:\n                        sub_links.add(sub_link)\n\n        return sub_links\n", "path": "haystack/connector/crawler.py"}]}