We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Datasource analysing refactoring
Couple thoughts about analysing:
History of analysing in 'upload ds' route:
1. Let come back to time when we work just with 'file' datasources. We dont run analysis after file uploaded. We do it if user want see analysis, or learn predictor.
2. We want add new feature: filter/sort for data in datasource in Scout 'data preview' screen. Doing in pandas.DataFrame is slow, so we decide use sqllite database. But how we can create database, if we dont know file columns types? If just copy csv to sqllite-db, then all  columns will be text, and all filters in Scout will be text. So we need analyse datasource right after it uploaded.
3. We add database-based datasources without any changes in datasource upload flow. How this flow looks now: route get from user query string and 'integration' name. Analyse process run. After it done, sqllite db created. 
Problems in this flow:
1. analysis can take long time
2. we save in sqllite db copy of data from real db.
3. if user will train predictor from that datasource after some time, predictor will be trained from current database data, not from sqllite data. But in 'datasource preview' will be still data from sqllite.
4. each time (not each really, it has some timeout), when user open 'datasource analysis', datasource analysis will be re-run.

How i see solve:
1. add statues for datasources. In this case, we can set status='analysing' for uploading datasource and make analysis in background. For datasources with this status should be disabled all interactions in Scout.
2. save datasource analysis in file in datasource folder. User should see date/time of last analysis and button 're-analyse' (probably just for db-based datasource, for file-based is make no sense). In this case we can open 'datasource analysis' screen without waiting.
3. dont make sqllite db for db-based datasources. User already have his data in database, we dont want make full copy of it.
4. point above mean we should disable 'datasource preview' screen for db-based datasources. Why: it will be difficult to make filter/sorting/pagination for that data. At least at this moment.


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `mindsdb/api/http/namespaces/datasource.py`
Content:
```
1 import datetime
2 import os
3 import threading
4 import tempfile
5 
6 import multipart
7 
8 import mindsdb
9 from dateutil.parser import parse
10 from flask import request, send_file
11 from flask_restx import Resource, abort     # 'abort' using to return errors as json: {'message': 'error text'}
12 from flask import current_app as ca
13 
14 from mindsdb.interfaces.datastore.sqlite_helpers import *
15 from mindsdb.api.http.namespaces.configs.datasources import ns_conf
16 from mindsdb.api.http.namespaces.entitites.datasources.datasource import (
17     datasource_metadata,
18     put_datasource_params
19 )
20 from mindsdb.api.http.namespaces.entitites.datasources.datasource_data import (
21     get_datasource_rows_params,
22     datasource_rows_metadata
23 )
24 from mindsdb.api.http.namespaces.entitites.datasources.datasource_files import (
25     put_datasource_file_params
26 )
27 from mindsdb.api.http.namespaces.entitites.datasources.datasource_missed_files import (
28     datasource_missed_files_metadata,
29     get_datasource_missed_files_params
30 )
31 
32 
33 @ns_conf.route('/')
34 class DatasourcesList(Resource):
35     @ns_conf.doc('get_datasources_list')
36     @ns_conf.marshal_list_with(datasource_metadata)
37     def get(self):
38         '''List all datasources'''
39         return ca.default_store.get_datasources()
40 
41 
42 @ns_conf.route('/<name>')
43 @ns_conf.param('name', 'Datasource name')
44 class Datasource(Resource):
45     @ns_conf.doc('get_datasource')
46     @ns_conf.marshal_with(datasource_metadata)
47     def get(self, name):
48         '''return datasource metadata'''
49         ds = ca.default_store.get_datasource(name)
50         if ds is not None:
51             return ds
52         return '', 404
53 
54     @ns_conf.doc('delete_datasource')
55     def delete(self, name):
56         '''delete datasource'''
57         try:
58             ca.default_store.delete_datasource(name)
59         except Exception as e:
60             print(e)
61             abort(400, str(e))
62         return '', 200
63 
64     @ns_conf.doc('put_datasource', params=put_datasource_params)
65     @ns_conf.marshal_with(datasource_metadata)
66     def put(self, name):
67         '''add new datasource'''
68         data = {}
69 
70         def on_field(field):
71             print(f'\n\n{field}\n\n')
72             name = field.field_name.decode()
73             value = field.value.decode()
74             data[name] = value
75 
76         file_object = None
77 
78         def on_file(file):
79             nonlocal file_object
80             data['file'] = file.file_name.decode()
81             file_object = file.file_object
82 
83         temp_dir_path = tempfile.mkdtemp(prefix='datasource_file_')
84 
85         if request.headers['Content-Type'].startswith('multipart/form-data'):
86             parser = multipart.create_form_parser(
87                 headers=request.headers,
88                 on_field=on_field,
89                 on_file=on_file,
90                 config={
91                     'UPLOAD_DIR': temp_dir_path.encode(),    # bytes required
92                     'UPLOAD_KEEP_FILENAME': True,
93                     'UPLOAD_KEEP_EXTENSIONS': True,
94                     'MAX_MEMORY_FILE_SIZE': 0
95                 }
96             )
97 
98             while True:
99                 chunk = request.stream.read(8192)
100                 if not chunk:
101                     break
102                 parser.write(chunk)
103             parser.finalize()
104             parser.close()
105 
106             if file_object is not None and not file_object.closed:
107                 file_object.close()
108         else:
109             data = request.json
110 
111         if 'query' in data:
112             source_type = request.json['integration_id']
113             ca.default_store.save_datasource(name, source_type, request.json)
114             os.rmdir(temp_dir_path)
115             return ca.default_store.get_datasource(name)
116 
117         ds_name = data['name'] if 'name' in data else name
118         source = data['source'] if 'source' in data else name
119         source_type = data['source_type']
120 
121         if source_type == 'file':
122             file_path = os.path.join(temp_dir_path, data['file'])
123         else:
124             file_path = None
125 
126         ca.default_store.save_datasource(ds_name, source_type, source, file_path)
127         os.rmdir(temp_dir_path)
128 
129         return ca.default_store.get_datasource(ds_name)
130 
131 
132 ds_analysis = {}
133 
134 
135 def analyzing_thread(name, default_store):
136     global ds_analysis
137     ds_analysis[name] = None
138     ds = default_store.get_datasource(name)
139     analysis = default_store.get_analysis(ds['name'])
140     ds_analysis[name] = {
141         'created_at': datetime.datetime.utcnow(),
142         'data': analysis
143     }
144 
145 
146 @ns_conf.route('/<name>/analyze')
147 @ns_conf.param('name', 'Datasource name')
148 class Analyze(Resource):
149     @ns_conf.doc('analyse_dataset')
150     def get(self, name):
151         global ds_analysis
152         if name in ds_analysis:
153             if ds_analysis[name] is None:
154                 return {'status': 'analyzing'}, 200
155             elif (datetime.datetime.utcnow() - ds_analysis[name]['created_at']) > datetime.timedelta(seconds=3600):
156                 del ds_analysis[name]
157             else:
158                 analysis = ds_analysis[name]['data']
159                 return analysis, 200
160 
161         ds = ca.default_store.get_datasource(name)
162         if ds is None:
163             print('No valid datasource given')
164             abort(400, 'No valid datasource given')
165 
166         x = threading.Thread(target=analyzing_thread, args=(name, ca.default_store))
167         x.start()
168         return {'status': 'analyzing'}, 200
169 
170 
171 @ns_conf.route('/<name>/analyze_subset')
172 @ns_conf.param('name', 'Datasource name')
173 class AnalyzeSubset(Resource):
174     @ns_conf.doc('analyse_datasubset')
175     def get(self, name):
176         ds = ca.default_store.get_datasource(name)
177         if ds is None:
178             print('No valid datasource given')
179             abort(400, 'No valid datasource given')
180 
181         where = []
182         for key, value in request.args.items():
183             if key.startswith('filter'):
184                 param = parse_filter(key, value)
185                 if param is None:
186                     abort(400, f'Not valid filter "{key}"')
187                 where.append(param)
188 
189         data_dict = ca.default_store.get_data(ds['name'], where)
190 
191         if data_dict['rowcount'] == 0:
192             return abort(400, 'Empty dataset after filters applying')
193 
194         return get_analysis(pd.DataFrame(data_dict['data'])), 200
195 
196 
197 @ns_conf.route('/<name>/data/')
198 @ns_conf.param('name', 'Datasource name')
199 class DatasourceData(Resource):
200     @ns_conf.doc('get_datasource_data', params=get_datasource_rows_params)
201     @ns_conf.marshal_with(datasource_rows_metadata)
202     def get(self, name):
203         '''return data rows'''
204         ds = ca.default_store.get_datasource(name)
205         if ds is None:
206             abort(400, 'No valid datasource given')
207 
208         params = {
209             'page[size]': None,
210             'page[offset]': None
211         }
212         where = []
213         for key, value in request.args.items():
214             if key == 'page[size]':
215                 params['page[size]'] = int(value)
216             if key == 'page[offset]':
217                 params['page[offset]'] = int(value)
218             elif key.startswith('filter'):
219                 param = parse_filter(key, value)
220                 if param is None:
221                     abort(400, f'Not valid filter "{key}"')
222                 where.append(param)
223 
224         data_dict = ca.default_store.get_data(name, where, params['page[size]'], params['page[offset]'])
225 
226         return data_dict, 200
227 
228 
229 @ns_conf.route('/<name>/download')
230 @ns_conf.param('name', 'Datasource name')
231 class DatasourceMissedFilesDownload(Resource):
232     @ns_conf.doc('get_datasource_download')
233     def get(self, name):
234         '''download uploaded file'''
235         ds = ca.default_store.get_datasource(name)
236         if not ds:
237             abort(404, "{} not found".format(name))
238         if not os.path.exists(ds['source']):
239             abort(404, "{} not found".format(name))
240 
241         return send_file(os.path.abspath(ds['source']), as_attachment=True)
242
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/mindsdb/api/http/namespaces/datasource.py b/mindsdb/api/http/namespaces/datasource.py
--- a/mindsdb/api/http/namespaces/datasource.py
+++ b/mindsdb/api/http/namespaces/datasource.py
@@ -152,8 +152,6 @@
         if name in ds_analysis:
             if ds_analysis[name] is None:
                 return {'status': 'analyzing'}, 200
-            elif (datetime.datetime.utcnow() - ds_analysis[name]['created_at']) > datetime.timedelta(seconds=3600):
-                del ds_analysis[name]
             else:
                 analysis = ds_analysis[name]['data']
                 return analysis, 200
@@ -167,6 +165,27 @@
         x.start()
         return {'status': 'analyzing'}, 200
 
+@ns_conf.route('/<name>/analyze_refresh')
+@ns_conf.param('name', 'Datasource name')
+class Analyze(Resource):
+    @ns_conf.doc('analyze_refresh_dataset')
+    def get(self, name):
+        global ds_analysis
+        if name in ds_analysis:
+            if ds_analysis[name] is None:
+                return {'status': 'analyzing'}, 200
+            else:
+                del ds_analysis[name]
+
+        ds = ca.default_store.get_datasource(name)
+        if ds is None:
+            print('No valid datasource given')
+            abort(400, 'No valid datasource given')
+
+        x = threading.Thread(target=analyzing_thread, args=(name, ca.default_store))
+        x.start()
+        return {'status': 'analyzing'}, 200
+
 
 @ns_conf.route('/<name>/analyze_subset')
 @ns_conf.param('name', 'Datasource name')

{"golden_diff": "diff --git a/mindsdb/api/http/namespaces/datasource.py b/mindsdb/api/http/namespaces/datasource.py\n--- a/mindsdb/api/http/namespaces/datasource.py\n+++ b/mindsdb/api/http/namespaces/datasource.py\n@@ -152,8 +152,6 @@\n         if name in ds_analysis:\n             if ds_analysis[name] is None:\n                 return {'status': 'analyzing'}, 200\n-            elif (datetime.datetime.utcnow() - ds_analysis[name]['created_at']) > datetime.timedelta(seconds=3600):\n-                del ds_analysis[name]\n             else:\n                 analysis = ds_analysis[name]['data']\n                 return analysis, 200\n@@ -167,6 +165,27 @@\n         x.start()\n         return {'status': 'analyzing'}, 200\n \n+@ns_conf.route('/<name>/analyze_refresh')\n+@ns_conf.param('name', 'Datasource name')\n+class Analyze(Resource):\n+    @ns_conf.doc('analyze_refresh_dataset')\n+    def get(self, name):\n+        global ds_analysis\n+        if name in ds_analysis:\n+            if ds_analysis[name] is None:\n+                return {'status': 'analyzing'}, 200\n+            else:\n+                del ds_analysis[name]\n+\n+        ds = ca.default_store.get_datasource(name)\n+        if ds is None:\n+            print('No valid datasource given')\n+            abort(400, 'No valid datasource given')\n+\n+        x = threading.Thread(target=analyzing_thread, args=(name, ca.default_store))\n+        x.start()\n+        return {'status': 'analyzing'}, 200\n+\n \n @ns_conf.route('/<name>/analyze_subset')\n @ns_conf.param('name', 'Datasource name')\n", "issue": "Datasource analysing refactoring\nCouple thoughts about analysing:\r\nHistory of analysing in 'upload ds' route:\r\n1. Let come back to time when we work just with 'file' datasources. We dont run analysis after file uploaded. We do it if user want see analysis, or learn predictor.\r\n2. We want add new feature: filter/sort for data in datasource in Scout 'data preview' screen. Doing in pandas.DataFrame is slow, so we decide use sqllite database. But how we can create database, if we dont know file columns types? If just copy csv to sqllite-db, then all  columns will be text, and all filters in Scout will be text. So we need analyse datasource right after it uploaded.\r\n3. We add database-based datasources without any changes in datasource upload flow. How this flow looks now: route get from user query string and 'integration' name. Analyse process run. After it done, sqllite db created. \r\nProblems in this flow:\r\n1. analysis can take long time\r\n2. we save in sqllite db copy of data from real db.\r\n3. if user will train predictor from that datasource after some time, predictor will be trained from current database data, not from sqllite data. But in 'datasource preview' will be still data from sqllite.\r\n4. each time (not each really, it has some timeout), when user open 'datasource analysis', datasource analysis will be re-run.\r\n\r\nHow i see solve:\r\n1. add statues for datasources. In this case, we can set status='analysing' for uploading datasource and make analysis in background. For datasources with this status should be disabled all interactions in Scout.\r\n2. save datasource analysis in file in datasource folder. User should see date/time of last analysis and button 're-analyse' (probably just for db-based datasource, for file-based is make no sense). In this case we can open 'datasource analysis' screen without waiting.\r\n3. dont make sqllite db for db-based datasources. User already have his data in database, we dont want make full copy of it.\r\n4. point above mean we should disable 'datasource preview' screen for db-based datasources. Why: it will be difficult to make filter/sorting/pagination for that data. At least at this moment.\r\n\n", "before_files": [{"content": "import datetime\nimport os\nimport threading\nimport tempfile\n\nimport multipart\n\nimport mindsdb\nfrom dateutil.parser import parse\nfrom flask import request, send_file\nfrom flask_restx import Resource, abort     # 'abort' using to return errors as json: {'message': 'error text'}\nfrom flask import current_app as ca\n\nfrom mindsdb.interfaces.datastore.sqlite_helpers import *\nfrom mindsdb.api.http.namespaces.configs.datasources import ns_conf\nfrom mindsdb.api.http.namespaces.entitites.datasources.datasource import (\n    datasource_metadata,\n    put_datasource_params\n)\nfrom mindsdb.api.http.namespaces.entitites.datasources.datasource_data import (\n    get_datasource_rows_params,\n    datasource_rows_metadata\n)\nfrom mindsdb.api.http.namespaces.entitites.datasources.datasource_files import (\n    put_datasource_file_params\n)\nfrom mindsdb.api.http.namespaces.entitites.datasources.datasource_missed_files import (\n    datasource_missed_files_metadata,\n    get_datasource_missed_files_params\n)\n\n\n@ns_conf.route('/')\nclass DatasourcesList(Resource):\n    @ns_conf.doc('get_datasources_list')\n    @ns_conf.marshal_list_with(datasource_metadata)\n    def get(self):\n        '''List all datasources'''\n        return ca.default_store.get_datasources()\n\n\n@ns_conf.route('/<name>')\n@ns_conf.param('name', 'Datasource name')\nclass Datasource(Resource):\n    @ns_conf.doc('get_datasource')\n    @ns_conf.marshal_with(datasource_metadata)\n    def get(self, name):\n        '''return datasource metadata'''\n        ds = ca.default_store.get_datasource(name)\n        if ds is not None:\n            return ds\n        return '', 404\n\n    @ns_conf.doc('delete_datasource')\n    def delete(self, name):\n        '''delete datasource'''\n        try:\n            ca.default_store.delete_datasource(name)\n        except Exception as e:\n            print(e)\n            abort(400, str(e))\n        return '', 200\n\n    @ns_conf.doc('put_datasource', params=put_datasource_params)\n    @ns_conf.marshal_with(datasource_metadata)\n    def put(self, name):\n        '''add new datasource'''\n        data = {}\n\n        def on_field(field):\n            print(f'\\n\\n{field}\\n\\n')\n            name = field.field_name.decode()\n            value = field.value.decode()\n            data[name] = value\n\n        file_object = None\n\n        def on_file(file):\n            nonlocal file_object\n            data['file'] = file.file_name.decode()\n            file_object = file.file_object\n\n        temp_dir_path = tempfile.mkdtemp(prefix='datasource_file_')\n\n        if request.headers['Content-Type'].startswith('multipart/form-data'):\n            parser = multipart.create_form_parser(\n                headers=request.headers,\n                on_field=on_field,\n                on_file=on_file,\n                config={\n                    'UPLOAD_DIR': temp_dir_path.encode(),    # bytes required\n                    'UPLOAD_KEEP_FILENAME': True,\n                    'UPLOAD_KEEP_EXTENSIONS': True,\n                    'MAX_MEMORY_FILE_SIZE': 0\n                }\n            )\n\n            while True:\n                chunk = request.stream.read(8192)\n                if not chunk:\n                    break\n                parser.write(chunk)\n            parser.finalize()\n            parser.close()\n\n            if file_object is not None and not file_object.closed:\n                file_object.close()\n        else:\n            data = request.json\n\n        if 'query' in data:\n            source_type = request.json['integration_id']\n            ca.default_store.save_datasource(name, source_type, request.json)\n            os.rmdir(temp_dir_path)\n            return ca.default_store.get_datasource(name)\n\n        ds_name = data['name'] if 'name' in data else name\n        source = data['source'] if 'source' in data else name\n        source_type = data['source_type']\n\n        if source_type == 'file':\n            file_path = os.path.join(temp_dir_path, data['file'])\n        else:\n            file_path = None\n\n        ca.default_store.save_datasource(ds_name, source_type, source, file_path)\n        os.rmdir(temp_dir_path)\n\n        return ca.default_store.get_datasource(ds_name)\n\n\nds_analysis = {}\n\n\ndef analyzing_thread(name, default_store):\n    global ds_analysis\n    ds_analysis[name] = None\n    ds = default_store.get_datasource(name)\n    analysis = default_store.get_analysis(ds['name'])\n    ds_analysis[name] = {\n        'created_at': datetime.datetime.utcnow(),\n        'data': analysis\n    }\n\n\n@ns_conf.route('/<name>/analyze')\n@ns_conf.param('name', 'Datasource name')\nclass Analyze(Resource):\n    @ns_conf.doc('analyse_dataset')\n    def get(self, name):\n        global ds_analysis\n        if name in ds_analysis:\n            if ds_analysis[name] is None:\n                return {'status': 'analyzing'}, 200\n            elif (datetime.datetime.utcnow() - ds_analysis[name]['created_at']) > datetime.timedelta(seconds=3600):\n                del ds_analysis[name]\n            else:\n                analysis = ds_analysis[name]['data']\n                return analysis, 200\n\n        ds = ca.default_store.get_datasource(name)\n        if ds is None:\n            print('No valid datasource given')\n            abort(400, 'No valid datasource given')\n\n        x = threading.Thread(target=analyzing_thread, args=(name, ca.default_store))\n        x.start()\n        return {'status': 'analyzing'}, 200\n\n\n@ns_conf.route('/<name>/analyze_subset')\n@ns_conf.param('name', 'Datasource name')\nclass AnalyzeSubset(Resource):\n    @ns_conf.doc('analyse_datasubset')\n    def get(self, name):\n        ds = ca.default_store.get_datasource(name)\n        if ds is None:\n            print('No valid datasource given')\n            abort(400, 'No valid datasource given')\n\n        where = []\n        for key, value in request.args.items():\n            if key.startswith('filter'):\n                param = parse_filter(key, value)\n                if param is None:\n                    abort(400, f'Not valid filter \"{key}\"')\n                where.append(param)\n\n        data_dict = ca.default_store.get_data(ds['name'], where)\n\n        if data_dict['rowcount'] == 0:\n            return abort(400, 'Empty dataset after filters applying')\n\n        return get_analysis(pd.DataFrame(data_dict['data'])), 200\n\n\n@ns_conf.route('/<name>/data/')\n@ns_conf.param('name', 'Datasource name')\nclass DatasourceData(Resource):\n    @ns_conf.doc('get_datasource_data', params=get_datasource_rows_params)\n    @ns_conf.marshal_with(datasource_rows_metadata)\n    def get(self, name):\n        '''return data rows'''\n        ds = ca.default_store.get_datasource(name)\n        if ds is None:\n            abort(400, 'No valid datasource given')\n\n        params = {\n            'page[size]': None,\n            'page[offset]': None\n        }\n        where = []\n        for key, value in request.args.items():\n            if key == 'page[size]':\n                params['page[size]'] = int(value)\n            if key == 'page[offset]':\n                params['page[offset]'] = int(value)\n            elif key.startswith('filter'):\n                param = parse_filter(key, value)\n                if param is None:\n                    abort(400, f'Not valid filter \"{key}\"')\n                where.append(param)\n\n        data_dict = ca.default_store.get_data(name, where, params['page[size]'], params['page[offset]'])\n\n        return data_dict, 200\n\n\n@ns_conf.route('/<name>/download')\n@ns_conf.param('name', 'Datasource name')\nclass DatasourceMissedFilesDownload(Resource):\n    @ns_conf.doc('get_datasource_download')\n    def get(self, name):\n        '''download uploaded file'''\n        ds = ca.default_store.get_datasource(name)\n        if not ds:\n            abort(404, \"{} not found\".format(name))\n        if not os.path.exists(ds['source']):\n            abort(404, \"{} not found\".format(name))\n\n        return send_file(os.path.abspath(ds['source']), as_attachment=True)\n", "path": "mindsdb/api/http/namespaces/datasource.py"}], "after_files": [{"content": "import datetime\nimport os\nimport threading\nimport tempfile\n\nimport multipart\n\nimport mindsdb\nfrom dateutil.parser import parse\nfrom flask import request, send_file\nfrom flask_restx import Resource, abort     # 'abort' using to return errors as json: {'message': 'error text'}\nfrom flask import current_app as ca\n\nfrom mindsdb.interfaces.datastore.sqlite_helpers import *\nfrom mindsdb.api.http.namespaces.configs.datasources import ns_conf\nfrom mindsdb.api.http.namespaces.entitites.datasources.datasource import (\n    datasource_metadata,\n    put_datasource_params\n)\nfrom mindsdb.api.http.namespaces.entitites.datasources.datasource_data import (\n    get_datasource_rows_params,\n    datasource_rows_metadata\n)\nfrom mindsdb.api.http.namespaces.entitites.datasources.datasource_files import (\n    put_datasource_file_params\n)\nfrom mindsdb.api.http.namespaces.entitites.datasources.datasource_missed_files import (\n    datasource_missed_files_metadata,\n    get_datasource_missed_files_params\n)\n\n\n@ns_conf.route('/')\nclass DatasourcesList(Resource):\n    @ns_conf.doc('get_datasources_list')\n    @ns_conf.marshal_list_with(datasource_metadata)\n    def get(self):\n        '''List all datasources'''\n        return ca.default_store.get_datasources()\n\n\n@ns_conf.route('/<name>')\n@ns_conf.param('name', 'Datasource name')\nclass Datasource(Resource):\n    @ns_conf.doc('get_datasource')\n    @ns_conf.marshal_with(datasource_metadata)\n    def get(self, name):\n        '''return datasource metadata'''\n        ds = ca.default_store.get_datasource(name)\n        if ds is not None:\n            return ds\n        return '', 404\n\n    @ns_conf.doc('delete_datasource')\n    def delete(self, name):\n        '''delete datasource'''\n        try:\n            ca.default_store.delete_datasource(name)\n        except Exception as e:\n            print(e)\n            abort(400, str(e))\n        return '', 200\n\n    @ns_conf.doc('put_datasource', params=put_datasource_params)\n    @ns_conf.marshal_with(datasource_metadata)\n    def put(self, name):\n        '''add new datasource'''\n        data = {}\n\n        def on_field(field):\n            print(f'\\n\\n{field}\\n\\n')\n            name = field.field_name.decode()\n            value = field.value.decode()\n            data[name] = value\n\n        file_object = None\n\n        def on_file(file):\n            nonlocal file_object\n            data['file'] = file.file_name.decode()\n            file_object = file.file_object\n\n        temp_dir_path = tempfile.mkdtemp(prefix='datasource_file_')\n\n        if request.headers['Content-Type'].startswith('multipart/form-data'):\n            parser = multipart.create_form_parser(\n                headers=request.headers,\n                on_field=on_field,\n                on_file=on_file,\n                config={\n                    'UPLOAD_DIR': temp_dir_path.encode(),    # bytes required\n                    'UPLOAD_KEEP_FILENAME': True,\n                    'UPLOAD_KEEP_EXTENSIONS': True,\n                    'MAX_MEMORY_FILE_SIZE': 0\n                }\n            )\n\n            while True:\n                chunk = request.stream.read(8192)\n                if not chunk:\n                    break\n                parser.write(chunk)\n            parser.finalize()\n            parser.close()\n\n            if file_object is not None and not file_object.closed:\n                file_object.close()\n        else:\n            data = request.json\n\n        if 'query' in data:\n            source_type = request.json['integration_id']\n            ca.default_store.save_datasource(name, source_type, request.json)\n            os.rmdir(temp_dir_path)\n            return ca.default_store.get_datasource(name)\n\n        ds_name = data['name'] if 'name' in data else name\n        source = data['source'] if 'source' in data else name\n        source_type = data['source_type']\n\n        if source_type == 'file':\n            file_path = os.path.join(temp_dir_path, data['file'])\n        else:\n            file_path = None\n\n        ca.default_store.save_datasource(ds_name, source_type, source, file_path)\n        os.rmdir(temp_dir_path)\n\n        return ca.default_store.get_datasource(ds_name)\n\n\nds_analysis = {}\n\n\ndef analyzing_thread(name, default_store):\n    global ds_analysis\n    ds_analysis[name] = None\n    ds = default_store.get_datasource(name)\n    analysis = default_store.get_analysis(ds['name'])\n    ds_analysis[name] = {\n        'created_at': datetime.datetime.utcnow(),\n        'data': analysis\n    }\n\n\n@ns_conf.route('/<name>/analyze')\n@ns_conf.param('name', 'Datasource name')\nclass Analyze(Resource):\n    @ns_conf.doc('analyse_dataset')\n    def get(self, name):\n        global ds_analysis\n        if name in ds_analysis:\n            if ds_analysis[name] is None:\n                return {'status': 'analyzing'}, 200\n            else:\n                analysis = ds_analysis[name]['data']\n                return analysis, 200\n\n        ds = ca.default_store.get_datasource(name)\n        if ds is None:\n            print('No valid datasource given')\n            abort(400, 'No valid datasource given')\n\n        x = threading.Thread(target=analyzing_thread, args=(name, ca.default_store))\n        x.start()\n        return {'status': 'analyzing'}, 200\n\n@ns_conf.route('/<name>/analyze_refresh')\n@ns_conf.param('name', 'Datasource name')\nclass Analyze(Resource):\n    @ns_conf.doc('analyze_refresh_dataset')\n    def get(self, name):\n        global ds_analysis\n        if name in ds_analysis:\n            if ds_analysis[name] is None:\n                return {'status': 'analyzing'}, 200\n            else:\n                del ds_analysis[name]\n\n        ds = ca.default_store.get_datasource(name)\n        if ds is None:\n            print('No valid datasource given')\n            abort(400, 'No valid datasource given')\n\n        x = threading.Thread(target=analyzing_thread, args=(name, ca.default_store))\n        x.start()\n        return {'status': 'analyzing'}, 200\n\n\n@ns_conf.route('/<name>/analyze_subset')\n@ns_conf.param('name', 'Datasource name')\nclass AnalyzeSubset(Resource):\n    @ns_conf.doc('analyse_datasubset')\n    def get(self, name):\n        ds = ca.default_store.get_datasource(name)\n        if ds is None:\n            print('No valid datasource given')\n            abort(400, 'No valid datasource given')\n\n        where = []\n        for key, value in request.args.items():\n            if key.startswith('filter'):\n                param = parse_filter(key, value)\n                if param is None:\n                    abort(400, f'Not valid filter \"{key}\"')\n                where.append(param)\n\n        data_dict = ca.default_store.get_data(ds['name'], where)\n\n        if data_dict['rowcount'] == 0:\n            return abort(400, 'Empty dataset after filters applying')\n\n        return get_analysis(pd.DataFrame(data_dict['data'])), 200\n\n\n@ns_conf.route('/<name>/data/')\n@ns_conf.param('name', 'Datasource name')\nclass DatasourceData(Resource):\n    @ns_conf.doc('get_datasource_data', params=get_datasource_rows_params)\n    @ns_conf.marshal_with(datasource_rows_metadata)\n    def get(self, name):\n        '''return data rows'''\n        ds = ca.default_store.get_datasource(name)\n        if ds is None:\n            abort(400, 'No valid datasource given')\n\n        params = {\n            'page[size]': None,\n            'page[offset]': None\n        }\n        where = []\n        for key, value in request.args.items():\n            if key == 'page[size]':\n                params['page[size]'] = int(value)\n            if key == 'page[offset]':\n                params['page[offset]'] = int(value)\n            elif key.startswith('filter'):\n                param = parse_filter(key, value)\n                if param is None:\n                    abort(400, f'Not valid filter \"{key}\"')\n                where.append(param)\n\n        data_dict = ca.default_store.get_data(name, where, params['page[size]'], params['page[offset]'])\n\n        return data_dict, 200\n\n\n@ns_conf.route('/<name>/download')\n@ns_conf.param('name', 'Datasource name')\nclass DatasourceMissedFilesDownload(Resource):\n    @ns_conf.doc('get_datasource_download')\n    def get(self, name):\n        '''download uploaded file'''\n        ds = ca.default_store.get_datasource(name)\n        if not ds:\n            abort(404, \"{} not found\".format(name))\n        if not os.path.exists(ds['source']):\n            abort(404, \"{} not found\".format(name))\n\n        return send_file(os.path.abspath(ds['source']), as_attachment=True)\n", "path": "mindsdb/api/http/namespaces/datasource.py"}]}