Load/Save FAISS index using fsspec
COLLABORATOR
Hi! Sure, feel free to submit a PR. Maybe if we want to be consistent with the existing API, it would be cleaner to directly add support for `fsspec` paths in `Dataset.load_faiss_index`/`Dataset.save_faiss_index` in the same manner as it was done in `Dataset.load_from_disk`/`Dataset.save_to_disk`.
2023-01-19T16:01:26Z
https://github.com/huggingface/datasets/issues/5428#issuecomment-1397211354
https://api.github.com/repos/huggingface/datasets/issues/5428
IC_kwDODunzps5TR8Da
https://api.github.com/repos/huggingface/datasets/issues/comments/1397211354/reactions
2023-01-19T16:01:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1397211354
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
That's a great idea! I'll do that instead. 
2023-01-19T16:34:14Z
https://github.com/huggingface/datasets/issues/5428#issuecomment-1397268207
https://api.github.com/repos/huggingface/datasets/issues/5428
IC_kwDODunzps5TSJ7v
https://api.github.com/repos/huggingface/datasets/issues/comments/1397268207/reactions
2023-01-19T16:34:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1397268207
https://avatars.githubusercontent.com/u/8976546?v=4
https://api.github.com/users/Dref360/events{/privacy}
https://api.github.com/users/Dref360/followers
https://api.github.com/users/Dref360/following{/other_user}
https://api.github.com/users/Dref360/gists{/gist_id}
https://github.com/Dref360
Dref360
MDQ6VXNlcjg5NzY1NDY=
https://api.github.com/users/Dref360/orgs
https://api.github.com/users/Dref360/received_events
https://api.github.com/users/Dref360/repos
https://api.github.com/users/Dref360/starred{/owner}{/repo}
https://api.github.com/users/Dref360/subscriptions
User
https://api.github.com/users/Dref360
public
### Feature request

From what I understand `faiss` already support this [link](https://github.com/facebookresearch/faiss/wiki/Index-IO,-cloning-and-hyper-parameter-tuning#generic-io-support)

I would like to use a stream as input to `Dataset.load_faiss_index` and `Dataset.save_faiss_index`.

### Motivation

In my case, I'm saving faiss index in cloud storage and use `fsspec` to load them. It would be ideal if I could send the stream directly instead of copying the file locally (or mounting the bucket) and then load the index.

### Your contribution

I can submit the PR