Add FAISS .range_search() method for retrieving all texts from dataset above similarity threshold
MEMBER
Hi ! You can access the faiss index with
```python
faiss_index = my_dataset.get_index("my_index_name").faiss_index
```
and then do whatever you want with it, e.g. query it using range_search:
```python
threshold = 0.95
limits, distances, indices = faiss_index.range_search(x=xq, thresh=threshold)

texts = dataset[indices]
```
2022-03-28T12:43:46Z
https://github.com/huggingface/datasets/issues/4029#issuecomment-1080603963
https://api.github.com/repos/huggingface/datasets/issues/4029
IC_kwDODunzps5AaLU7
https://api.github.com/repos/huggingface/datasets/issues/comments/1080603963/reactions
2022-03-28T12:43:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1080603963
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
wow, that's great, thank you for the explanation. (if that's not already in the documentation, could be worth adding it)

which type of faiss index is Datasets using? I looked into faiss recently and I understand that there are several different types of indexes and the choice is important, e.g. regarding which distance metric you use (euclidian vs. cosine/dot product), the size of my dataset etc. can I chose the type of index somehow as well?
2022-03-28T19:47:10Z
https://github.com/huggingface/datasets/issues/4029#issuecomment-1081068563
https://api.github.com/repos/huggingface/datasets/issues/4029
IC_kwDODunzps5Ab8wT
https://api.github.com/repos/huggingface/datasets/issues/comments/1081068563/reactions
2022-03-28T19:50:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1081068563
https://avatars.githubusercontent.com/u/41862082?v=4
https://api.github.com/users/MoritzLaurer/events{/privacy}
https://api.github.com/users/MoritzLaurer/followers
https://api.github.com/users/MoritzLaurer/following{/other_user}
https://api.github.com/users/MoritzLaurer/gists{/gist_id}
https://github.com/MoritzLaurer
MoritzLaurer
MDQ6VXNlcjQxODYyMDgy
https://api.github.com/users/MoritzLaurer/orgs
https://api.github.com/users/MoritzLaurer/received_events
https://api.github.com/users/MoritzLaurer/repos
https://api.github.com/users/MoritzLaurer/starred{/owner}{/repo}
https://api.github.com/users/MoritzLaurer/subscriptions
User
https://api.github.com/users/MoritzLaurer
public
MEMBER
`Dataset.add_faiss_index` has a `string_factory` parameter, used to set the type of index (see the faiss documentation about [index factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory)). Alternatively, you can pass an index you've defined yourself using faiss with the `custom_index` parameter of `Dataset.add_faiss_index` 

Here is the full documentation of `Dataset.add_faiss_index`: https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes#datasets.Dataset.add_faiss_index
2022-03-29T09:49:17Z
https://github.com/huggingface/datasets/issues/4029#issuecomment-1081662010
https://api.github.com/repos/huggingface/datasets/issues/4029
IC_kwDODunzps5AeNo6
https://api.github.com/repos/huggingface/datasets/issues/comments/1081662010/reactions
2022-03-29T09:49:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1081662010
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
great thanks, I will try it out
2022-03-29T19:57:28Z
https://github.com/huggingface/datasets/issues/4029#issuecomment-1082316266
https://api.github.com/repos/huggingface/datasets/issues/4029
IC_kwDODunzps5AgtXq
https://api.github.com/repos/huggingface/datasets/issues/comments/1082316266/reactions
2022-03-29T19:57:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1082316266
https://avatars.githubusercontent.com/u/41862082?v=4
https://api.github.com/users/MoritzLaurer/events{/privacy}
https://api.github.com/users/MoritzLaurer/followers
https://api.github.com/users/MoritzLaurer/following{/other_user}
https://api.github.com/users/MoritzLaurer/gists{/gist_id}
https://github.com/MoritzLaurer
MoritzLaurer
MDQ6VXNlcjQxODYyMDgy
https://api.github.com/users/MoritzLaurer/orgs
https://api.github.com/users/MoritzLaurer/received_events
https://api.github.com/users/MoritzLaurer/repos
https://api.github.com/users/MoritzLaurer/starred{/owner}{/repo}
https://api.github.com/users/MoritzLaurer/subscriptions
User
https://api.github.com/users/MoritzLaurer
public
**Is your feature request related to a problem? Please describe.**
I would like to retrieve all texts from a dataset, which are semantically similar to a specific input text (query), above a certain (cosine) similarity threshold. My dataset is very large (Wikipedia), so I need to use Datasets and FAISS for this. I would like to be able to repeat many different queries on the dataset quickly. 

**Describe the solution you'd like**
dataset objects currently have the .get_nearest_examples() method for text retrieval via FAISS. But this only allows retrieving a specific number of K texts instead of everything above a specified similarity threshold. 
It would be great if HF Datasets would also support the FAISS method .range_search() for retrieving texts above a certain similarity threshold. 
see details here: https://github.com/facebookresearch/faiss/issues/1273

**Describe alternatives you've considered**
I've considered using native FAISS, but doing this via HF datasets would be better. My assumption is that Dataset features like dataset streaming make it easier to work with large datasets


**Additional context**
The concrete use-case is: I have a large dataset (wikipedia) and I would like to retrieve all paragraphs which are similar to a query. I will use sentence-transformers for encoding the texts. 
