Scores from Index at extra positions are not filtered out
COLLABORATOR
Hi! Yes, that makes sense! Would you like to submit a PR to fix this?
2022-03-18T16:14:52Z
https://github.com/huggingface/datasets/issues/3961#issuecomment-1072565992
https://api.github.com/repos/huggingface/datasets/issues/3961
IC_kwDODunzps4_7g7o
https://api.github.com/repos/huggingface/datasets/issues/comments/1072565992/reactions
2022-03-18T16:14:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1072565992
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Created PR https://github.com/huggingface/datasets/pull/3971
2022-03-19T18:47:49Z
https://github.com/huggingface/datasets/issues/3961#issuecomment-1073061489
https://api.github.com/repos/huggingface/datasets/issues/3961
IC_kwDODunzps4_9Z5x
https://api.github.com/repos/huggingface/datasets/issues/comments/1073061489/reactions
2022-03-19T18:47:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1073061489
https://avatars.githubusercontent.com/u/36671559?v=4
https://api.github.com/users/vishalsrao/events{/privacy}
https://api.github.com/users/vishalsrao/followers
https://api.github.com/users/vishalsrao/following{/other_user}
https://api.github.com/users/vishalsrao/gists{/gist_id}
https://github.com/vishalsrao
vishalsrao
MDQ6VXNlcjM2NjcxNTU5
https://api.github.com/users/vishalsrao/orgs
https://api.github.com/users/vishalsrao/received_events
https://api.github.com/users/vishalsrao/repos
https://api.github.com/users/vishalsrao/starred{/owner}{/repo}
https://api.github.com/users/vishalsrao/subscriptions
User
https://api.github.com/users/vishalsrao
public
If a FAISS index has fewer records than the requested number of top results (k), then it returns -1 in indices for the additional positions. The get_nearest_examples method only filters out the extra results from the dataset samples. It would be better to filter out extra scores too.

Reference: https://github.com/huggingface/datasets/blob/2.0.0/src/datasets/search.py#L693
