dataset.search_batch() function outputs all -1 indices sometime.
NONE
Actually, I found the answer [here](https://github.com/facebookresearch/faiss/wiki/FAQ#what-does-it-mean-when-a-search-returns--1-ids). 

So we have to do some modifications to the code for instances where the index doesn't retrieve any IDs.
2021-04-06T23:02:41Z
https://github.com/huggingface/datasets/issues/2175#issuecomment-814488623
https://api.github.com/repos/huggingface/datasets/issues/2175
MDEyOklzc3VlQ29tbWVudDgxNDQ4ODYyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/814488623/reactions
2021-04-06T23:02:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814488623
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
NONE
@lhoestq  @patrickvonplaten 

I also found another short bug in the retrieval part.   Especially, when retrieving documents. If Faiss returns the -1 as the index, the retriever will always use the last element in the dataset.

please check [def get_doc_dicts function](https://github.com/huggingface/transformers/blob/master/src/transformers/models/rag/retrieval_rag.py#L222)


Does the use of the HNSW guarantee to retrieve valid indexes always? 


2021-04-07T06:08:54Z
https://github.com/huggingface/datasets/issues/2175#issuecomment-814630559
https://api.github.com/repos/huggingface/datasets/issues/2175
MDEyOklzc3VlQ29tbWVudDgxNDYzMDU1OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/814630559/reactions
2021-04-07T06:08:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814630559
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
MEMBER
Hi !
No it happens sometimes to return -1, especially if your dataset is small.
If your dataset is big enough it shouldn't happen in my experience.

Ideally we should ignore all the -1 that are returned. It should be possible to change that in RAG's code 
2021-04-07T09:09:11Z
https://github.com/huggingface/datasets/issues/2175#issuecomment-814746509
https://api.github.com/repos/huggingface/datasets/issues/2175
MDEyOklzc3VlQ29tbWVudDgxNDc0NjUwOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/814746509/reactions
2021-04-07T09:10:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814746509
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I also checked with some indexes it returns more -1s. Specially with IVF
when nprobr is very low. It doesn't happen when using HNSW though. But at
the moment if it happens, dataset will always return the last element.
Maybe we should change it to repeat the most last valid retrieved doc id.
What do you think?

On Wed, Apr 7, 2021, 21:09 Quentin Lhoest ***@***.***> wrote:

> Hi !
> No it happens sometimes to return -1, especially if your dataset is small.
> If your dataset is big enough it shouldn't happen.
>
> Ideally we should ignore all the -1 that are returned. It should be
> possible to change that in RAG's code
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2175#issuecomment-814746509>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGTENOTLBEZTXEO2RS3THQOMPANCNFSM42PRVYDA>
> .
>

2021-04-07T09:12:54Z
https://github.com/huggingface/datasets/issues/2175#issuecomment-814749265
https://api.github.com/repos/huggingface/datasets/issues/2175
MDEyOklzc3VlQ29tbWVudDgxNDc0OTI2NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/814749265/reactions
2021-04-07T09:12:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814749265
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
MEMBER
That would be an easy way to workaround this issue. Feel free to open a PR on `transformers` and ping me ! :)
2021-04-07T09:16:32Z
https://github.com/huggingface/datasets/issues/2175#issuecomment-814752589
https://api.github.com/repos/huggingface/datasets/issues/2175
MDEyOklzc3VlQ29tbWVudDgxNDc1MjU4OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/814752589/reactions
2021-04-07T09:16:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814752589
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Sure.  Will push everything together with RAG end to end. :) thanks a lot.

On Wed, Apr 7, 2021, 21:16 Quentin Lhoest ***@***.***> wrote:

> That would be an easy way to workaround this issue. Feel free to open a PR
> on transformers and ping me ! :)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2175#issuecomment-814752589>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGWLROCGARKN7WOJYSTTHQPH5ANCNFSM42PRVYDA>
> .
>

2021-04-07T09:23:19Z
https://github.com/huggingface/datasets/issues/2175#issuecomment-814758365
https://api.github.com/repos/huggingface/datasets/issues/2175
MDEyOklzc3VlQ29tbWVudDgxNDc1ODM2NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/814758365/reactions
2021-04-07T09:23:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814758365
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
I am working with RAG and playing around with different faiss indexes. At the moment I use **index = faiss.index_factory(768, "IVF65536_HNSW32,Flat")**.

During the retrieval phase exactly in [this line of retrieval_rag.py](https://github.com/huggingface/transformers/blob/master/src/transformers/models/rag/retrieval_rag.py#L231) an error issue when all retrieved indices are -1.  Please refer to the screenshot of a PID worker. 

![image](https://user-images.githubusercontent.com/16892570/113782387-37a67600-9786-11eb-9c29-acad661a9648.png)


Here, my retrieve batch size is 2 and n_docs is 5. I can solve this by working around np. stack, but I want to ask, why we get an output index of -1. Do you have any idea :) ?

Is this a problem of the index, where the faiss can't find any similar vector?
Is there documentation on the output index being -1?

@lhoestq 
 