add_faisis_index  gets very slow when doing it interatively  
MEMBER
I think faiss automatically sets the number of threads to use to build the index.
Can you check how many CPU cores are being used when you build the index in `use_own_knowleldge_dataset` as compared to this script ? Are there other programs running (maybe for rank>0) ?
2021-03-16T16:18:51Z
https://github.com/huggingface/datasets/issues/2046#issuecomment-800404322
https://api.github.com/repos/huggingface/datasets/issues/2046
MDEyOklzc3VlQ29tbWVudDgwMDQwNDMyMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/800404322/reactions
2021-03-16T16:18:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800404322
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi,
 I am running the add_faiss_index during the training process of the RAG from the master process (rank 0). But at the exact moment, I do not run any other process since I do it in every 5000 training steps. 
 
 I think what you say is correct. It depends on the number of CPU cores. I did an experiment to compare the time taken to finish the add_faiss_index process on use_own_knowleldge_dataset.py vs the training loop thing.  The training loop thing takes 40 mins more. It might be natural right? 
 
 
 at the moment it uses around 40 cores of a 96 core machine (I am fine-tuning the entire process).  
2021-03-16T20:57:32Z
https://github.com/huggingface/datasets/issues/2046#issuecomment-800599596
https://api.github.com/repos/huggingface/datasets/issues/2046
MDEyOklzc3VlQ29tbWVudDgwMDU5OTU5Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/800599596/reactions
2021-03-17T05:03:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800599596
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
MEMBER
Can you try to set the number of threads manually ?
If you set the same number of threads for both the `use_own_knowledge_dataset.py` and RAG training, it should take the same amount of time.
You can see how to set the number of thread in the faiss wiki: https://github.com/facebookresearch/faiss/wiki/Threads-and-asynchronous-calls
2021-03-17T10:16:34Z
https://github.com/huggingface/datasets/issues/2046#issuecomment-800963855
https://api.github.com/repos/huggingface/datasets/issues/2046
MDEyOklzc3VlQ29tbWVudDgwMDk2Mzg1NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/800963855/reactions
2021-03-17T10:16:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800963855
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Ok, I will report the details too soon. I am the first one on the list and currently add_index being computed for the 3rd time in the loop. Actually seems like the time is taken to complete each interaction is the same, but around 1 hour more compared to running it without the training loop. A the moment this takes 5hrs and 30 mins.  If there is any way to faster the process, an end-to-end rag will be perfect. So I will also try out with different thread numbers too.  

![image](https://user-images.githubusercontent.com/16892570/111453464-798c5f80-8778-11eb-86d0-19d212f58e38.png)

2021-03-17T10:32:32Z
https://github.com/huggingface/datasets/issues/2046#issuecomment-800973651
https://api.github.com/repos/huggingface/datasets/issues/2046
MDEyOklzc3VlQ29tbWVudDgwMDk3MzY1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/800973651/reactions
2021-03-17T10:32:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800973651
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
NONE
@lhoestq  on a different note, I read about using Faiss-GPU, but the documentation says we should use it when the dataset has the ability to fit into the GPU memory. Although this might work, in the long-term this is not that practical for me.

https://github.com/matsui528/faiss_tips
2021-03-17T10:38:00Z
https://github.com/huggingface/datasets/issues/2046#issuecomment-800977223
https://api.github.com/repos/huggingface/datasets/issues/2046
MDEyOklzc3VlQ29tbWVudDgwMDk3NzIyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/800977223/reactions
2021-03-17T10:54:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800977223
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
NONE
@lhoestq 

Hi,  I executed the **use_own_dataset.py** script independently and ask a few of my friends to run their programs in the HPC machine at the same time.  

 Once there are so many other processes are running the add_index function gets slows down naturally.  So basically the speed of the add_index depends entirely on the number of CPU processes.  Then I set the number of threads as you have mentioned and got actually the same time for RAG training and independat running.  So you are correct! :) 

 
 Then I added this [issue in Faiss repostiary](https://github.com/facebookresearch/faiss/issues/1767). I got an answer saying our current **IndexHNSWFlat** can get slow for 30 million vectors and it would be better to use alternatives. What do you think?
2021-03-18T08:53:45Z
https://github.com/huggingface/datasets/issues/2046#issuecomment-801747003
https://api.github.com/repos/huggingface/datasets/issues/2046
MDEyOklzc3VlQ29tbWVudDgwMTc0NzAwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/801747003/reactions
2021-03-18T09:01:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/801747003
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
MEMBER
It's a matter of tradeoffs.
HSNW is fast at query time but takes some time to build.
A flat index is flat to build but is "slow" at query time.
An IVF index is probably a good choice for you: fast building and fast queries (but still slower queries than HSNW).

Note that for an IVF index you would need to have an `nprobe` parameter (number of cells to visit for one query, there are `nlist` in total) that is not too small in order to have good retrieval accuracy, but not too big otherwise the queries will take too much time. From the faiss documentation:
> The nprobe parameter is always a way of adjusting the tradeoff between speed and accuracy of the result. Setting nprobe = nlist gives the same result as the brute-force search (but slower).

From my experience with indexes on DPR embeddings, setting nprobe around 1/4 of nlist gives really good retrieval accuracy and there's no need to have a value higher than that (or you would need to brute-force in order to see a difference).
2021-03-19T11:19:06Z
https://github.com/huggingface/datasets/issues/2046#issuecomment-802759596
https://api.github.com/repos/huggingface/datasets/issues/2046
MDEyOklzc3VlQ29tbWVudDgwMjc1OTU5Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/802759596/reactions
2021-03-19T11:19:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/802759596
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq 

Thanks a lot for sharing all this prior knowledge. 

Just asking what would be a good nlist of parameters for 30 million embeddings?
2021-03-21T01:20:29Z
https://github.com/huggingface/datasets/issues/2046#issuecomment-803495576
https://api.github.com/repos/huggingface/datasets/issues/2046
MDEyOklzc3VlQ29tbWVudDgwMzQ5NTU3Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/803495576/reactions
2021-03-22T02:29:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/803495576
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
MEMBER
When IVF is used alone, nlist should be between `4*sqrt(n)` and `16*sqrt(n)`.
For more details take a look at [this section of the Faiss wiki](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index#how-big-is-the-dataset)
2021-03-22T09:42:53Z
https://github.com/huggingface/datasets/issues/2046#issuecomment-803920065
https://api.github.com/repos/huggingface/datasets/issues/2046
MDEyOklzc3VlQ29tbWVudDgwMzkyMDA2NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/803920065/reactions
2021-03-22T09:42:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/803920065
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks a lot. I was lost with calling the index from class and using faiss_index_factory. 
2021-03-22T09:58:24Z
https://github.com/huggingface/datasets/issues/2046#issuecomment-803932521
https://api.github.com/repos/huggingface/datasets/issues/2046
MDEyOklzc3VlQ29tbWVudDgwMzkzMjUyMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/803932521/reactions
2021-03-22T09:58:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/803932521
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
NONE
@lhoestq  Thanks a lot for the help you have given to solve this issue. As per my experiments, IVF index suits well for my case and it is a lot faster. The use of this can make the entire RAG end-to-end trainable lot faster.  So I will close this issue. Will do the final PR soon. 
2021-03-24T22:29:11Z
https://github.com/huggingface/datasets/issues/2046#issuecomment-806226118
https://api.github.com/repos/huggingface/datasets/issues/2046
MDEyOklzc3VlQ29tbWVudDgwNjIyNjExOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/806226118/reactions
2021-03-24T22:29:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/806226118
https://avatars.githubusercontent.com/u/16892570?v=4
https://api.github.com/users/shamanez/events{/privacy}
https://api.github.com/users/shamanez/followers
https://api.github.com/users/shamanez/following{/other_user}
https://api.github.com/users/shamanez/gists{/gist_id}
https://github.com/shamanez
shamanez
MDQ6VXNlcjE2ODkyNTcw
https://api.github.com/users/shamanez/orgs
https://api.github.com/users/shamanez/received_events
https://api.github.com/users/shamanez/repos
https://api.github.com/users/shamanez/starred{/owner}{/repo}
https://api.github.com/users/shamanez/subscriptions
User
https://api.github.com/users/shamanez
public
As the below code suggests, I want to run add_faisis_index in every nth interaction from the training loop. I have 7.2 million documents. Usually, it takes 2.5 hours (if I run an as a separate process similar to the script given in rag/use_own_knowleldge_dataset.py). Now, this takes usually 5hrs. Is this normal?   Any way to make this process faster? 

@lhoestq 

```
   def training_step(self, batch, batch_idx) -> Dict:

    
        if (not batch_idx==0) and (batch_idx%5==0):

            print("******************************************************")
            ctx_encoder=self.trainer.model.module.module.model.rag.ctx_encoder
            model_copy =type(ctx_encoder)(self.config_dpr) # get a new instance  #this will be load in the CPU
            model_copy.load_state_dict(ctx_encoder.state_dict()) # copy weights and stuff


            list_of_gpus = ['cuda:2','cuda:3']
            c_dir='/custom/cache/dir'

            kb_dataset = load_dataset("csv", data_files=[self.custom_config.csv_path], split="train", delimiter="\t", column_names=["title", "text"],cache_dir=c_dir) 

            print(kb_dataset)

      
            n=len(list_of_gpus) #nunber of dedicated GPUs
            kb_list=[kb_dataset.shard(n, i, contiguous=True) for i in range(n)]

            #kb_dataset.save_to_disk('/hpc/gsir059/MY-Test/RAY/transformers/examples/research_projects/rag/haha-dir')


            print(self.trainer.global_rank)
            dataset_shards = self.re_encode_kb(model_copy.to(device=list_of_gpus[self.trainer.global_rank]),kb_list[self.trainer.global_rank])
            output = [None for _ in list_of_gpus]

            #self.trainer.accelerator_connector.accelerator.barrier("embedding_process")
            dist.all_gather_object(output, dataset_shards)
            

            #This creation and re-initlaization of the new index
            if (self.trainer.global_rank==0):  #saving will be done in the main process 
           
                combined_dataset = concatenate_datasets(output)
    
                passages_path =self.config.passages_path

                logger.info("saving the dataset with  ")
                #combined_dataset.save_to_disk('/hpc/gsir059/MY-Test/RAY/transformers/examples/research_projects/rag/MY-Passage')
                combined_dataset.save_to_disk(passages_path)
                logger.info("Add faiss index to the dataset that consist of embeddings") 

    
                embedding_dataset=combined_dataset
                index = faiss.IndexHNSWFlat(768, 128, faiss.METRIC_INNER_PRODUCT)
                embedding_dataset.add_faiss_index("embeddings", custom_index=index)

                embedding_dataset.get_index("embeddings").save(self.config.index_path)

