Loading a faiss index KeyError
MEMBER
In your code `dataset2` doesn't contain the "embeddings" column, since it is created from the pandas DataFrame with columns "text" and "label".

Therefore when you call `dataset2[embeddings_name]`, you get a `KeyError`.

If you want the "embeddings" column back, you can create `dataset2` with
```python
dataset2 = load_from_disk(dataset_filename)
```
where `dataset_filename` is the place where you saved you dataset with the embeddings in the first place.
2021-03-11T14:51:44Z
https://github.com/huggingface/datasets/issues/2029#issuecomment-796792705
https://api.github.com/repos/huggingface/datasets/issues/2029
MDEyOklzc3VlQ29tbWVudDc5Njc5MjcwNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/796792705/reactions
2021-03-11T14:51:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/796792705
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Ok in that case HF should fix their misleading example at https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index 

I copy-pasted it here.

> When you are done with your queries you can save your index on disk:
> 
> ```python
> ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss')
> ```
> Then reload it later:
> 
> ```python
> ds = load_dataset('crime_and_punish', split='train[:100]')
> ds.load_faiss_index('embeddings', 'my_index.faiss')
> ```
2021-03-11T15:52:04Z
https://github.com/huggingface/datasets/issues/2029#issuecomment-796836390
https://api.github.com/repos/huggingface/datasets/issues/2029
MDEyOklzc3VlQ29tbWVudDc5NjgzNjM5MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/796836390/reactions
2021-03-11T15:52:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/796836390
https://avatars.githubusercontent.com/u/24982805?v=4
https://api.github.com/users/nbroad1881/events{/privacy}
https://api.github.com/users/nbroad1881/followers
https://api.github.com/users/nbroad1881/following{/other_user}
https://api.github.com/users/nbroad1881/gists{/gist_id}
https://github.com/nbroad1881
nbroad1881
MDQ6VXNlcjI0OTgyODA1
https://api.github.com/users/nbroad1881/orgs
https://api.github.com/users/nbroad1881/received_events
https://api.github.com/users/nbroad1881/repos
https://api.github.com/users/nbroad1881/starred{/owner}{/repo}
https://api.github.com/users/nbroad1881/subscriptions
User
https://api.github.com/users/nbroad1881
public
MEMBER
Hi !

The code of the example is valid.
An index is a search engine, it's not considered a column of a dataset.
When you do `ds.load_faiss_index("embeddings", 'my_index.faiss')`, it attaches an index named "embeddings" to the dataset but it doesn't re-add the "embeddings" column. You can list the indexes of a dataset by using `ds.list_indexes()`.

If I understand correctly by reading this example you thought that it was re-adding the "embeddings" column.
This looks misleading indeed, and we should add a note to make it more explicit that it doesn't store the column that was used to build the index.

Feel free to open a PR to suggest an improvement on the documentation if you want to contribute :)
2021-03-11T17:20:15Z
https://github.com/huggingface/datasets/issues/2029#issuecomment-796903025
https://api.github.com/repos/huggingface/datasets/issues/2029
MDEyOklzc3VlQ29tbWVudDc5NjkwMzAyNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/796903025/reactions
2021-03-11T17:20:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/796903025
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> If I understand correctly by reading this example you thought that it was re-adding the "embeddings" column.
Yes. I was trying to use the dataset in RAG and it complained that the dataset didn't have the right columns. No problems when loading the dataset with `load_from_disk` and then doing `load_faiss_index`

What I learned was
1. column and index are different
2. loading the index does not create a column
3. the column is not needed to be able to use the index
4. RAG needs both the embeddings column and the index

If I can come up with a way to articulate this in the right spot in the docs, I'll open a PR
2021-03-12T00:21:09Z
https://github.com/huggingface/datasets/issues/2029#issuecomment-797144831
https://api.github.com/repos/huggingface/datasets/issues/2029
MDEyOklzc3VlQ29tbWVudDc5NzE0NDgzMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/797144831/reactions
2021-03-12T00:21:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797144831
https://avatars.githubusercontent.com/u/24982805?v=4
https://api.github.com/users/nbroad1881/events{/privacy}
https://api.github.com/users/nbroad1881/followers
https://api.github.com/users/nbroad1881/following{/other_user}
https://api.github.com/users/nbroad1881/gists{/gist_id}
https://github.com/nbroad1881
nbroad1881
MDQ6VXNlcjI0OTgyODA1
https://api.github.com/users/nbroad1881/orgs
https://api.github.com/users/nbroad1881/received_events
https://api.github.com/users/nbroad1881/repos
https://api.github.com/users/nbroad1881/starred{/owner}{/repo}
https://api.github.com/users/nbroad1881/subscriptions
User
https://api.github.com/users/nbroad1881
public
I've recently been testing out RAG and DPR embeddings, and I've run into an issue that is not apparent in the documentation.

The basic steps are:

1. Create a dataset (dataset1)
2. Create an embeddings column using DPR
3. Add a faiss index to the dataset
4. Save faiss index to a file
5. Create a new dataset (dataset2) with the same text and label information as dataset1
6. Try to load the faiss index from file to dataset2
7. Get `KeyError: "Column embeddings not in the dataset"`

I've made a colab notebook that should show exactly what I did. Please switch to GPU runtime; I didn't check on CPU.

https://colab.research.google.com/drive/1X0S9ZuZ8k0ybcoei4w7so6dS_WrABmIx?usp=sharing

Ubuntu Version
VERSION="18.04.5 LTS (Bionic Beaver)"

datasets==1.4.1
faiss==1.5.3
faiss-gpu==1.7.0
torch==1.8.0+cu101
transformers==4.3.3

NVIDIA-SMI 460.56
Driver Version: 460.32.03
CUDA Version: 11.2    
Tesla K80           

I was basically following the steps here: https://huggingface.co/docs/datasets/faiss_and_ea.html#adding-a-faiss-index

I included the exact code from the documentation at the end of the notebook to show that they don't work either.
