Error "in void don't know how to serialize this type of index" when saving index to disk when device=0 (GPU)
MEMBER
Hi @corticalstack ! Thanks for reporting. Indeed in the recent versions of Faiss we must use `getDevice` to check if the index in on GPU.

I'm opening a PR
2021-02-10T17:31:47Z
https://github.com/huggingface/datasets/issues/1859#issuecomment-776882967
https://api.github.com/repos/huggingface/datasets/issues/1859
MDEyOklzc3VlQ29tbWVudDc3Njg4Mjk2Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/776882967/reactions
2021-02-10T17:31:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/776882967
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I fixed this issue. It should work fine now.
Feel free to try it out by installing `datasets` from source.
Otherwise you can wait for the next release of `datasets` (in a few days)
2021-02-10T18:17:40Z
https://github.com/huggingface/datasets/issues/1859#issuecomment-776911178
https://api.github.com/repos/huggingface/datasets/issues/1859
MDEyOklzc3VlQ29tbWVudDc3NjkxMTE3OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/776911178/reactions
2021-02-10T18:17:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/776911178
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for such a quick fix and merge to master, pip installed git master, tested all OK
2021-02-10T18:31:51Z
https://github.com/huggingface/datasets/issues/1859#issuecomment-776920219
https://api.github.com/repos/huggingface/datasets/issues/1859
MDEyOklzc3VlQ29tbWVudDc3NjkyMDIxOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/776920219/reactions
2021-02-10T18:32:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/776920219
https://avatars.githubusercontent.com/u/3995321?v=4
https://api.github.com/users/corticalstack/events{/privacy}
https://api.github.com/users/corticalstack/followers
https://api.github.com/users/corticalstack/following{/other_user}
https://api.github.com/users/corticalstack/gists{/gist_id}
https://github.com/corticalstack
corticalstack
MDQ6VXNlcjM5OTUzMjE=
https://api.github.com/users/corticalstack/orgs
https://api.github.com/users/corticalstack/received_events
https://api.github.com/users/corticalstack/repos
https://api.github.com/users/corticalstack/starred{/owner}{/repo}
https://api.github.com/users/corticalstack/subscriptions
User
https://api.github.com/users/corticalstack
public
Error serializing faiss index.  Error as follows:

`Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) at /home/conda/feedstock_root/build_artifacts/faiss-split_1612472484670/work/faiss/impl/index_write.cpp:453: don't know how to serialize this type of index`


Note:

`torch.cuda.is_available()` reports:

```
Cuda is available
cuda:0

```

Adding index, device=0 for GPU.

`dataset.add_faiss_index(column='embeddings', index_name='idx_embeddings', device=0)`

However, during a quick debug, self.faiss_index has no attr "device" when checked in` search.py, method save`, so fails to transform gpu index to cpu index.  If I add index without device, index is saved OK.


```
def save(self, file: str):
        """Serialize the FaissIndex on disk"""
        import faiss  # noqa: F811

        if (
            hasattr(self.faiss_index, "device")
            and self.faiss_index.device is not None
            and self.faiss_index.device > -1
        ):
            index = faiss.index_gpu_to_cpu(self.faiss_index)
        else:
            index = self.faiss_index
        faiss.write_index(index, file)
```
