Multi-GPU support for `FaissIndex`
CONTRIBUTOR
Any love?
2022-02-15T08:01:31Z
https://github.com/huggingface/datasets/pull/3721#issuecomment-1039971537
https://api.github.com/repos/huggingface/datasets/issues/3721
IC_kwDODunzps49_LTR
https://api.github.com/repos/huggingface/datasets/issues/comments/1039971537/reactions
2022-02-15T08:37:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1039971537
https://avatars.githubusercontent.com/u/32859905?v=4
https://api.github.com/users/rentruewang/events{/privacy}
https://api.github.com/users/rentruewang/followers
https://api.github.com/users/rentruewang/following{/other_user}
https://api.github.com/users/rentruewang/gists{/gist_id}
https://github.com/rentruewang
rentruewang
MDQ6VXNlcjMyODU5OTA1
https://api.github.com/users/rentruewang/orgs
https://api.github.com/users/rentruewang/received_events
https://api.github.com/users/rentruewang/repos
https://api.github.com/users/rentruewang/starred{/owner}{/repo}
https://api.github.com/users/rentruewang/subscriptions
User
https://api.github.com/users/rentruewang
public
CONTRIBUTOR
Hi, any update?
2022-02-18T17:18:09Z
https://github.com/huggingface/datasets/pull/3721#issuecomment-1044868330
https://api.github.com/repos/huggingface/datasets/issues/3721
IC_kwDODunzps4-R2zq
https://api.github.com/repos/huggingface/datasets/issues/comments/1044868330/reactions
2022-02-18T17:18:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1044868330
https://avatars.githubusercontent.com/u/32859905?v=4
https://api.github.com/users/rentruewang/events{/privacy}
https://api.github.com/users/rentruewang/followers
https://api.github.com/users/rentruewang/following{/other_user}
https://api.github.com/users/rentruewang/gists{/gist_id}
https://github.com/rentruewang
rentruewang
MDQ6VXNlcjMyODU5OTA1
https://api.github.com/users/rentruewang/orgs
https://api.github.com/users/rentruewang/received_events
https://api.github.com/users/rentruewang/repos
https://api.github.com/users/rentruewang/starred{/owner}{/repo}
https://api.github.com/users/rentruewang/subscriptions
User
https://api.github.com/users/rentruewang
public
CONTRIBUTOR
@albertvillanova Sorry for bothering you again, quick follow up: is there anything else you want me to add / modify?
2022-02-23T07:33:23Z
https://github.com/huggingface/datasets/pull/3721#issuecomment-1048512388
https://api.github.com/repos/huggingface/datasets/issues/3721
IC_kwDODunzps4-fweE
https://api.github.com/repos/huggingface/datasets/issues/comments/1048512388/reactions
2022-02-23T07:35:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1048512388
https://avatars.githubusercontent.com/u/32859905?v=4
https://api.github.com/users/rentruewang/events{/privacy}
https://api.github.com/users/rentruewang/followers
https://api.github.com/users/rentruewang/following{/other_user}
https://api.github.com/users/rentruewang/gists{/gist_id}
https://github.com/rentruewang
rentruewang
MDQ6VXNlcjMyODU5OTA1
https://api.github.com/users/rentruewang/orgs
https://api.github.com/users/rentruewang/received_events
https://api.github.com/users/rentruewang/repos
https://api.github.com/users/rentruewang/starred{/owner}{/repo}
https://api.github.com/users/rentruewang/subscriptions
User
https://api.github.com/users/rentruewang
public
MEMBER
Hi @rentruewang , we updated the documentation on `master`, could you merge `master` into your branch please ?
2022-03-07T10:20:24Z
https://github.com/huggingface/datasets/pull/3721#issuecomment-1060442497
https://api.github.com/repos/huggingface/datasets/issues/3721
IC_kwDODunzps4_NRGB
https://api.github.com/repos/huggingface/datasets/issues/comments/1060442497/reactions
2022-03-07T10:20:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1060442497
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq I've merge `huggingface/datasets/master` into this PR. Please review. Thanks! 🤗

Edit: Umm... I was experimenting with what renaming a branch would do to a pull request. Please ignore the `closed this PR` down below. 🤗
2022-03-07T16:18:12Z
https://github.com/huggingface/datasets/pull/3721#issuecomment-1060868860
https://api.github.com/repos/huggingface/datasets/issues/3721
IC_kwDODunzps4_O5L8
https://api.github.com/repos/huggingface/datasets/issues/comments/1060868860/reactions
2022-03-07T16:22:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1060868860
https://avatars.githubusercontent.com/u/32859905?v=4
https://api.github.com/users/rentruewang/events{/privacy}
https://api.github.com/users/rentruewang/followers
https://api.github.com/users/rentruewang/following{/other_user}
https://api.github.com/users/rentruewang/gists{/gist_id}
https://github.com/rentruewang
rentruewang
MDQ6VXNlcjMyODU5OTA1
https://api.github.com/users/rentruewang/orgs
https://api.github.com/users/rentruewang/received_events
https://api.github.com/users/rentruewang/repos
https://api.github.com/users/rentruewang/starred{/owner}{/repo}
https://api.github.com/users/rentruewang/subscriptions
User
https://api.github.com/users/rentruewang
public
Per #3716 , current implementation does not take into consideration that `faiss` can run on multiple GPUs.

In this commit, I provided multi-GPU support for `FaissIndex` by modifying the device management in `IndexableMixin.add_faiss_index` and `FaissIndex.load`.

Now users are able to pass in
1. a positive integer (as usual) to use 1 GPU
2. a negative integer `-1` to use all GPUs
3. a list of integers e.g. `[0, 1]` to run only on those GPUs
4. Of course, passing in nothing still runs on CPU.

This closes: #3716 