Support DataLoader with num_workers > 0 in streaming mode
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-05-19T15:14:01Z
https://github.com/huggingface/datasets/pull/4375#issuecomment-1131854079
https://api.github.com/repos/huggingface/datasets/issues/4375
IC_kwDODunzps5Ddrj_
https://api.github.com/repos/huggingface/datasets/issues/comments/1131854079/reactions
2022-06-10T20:56:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1131854079
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Alright this is finally ready for review ! It's quite long I'm sorry, but it's not easy to disentangle everything ^^'

The main additions are in
- src/datasets/formatting/dataset_wrappers/torch_iterable_dataset.py
- src/datasets/iterable_dataset.py
- src/datasets/utils/patching.py
2022-06-07T13:18:04Z
https://github.com/huggingface/datasets/pull/4375#issuecomment-1148662004
https://api.github.com/repos/huggingface/datasets/issues/4375
IC_kwDODunzps5EdzD0
https://api.github.com/repos/huggingface/datasets/issues/comments/1148662004/reactions
2022-06-07T13:18:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1148662004
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Added some comments and an error when lists have different lengths for sharding :)
2022-06-09T10:03:57Z
https://github.com/huggingface/datasets/pull/4375#issuecomment-1150927796
https://api.github.com/repos/huggingface/datasets/issues/4375
IC_kwDODunzps5EmcO0
https://api.github.com/repos/huggingface/datasets/issues/comments/1150927796/reactions
2022-06-09T10:03:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1150927796
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Let's resolve the merge conflict and the CI error (if it's related to the changes), and I can review the PR again.
2022-06-09T10:44:26Z
https://github.com/huggingface/datasets/pull/4375#issuecomment-1150964740
https://api.github.com/repos/huggingface/datasets/issues/4375
IC_kwDODunzps5EmlQE
https://api.github.com/repos/huggingface/datasets/issues/comments/1150964740/reactions
2022-06-09T10:44:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1150964740
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Feel free to review again :) The CI fail is unrelated to this PR and will be fixed by https://github.com/huggingface/datasets/pull/4472 (the hub now returns 401 instead of 404 for unauthenticated requests to non-existing repos)
2022-06-10T12:30:15Z
https://github.com/huggingface/datasets/pull/4375#issuecomment-1152308377
https://api.github.com/repos/huggingface/datasets/issues/4375
IC_kwDODunzps5ErtSZ
https://api.github.com/repos/huggingface/datasets/issues/comments/1152308377/reactions
2022-06-10T12:39:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1152308377
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
CI failures are unrelated to this PR - merging :)

(CI fails are a mix of pip install fails and Hub fails)
2022-06-10T20:47:18Z
https://github.com/huggingface/datasets/pull/4375#issuecomment-1152727259
https://api.github.com/repos/huggingface/datasets/issues/4375
IC_kwDODunzps5EtTjb
https://api.github.com/repos/huggingface/datasets/issues/comments/1152727259/reactions
2022-06-10T20:47:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1152727259
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq you're our hero :)
2022-07-04T16:05:13Z
https://github.com/huggingface/datasets/pull/4375#issuecomment-1173969331
https://api.github.com/repos/huggingface/datasets/issues/4375
IC_kwDODunzps5F-Vmz
https://api.github.com/repos/huggingface/datasets/issues/comments/1173969331/reactions
2022-07-04T16:05:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1173969331
https://avatars.githubusercontent.com/u/3491902?v=4
https://api.github.com/users/justheuristic/events{/privacy}
https://api.github.com/users/justheuristic/followers
https://api.github.com/users/justheuristic/following{/other_user}
https://api.github.com/users/justheuristic/gists{/gist_id}
https://github.com/justheuristic
justheuristic
MDQ6VXNlcjM0OTE5MDI=
https://api.github.com/users/justheuristic/orgs
https://api.github.com/users/justheuristic/received_events
https://api.github.com/users/justheuristic/repos
https://api.github.com/users/justheuristic/starred{/owner}{/repo}
https://api.github.com/users/justheuristic/subscriptions
User
https://api.github.com/users/justheuristic
public
### Issue

It's currently not possible to properly stream a dataset using multiple `torch.utils.data.DataLoader` workers:

- the `TorchIterableDataset` can't be pickled and passed to the subprocesses: https://github.com/huggingface/datasets/issues/3950
- streaming extension is failing: https://github.com/huggingface/datasets/issues/3951
- `fsspec` doesn't work out of the box in subprocesses

### Solution in this PR

I fixed these to enable passing an `IterableDataset` to a  `torch.utils.data.DataLoader` with `num_workers > 0`.

I also had to shard the `IterableDataset` to give each worker a shard, otherwise data would be duplicated. This is implemented in `TorchIterableDataset.__iter__` and uses the new `IterableDataset._iter_shard(shard_idx)` method

I also had to do a few changes the patching that enable streaming in dataset scripts:
- the patches are now always applied - not just for streaming mode. They're applied when a builder is instantiated
- I improved it to also check for renamed modules or attributes (ex: pandas vs pd)
- I grouped all the patches of pathlib.Path into a class `xPath`, so that `Path` outside of dataset scripts stay unchanged - otherwise I didn't change the content of the extended Path methods for streaming
- I fixed a bug with the `pd.read_csv` patch, opening the file in "rb" mode was missing and causing some datasets to not work in streaming mode, and compression inference was missing

### A few details regarding `fsspec` in multiprocessing

From https://github.com/fsspec/filesystem_spec/pull/963#issuecomment-1131709948 :
> Non-async instances might be safe in the forked child, if they hold no open files/sockets etc.; I'm not sure any implementations pass this test!
> If any async instance has been created, the newly forked processes must:
> 1. discard references to locks, threads and event loops and make new ones
> 2. not use any async fsspec instances from the parent process
> 3. clear all class instance caches

Therefore in a DataLoader's worker, I clear the reference to the loop and thread (1). We should be fine for 2 and 3 already since we don't use fsspec class instances from the parent process.

Fix https://github.com/huggingface/datasets/issues/3950
Fix https://github.com/huggingface/datasets/issues/3951

TODO:
- [x] fix tests