Support torch dataloader without torch formatting
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-12-13T19:45:13Z
https://github.com/huggingface/datasets/pull/5357#issuecomment-1349598106
https://api.github.com/repos/huggingface/datasets/issues/5357
IC_kwDODunzps5QcTua
https://api.github.com/repos/huggingface/datasets/issues/comments/1349598106/reactions
2022-12-15T19:19:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1349598106
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Need some more time to fix the tests, especially with pickle
2022-12-14T14:13:14Z
https://github.com/huggingface/datasets/pull/5357#issuecomment-1351445361
https://api.github.com/repos/huggingface/datasets/issues/5357
IC_kwDODunzps5QjWtx
https://api.github.com/repos/huggingface/datasets/issues/comments/1351445361/reactions
2022-12-14T14:13:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1351445361
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> And I actually don't quite understand the idea - what's the motivation behind making only IterableDataset compatible with torch DataLoader without setting the format explicitly?

Setting the format to pytorch = set the output types of the dataset to be pytorch tensors. However sometimes your dataset is not made of tensors but you still want to be able to use a pytorch DataLoader
2022-12-15T16:12:56Z
https://github.com/huggingface/datasets/pull/5357#issuecomment-1353337770
https://api.github.com/repos/huggingface/datasets/issues/5357
IC_kwDODunzps5Qqkuq
https://api.github.com/repos/huggingface/datasets/issues/comments/1353337770/reactions
2022-12-15T16:12:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1353337770
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
A bit more context. 

The arrow-backed `Dataset` supports `DataLoader(ds)` (even if the format is not "torch"), and we want to be able to do the same with `IterableDataset` for consistency. However, this is when the PyTorch internals come into play - an iterable dataset needs to be an instance of `torch.utils.data.IterableDataset` due to [this](https://github.com/pytorch/pytorch/blob/abc54f93145830b502400faa92bec86e05422fbd/torch/utils/data/dataloader.py#L276) check (notice there is no check for the map-style version). Hence the explicit subclassing in this PR.
2022-12-15T16:36:50Z
https://github.com/huggingface/datasets/pull/5357#issuecomment-1353370108
https://api.github.com/repos/huggingface/datasets/issues/5357
IC_kwDODunzps5Qqsn8
https://api.github.com/repos/huggingface/datasets/issues/comments/1353370108/reactions
2022-12-15T16:36:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1353370108
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Exactly :) Btw I just took your comments into account @polinaeterna , so feel free to review again
2022-12-15T17:51:43Z
https://github.com/huggingface/datasets/pull/5357#issuecomment-1353486025
https://api.github.com/repos/huggingface/datasets/issues/5357
IC_kwDODunzps5QrI7J
https://api.github.com/repos/huggingface/datasets/issues/comments/1353486025/reactions
2022-12-15T17:51:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1353486025
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq just checking, does this change still preserve the fix to the "data duplicate when setting num_works > 1 with streaming data" issue from before?

https://github.com/huggingface/datasets/issues/3423
2023-01-04T04:46:34Z
https://github.com/huggingface/datasets/pull/5357#issuecomment-1370483615
https://api.github.com/repos/huggingface/datasets/issues/5357
IC_kwDODunzps5Rr-uf
https://api.github.com/repos/huggingface/datasets/issues/comments/1370483615/reactions
2023-01-04T04:46:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1370483615
https://avatars.githubusercontent.com/u/10822753?v=4
https://api.github.com/users/corbyrosset/events{/privacy}
https://api.github.com/users/corbyrosset/followers
https://api.github.com/users/corbyrosset/following{/other_user}
https://api.github.com/users/corbyrosset/gists{/gist_id}
https://github.com/corbyrosset
corbyrosset
MDQ6VXNlcjEwODIyNzUz
https://api.github.com/users/corbyrosset/orgs
https://api.github.com/users/corbyrosset/received_events
https://api.github.com/users/corbyrosset/repos
https://api.github.com/users/corbyrosset/starred{/owner}{/repo}
https://api.github.com/users/corbyrosset/subscriptions
User
https://api.github.com/users/corbyrosset
public
MEMBER
Yes :)
2023-01-04T12:45:40Z
https://github.com/huggingface/datasets/pull/5357#issuecomment-1370888148
https://api.github.com/repos/huggingface/datasets/issues/5357
IC_kwDODunzps5RthfU
https://api.github.com/repos/huggingface/datasets/issues/comments/1370888148/reactions
2023-01-04T12:45:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1370888148
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
In https://github.com/huggingface/datasets/pull/5084 we make the torch formatting consistent with the map-style datasets formatting: a torch formatted iterable dataset will yield torch tensors.

The previous behavior of the torch formatting for iterable dataset was simply to make the iterable dataset inherit from `torch.utils.data.Dataset` to make it work in a torch DataLoader. However ideally an unformatted dataset should also work with a DataLoader. To fix that, `datasets.IterableDataset` should inherit from `torch.utils.data.IterableDataset`.

Since we don't want to import torch on startup, I created this PR to dynamically make the `datasets.IterableDataset` class inherit form the torch one when a `datasets.IterableDataset` is instantiated and if PyTorch is available.

```python
>>> from datasets import load_dataset
>>> ds = load_dataset("c4", "en", streaming=True, split="train")
>>> import torch.utils.data
>>> isinstance(ds, torch.utils.data.IterableDataset)
True
>>> dataloader = torch.utils.data.DataLoader(ds, batch_size=32, num_workers=4)
>>> for example in dataloader:
...:     ...
```