How to convert datasets.arrow_dataset.Dataset to torch.utils.data.Dataset
COLLABORATOR
Hi,

a HF dataset can be converted to a Torch Dataset with a simple wrapper as follows:
```python
from torch.utils.data import Dataset
 
class HFDataset(Dataset):
    def __init__(self, dset):
        self.dset = dset

    def __getitem__(self, idx):
        return self.dset[idx]

    def __len__(self):
        return len(self.dset)

train_ds = HFDataset(train_ds)
```
@lhoestq Since the Arrow Dataset already provides `__getitem__` and `__len__`, I think we could use the [virtual subclass](https://docs.python.org/3/library/abc.html#abc.ABCMeta.register) mechanism from the `abc` module to elegantly solve this issue. This mechanism would allow the Arrow Dataset to be used in place of the Torch Dataset because the `isinstance(instance of Arrow Dataset, TorchDataset)` check would return True (DeepSpeed has this check [here](https://github.com/microsoft/DeepSpeed/blob/ab5534fc4c0f8ca21ada321f9730d723aa31288b/deepspeed/runtime/engine.py#L823)).

And it requires a minimal change in the `arrow_dataset.py` file:
```python
if config.TORCH_AVAILABLE:
    from torch.utils.data import Dataset as TorchDataset
    TorchDataset.register(Dataset)
```
2021-04-04T16:19:58Z
https://github.com/huggingface/datasets/issues/2165#issuecomment-813059718
https://api.github.com/repos/huggingface/datasets/issues/2165
MDEyOklzc3VlQ29tbWVudDgxMzA1OTcxOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/813059718/reactions
2021-04-04T16:19:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/813059718
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Interesting ! Thanks for sharing this @mariosasko . I like the idea
This looks like something we should add IMO
2021-04-07T14:28:36Z
https://github.com/huggingface/datasets/issues/2165#issuecomment-814961327
https://api.github.com/repos/huggingface/datasets/issues/2165
MDEyOklzc3VlQ29tbWVudDgxNDk2MTMyNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/814961327/reactions
2021-04-07T14:28:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814961327
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@mariosasko 
Thx for your code!
It perfectly works with a small modification for HF NLP dataset:
```
original_ds = nlp.load_dataset('scientific_papers', 'arxiv')
train_ds = HFDataset(train_ds['train']) # needs splitting
```
2021-04-07T15:06:04Z
https://github.com/huggingface/datasets/issues/2165#issuecomment-814990976
https://api.github.com/repos/huggingface/datasets/issues/2165
MDEyOklzc3VlQ29tbWVudDgxNDk5MDk3Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/814990976/reactions
2021-04-07T15:06:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814990976
https://avatars.githubusercontent.com/u/24562381?v=4
https://api.github.com/users/y-rokutan/events{/privacy}
https://api.github.com/users/y-rokutan/followers
https://api.github.com/users/y-rokutan/following{/other_user}
https://api.github.com/users/y-rokutan/gists{/gist_id}
https://github.com/y-rokutan
y-rokutan
MDQ6VXNlcjI0NTYyMzgx
https://api.github.com/users/y-rokutan/orgs
https://api.github.com/users/y-rokutan/received_events
https://api.github.com/users/y-rokutan/repos
https://api.github.com/users/y-rokutan/starred{/owner}{/repo}
https://api.github.com/users/y-rokutan/subscriptions
User
https://api.github.com/users/y-rokutan
public
COLLABORATOR
@lhoestq Sadly, from Python 3.7 onwards `torch.utils.data.Dataset` doesn't support the virtual subclass mechanism due to `typing.Generic` type no longer having `abc.ABCMeta` as its metaclass.

With that in mind, another option is to remove a direct type check (`isinstance(dataset, torch.utils.data.Dataset)`) in `deepspeed.initalize` and to rewrite the checks in a manner similar to `torch.utils.data.DataLoader` ([link](https://github.com/pytorch/pytorch/blob/b80c6f863f2327c712c478f67c248b94d66b65ac/torch/utils/data/dataloader.py#L197-L239)). This is exactly why the `DataLoader` works with arbitrary objects that provide `__getitem__` and `__len__` (and in our case, the `ArrowDataset`). By doing so, their code wouldn't be any stricter in comparison to the `DataLoader`.

So if you agree, I can open an issue in their repo and fix this if they like the idea.
2021-04-11T20:36:28Z
https://github.com/huggingface/datasets/issues/2165#issuecomment-817368840
https://api.github.com/repos/huggingface/datasets/issues/2165
MDEyOklzc3VlQ29tbWVudDgxNzM2ODg0MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/817368840/reactions
2021-04-11T20:36:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/817368840
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
That makes sense ! Feel free to open an issue on their repo and discuss this idea
2021-04-12T13:36:09Z
https://github.com/huggingface/datasets/issues/2165#issuecomment-817817762
https://api.github.com/repos/huggingface/datasets/issues/2165
MDEyOklzc3VlQ29tbWVudDgxNzgxNzc2Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/817817762/reactions
2021-04-12T13:36:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/817817762
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
@y-rokutan Hi, now if you install `deepspeed` from master (this feature will be available in the next official release), the code should work without subclassing. Let us know if you still have any issues.
2021-05-02T11:01:05Z
https://github.com/huggingface/datasets/issues/2165#issuecomment-830790575
https://api.github.com/repos/huggingface/datasets/issues/2165
MDEyOklzc3VlQ29tbWVudDgzMDc5MDU3NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/830790575/reactions
2021-05-02T11:01:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/830790575
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Worth mentioning that any function that expects a `torch..Dataset` (like `torch..DataLoader`) will fail a mypy-esque typecheck if a `datasets.Dataset` is passed, even though it implements the interface correctly (I think). The virtual subclass idea was a good one- I wonder if there's another workaround given the Generic issue. What we're really talking about is something similar to the structural subtyping semantics that `typing.Protocol` defines. If `torch..DataLoader` accepted anything that supports `__getitem__` and `__len__` methods this would be much easier. Not sure if there's a way to do this without the wrapper from the perspective of `datasets`.
2021-08-24T15:55:35Z
https://github.com/huggingface/datasets/issues/2165#issuecomment-904765936
https://api.github.com/repos/huggingface/datasets/issues/2165
IC_kwDODunzps417aHw
https://api.github.com/repos/huggingface/datasets/issues/comments/904765936/reactions
2021-08-24T15:55:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/904765936
https://avatars.githubusercontent.com/u/9934829?v=4
https://api.github.com/users/aronszanto/events{/privacy}
https://api.github.com/users/aronszanto/followers
https://api.github.com/users/aronszanto/following{/other_user}
https://api.github.com/users/aronszanto/gists{/gist_id}
https://github.com/aronszanto
aronszanto
MDQ6VXNlcjk5MzQ4Mjk=
https://api.github.com/users/aronszanto/orgs
https://api.github.com/users/aronszanto/received_events
https://api.github.com/users/aronszanto/repos
https://api.github.com/users/aronszanto/starred{/owner}{/repo}
https://api.github.com/users/aronszanto/subscriptions
User
https://api.github.com/users/aronszanto
public
Hi,

I'm trying to pretraine deep-speed model using HF arxiv dataset like:
```
train_ds = nlp.load_dataset('scientific_papers', 'arxiv')
train_ds.set_format(
        type="torch",
        columns=["input_ids", "attention_mask", "global_attention_mask", "labels"],
    )
engine, _, _, _ = deepspeed.initialize(
    args=args,
    model=model,
    model_parameters=[p for p in model.parameters() if p.requires_grad],
    training_data=train_ds)
```
but deepspeed.initialize accepts torch.utils.data.Dataset only. How can I convert HF-style dataset to torch-style dataset?
