Improve torch formatting performance
MEMBER
That’s interesting thanks, let’s see what we can do. Can you detail your last sentence? I’m not sure I understand it well.
2021-06-14T13:45:58Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-860697462
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2MDY5NzQ2Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/860697462/reactions
2021-06-14T13:45:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/860697462
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Hi ! I just re-ran a quick benchmark and using `to_numpy()` seems to be faster now:

```python
import pyarrow as pa  # I used pyarrow 3.0.0
import numpy as np

n, max_length = 1_000, 512
low, high, size = 0, 2 << 16, (n, max_length)

table = pa.Table.from_pydict({
    "input_ids": np.random.default_rng(42).integers(low=low, high=high, size=size).tolist()
})


%%timeit
_ = table.to_pandas()["input_ids"].to_numpy()
# 1.44 ms ± 80.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
_ = table["input_ids"].to_pandas().to_numpy()
# 461 µs ± 14.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
_ = table["input_ids"].to_numpy()
# 317 µs ± 5.06 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

Currently the conversion from arrow to numpy is done in the NumpyArrowExtractor here:

https://github.com/huggingface/datasets/blob/d6d0ede9486ffad7944642ca9a326e058b676788/src/datasets/formatting/formatting.py#L143-L166

Let's update the NumpyArrowExtractor to call `to_numpy` directly and see how our github benchmarks evolve ?__
2021-06-14T14:47:42Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-860745032
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2MDc0NTAzMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/860745032/reactions
2021-06-14T14:48:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/860745032
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Sounds like a plan @lhoestq  If you create a PR I'll pick it up and try it out right away! 
2021-06-14T15:05:11Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-860759049
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2MDc1OTA0OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/860759049/reactions
2021-06-14T15:05:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/860759049
https://avatars.githubusercontent.com/u/458335?v=4
https://api.github.com/users/vblagoje/events{/privacy}
https://api.github.com/users/vblagoje/followers
https://api.github.com/users/vblagoje/following{/other_user}
https://api.github.com/users/vblagoje/gists{/gist_id}
https://github.com/vblagoje
vblagoje
MDQ6VXNlcjQ1ODMzNQ==
https://api.github.com/users/vblagoje/orgs
https://api.github.com/users/vblagoje/received_events
https://api.github.com/users/vblagoje/repos
https://api.github.com/users/vblagoje/starred{/owner}{/repo}
https://api.github.com/users/vblagoje/subscriptions
User
https://api.github.com/users/vblagoje
public
CONTRIBUTOR
@lhoestq I can also prepare the PR, just lmk. 
2021-06-14T17:36:29Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-860865439
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2MDg2NTQzOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/860865439/reactions
2021-06-14T17:36:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/860865439
https://avatars.githubusercontent.com/u/458335?v=4
https://api.github.com/users/vblagoje/events{/privacy}
https://api.github.com/users/vblagoje/followers
https://api.github.com/users/vblagoje/following{/other_user}
https://api.github.com/users/vblagoje/gists{/gist_id}
https://github.com/vblagoje
vblagoje
MDQ6VXNlcjQ1ODMzNQ==
https://api.github.com/users/vblagoje/orgs
https://api.github.com/users/vblagoje/received_events
https://api.github.com/users/vblagoje/repos
https://api.github.com/users/vblagoje/starred{/owner}{/repo}
https://api.github.com/users/vblagoje/subscriptions
User
https://api.github.com/users/vblagoje
public
MEMBER
I’m not exactly sure how to read the graph but it seems that to_categorical take a lot of time here. Could you share more informations on the features/stats of your datasets so we could maybe design a synthetic datasets that looks more similar for debugging testing?
2021-06-14T17:55:44Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-860877573
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2MDg3NzU3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/860877573/reactions
2021-06-14T17:55:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/860877573
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
I created https://github.com/huggingface/datasets/pull/2505 if you want to play with it @vblagoje 
2021-06-15T10:12:23Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-861373578
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2MTM3MzU3OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/861373578/reactions
2021-06-15T10:12:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/861373578
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> I’m not exactly sure how to read the graph but it seems that to_categorical take a lot of time here. Could you share more informations on the features/stats of your datasets so we could maybe design a synthetic datasets that looks more similar for debugging testing?

@thomwolf starting from the top, each rectangle represents the cumulative amount of it takes to execute the method call. Therefore, format_batch in torch_formatter.py takes ~20 sec, and the largest portion of that call is taken by to_pandas call and the smaller portion (grey rectangle) by the other method invocation(s) in format_batch (series_to_numpy etc). 

Features of the dataset are BERT pre-training model input columns i.e:
```
f = Features({            
 "input_ids": Sequence(feature=Value(dtype="int32")),           
 "attention_mask": Sequence(feature=Value(dtype="int8")),            
 "token_type_ids": Sequence(feature=Value(dtype="int8")),           
 "labels": Sequence(feature=Value(dtype="int32")),           
 "next_sentence_label": Value(dtype="int8")
})
```

I'll work with @lhoestq till we get to the bottom of this one. 
 
2021-06-15T10:51:52Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-861397862
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2MTM5Nzg2Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/861397862/reactions
2021-06-15T10:52:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/861397862
https://avatars.githubusercontent.com/u/458335?v=4
https://api.github.com/users/vblagoje/events{/privacy}
https://api.github.com/users/vblagoje/followers
https://api.github.com/users/vblagoje/following{/other_user}
https://api.github.com/users/vblagoje/gists{/gist_id}
https://github.com/vblagoje
vblagoje
MDQ6VXNlcjQ1ODMzNQ==
https://api.github.com/users/vblagoje/orgs
https://api.github.com/users/vblagoje/received_events
https://api.github.com/users/vblagoje/repos
https://api.github.com/users/vblagoje/starred{/owner}{/repo}
https://api.github.com/users/vblagoje/subscriptions
User
https://api.github.com/users/vblagoje
public
CONTRIBUTOR
@lhoestq the proposed branch is faster, but overall training speedup is a few percentage points. I couldn't figure out how to include the GitHub branch into setup.py, so I couldn't start NVidia optimized Docker-based pre-training run.  But on bare metal, there is a slight improvement. I'll do some more performance traces. 
2021-06-15T12:54:59Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-861473459
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2MTQ3MzQ1OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/861473459/reactions
2021-06-15T12:55:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/861473459
https://avatars.githubusercontent.com/u/458335?v=4
https://api.github.com/users/vblagoje/events{/privacy}
https://api.github.com/users/vblagoje/followers
https://api.github.com/users/vblagoje/following{/other_user}
https://api.github.com/users/vblagoje/gists{/gist_id}
https://github.com/vblagoje
vblagoje
MDQ6VXNlcjQ1ODMzNQ==
https://api.github.com/users/vblagoje/orgs
https://api.github.com/users/vblagoje/received_events
https://api.github.com/users/vblagoje/repos
https://api.github.com/users/vblagoje/starred{/owner}{/repo}
https://api.github.com/users/vblagoje/subscriptions
User
https://api.github.com/users/vblagoje
public
MEMBER
Hi @vblagoje, to install Datasets from @lhoestq PR reference #2505, you can use:
```shell
pip install git+ssh://git@github.com/huggingface/datasets.git@refs/pull/2505/head#egg=datasets
```
2021-06-16T08:55:24Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-862181573
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2MjE4MTU3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/862181573/reactions
2021-06-16T08:55:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/862181573
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
Hey @albertvillanova yes thank you, I am aware, I can easily pull it from a terminal command line but then I can't automate docker image builds as dependencies are picked up from setup.py and for some reason setup.py doesn't accept this string format.
2021-06-16T09:04:26Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-862187724
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2MjE4NzcyNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/862187724/reactions
2021-06-16T09:04:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/862187724
https://avatars.githubusercontent.com/u/458335?v=4
https://api.github.com/users/vblagoje/events{/privacy}
https://api.github.com/users/vblagoje/followers
https://api.github.com/users/vblagoje/following{/other_user}
https://api.github.com/users/vblagoje/gists{/gist_id}
https://github.com/vblagoje
vblagoje
MDQ6VXNlcjQ1ODMzNQ==
https://api.github.com/users/vblagoje/orgs
https://api.github.com/users/vblagoje/received_events
https://api.github.com/users/vblagoje/repos
https://api.github.com/users/vblagoje/starred{/owner}{/repo}
https://api.github.com/users/vblagoje/subscriptions
User
https://api.github.com/users/vblagoje
public
MEMBER
@vblagoje in that case, you can add this to your `setup.py`:
```python
    install_requires=[
        "datasets @ git+ssh://git@github.com/huggingface/datasets.git@refs/pull/2505/head",
```
2021-06-16T09:43:45Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-862213983
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2MjIxMzk4Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/862213983/reactions
2021-06-16T09:43:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/862213983
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
@lhoestq @thomwolf @albertvillanova The new approach is definitely faster, dataloader now takes less than 3% cumulative time (pink rectangle two rectangles to the right of tensor.py backward invocation)

![Screen Shot 2021-06-16 at 3 05 06 PM](https://user-images.githubusercontent.com/458335/122224432-19de4700-ce82-11eb-982f-d45d4bcc1e41.png)

When we drill down into dataloader next invocation we get:

![Screen Shot 2021-06-16 at 3 09 56 PM](https://user-images.githubusercontent.com/458335/122224976-a1c45100-ce82-11eb-8d40-59194740d616.png)

And finally format_batch:

![Screen Shot 2021-06-16 at 3 11 07 PM](https://user-images.githubusercontent.com/458335/122225132-cae4e180-ce82-11eb-8a16-967ab7c1c2aa.png)


Not sure this could be further improved but this is definitely a decent step forward.


2021-06-16T13:13:01Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-862368831
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2MjM2ODgzMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/862368831/reactions
2021-06-16T13:13:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/862368831
https://avatars.githubusercontent.com/u/458335?v=4
https://api.github.com/users/vblagoje/events{/privacy}
https://api.github.com/users/vblagoje/followers
https://api.github.com/users/vblagoje/following{/other_user}
https://api.github.com/users/vblagoje/gists{/gist_id}
https://github.com/vblagoje
vblagoje
MDQ6VXNlcjQ1ODMzNQ==
https://api.github.com/users/vblagoje/orgs
https://api.github.com/users/vblagoje/received_events
https://api.github.com/users/vblagoje/repos
https://api.github.com/users/vblagoje/starred{/owner}{/repo}
https://api.github.com/users/vblagoje/subscriptions
User
https://api.github.com/users/vblagoje
public
CONTRIBUTOR
> ```python
> datasets @ git+ssh://git@github.com/huggingface/datasets.git@refs/pull/2505/head
> ```

@albertvillanova how would I replace datasets dependency in https://github.com/huggingface/transformers/blob/master/setup.py as the above approach is not working. 
2021-06-19T07:34:44Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-864370148
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2NDM3MDE0OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/864370148/reactions
2021-06-19T07:34:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/864370148
https://avatars.githubusercontent.com/u/458335?v=4
https://api.github.com/users/vblagoje/events{/privacy}
https://api.github.com/users/vblagoje/followers
https://api.github.com/users/vblagoje/following{/other_user}
https://api.github.com/users/vblagoje/gists{/gist_id}
https://github.com/vblagoje
vblagoje
MDQ6VXNlcjQ1ODMzNQ==
https://api.github.com/users/vblagoje/orgs
https://api.github.com/users/vblagoje/received_events
https://api.github.com/users/vblagoje/repos
https://api.github.com/users/vblagoje/starred{/owner}{/repo}
https://api.github.com/users/vblagoje/subscriptions
User
https://api.github.com/users/vblagoje
public
MEMBER
@vblagoje I tested my proposed approach before posting it here and it worked for me. 

Is it not working in your case because of the SSH protocol? In that case you could try the same approach but using HTTPS:
```
"datasets @ git+https://github.com/huggingface/datasets.git@refs/pull/2505/head",
``` 
2021-06-21T09:19:59Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-864876980
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2NDg3Njk4MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/864876980/reactions
2021-06-21T09:21:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/864876980
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Also note the blanks before and after the `@`.
2021-06-21T09:20:51Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-864877627
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2NDg3NzYyNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/864877627/reactions
2021-06-21T09:20:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/864877627
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
@albertvillanova of course it works. Apologies. I needed to change datasets in all deps references , like [here](https://github.com/huggingface/transformers/blob/master/setup.py#L235) for example. 
2021-06-21T09:54:21Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-864899905
https://api.github.com/repos/huggingface/datasets/issues/2498
MDEyOklzc3VlQ29tbWVudDg2NDg5OTkwNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/864899905/reactions
2021-06-21T09:54:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/864899905
https://avatars.githubusercontent.com/u/458335?v=4
https://api.github.com/users/vblagoje/events{/privacy}
https://api.github.com/users/vblagoje/followers
https://api.github.com/users/vblagoje/following{/other_user}
https://api.github.com/users/vblagoje/gists{/gist_id}
https://github.com/vblagoje
vblagoje
MDQ6VXNlcjQ1ODMzNQ==
https://api.github.com/users/vblagoje/orgs
https://api.github.com/users/vblagoje/received_events
https://api.github.com/users/vblagoje/repos
https://api.github.com/users/vblagoje/starred{/owner}{/repo}
https://api.github.com/users/vblagoje/subscriptions
User
https://api.github.com/users/vblagoje
public
NONE
Is time spent casting an issue here? See https://github.com/huggingface/datasets/issues/4676 that Datasets can spend huge amounts of time repeatedly casting to Python objects.
2022-07-15T17:12:04Z
https://github.com/huggingface/datasets/issues/2498#issuecomment-1185740393
https://api.github.com/repos/huggingface/datasets/issues/2498
IC_kwDODunzps5GrPZp
https://api.github.com/repos/huggingface/datasets/issues/comments/1185740393/reactions
2022-07-15T17:12:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1185740393
https://avatars.githubusercontent.com/u/662612?v=4
https://api.github.com/users/srobertjames/events{/privacy}
https://api.github.com/users/srobertjames/followers
https://api.github.com/users/srobertjames/following{/other_user}
https://api.github.com/users/srobertjames/gists{/gist_id}
https://github.com/srobertjames
srobertjames
MDQ6VXNlcjY2MjYxMg==
https://api.github.com/users/srobertjames/orgs
https://api.github.com/users/srobertjames/received_events
https://api.github.com/users/srobertjames/repos
https://api.github.com/users/srobertjames/starred{/owner}{/repo}
https://api.github.com/users/srobertjames/subscriptions
User
https://api.github.com/users/srobertjames
public
**Is your feature request related to a problem? Please describe.**
It would be great, if possible, to further improve read performance of raw encoded datasets and their subsequent conversion to torch tensors. 

A bit more background.  I am working on LM pre-training using HF ecosystem. We use encoded HF Wikipedia and BookCorpus datasets. The training machines are similar to DGX-1 workstations. We use HF trainer torch.distributed training approach on a single machine with 8 GPUs.

The current performance is about 30% slower than NVidia optimized BERT [examples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling) baseline. Quite a bit of customized code and training loop tricks were used to achieve the baseline performance. It would be great to achieve the same performance while using nothing more than off the shelf HF ecosystem. Perhaps, in the future, with @stas00 work on deepspeed integration, it could even be exceeded. 

**Describe the solution you'd like**
Using profiling tools we've observed that appx. 25% of cumulative run time is spent on data loader next call.
![dataloader_next](https://user-images.githubusercontent.com/458335/121895543-59742a00-ccee-11eb-85fb-f07715e3f1f6.png)

As you can observe most of the data loader next call is spent in HF datasets torch_formatter.py format_batch call. 

Digging a bit deeper into format_batch we can see the following profiler data:
![torch_formatter](https://user-images.githubusercontent.com/458335/121895944-c7b8ec80-ccee-11eb-95d5-5875c5716c30.png)

Once again, a lot of time is spent in pyarrow table conversion to pandas which seems like an intermediary step. Offline @lhoestq told me that this approach was, for some unknown reason, faster than direct to numpy conversion. 

**Describe alternatives you've considered**
I am not familiar with pyarrow and have not yet considered the alternatives to the current approach. 

Most of the online advice around data loader performance improvements revolve around increasing number of workers, using pin memory for copying tensors from host device to gpus but we've already tried these avenues without much performance improvement.  Weights & Biases dashboard for the pre-training task reports CPU utilization of ~ 10%, GPUs are completely saturated (GPU utilization is above 95% on all GPUs), while disk utilization is above 90%. 

