trainer became extremely slow after reload dataset by `load_from_disk`
NONE
I tried to make the dataset much more smaller (100000 rows) , then the speed became `33.88it/s` from`8.62s/it`.  It's nearly 200 times... Do you have any idea? Thank you!
2022-05-23T14:22:48Z
https://github.com/huggingface/datasets/issues/4394#issuecomment-1134744017
https://api.github.com/repos/huggingface/datasets/issues/4394
IC_kwDODunzps5DotHR
https://api.github.com/repos/huggingface/datasets/issues/comments/1134744017/reactions
2022-05-23T15:36:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1134744017
https://avatars.githubusercontent.com/u/50416856?v=4
https://api.github.com/users/conan1024hao/events{/privacy}
https://api.github.com/users/conan1024hao/followers
https://api.github.com/users/conan1024hao/following{/other_user}
https://api.github.com/users/conan1024hao/gists{/gist_id}
https://github.com/conan1024hao
conan1024hao
MDQ6VXNlcjUwNDE2ODU2
https://api.github.com/users/conan1024hao/orgs
https://api.github.com/users/conan1024hao/received_events
https://api.github.com/users/conan1024hao/repos
https://api.github.com/users/conan1024hao/starred{/owner}{/repo}
https://api.github.com/users/conan1024hao/subscriptions
User
https://api.github.com/users/conan1024hao
public
NONE
Similar issue: https://github.com/huggingface/transformers/issues/8818

I changed `RandomSampler` to  `SequentialSampler` in the `trainer.py`, but the speed didn't become faster.
2022-05-23T14:32:46Z
https://github.com/huggingface/datasets/issues/4394#issuecomment-1134755807
https://api.github.com/repos/huggingface/datasets/issues/4394
IC_kwDODunzps5Dov_f
https://api.github.com/repos/huggingface/datasets/issues/comments/1134755807/reactions
2022-05-23T15:34:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1134755807
https://avatars.githubusercontent.com/u/50416856?v=4
https://api.github.com/users/conan1024hao/events{/privacy}
https://api.github.com/users/conan1024hao/followers
https://api.github.com/users/conan1024hao/following{/other_user}
https://api.github.com/users/conan1024hao/gists{/gist_id}
https://github.com/conan1024hao
conan1024hao
MDQ6VXNlcjUwNDE2ODU2
https://api.github.com/users/conan1024hao/orgs
https://api.github.com/users/conan1024hao/received_events
https://api.github.com/users/conan1024hao/repos
https://api.github.com/users/conan1024hao/starred{/owner}{/repo}
https://api.github.com/users/conan1024hao/subscriptions
User
https://api.github.com/users/conan1024hao
public
NONE
I changed
```
tokenized_datasets = load_from_disk(
        "/pathto/dataset"
    )
```
to
```
tokenized_datasets = load_from_disk(
        "/pathto/dataset", keep_in_memory=True
    )
```
and obtained normal speed. It's seems that the problem is on the os's speed limit.
2022-05-23T16:02:41Z
https://github.com/huggingface/datasets/issues/4394#issuecomment-1134861269
https://api.github.com/repos/huggingface/datasets/issues/4394
IC_kwDODunzps5DpJvV
https://api.github.com/repos/huggingface/datasets/issues/comments/1134861269/reactions
2022-05-23T16:02:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1134861269
https://avatars.githubusercontent.com/u/50416856?v=4
https://api.github.com/users/conan1024hao/events{/privacy}
https://api.github.com/users/conan1024hao/followers
https://api.github.com/users/conan1024hao/following{/other_user}
https://api.github.com/users/conan1024hao/gists{/gist_id}
https://github.com/conan1024hao
conan1024hao
MDQ6VXNlcjUwNDE2ODU2
https://api.github.com/users/conan1024hao/orgs
https://api.github.com/users/conan1024hao/received_events
https://api.github.com/users/conan1024hao/repos
https://api.github.com/users/conan1024hao/starred{/owner}{/repo}
https://api.github.com/users/conan1024hao/subscriptions
User
https://api.github.com/users/conan1024hao
public
MEMBER
Hi ! Currently `save_to_disk` saves one big Arrow file, which causes some slow downs. This has been discussed in #3735 and we'll implement sharding pretty soon to solve this

For now you can try splitting and saving your dataset in several Arrow files. Then you can load them one by one and use `concatenate_datasets` to have your big dataset again and hopefully with a better speed
2022-06-06T16:06:29Z
https://github.com/huggingface/datasets/issues/4394#issuecomment-1147619423
https://api.github.com/repos/huggingface/datasets/issues/4394
IC_kwDODunzps5EZ0hf
https://api.github.com/repos/huggingface/datasets/issues/comments/1147619423/reactions
2022-06-06T16:08:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1147619423
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Any update on fixing this? The issue still seems to be present.
2023-11-23T07:40:30Z
https://github.com/huggingface/datasets/issues/4394#issuecomment-1823925909
https://api.github.com/repos/huggingface/datasets/issues/4394
IC_kwDODunzps5stuaV
https://api.github.com/repos/huggingface/datasets/issues/comments/1823925909/reactions
2023-11-23T07:40:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1823925909
https://avatars.githubusercontent.com/u/1286898?v=4
https://api.github.com/users/Ubadub/events{/privacy}
https://api.github.com/users/Ubadub/followers
https://api.github.com/users/Ubadub/following{/other_user}
https://api.github.com/users/Ubadub/gists{/gist_id}
https://github.com/Ubadub
Ubadub
MDQ6VXNlcjEyODY4OTg=
https://api.github.com/users/Ubadub/orgs
https://api.github.com/users/Ubadub/received_events
https://api.github.com/users/Ubadub/repos
https://api.github.com/users/Ubadub/starred{/owner}{/repo}
https://api.github.com/users/Ubadub/subscriptions
User
https://api.github.com/users/Ubadub
public
## Describe the bug
Due to memory problem, I need to save my tokenized datasets locally by CPU and reload it by multi GPU for running training script. However, after I reload it by `load_from_disk` and start training, the speed is extremely slow. It says I need about 1500 hours with 8 A100 cards. Before this, I can run the whole script in one day with a single A100 card.
Since I am try to pre-train a BERT, **my dataset is very large(29058165 rows)**

## Steps to reproduce the bug
```python
tokenized_datasets.save_to_disk(
        "/pathto/dataset"
    )
tokenized_datasets = load_from_disk(
        "/pathto/dataset"
    )
trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=tokenized_datasets["train"] if training_args.do_train else None,
        eval_dataset=tokenized_datasets["validation"]
        if training_args.do_eval
        else None,
        tokenizer=tokenizer,
        data_collator=data_collator,
    )
 train_result = trainer.train(resume_from_checkpoint=checkpoint)
```

## Expected results
Without the save and reload process, I only need about one day to run the whole script with one A100 card.

## Actual results
```
[INFO|trainer.py:1290] 2022-05-23 22:49:46,266 >> ***** Running training *****
[INFO|trainer.py:1291] 2022-05-23 22:49:46,266 >>   Num examples = 29058165
[INFO|trainer.py:1292] 2022-05-23 22:49:46,266 >>   Num Epochs = 5
[INFO|trainer.py:1293] 2022-05-23 22:49:46,266 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:1294] 2022-05-23 22:49:46,266 >>   Total train batch size (w. parallel, distributed & accumulation) = 256
[INFO|trainer.py:1295] 2022-05-23 22:49:46,266 >>   Gradient Accumulation steps = 2
[INFO|trainer.py:1296] 2022-05-23 22:49:46,266 >>   Total optimization steps = 567540
  0%|          | 1/567540 [00:09<1544:49:04,  9.80s/it]
  0%|          | 2/567540 [00:17<1320:00:17,  8.37s/it]
  0%|          | 3/567540 [00:26<1393:10:17,  8.84s/it]
  0%|          | 4/567540 [00:34<1344:56:33,  8.53s/it]
  0%|          | 5/567540 [00:43<1359:36:12,  8.62s/it]
```

## Environment info
```
torch              1.11.0+cu113
torchaudio         0.11.0+cu113
torchvision        0.12.0+cu113
transformers       4.18.0
datasets           2.2.2
```