Load Data Sets Too Slow In Train Seq2seq Model
MEMBER
Hi ! you can speed it up using multiprocessing by passing `num_proc=` to `load_dataset()`
2023-06-14T12:22:00Z
https://github.com/huggingface/datasets/issues/5941#issuecomment-1591088082
https://api.github.com/repos/huggingface/datasets/issues/5941
IC_kwDODunzps5e1hPS
https://api.github.com/repos/huggingface/datasets/issues/comments/1591088082/reactions
2023-06-14T12:22:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1591088082
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
already did，but not useful for step Generating train split，it works in step "Resolving data files" & "Downloading data files" 
2023-06-15T00:53:46Z
https://github.com/huggingface/datasets/issues/5941#issuecomment-1592173628
https://api.github.com/repos/huggingface/datasets/issues/5941
IC_kwDODunzps5e5qQ8
https://api.github.com/repos/huggingface/datasets/issues/comments/1592173628/reactions
2023-06-15T00:53:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1592173628
https://avatars.githubusercontent.com/u/19569322?v=4
https://api.github.com/users/xyx361100238/events{/privacy}
https://api.github.com/users/xyx361100238/followers
https://api.github.com/users/xyx361100238/following{/other_user}
https://api.github.com/users/xyx361100238/gists{/gist_id}
https://github.com/xyx361100238
xyx361100238
MDQ6VXNlcjE5NTY5MzIy
https://api.github.com/users/xyx361100238/orgs
https://api.github.com/users/xyx361100238/received_events
https://api.github.com/users/xyx361100238/repos
https://api.github.com/users/xyx361100238/starred{/owner}{/repo}
https://api.github.com/users/xyx361100238/subscriptions
User
https://api.github.com/users/xyx361100238
public
NONE
@mariosasko some advice ， thanks！
2023-06-15T00:55:58Z
https://github.com/huggingface/datasets/issues/5941#issuecomment-1592174718
https://api.github.com/repos/huggingface/datasets/issues/5941
IC_kwDODunzps5e5qh-
https://api.github.com/repos/huggingface/datasets/issues/comments/1592174718/reactions
2023-06-15T00:55:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1592174718
https://avatars.githubusercontent.com/u/19569322?v=4
https://api.github.com/users/xyx361100238/events{/privacy}
https://api.github.com/users/xyx361100238/followers
https://api.github.com/users/xyx361100238/following{/other_user}
https://api.github.com/users/xyx361100238/gists{/gist_id}
https://github.com/xyx361100238
xyx361100238
MDQ6VXNlcjE5NTY5MzIy
https://api.github.com/users/xyx361100238/orgs
https://api.github.com/users/xyx361100238/received_events
https://api.github.com/users/xyx361100238/repos
https://api.github.com/users/xyx361100238/starred{/owner}{/repo}
https://api.github.com/users/xyx361100238/subscriptions
User
https://api.github.com/users/xyx361100238
public
NONE
I met the same problem,  terrible  experience
2023-06-17T17:00:58Z
https://github.com/huggingface/datasets/issues/5941#issuecomment-1595807119
https://api.github.com/repos/huggingface/datasets/issues/5941
IC_kwDODunzps5fHhWP
https://api.github.com/repos/huggingface/datasets/issues/comments/1595807119/reactions
2023-06-17T17:00:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1595807119
https://avatars.githubusercontent.com/u/14243491?v=4
https://api.github.com/users/mengban/events{/privacy}
https://api.github.com/users/mengban/followers
https://api.github.com/users/mengban/following{/other_user}
https://api.github.com/users/mengban/gists{/gist_id}
https://github.com/mengban
mengban
MDQ6VXNlcjE0MjQzNDkx
https://api.github.com/users/mengban/orgs
https://api.github.com/users/mengban/received_events
https://api.github.com/users/mengban/repos
https://api.github.com/users/mengban/starred{/owner}{/repo}
https://api.github.com/users/mengban/subscriptions
User
https://api.github.com/users/mengban
public
NONE
@mariosasko 
2023-07-13T01:32:53Z
https://github.com/huggingface/datasets/issues/5941#issuecomment-1633407402
https://api.github.com/repos/huggingface/datasets/issues/5941
IC_kwDODunzps5hW9Gq
https://api.github.com/repos/huggingface/datasets/issues/comments/1633407402/reactions
2023-07-13T01:32:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1633407402
https://avatars.githubusercontent.com/u/19569322?v=4
https://api.github.com/users/xyx361100238/events{/privacy}
https://api.github.com/users/xyx361100238/followers
https://api.github.com/users/xyx361100238/following{/other_user}
https://api.github.com/users/xyx361100238/gists{/gist_id}
https://github.com/xyx361100238
xyx361100238
MDQ6VXNlcjE5NTY5MzIy
https://api.github.com/users/xyx361100238/orgs
https://api.github.com/users/xyx361100238/received_events
https://api.github.com/users/xyx361100238/repos
https://api.github.com/users/xyx361100238/starred{/owner}{/repo}
https://api.github.com/users/xyx361100238/subscriptions
User
https://api.github.com/users/xyx361100238
public
COLLABORATOR
We need more info about the issue to provide help. 

Can you interrupt the process (with `num_proc=None`) after the `load_dataset` call when the slowdown occurs? So we can know what part of the code is causing it.

The `audiofolder` \ `imagefolder` with metadata is not performant for large datasets. Luckily, we can make them much faster if drop the nested metadata files feature (not that useful). I plan to work on this soon.

In the meantime, it's better to use `Dataset.from_generator` (requires replacing the `load_dataset` calls in the transformers script with `Dataset.from_generator`) or write a dataset loading script for large datasets.
2023-07-13T19:15:10Z
https://github.com/huggingface/datasets/issues/5941#issuecomment-1634763738
https://api.github.com/repos/huggingface/datasets/issues/5941
IC_kwDODunzps5hcIPa
https://api.github.com/repos/huggingface/datasets/issues/comments/1634763738/reactions
2023-07-13T19:15:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1634763738
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Can you interrupt the process (with num_proc=None) after the load_dataset call when the slowdown occurs? So we can know what part of the code is causing it.
（I'll try this operation）
The audiofolder \ imagefolder with metadata is not performant for large datasets. Luckily, we can make them much faster if drop the nested metadata files feature (not that useful). I plan to work on this soon.
(My data is indeed a bit large, exceeding 10000 hours of audio data. Looking forward to your improvement work very much)

In the meantime, it's better to use Dataset.from_generator (requires replacing the load_dataset calls in the transformers script with Dataset.from_generator) or write a dataset loading script for large datasets.
（I want to use Dataset.from_generator instead of  load_dataset ，where can i found sample code to load audio&label dataset， I was to do asr task）
2023-07-18T02:03:44Z
https://github.com/huggingface/datasets/issues/5941#issuecomment-1639166049
https://api.github.com/repos/huggingface/datasets/issues/5941
IC_kwDODunzps5hs7Bh
https://api.github.com/repos/huggingface/datasets/issues/comments/1639166049/reactions
2023-07-18T02:03:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1639166049
https://avatars.githubusercontent.com/u/19569322?v=4
https://api.github.com/users/xyx361100238/events{/privacy}
https://api.github.com/users/xyx361100238/followers
https://api.github.com/users/xyx361100238/following{/other_user}
https://api.github.com/users/xyx361100238/gists{/gist_id}
https://github.com/xyx361100238
xyx361100238
MDQ6VXNlcjE5NTY5MzIy
https://api.github.com/users/xyx361100238/orgs
https://api.github.com/users/xyx361100238/received_events
https://api.github.com/users/xyx361100238/repos
https://api.github.com/users/xyx361100238/starred{/owner}{/repo}
https://api.github.com/users/xyx361100238/subscriptions
User
https://api.github.com/users/xyx361100238
public
NONE
Can you interrupt the process (with num_proc=None) after the load_dataset call when the slowdown occurs? So we can know what part of the code is causing it.
================================================================================
Here is the log：
[load_dataset.log](https://github.com/huggingface/datasets/files/12169362/load_dataset.log)
（The larger my training data, the slower it loads）
![image](https://github.com/huggingface/datasets/assets/19569322/381b73e4-0a54-4240-b95e-cb8164584047)


2023-07-26T07:47:23Z
https://github.com/huggingface/datasets/issues/5941#issuecomment-1651160451
https://api.github.com/repos/huggingface/datasets/issues/5941
IC_kwDODunzps5iarWD
https://api.github.com/repos/huggingface/datasets/issues/comments/1651160451/reactions
2023-07-26T07:47:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1651160451
https://avatars.githubusercontent.com/u/19569322?v=4
https://api.github.com/users/xyx361100238/events{/privacy}
https://api.github.com/users/xyx361100238/followers
https://api.github.com/users/xyx361100238/following{/other_user}
https://api.github.com/users/xyx361100238/gists{/gist_id}
https://github.com/xyx361100238
xyx361100238
MDQ6VXNlcjE5NTY5MzIy
https://api.github.com/users/xyx361100238/orgs
https://api.github.com/users/xyx361100238/received_events
https://api.github.com/users/xyx361100238/repos
https://api.github.com/users/xyx361100238/starred{/owner}{/repo}
https://api.github.com/users/xyx361100238/subscriptions
User
https://api.github.com/users/xyx361100238
public
NONE
In the meantime, it's better to use Dataset.from_generator (requires replacing the load_dataset calls in the transformers script with Dataset.from_generator) or write a dataset loading script for large datasets.
================================================================================
I tried ‘Dataset. from_generator’ implements data loading, but the testing results show no improvement
2023-07-26T07:49:35Z
https://github.com/huggingface/datasets/issues/5941#issuecomment-1651163443
https://api.github.com/repos/huggingface/datasets/issues/5941
IC_kwDODunzps5iasEz
https://api.github.com/repos/huggingface/datasets/issues/comments/1651163443/reactions
2023-07-26T07:49:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1651163443
https://avatars.githubusercontent.com/u/19569322?v=4
https://api.github.com/users/xyx361100238/events{/privacy}
https://api.github.com/users/xyx361100238/followers
https://api.github.com/users/xyx361100238/following{/other_user}
https://api.github.com/users/xyx361100238/gists{/gist_id}
https://github.com/xyx361100238
xyx361100238
MDQ6VXNlcjE5NTY5MzIy
https://api.github.com/users/xyx361100238/orgs
https://api.github.com/users/xyx361100238/received_events
https://api.github.com/users/xyx361100238/repos
https://api.github.com/users/xyx361100238/starred{/owner}{/repo}
https://api.github.com/users/xyx361100238/subscriptions
User
https://api.github.com/users/xyx361100238
public
NONE
I have already solved this problem, referring to #5990 : read audio frist, then use data_generator to change format .
2023-08-15T02:52:14Z
https://github.com/huggingface/datasets/issues/5941#issuecomment-1678355598
https://api.github.com/repos/huggingface/datasets/issues/5941
IC_kwDODunzps5kCayO
https://api.github.com/repos/huggingface/datasets/issues/comments/1678355598/reactions
2023-08-15T02:52:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1678355598
https://avatars.githubusercontent.com/u/19569322?v=4
https://api.github.com/users/xyx361100238/events{/privacy}
https://api.github.com/users/xyx361100238/followers
https://api.github.com/users/xyx361100238/following{/other_user}
https://api.github.com/users/xyx361100238/gists{/gist_id}
https://github.com/xyx361100238
xyx361100238
MDQ6VXNlcjE5NTY5MzIy
https://api.github.com/users/xyx361100238/orgs
https://api.github.com/users/xyx361100238/received_events
https://api.github.com/users/xyx361100238/repos
https://api.github.com/users/xyx361100238/starred{/owner}{/repo}
https://api.github.com/users/xyx361100238/subscriptions
User
https://api.github.com/users/xyx361100238
public
### Describe the bug


step 'Generating train split' in load_dataset is too slow：
![image](https://github.com/huggingface/datasets/assets/19569322/d9b08eee-95fe-4741-a346-b70416c948f8)


### Steps to reproduce the bug

Data： own data，16K16B Mono wav
Oficial Script:[ run_speech_recognition_seq2seq.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py)
Add Code：
    if data_args.data_path is not None:
        print(data_args.data_path)
        raw_datasets = load_dataset("audiofolder", data_dir=data_args.data_path, cache_dir=model_args.cache_dir)
        raw_datasets = raw_datasets.cast_column("audio", Audio(sampling_rate=16000))
        raw_datasets = raw_datasets["train"].train_test_split(test_size=0.005, shuffle=True)
 （change cache_dir to other path ，ex:/DATA/cache）

### Expected behavior

load data fast,at least 1000+
`Generating train split: 387875 examples [32:24:45, 1154.83 examples/s]`

### Environment info


- `transformers` version: 4.28.0.dev0
- Platform: Linux-5.4.0-149-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.13.2
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>