Very slow data loading on large dataset
MEMBER
When you load a text file for the first time with `nlp`, the file is converted into Apache Arrow format. Arrow allows to use memory-mapping, which means that you can load an arbitrary large dataset.

Note that as soon as the conversion has been done once, the next time you'll load the dataset it will be much faster.

However for a 1TB dataset, the conversion can indeed take time. You could try to load parts of it in parallel, and then use `nlp.concatenate_datasets` to get your full dataset.
2020-08-31T13:04:10Z
https://github.com/huggingface/datasets/issues/546#issuecomment-683764310
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4Mzc2NDMxMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/683764310/reactions
2020-08-31T13:04:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/683764310
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Humm, we can give a look at these large scale datasets indeed.

Do you mind sharing a few stats on your dataset so I can try to test on a similar one?

In particular some orders of magnitudes for the number of files, number of lines per files, line lengths.
2020-08-31T13:04:19Z
https://github.com/huggingface/datasets/issues/546#issuecomment-683764380
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4Mzc2NDM4MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/683764380/reactions
2020-08-31T13:04:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/683764380
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
@lhoestq Yes, I understand that the first time requires more time. The concatenate_datasets seems to be a workaround, but I believe a multi-processing method should be integrated into load_dataset to make it easier and more efficient for users.

@thomwolf Sure, here are the statistics:
Number of lines: 4.2 Billion
Number of files: 6K
Number of tokens: 800 Billion
The number of lines is distributed equally across these 6k files.
The line length varies between 100 tokens to 40k tokens.

2020-08-31T13:57:16Z
https://github.com/huggingface/datasets/issues/546#issuecomment-683795392
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4Mzc5NTM5Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/683795392/reactions
2020-08-31T13:57:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/683795392
https://avatars.githubusercontent.com/u/6087313?v=4
https://api.github.com/users/agemagician/events{/privacy}
https://api.github.com/users/agemagician/followers
https://api.github.com/users/agemagician/following{/other_user}
https://api.github.com/users/agemagician/gists{/gist_id}
https://github.com/agemagician
agemagician
MDQ6VXNlcjYwODczMTM=
https://api.github.com/users/agemagician/orgs
https://api.github.com/users/agemagician/received_events
https://api.github.com/users/agemagician/repos
https://api.github.com/users/agemagician/starred{/owner}{/repo}
https://api.github.com/users/agemagician/subscriptions
User
https://api.github.com/users/agemagician
public
MEMBER
@agemagician you can give a try at a multithreaded version if you want (currently on the #548).

To test it, you just need to copy the new `text` processing script which is [here](https://github.com/huggingface/nlp/blob/07d92a82b7594498ff702f3cca55c074e2052257/datasets/text/text.py) somewhere on your drive and give it's local path instead of `text` to `load_dataset`. E.g. in your example:
```python
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('./datasets/text.py',   # path to where you've dowloaded the multi-threaded text loading script
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```
2020-09-01T09:24:06Z
https://github.com/huggingface/datasets/issues/546#issuecomment-684648927
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4NDY0ODkyNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/684648927/reactions
2020-09-01T09:24:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/684648927
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
I have already generated the dataset, but now I tried to reload it and it is still very slow.

I also have installed your commit and it is slow, even after the dataset was already generated.
`pip install git+https://github.com/huggingface/nlp.git@07d92a82b7594498ff702f3cca55c074e2052257`

It uses only a single thread.

Did I miss something ?
2020-09-01T10:39:04Z
https://github.com/huggingface/datasets/issues/546#issuecomment-684757825
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4NDc1NzgyNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/684757825/reactions
2020-09-01T10:39:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/684757825
https://avatars.githubusercontent.com/u/6087313?v=4
https://api.github.com/users/agemagician/events{/privacy}
https://api.github.com/users/agemagician/followers
https://api.github.com/users/agemagician/following{/other_user}
https://api.github.com/users/agemagician/gists{/gist_id}
https://github.com/agemagician
agemagician
MDQ6VXNlcjYwODczMTM=
https://api.github.com/users/agemagician/orgs
https://api.github.com/users/agemagician/received_events
https://api.github.com/users/agemagician/repos
https://api.github.com/users/agemagician/starred{/owner}{/repo}
https://api.github.com/users/agemagician/subscriptions
User
https://api.github.com/users/agemagician
public
MEMBER
As mentioned in #548 , each time you call `load_dataset` with `data_files=`, they are hashed to get the cache directory name. Hashing can be too slow with 1TB of data. I feel like we should have a faster way of getting a hash that identifies the input data files
2020-09-01T10:42:19Z
https://github.com/huggingface/datasets/issues/546#issuecomment-684759767
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4NDc1OTc2Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/684759767/reactions
2020-09-01T10:42:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/684759767
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I believe this is really a very important feature, otherwise, we will still have the issue of too slow loading problems even if the data cache generation is fast.
2020-09-01T11:12:07Z
https://github.com/huggingface/datasets/issues/546#issuecomment-684775270
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4NDc3NTI3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/684775270/reactions
2020-09-01T11:12:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/684775270
https://avatars.githubusercontent.com/u/6087313?v=4
https://api.github.com/users/agemagician/events{/privacy}
https://api.github.com/users/agemagician/followers
https://api.github.com/users/agemagician/following{/other_user}
https://api.github.com/users/agemagician/gists{/gist_id}
https://github.com/agemagician
agemagician
MDQ6VXNlcjYwODczMTM=
https://api.github.com/users/agemagician/orgs
https://api.github.com/users/agemagician/received_events
https://api.github.com/users/agemagician/repos
https://api.github.com/users/agemagician/starred{/owner}{/repo}
https://api.github.com/users/agemagician/subscriptions
User
https://api.github.com/users/agemagician
public
MEMBER
Hmm ok then maybe it's the hashing step indeed.

Let's see if we can improve this as well.

(you will very likely have to regenerate your dataset if we change this part of the lib though since I expect modifications on this part of the lib to results in new hashes)
2020-09-01T11:37:07Z
https://github.com/huggingface/datasets/issues/546#issuecomment-684787914
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4NDc4NzkxNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/684787914/reactions
2020-09-01T11:37:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/684787914
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Also, @agemagician you have to follow the step I indicate in my previous message [here](https://github.com/huggingface/nlp/issues/546#issuecomment-684648927) to use the new text loading script.

Just doing `pip install git+https://github.com/huggingface/nlp.git@07d92a82b7594498ff702f3cca55c074e2052257` like you did won't use the new script (they are not inside the library but hosted on our hub).
2020-09-01T11:38:39Z
https://github.com/huggingface/datasets/issues/546#issuecomment-684788702
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4NDc4ODcwMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/684788702/reactions
2020-09-01T11:38:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/684788702
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
No problem, I will regenerate it. This will make us see if we solved both issues and now both the data generation step, as well as the hashing step, is fast.
2020-09-01T11:39:11Z
https://github.com/huggingface/datasets/issues/546#issuecomment-684789023
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4NDc4OTAyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/684789023/reactions
2020-09-01T11:39:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/684789023
https://avatars.githubusercontent.com/u/6087313?v=4
https://api.github.com/users/agemagician/events{/privacy}
https://api.github.com/users/agemagician/followers
https://api.github.com/users/agemagician/following{/other_user}
https://api.github.com/users/agemagician/gists{/gist_id}
https://github.com/agemagician
agemagician
MDQ6VXNlcjYwODczMTM=
https://api.github.com/users/agemagician/orgs
https://api.github.com/users/agemagician/received_events
https://api.github.com/users/agemagician/repos
https://api.github.com/users/agemagician/starred{/owner}{/repo}
https://api.github.com/users/agemagician/subscriptions
User
https://api.github.com/users/agemagician
public
NONE
Any news for the hashing ?
2020-09-03T19:50:31Z
https://github.com/huggingface/datasets/issues/546#issuecomment-686720267
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4NjcyMDI2Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/686720267/reactions
2020-09-03T19:50:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/686720267
https://avatars.githubusercontent.com/u/6087313?v=4
https://api.github.com/users/agemagician/events{/privacy}
https://api.github.com/users/agemagician/followers
https://api.github.com/users/agemagician/following{/other_user}
https://api.github.com/users/agemagician/gists{/gist_id}
https://github.com/agemagician
agemagician
MDQ6VXNlcjYwODczMTM=
https://api.github.com/users/agemagician/orgs
https://api.github.com/users/agemagician/received_events
https://api.github.com/users/agemagician/repos
https://api.github.com/users/agemagician/starred{/owner}{/repo}
https://api.github.com/users/agemagician/subscriptions
User
https://api.github.com/users/agemagician
public
MEMBER
I'm working on it today :)
2020-09-04T08:35:41Z
https://github.com/huggingface/datasets/issues/546#issuecomment-687009730
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4NzAwOTczMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/687009730/reactions
2020-09-04T08:35:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/687009730
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Ok so now the text files won't be hashed.

I also updated #548 to include this change.
Let us know if it helps @agemagician :)
2020-09-04T15:06:24Z
https://github.com/huggingface/datasets/issues/546#issuecomment-687207109
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4NzIwNzEwOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/687207109/reactions
2020-09-04T15:06:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/687207109
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Perfect thanks for your amazing work.
2020-09-08T12:44:59Z
https://github.com/huggingface/datasets/issues/546#issuecomment-688841784
https://api.github.com/repos/huggingface/datasets/issues/546
MDEyOklzc3VlQ29tbWVudDY4ODg0MTc4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/688841784/reactions
2020-09-08T12:44:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688841784
https://avatars.githubusercontent.com/u/6087313?v=4
https://api.github.com/users/agemagician/events{/privacy}
https://api.github.com/users/agemagician/followers
https://api.github.com/users/agemagician/following{/other_user}
https://api.github.com/users/agemagician/gists{/gist_id}
https://github.com/agemagician
agemagician
MDQ6VXNlcjYwODczMTM=
https://api.github.com/users/agemagician/orgs
https://api.github.com/users/agemagician/received_events
https://api.github.com/users/agemagician/repos
https://api.github.com/users/agemagician/starred{/owner}{/repo}
https://api.github.com/users/agemagician/subscriptions
User
https://api.github.com/users/agemagician
public
NONE
Right now, for caching 18Gb data, it is taking 1 hour 10 minute. Is that proper expected time? @lhoestq @agemagician 
In this rate (assuming large file will caching at the same rate) caching full mC4 (27TB) requires a month (~26 days). 

2021-10-05T21:34:50Z
https://github.com/huggingface/datasets/issues/546#issuecomment-934879105
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps43uR-B
https://api.github.com/repos/huggingface/datasets/issues/comments/934879105/reactions
2021-10-05T21:38:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/934879105
https://avatars.githubusercontent.com/u/32699797?v=4
https://api.github.com/users/sbmaruf/events{/privacy}
https://api.github.com/users/sbmaruf/followers
https://api.github.com/users/sbmaruf/following{/other_user}
https://api.github.com/users/sbmaruf/gists{/gist_id}
https://github.com/sbmaruf
sbmaruf
MDQ6VXNlcjMyNjk5Nzk3
https://api.github.com/users/sbmaruf/orgs
https://api.github.com/users/sbmaruf/received_events
https://api.github.com/users/sbmaruf/repos
https://api.github.com/users/sbmaruf/starred{/owner}{/repo}
https://api.github.com/users/sbmaruf/subscriptions
User
https://api.github.com/users/sbmaruf
public
MEMBER
Hi ! Currently it is that slow because we haven't implemented parallelism for the dataset generation yet.
Though we will definitely work on this :)

For now I'd recommend loading the dataset shard by shard in parallel, and then concatenate them:
```python
# in one process, load first 100 files for english
shard1 = load_dataset("allenai/c4", data_files="multilingual/c4-en.tfrecord-000**.json.gz")
# in another process load next 100 files for english
shard2 = load_dataset("allenai/c4", data_files="multilingual/c4-en.tfrecord-001**.json.gz")

# finally
concatenate_datasets([shard1, shard2, ...])
2021-10-06T10:00:16Z
https://github.com/huggingface/datasets/issues/546#issuecomment-935884005
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps43yHTl
https://api.github.com/repos/huggingface/datasets/issues/comments/935884005/reactions
2021-10-08T15:12:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/935884005
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for the help..!!!
2021-10-07T17:34:33Z
https://github.com/huggingface/datasets/issues/546#issuecomment-938008637
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps436OA9
https://api.github.com/repos/huggingface/datasets/issues/comments/938008637/reactions
2021-10-07T17:34:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/938008637
https://avatars.githubusercontent.com/u/32699797?v=4
https://api.github.com/users/sbmaruf/events{/privacy}
https://api.github.com/users/sbmaruf/followers
https://api.github.com/users/sbmaruf/following{/other_user}
https://api.github.com/users/sbmaruf/gists{/gist_id}
https://github.com/sbmaruf
sbmaruf
MDQ6VXNlcjMyNjk5Nzk3
https://api.github.com/users/sbmaruf/orgs
https://api.github.com/users/sbmaruf/received_events
https://api.github.com/users/sbmaruf/repos
https://api.github.com/users/sbmaruf/starred{/owner}{/repo}
https://api.github.com/users/sbmaruf/subscriptions
User
https://api.github.com/users/sbmaruf
public
NONE
Sorry to write on a closed issue but, has there been any progress on parallelizing the `load_dataset` function?
2022-04-21T14:12:14Z
https://github.com/huggingface/datasets/issues/546#issuecomment-1105268504
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps5B4Q8Y
https://api.github.com/repos/huggingface/datasets/issues/comments/1105268504/reactions
2022-04-21T14:12:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1105268504
https://avatars.githubusercontent.com/u/44896790?v=4
https://api.github.com/users/PauBadiaM/events{/privacy}
https://api.github.com/users/PauBadiaM/followers
https://api.github.com/users/PauBadiaM/following{/other_user}
https://api.github.com/users/PauBadiaM/gists{/gist_id}
https://github.com/PauBadiaM
PauBadiaM
MDQ6VXNlcjQ0ODk2Nzkw
https://api.github.com/users/PauBadiaM/orgs
https://api.github.com/users/PauBadiaM/received_events
https://api.github.com/users/PauBadiaM/repos
https://api.github.com/users/PauBadiaM/starred{/owner}{/repo}
https://api.github.com/users/PauBadiaM/subscriptions
User
https://api.github.com/users/PauBadiaM
public
MEMBER
Hi ! No but this is in our plans (probably a few weeks)
2022-04-21T16:44:32Z
https://github.com/huggingface/datasets/issues/546#issuecomment-1105457887
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps5B4_Lf
https://api.github.com/repos/huggingface/datasets/issues/comments/1105457887/reactions
2022-04-21T16:44:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1105457887
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I'm literally crying waiting for the trainer to restart from checkpoint. It's getting stuck at `get_train_dataloader` and I think this is to do with the same issue... has there been any progress on this?
2022-06-17T16:12:32Z
https://github.com/huggingface/datasets/issues/546#issuecomment-1159029192
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps5FFWHI
https://api.github.com/repos/huggingface/datasets/issues/comments/1159029192/reactions
2022-06-17T16:12:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1159029192
https://avatars.githubusercontent.com/u/30216068?v=4
https://api.github.com/users/alexcoca/events{/privacy}
https://api.github.com/users/alexcoca/followers
https://api.github.com/users/alexcoca/following{/other_user}
https://api.github.com/users/alexcoca/gists{/gist_id}
https://github.com/alexcoca
alexcoca
MDQ6VXNlcjMwMjE2MDY4
https://api.github.com/users/alexcoca/orgs
https://api.github.com/users/alexcoca/received_events
https://api.github.com/users/alexcoca/repos
https://api.github.com/users/alexcoca/starred{/owner}{/repo}
https://api.github.com/users/alexcoca/subscriptions
User
https://api.github.com/users/alexcoca
public
MEMBER
> I'm literally crying waiting for the trainer to restart from checkpoint. It's getting stuck at get_train_dataloader and I think this is to do with the same issue...

Once the dataset is cached once, it's not regenerated again. Your issue seems different
2022-06-17T16:32:58Z
https://github.com/huggingface/datasets/issues/546#issuecomment-1159045143
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps5FFaAX
https://api.github.com/repos/huggingface/datasets/issues/comments/1159045143/reactions
2022-06-17T16:32:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1159045143
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
hmmm, yes. I'll come back with details on this, fairly easy to reproduce. Takes about 30 minutes to get from checkpoint loading to starting training...
2022-06-17T17:06:51Z
https://github.com/huggingface/datasets/issues/546#issuecomment-1159075546
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps5FFhba
https://api.github.com/repos/huggingface/datasets/issues/comments/1159075546/reactions
2022-06-17T17:06:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1159075546
https://avatars.githubusercontent.com/u/30216068?v=4
https://api.github.com/users/alexcoca/events{/privacy}
https://api.github.com/users/alexcoca/followers
https://api.github.com/users/alexcoca/following{/other_user}
https://api.github.com/users/alexcoca/gists{/gist_id}
https://github.com/alexcoca
alexcoca
MDQ6VXNlcjMwMjE2MDY4
https://api.github.com/users/alexcoca/orgs
https://api.github.com/users/alexcoca/received_events
https://api.github.com/users/alexcoca/repos
https://api.github.com/users/alexcoca/starred{/owner}{/repo}
https://api.github.com/users/alexcoca/subscriptions
User
https://api.github.com/users/alexcoca
public
NONE
@lhoestq yo, any news in making it possible to download large datasets faster?
2023-12-19T04:34:54Z
https://github.com/huggingface/datasets/issues/546#issuecomment-1862103265
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps5u_XDh
https://api.github.com/repos/huggingface/datasets/issues/comments/1862103265/reactions
2023-12-19T04:34:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1862103265
https://avatars.githubusercontent.com/u/17604849?v=4
https://api.github.com/users/kopyl/events{/privacy}
https://api.github.com/users/kopyl/followers
https://api.github.com/users/kopyl/following{/other_user}
https://api.github.com/users/kopyl/gists{/gist_id}
https://github.com/kopyl
kopyl
MDQ6VXNlcjE3NjA0ODQ5
https://api.github.com/users/kopyl/orgs
https://api.github.com/users/kopyl/received_events
https://api.github.com/users/kopyl/repos
https://api.github.com/users/kopyl/starred{/owner}{/repo}
https://api.github.com/users/kopyl/subscriptions
User
https://api.github.com/users/kopyl
public
NONE
@lhoestq For some reason [setting num_proc](https://discuss.huggingface.co/t/how-can-i-multithreadedly-download-a-huggingface-dataset/56178/2?u=kopyl) does not work at all... My dataset has 58 parquet files and i was hoping passing `num_proc` to `load_dataset` would spawn 58 Python processes each downloading its own parquet so I can load my dataset in 1 minutes instead of 50...
2023-12-19T06:30:22Z
https://github.com/huggingface/datasets/issues/546#issuecomment-1862198009
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps5u_uL5
https://api.github.com/repos/huggingface/datasets/issues/comments/1862198009/reactions
2023-12-19T06:30:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1862198009
https://avatars.githubusercontent.com/u/17604849?v=4
https://api.github.com/users/kopyl/events{/privacy}
https://api.github.com/users/kopyl/followers
https://api.github.com/users/kopyl/following{/other_user}
https://api.github.com/users/kopyl/gists{/gist_id}
https://github.com/kopyl
kopyl
MDQ6VXNlcjE3NjA0ODQ5
https://api.github.com/users/kopyl/orgs
https://api.github.com/users/kopyl/received_events
https://api.github.com/users/kopyl/repos
https://api.github.com/users/kopyl/starred{/owner}{/repo}
https://api.github.com/users/kopyl/subscriptions
User
https://api.github.com/users/kopyl
public
MEMBER
It does spawn `num_proc` processes. Note that when you download in parallel you're often bounded by your bandwidth at one point, so 50 processes is unlikely to get you a x50 download speed up but a bit less
2023-12-19T10:29:37Z
https://github.com/huggingface/datasets/issues/546#issuecomment-1862503821
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps5vA42N
https://api.github.com/repos/huggingface/datasets/issues/comments/1862503821/reactions
2023-12-19T10:29:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1862503821
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> It does spawn `num_proc` processes. Note that when you download in parallel you're often bounded by your bandwidth at one point, so 50 processes is unlikely to get you a x50 download speed up but a bit less

why then i see only 1 parquet file download progress bar at a time?
2023-12-25T14:55:16Z
https://github.com/huggingface/datasets/issues/546#issuecomment-1869016554
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps5vZu3q
https://api.github.com/repos/huggingface/datasets/issues/comments/1869016554/reactions
2023-12-25T14:55:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1869016554
https://avatars.githubusercontent.com/u/17604849?v=4
https://api.github.com/users/kopyl/events{/privacy}
https://api.github.com/users/kopyl/followers
https://api.github.com/users/kopyl/following{/other_user}
https://api.github.com/users/kopyl/gists{/gist_id}
https://github.com/kopyl
kopyl
MDQ6VXNlcjE3NjA0ODQ5
https://api.github.com/users/kopyl/orgs
https://api.github.com/users/kopyl/received_events
https://api.github.com/users/kopyl/repos
https://api.github.com/users/kopyl/starred{/owner}{/repo}
https://api.github.com/users/kopyl/subscriptions
User
https://api.github.com/users/kopyl
public
MEMBER
Ah indeed parallel downloads are not enabled yet for some datasets.
I opened https://github.com/huggingface/datasets/pull/6551 to fix this.

2024-01-02T18:07:13Z
https://github.com/huggingface/datasets/issues/546#issuecomment-1874368911
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps5vuJmP
https://api.github.com/repos/huggingface/datasets/issues/comments/1874368911/reactions
2024-01-02T18:07:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1874368911
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq thank you very much, i though for a second i'm just tripping.
2024-01-02T20:26:23Z
https://github.com/huggingface/datasets/issues/546#issuecomment-1874510258
https://api.github.com/repos/huggingface/datasets/issues/546
IC_kwDODunzps5vusGy
https://api.github.com/repos/huggingface/datasets/issues/comments/1874510258/reactions
2024-01-02T20:26:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1874510258
https://avatars.githubusercontent.com/u/17604849?v=4
https://api.github.com/users/kopyl/events{/privacy}
https://api.github.com/users/kopyl/followers
https://api.github.com/users/kopyl/following{/other_user}
https://api.github.com/users/kopyl/gists{/gist_id}
https://github.com/kopyl
kopyl
MDQ6VXNlcjE3NjA0ODQ5
https://api.github.com/users/kopyl/orgs
https://api.github.com/users/kopyl/received_events
https://api.github.com/users/kopyl/repos
https://api.github.com/users/kopyl/starred{/owner}{/repo}
https://api.github.com/users/kopyl/subscriptions
User
https://api.github.com/users/kopyl
public
I made a simple python script to check the NLP library speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

Is there something that I am missing ?