Possible Bug: Small training/dataset file creates gigantic output
MEMBER
The preprocessing tokenizes the input text. Tokenization outputs `input_ids`, `attention_mask`, `token_type_ids` and `special_tokens_mask`. All those are of length`max_seq_length` because of padding. Therefore for each sample it generate 4 *`max_seq_length` integers. Currently they're all saved as int64. This is why the tokenization takes so much space.

I'm sure we can optimize that though
What do you think @sgugger ?
2020-11-17T14:10:58Z
https://github.com/huggingface/datasets/issues/861#issuecomment-728953220
https://api.github.com/repos/huggingface/datasets/issues/861
MDEyOklzc3VlQ29tbWVudDcyODk1MzIyMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/728953220/reactions
2020-11-17T14:40:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/728953220
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
First I think we should disable padding in the dataset processing and let the data collator do it.

Then I'm wondering if you need attention_mask and token_type_ids at this point ?

Finally we can also specify the output feature types at this line https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py#L280 to use more optimized integer precisions for the output. Maybe something like:
- input_ids: uint16 or uint32
- token_type_ids: uint8 or bool
- attention_mask: bool
- special_tokens_mask: bool

Also IMO these changes are all on the `transformers` side. Maybe we should discuss on the `transformers` repo
2020-11-17T15:16:04Z
https://github.com/huggingface/datasets/issues/861#issuecomment-728995695
https://api.github.com/repos/huggingface/datasets/issues/861
MDEyOklzc3VlQ29tbWVudDcyODk5NTY5NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/728995695/reactions
2021-02-05T13:11:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/728995695
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> First I think we should disable padding in the dataset processing and let the data collator do it.

No, you can't do that on TPUs as dynamic shapes will result in a very slow training. The script can however be tweaked to use the `PaddingDataCollator` with a fixed max length instead of dynamic batching.

For the other optimizations, they can be done by changing the script directly for each user's use case. Not sure we can find something that is general enough to be in transformers or the examples script.
2020-11-17T15:38:15Z
https://github.com/huggingface/datasets/issues/861#issuecomment-729011564
https://api.github.com/repos/huggingface/datasets/issues/861
MDEyOklzc3VlQ29tbWVudDcyOTAxMTU2NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/729011564/reactions
2020-11-17T15:38:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/729011564
https://avatars.githubusercontent.com/u/35901082?v=4
https://api.github.com/users/sgugger/events{/privacy}
https://api.github.com/users/sgugger/followers
https://api.github.com/users/sgugger/following{/other_user}
https://api.github.com/users/sgugger/gists{/gist_id}
https://github.com/sgugger
sgugger
MDQ6VXNlcjM1OTAxMDgy
https://api.github.com/users/sgugger/orgs
https://api.github.com/users/sgugger/received_events
https://api.github.com/users/sgugger/repos
https://api.github.com/users/sgugger/starred{/owner}{/repo}
https://api.github.com/users/sgugger/subscriptions
User
https://api.github.com/users/sgugger
public
MEMBER
Oh yes right..
Do you think that a lazy map feature on the `datasets` side could help to avoid storing padded tokenized texts then ?
2020-11-17T15:58:54Z
https://github.com/huggingface/datasets/issues/861#issuecomment-729024709
https://api.github.com/repos/huggingface/datasets/issues/861
MDEyOklzc3VlQ29tbWVudDcyOTAyNDcwOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/729024709/reactions
2020-11-17T15:58:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/729024709
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I think I can do the tweak mentioned above with the data collator as short fix (but fully focused on v4 right now so that will be for later this week, beginning of next week :-) ).
If it doesn't hurt performance to tokenize on the fly, that would clearly be the long-term solution however!
2020-11-17T16:55:29Z
https://github.com/huggingface/datasets/issues/861#issuecomment-729060582
https://api.github.com/repos/huggingface/datasets/issues/861
MDEyOklzc3VlQ29tbWVudDcyOTA2MDU4Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/729060582/reactions
2020-11-17T16:55:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/729060582
https://avatars.githubusercontent.com/u/35901082?v=4
https://api.github.com/users/sgugger/events{/privacy}
https://api.github.com/users/sgugger/followers
https://api.github.com/users/sgugger/following{/other_user}
https://api.github.com/users/sgugger/gists{/gist_id}
https://github.com/sgugger
sgugger
MDQ6VXNlcjM1OTAxMDgy
https://api.github.com/users/sgugger/orgs
https://api.github.com/users/sgugger/received_events
https://api.github.com/users/sgugger/repos
https://api.github.com/users/sgugger/starred{/owner}{/repo}
https://api.github.com/users/sgugger/subscriptions
User
https://api.github.com/users/sgugger
public
NONE
> Hey guys,
> 
> I was trying to create a new bert model from scratch via _huggingface transformers + tokenizers + dataets_ (actually using this example script by your team: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). It was supposed to be a first test with a small 5 GB raw text file but I can't even end the preprocessing handled by datasets because this tiny 5 GB text file becomes more than 1 TB when processing. My system was running out of space and crashed prematurely.
> 
> I've done training from scratch via Google's bert repo in the past and I can remember that the resulting pretraining data can become quite big. But 5 GB becoming 1 TB was never the case. Is this considered normal or is it a bug?
> 
> I've used the following CMD:
> `python xla_spawn.py --num_cores=8 run_mlm.py --model_type bert --config_name config.json --tokenizer_name tokenizer.json --train_file dataset_full.txt --do_train --output_dir out --max_steps 500000 --save_steps 2500 --save_total_limit 2 --prediction_loss_only --line_by_line --max_seq_length 128 --pad_to_max_length --preprocessing_num_workers 16 --per_device_train_batch_size 128 --overwrite_output_dir --debug`

It's actually because of the parameter 'preprocessing_num_worker' when using TPU. 
I am also planning to have my model trained on the google TPU with a 11gb text corpus. With x8 cores enabled, each TPU core has its own dataset.  When not using distributed training, the preprocessed file is about 77gb. On the opposite, if enable xla, the file produced will easily consume all my free space(more than 220gb, I think it will be, in the end, around 600gb ). 
So I think that's maybe where the problem came from. 

Is there any possibility that all of the cores share the same preprocess dataset?

@sgugger @RammMaschine 
2021-01-04T23:39:10Z
https://github.com/huggingface/datasets/issues/861#issuecomment-754287234
https://api.github.com/repos/huggingface/datasets/issues/861
MDEyOklzc3VlQ29tbWVudDc1NDI4NzIzNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/754287234/reactions
2021-01-04T23:42:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/754287234
https://avatars.githubusercontent.com/u/31885058?v=4
https://api.github.com/users/CharleoY/events{/privacy}
https://api.github.com/users/CharleoY/followers
https://api.github.com/users/CharleoY/following{/other_user}
https://api.github.com/users/CharleoY/gists{/gist_id}
https://github.com/CharleoY
CharleoY
MDQ6VXNlcjMxODg1MDU4
https://api.github.com/users/CharleoY/orgs
https://api.github.com/users/CharleoY/received_events
https://api.github.com/users/CharleoY/repos
https://api.github.com/users/CharleoY/starred{/owner}{/repo}
https://api.github.com/users/CharleoY/subscriptions
User
https://api.github.com/users/CharleoY
public
MEMBER
Hi @NebelAI, we have optimized Datasets' disk usage in the latest release v1.5.

Feel free to update your Datasets version
```shell
pip install -U datasets
```
and see if it better suits your needs.
2021-03-22T12:04:54Z
https://github.com/huggingface/datasets/issues/861#issuecomment-804011287
https://api.github.com/repos/huggingface/datasets/issues/861
MDEyOklzc3VlQ29tbWVudDgwNDAxMTI4Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/804011287/reactions
2021-03-22T12:04:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/804011287
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Hey guys,

I was trying to create a new bert model from scratch via _huggingface transformers + tokenizers + dataets_ (actually using this example script by your team: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py). It was supposed to be a first test with a small 5 GB raw text file but I can't even end the preprocessing handled by datasets because this tiny 5 GB text file becomes more than 1 TB when processing. My system was running out of space and crashed prematurely.

I've done training from scratch via Google's bert repo in the past and I can remember that the resulting pretraining data can become quite big. But 5 GB becoming 1 TB was never the case. Is this considered normal or is it a bug?

I've used the following CMD:
`python xla_spawn.py --num_cores=8 run_mlm.py --model_type bert --config_name config.json --tokenizer_name tokenizer.json --train_file dataset_full.txt --do_train --output_dir out --max_steps 500000 --save_steps 2500 --save_total_limit 2 --prediction_loss_only --line_by_line --max_seq_length 128 --pad_to_max_length --preprocessing_num_workers 16 --per_device_train_batch_size 128 --overwrite_output_dir --debug`

