How to not load huggingface datasets into memory 
NONE
So maybe a summary here: 
If I could fit a large model with batch_size = X into memory, is there a way I could train this model for huge datasets with keeping setting the same? thanks 
2021-03-08T12:44:01Z
https://github.com/huggingface/datasets/issues/2007#issuecomment-792731079
https://api.github.com/repos/huggingface/datasets/issues/2007
MDEyOklzc3VlQ29tbWVudDc5MjczMTA3OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/792731079/reactions
2021-03-08T12:44:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/792731079
https://avatars.githubusercontent.com/u/79165106?v=4
https://api.github.com/users/dorost1234/events{/privacy}
https://api.github.com/users/dorost1234/followers
https://api.github.com/users/dorost1234/following{/other_user}
https://api.github.com/users/dorost1234/gists{/gist_id}
https://github.com/dorost1234
dorost1234
MDQ6VXNlcjc5MTY1MTA2
https://api.github.com/users/dorost1234/orgs
https://api.github.com/users/dorost1234/received_events
https://api.github.com/users/dorost1234/repos
https://api.github.com/users/dorost1234/starred{/owner}{/repo}
https://api.github.com/users/dorost1234/subscriptions
User
https://api.github.com/users/dorost1234
public
MEMBER
The `datastets` library doesn't load datasets into memory. Therefore you can load a dataset that is terabytes big without filling up your RAM.

The only thing that's loaded into memory during training is the batch used in the training step.
So as long as your model works with batch_size = X, then you can load an even bigger dataset and it will work as well with the same batch_size.

Note that you still have to take into account that some batches take more memory than others, depending on the texts lengths. If it works for a batch with batch_size = X and with texts of maximum length, then it will work for all batches.

In your case I guess that there are a few long sentences in the dataset. For those long sentences you get a memory error on your GPU because they're too long. By passing `max_train_samples` you may have taken a subset of the dataset that only contain short sentences. That's probably why in your case it worked only when you set `max_train_samples`.
I'd suggest you to reduce the batch size so that the batches with long sentences can be loaded on the GPU.

Let me know if that helps or if you have other questions
2021-03-08T16:11:19Z
https://github.com/huggingface/datasets/issues/2007#issuecomment-792864902
https://api.github.com/repos/huggingface/datasets/issues/2007
MDEyOklzc3VlQ29tbWVudDc5Mjg2NDkwMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/792864902/reactions
2021-03-08T16:11:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/792864902
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Hi
I am running this example from transformers library version 4.3.3:
(Here is the full documentation https://github.com/huggingface/transformers/issues/8771 but the running command should work out of the box)

 USE_TF=0  deepspeed  run_seq2seq.py --model_name_or_path google/mt5-base --dataset_name wmt16 --dataset_config_name ro-en --source_prefix "translate English to Romanian: " --task translation_en_to_ro   --output_dir /test/test_large  --do_train --do_eval --predict_with_generate  --max_train_samples 500   --max_val_samples 500  --max_source_length 128 --max_target_length 128 --sortish_sampler --per_device_train_batch_size 8   --val_max_target_length 128 --deepspeed ds_config.json --num_train_epochs 1 --eval_steps 25000 --warmup_steps 500 --overwrite_output_dir

(Here please find the script: https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_seq2seq.py)

If you do not pass max_train_samples in above command to load the full dataset, then I get memory issue on a gpu with 24 GigBytes of memory.
 
I need to train large-scale mt5 model on large-scale datasets of wikipedia (multiple of them concatenated or other datasets in multiple languages like OPUS), could you help me how I can avoid loading the full data into memory? to make the scripts not related to data size? 

In above example, I was hoping the script could work without relying on dataset size, so I can still train the model without subsampling training set.

thank you so much @lhoestq for your great help in advance


