CUDA out of memory
MEMBER
Could you provide more details ? What's the code you ran ?
2020-10-23T14:08:36Z
https://github.com/huggingface/datasets/issues/757#issuecomment-715365484
https://api.github.com/repos/huggingface/datasets/issues/757
MDEyOklzc3VlQ29tbWVudDcxNTM2NTQ4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/715365484/reactions
2020-10-23T14:08:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/715365484
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
```python
tokenizer = FunnelTokenizer.from_pretrained('funnel-transformer/small')

def tokenize(batch):
    return tokenizer(batch['text'], padding='max_length', truncation=True,max_length=512)

dataset = load_dataset("bookcorpus",split='train[:1000]').shuffle()
dataset = dataset.map(tokenize, batched=True, batch_size=512)

# dataset = LineByLineTextDataset(
#     tokenizer=tokenizer,
#     file_path="./wiki1000.txt",
#     block_size=128
# )

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

config=FunnelConfig(
    return_dict=True
)

model= FunnelForMaskedLM(config=config)

training_args = TrainingArguments(
    output_dir="./checkpoints",
    overwrite_output_dir=True,
    do_train=True,
    num_train_epochs=1,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    save_steps=10000,
    logging_dir='./ptlogs'
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)
trainer.train()
```
2020-10-24T10:49:09Z
https://github.com/huggingface/datasets/issues/757#issuecomment-715897044
https://api.github.com/repos/huggingface/datasets/issues/757
MDEyOklzc3VlQ29tbWVudDcxNTg5NzA0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/715897044/reactions
2020-10-24T10:50:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/715897044
https://avatars.githubusercontent.com/u/47059217?v=4
https://api.github.com/users/li1117heex/events{/privacy}
https://api.github.com/users/li1117heex/followers
https://api.github.com/users/li1117heex/following{/other_user}
https://api.github.com/users/li1117heex/gists{/gist_id}
https://github.com/li1117heex
li1117heex
MDQ6VXNlcjQ3MDU5MjE3
https://api.github.com/users/li1117heex/orgs
https://api.github.com/users/li1117heex/received_events
https://api.github.com/users/li1117heex/repos
https://api.github.com/users/li1117heex/starred{/owner}{/repo}
https://api.github.com/users/li1117heex/subscriptions
User
https://api.github.com/users/li1117heex
public
NONE
`RuntimeError: CUDA out of memory. Tried to allocate 954.00 MiB (GPU 0; 15.90 GiB total capacity; 14.35 GiB already allocated; 753.75 MiB free; 14.39 GiB reserved in total by PyTorch)
Exception raised from malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:272 (most recent call first):`

part of error output
2020-10-24T10:51:57Z
https://github.com/huggingface/datasets/issues/757#issuecomment-715897373
https://api.github.com/repos/huggingface/datasets/issues/757
MDEyOklzc3VlQ29tbWVudDcxNTg5NzM3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/715897373/reactions
2020-10-24T10:51:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/715897373
https://avatars.githubusercontent.com/u/47059217?v=4
https://api.github.com/users/li1117heex/events{/privacy}
https://api.github.com/users/li1117heex/followers
https://api.github.com/users/li1117heex/following{/other_user}
https://api.github.com/users/li1117heex/gists{/gist_id}
https://github.com/li1117heex
li1117heex
MDQ6VXNlcjQ3MDU5MjE3
https://api.github.com/users/li1117heex/orgs
https://api.github.com/users/li1117heex/received_events
https://api.github.com/users/li1117heex/repos
https://api.github.com/users/li1117heex/starred{/owner}{/repo}
https://api.github.com/users/li1117heex/subscriptions
User
https://api.github.com/users/li1117heex
public
NONE
from funnel model to bert model : error still happened

from your dataset to LineByLineTextDataset : error disapeared
2020-10-24T10:53:38Z
https://github.com/huggingface/datasets/issues/757#issuecomment-715897553
https://api.github.com/repos/huggingface/datasets/issues/757
MDEyOklzc3VlQ29tbWVudDcxNTg5NzU1Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/715897553/reactions
2020-10-24T10:53:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/715897553
https://avatars.githubusercontent.com/u/47059217?v=4
https://api.github.com/users/li1117heex/events{/privacy}
https://api.github.com/users/li1117heex/followers
https://api.github.com/users/li1117heex/following{/other_user}
https://api.github.com/users/li1117heex/gists{/gist_id}
https://github.com/li1117heex
li1117heex
MDQ6VXNlcjQ3MDU5MjE3
https://api.github.com/users/li1117heex/orgs
https://api.github.com/users/li1117heex/received_events
https://api.github.com/users/li1117heex/repos
https://api.github.com/users/li1117heex/starred{/owner}{/repo}
https://api.github.com/users/li1117heex/subscriptions
User
https://api.github.com/users/li1117heex
public
NONE
notice i just loaded 1000 rows of data
2020-10-24T10:54:28Z
https://github.com/huggingface/datasets/issues/757#issuecomment-715897657
https://api.github.com/repos/huggingface/datasets/issues/757
MDEyOklzc3VlQ29tbWVudDcxNTg5NzY1Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/715897657/reactions
2020-10-24T10:54:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/715897657
https://avatars.githubusercontent.com/u/47059217?v=4
https://api.github.com/users/li1117heex/events{/privacy}
https://api.github.com/users/li1117heex/followers
https://api.github.com/users/li1117heex/following{/other_user}
https://api.github.com/users/li1117heex/gists{/gist_id}
https://github.com/li1117heex
li1117heex
MDQ6VXNlcjQ3MDU5MjE3
https://api.github.com/users/li1117heex/orgs
https://api.github.com/users/li1117heex/received_events
https://api.github.com/users/li1117heex/repos
https://api.github.com/users/li1117heex/starred{/owner}{/repo}
https://api.github.com/users/li1117heex/subscriptions
User
https://api.github.com/users/li1117heex
public
NONE
the error happens when executing loss.backward()
2020-10-24T11:44:57Z
https://github.com/huggingface/datasets/issues/757#issuecomment-715903284
https://api.github.com/repos/huggingface/datasets/issues/757
MDEyOklzc3VlQ29tbWVudDcxNTkwMzI4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/715903284/reactions
2020-10-24T11:44:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/715903284
https://avatars.githubusercontent.com/u/47059217?v=4
https://api.github.com/users/li1117heex/events{/privacy}
https://api.github.com/users/li1117heex/followers
https://api.github.com/users/li1117heex/following{/other_user}
https://api.github.com/users/li1117heex/gists{/gist_id}
https://github.com/li1117heex
li1117heex
MDQ6VXNlcjQ3MDU5MjE3
https://api.github.com/users/li1117heex/orgs
https://api.github.com/users/li1117heex/received_events
https://api.github.com/users/li1117heex/repos
https://api.github.com/users/li1117heex/starred{/owner}{/repo}
https://api.github.com/users/li1117heex/subscriptions
User
https://api.github.com/users/li1117heex
public
MEMBER
Since you're using a data collator you don't need to tokenizer the dataset using `map`. Could you try not to use `map` and only the data collator instead ? The data collator is supposed to pad to the longest sequence in each batch afaik, instead of padding to 512.

Also cc @sgugger 
2020-10-26T09:39:18Z
https://github.com/huggingface/datasets/issues/757#issuecomment-716430613
https://api.github.com/repos/huggingface/datasets/issues/757
MDEyOklzc3VlQ29tbWVudDcxNjQzMDYxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/716430613/reactions
2020-10-26T09:39:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/716430613
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Closing this one.
Feel free to re-open if you have other questions about this issue
2020-12-23T14:06:29Z
https://github.com/huggingface/datasets/issues/757#issuecomment-750315073
https://api.github.com/repos/huggingface/datasets/issues/757
MDEyOklzc3VlQ29tbWVudDc1MDMxNTA3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/750315073/reactions
2020-12-23T14:06:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/750315073
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
In your dataset ,cuda run out of memory as long as the trainer begins:
however, without changing any other element/parameter,just switch dataset to `LineByLineTextDataset`,everything becames OK.
