datasets map() handles all data at a stroke and takes long time
MEMBER
Hi ! Interesting question ;)

> Which is better? Process in map() or in data-collator

As you said, both can be used in practice: map() if you want to preprocess before training, or a data-collator (or the equivalent `dataset.set_transform`) if you want to preprocess on-the-fly during training. Both options are great and really depend on your case.

To choose between the two, here are IMO the main caveats of each approach:
- if your preprocessing takes too much CPU for example, using a data-collator may slow down your training and your GPUs may not work at full speed
- on the other hand, map() may take a lot of time and disk space to run if your dataset is too big.

> Why huggingface advises map() function? There should be some advantages to using map()

To get the best throughput when training a model, it is often recommended to preprocess your dataset before training. Note that preprocessing may include other steps before tokenization such as data filtering, cleaning, chunking etc. which are often done before training.
2022-08-30T09:28:40Z
https://github.com/huggingface/datasets/issues/4912#issuecomment-1231408405
https://api.github.com/repos/huggingface/datasets/issues/4912
IC_kwDODunzps5JZc0V
https://api.github.com/repos/huggingface/datasets/issues/comments/1231408405/reactions
2022-08-30T09:30:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1231408405
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for your clear explanation @lhoestq ! 
> *  if your preprocessing takes too much CPU for example, using a data-collator may slow down your training and your GPUs may not work at full speed
> * on the other hand, map() may take a lot of time and disk space to run if your dataset is too big.

I really agree with you. There should be some trade-off between processing before and during the train loop.
Besides, I find `map()` function can cache the results once it has been executed. Very useful!
2022-08-31T05:53:41Z
https://github.com/huggingface/datasets/issues/4912#issuecomment-1232494105
https://api.github.com/repos/huggingface/datasets/issues/4912
IC_kwDODunzps5Jdl4Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1232494105/reactions
2022-08-31T05:53:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1232494105
https://avatars.githubusercontent.com/u/40711748?v=4
https://api.github.com/users/BruceStayHungry/events{/privacy}
https://api.github.com/users/BruceStayHungry/followers
https://api.github.com/users/BruceStayHungry/following{/other_user}
https://api.github.com/users/BruceStayHungry/gists{/gist_id}
https://github.com/BruceStayHungry
BruceStayHungry
MDQ6VXNlcjQwNzExNzQ4
https://api.github.com/users/BruceStayHungry/orgs
https://api.github.com/users/BruceStayHungry/received_events
https://api.github.com/users/BruceStayHungry/repos
https://api.github.com/users/BruceStayHungry/starred{/owner}{/repo}
https://api.github.com/users/BruceStayHungry/subscriptions
User
https://api.github.com/users/BruceStayHungry
public
MEMBER
I'm closing this issue if you don't mind, feel free to reopen if needed ;)
2022-09-06T09:23:35Z
https://github.com/huggingface/datasets/issues/4912#issuecomment-1237892881
https://api.github.com/repos/huggingface/datasets/issues/4912
IC_kwDODunzps5JyL8R
https://api.github.com/repos/huggingface/datasets/issues/comments/1237892881/reactions
2022-09-06T09:23:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1237892881
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq How to preprocess on-the-fly during training？my data is about 1w hours, when I use map to  preprocess, and It's not finished yet， but all disk space(2T) is full.
2023-04-04T03:47:49Z
https://github.com/huggingface/datasets/issues/4912#issuecomment-1495305693
https://api.github.com/repos/huggingface/datasets/issues/4912
IC_kwDODunzps5ZII3d
https://api.github.com/repos/huggingface/datasets/issues/comments/1495305693/reactions
2023-04-04T03:48:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1495305693
https://avatars.githubusercontent.com/u/38179632?v=4
https://api.github.com/users/ben-8543/events{/privacy}
https://api.github.com/users/ben-8543/followers
https://api.github.com/users/ben-8543/following{/other_user}
https://api.github.com/users/ben-8543/gists{/gist_id}
https://github.com/ben-8543
ben-8543
MDQ6VXNlcjM4MTc5NjMy
https://api.github.com/users/ben-8543/orgs
https://api.github.com/users/ben-8543/received_events
https://api.github.com/users/ben-8543/repos
https://api.github.com/users/ben-8543/starred{/owner}{/repo}
https://api.github.com/users/ben-8543/subscriptions
User
https://api.github.com/users/ben-8543
public
MEMBER
Hi ! You can do that using `set_transform`, see https://huggingface.co/docs/datasets/process#format-transform for more info :)
2023-04-04T09:35:46Z
https://github.com/huggingface/datasets/issues/4912#issuecomment-1495654155
https://api.github.com/repos/huggingface/datasets/issues/4912
IC_kwDODunzps5ZJd8L
https://api.github.com/repos/huggingface/datasets/issues/comments/1495654155/reactions
2023-04-04T09:35:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1495654155
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
unfortunately , it not work.
2023-04-05T01:43:39Z
https://github.com/huggingface/datasets/issues/4912#issuecomment-1496808540
https://api.github.com/repos/huggingface/datasets/issues/4912
IC_kwDODunzps5ZN3xc
https://api.github.com/repos/huggingface/datasets/issues/comments/1496808540/reactions
2023-04-05T01:43:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1496808540
https://avatars.githubusercontent.com/u/38179632?v=4
https://api.github.com/users/ben-8543/events{/privacy}
https://api.github.com/users/ben-8543/followers
https://api.github.com/users/ben-8543/following{/other_user}
https://api.github.com/users/ben-8543/gists{/gist_id}
https://github.com/ben-8543
ben-8543
MDQ6VXNlcjM4MTc5NjMy
https://api.github.com/users/ben-8543/orgs
https://api.github.com/users/ben-8543/received_events
https://api.github.com/users/ben-8543/repos
https://api.github.com/users/ben-8543/starred{/owner}{/repo}
https://api.github.com/users/ben-8543/subscriptions
User
https://api.github.com/users/ben-8543
public
MEMBER
Could you share more details ?
2023-04-06T09:43:58Z
https://github.com/huggingface/datasets/issues/4912#issuecomment-1498778966
https://api.github.com/repos/huggingface/datasets/issues/4912
IC_kwDODunzps5ZVY1W
https://api.github.com/repos/huggingface/datasets/issues/comments/1498778966/reactions
2023-04-06T09:43:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1498778966
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
**1. Background**

Huggingface datasets package advises using `map()`  to process data in batches. In the example code on pretraining masked language model, they use `map()` to tokenize all data at a stroke before the train loop. 

The corresponding code:
```
with accelerator.main_process_first():
    tokenized_datasets = raw_datasets.map(
        tokenize_function,
        batched=True,
        num_proc=args.preprocessing_num_workers,
        remove_columns=column_names,
        load_from_cache_file=not args.overwrite_cache,
        desc="Running tokenizer on every text in dataset"
    )
```

**2. The problem**

Thus, when I try the same pertaining code with a much larger corpus, it takes quite a long time to tokenize.

Also, we can choose to tokenize data in `data-collator`. In this way, the program only tokenizes one batch in the next training step and avoids getting stuck in tokenization.

**3. My question**

As described above, my questions are:
* **Which is better? Process in `map()` or in `data-collator`**
* **Why huggingface advises `map()` function?** There should be some advantages to using `map()`


Thanks for your answers!