[Question & Bug Report] Can we preprocess a dataset on the fly?
MEMBER
We are very actively working on this. How does your dataset look like in practice (number/size/type of files)?
2021-01-24T10:13:14Z
https://github.com/huggingface/datasets/issues/1776#issuecomment-766322913
https://api.github.com/repos/huggingface/datasets/issues/1776
MDEyOklzc3VlQ29tbWVudDc2NjMyMjkxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/766322913/reactions
2021-01-24T10:13:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/766322913
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
It's a text file with many lines (about 1B) of Chinese sentences. I use it to train language model using https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm_wwm.py
2021-01-24T13:04:56Z
https://github.com/huggingface/datasets/issues/1776#issuecomment-766344800
https://api.github.com/repos/huggingface/datasets/issues/1776
MDEyOklzc3VlQ29tbWVudDc2NjM0NDgwMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/766344800/reactions
2021-01-24T13:04:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/766344800
https://avatars.githubusercontent.com/u/14048129?v=4
https://api.github.com/users/shuaihuaiyi/events{/privacy}
https://api.github.com/users/shuaihuaiyi/followers
https://api.github.com/users/shuaihuaiyi/following{/other_user}
https://api.github.com/users/shuaihuaiyi/gists{/gist_id}
https://github.com/shuaihuaiyi
shuaihuaiyi
MDQ6VXNlcjE0MDQ4MTI5
https://api.github.com/users/shuaihuaiyi/orgs
https://api.github.com/users/shuaihuaiyi/received_events
https://api.github.com/users/shuaihuaiyi/repos
https://api.github.com/users/shuaihuaiyi/starred{/owner}{/repo}
https://api.github.com/users/shuaihuaiyi/subscriptions
User
https://api.github.com/users/shuaihuaiyi
public
MEMBER
Indeed I will submit a PR in a fez days to enable processing on-the-fly :)
This can be useful in language modeling for tokenization, padding etc.

2021-01-24T15:05:18Z
https://github.com/huggingface/datasets/issues/1776#issuecomment-766362508
https://api.github.com/repos/huggingface/datasets/issues/1776
MDEyOklzc3VlQ29tbWVudDc2NjM2MjUwOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/766362508/reactions
2021-01-24T15:05:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/766362508
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
any update on this issue? ...really look forward to use it 
2021-02-06T01:45:03Z
https://github.com/huggingface/datasets/issues/1776#issuecomment-774374964
https://api.github.com/repos/huggingface/datasets/issues/1776
MDEyOklzc3VlQ29tbWVudDc3NDM3NDk2NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/774374964/reactions
2021-02-06T01:45:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774374964
https://avatars.githubusercontent.com/u/56231298?v=4
https://api.github.com/users/acul3/events{/privacy}
https://api.github.com/users/acul3/followers
https://api.github.com/users/acul3/following{/other_user}
https://api.github.com/users/acul3/gists{/gist_id}
https://github.com/acul3
acul3
MDQ6VXNlcjU2MjMxMjk4
https://api.github.com/users/acul3/orgs
https://api.github.com/users/acul3/received_events
https://api.github.com/users/acul3/repos
https://api.github.com/users/acul3/starred{/owner}{/repo}
https://api.github.com/users/acul3/subscriptions
User
https://api.github.com/users/acul3
public
CONTRIBUTOR
Hi @acul3,

Please look at the discussion on a related Issue #1825. I think using `set_transform` after building from source should do.
2021-02-06T06:36:30Z
https://github.com/huggingface/datasets/issues/1776#issuecomment-774410299
https://api.github.com/repos/huggingface/datasets/issues/1776
MDEyOklzc3VlQ29tbWVudDc3NDQxMDI5OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/774410299/reactions
2021-02-06T11:35:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774410299
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
NONE
@gchhablani thank you so much

will try look at it
2021-02-06T11:21:55Z
https://github.com/huggingface/datasets/issues/1776#issuecomment-774457056
https://api.github.com/repos/huggingface/datasets/issues/1776
MDEyOklzc3VlQ29tbWVudDc3NDQ1NzA1Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/774457056/reactions
2021-02-06T11:21:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774457056
https://avatars.githubusercontent.com/u/56231298?v=4
https://api.github.com/users/acul3/events{/privacy}
https://api.github.com/users/acul3/followers
https://api.github.com/users/acul3/following{/other_user}
https://api.github.com/users/acul3/gists{/gist_id}
https://github.com/acul3
acul3
MDQ6VXNlcjU2MjMxMjk4
https://api.github.com/users/acul3/orgs
https://api.github.com/users/acul3/received_events
https://api.github.com/users/acul3/repos
https://api.github.com/users/acul3/starred{/owner}{/repo}
https://api.github.com/users/acul3/subscriptions
User
https://api.github.com/users/acul3
public
I know we can use `Datasets.map` to preprocess a dataset, but I'm using it with very large corpus which generates huge cache file (several TB cache from a 400 GB text file). I have no disk large enough to save it.  Can we preprocess a dataset on the fly without generating cache?

BTW, I tried raising `writer_batch_size`. Seems that argument doesn't have any effect when it's larger than `batch_size`, because you are saving all the batch instantly after it's processed. Please check the following code:

https://github.com/huggingface/datasets/blob/0281f9d881f3a55c89aeaa642f1ba23444b64083/src/datasets/arrow_dataset.py#L1532