Adjusting chunk size for streaming datasets
MEMBER
Hi ! Data streaming uses `fsspec` to read the data files progressively. IIRC the block size for buffering is 5MiB by default. So every time you finish iterating over a block, it downloads the next one. You can still try to increase the `fsspec` block size for buffering if it can help. To do so you just need to increase `fsspec.spec.AbstractBufferedFile.DEFAULT_BLOCK_SIZE `

Currently this is unfortunately done in a single thread, so it blocks the processing to download and uncompress the next block. At one point it would be nice to be able to do that in parallel !
2022-01-05T16:12:22Z
https://github.com/huggingface/datasets/issues/3499#issuecomment-1005864206
https://api.github.com/repos/huggingface/datasets/issues/3499
IC_kwDODunzps479EUO
https://api.github.com/repos/huggingface/datasets/issues/comments/1005864206/reactions
2022-01-05T16:12:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1005864206
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi! Thanks for the help, I will try it :)
2022-01-07T15:39:21Z
https://github.com/huggingface/datasets/issues/3499#issuecomment-1007505253
https://api.github.com/repos/huggingface/datasets/issues/3499
IC_kwDODunzps48DU9l
https://api.github.com/repos/huggingface/datasets/issues/comments/1007505253/reactions
2022-01-07T15:39:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1007505253
https://avatars.githubusercontent.com/u/3775944?v=4
https://api.github.com/users/JoelNiklaus/events{/privacy}
https://api.github.com/users/JoelNiklaus/followers
https://api.github.com/users/JoelNiklaus/following{/other_user}
https://api.github.com/users/JoelNiklaus/gists{/gist_id}
https://github.com/JoelNiklaus
JoelNiklaus
MDQ6VXNlcjM3NzU5NDQ=
https://api.github.com/users/JoelNiklaus/orgs
https://api.github.com/users/JoelNiklaus/received_events
https://api.github.com/users/JoelNiklaus/repos
https://api.github.com/users/JoelNiklaus/starred{/owner}{/repo}
https://api.github.com/users/JoelNiklaus/subscriptions
User
https://api.github.com/users/JoelNiklaus
public
**Is your feature request related to a problem? Please describe.**
I want to use mc4 which I cannot save locally, so I stream it. However, I want to process the entire dataset and filter some documents from it. With the current chunk size of around 1000 documents (right?) I hit a performance bottleneck because of the frequent decompressing. 

**Describe the solution you'd like**
I would appreciate a parameter in the load_dataset function, that allows me to set the chunksize myself (to a value like 100'000 in my case). Like that, I hope to improve the processing time.

