feat: use content defined chunking 
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7589). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2025-05-29T18:40:28Z
https://github.com/huggingface/datasets/pull/7589#issuecomment-2920258514
https://api.github.com/repos/huggingface/datasets/issues/7589
IC_kwDODunzps6uD5vS
https://api.github.com/repos/huggingface/datasets/issues/comments/2920258514/reactions
2025-05-29T18:40:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2920258514
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Need to set `DEFAULT_MAX_BATCH_SIZE = 1024 * 1024`
2025-06-16T10:08:55Z
https://github.com/huggingface/datasets/pull/7589#issuecomment-2975933002
https://api.github.com/repos/huggingface/datasets/issues/7589
IC_kwDODunzps6xYSJK
https://api.github.com/repos/huggingface/datasets/issues/comments/2975933002/reactions
2025-06-16T10:08:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2975933002
https://avatars.githubusercontent.com/u/961747?v=4
https://api.github.com/users/kszucs/events{/privacy}
https://api.github.com/users/kszucs/followers
https://api.github.com/users/kszucs/following{/other_user}
https://api.github.com/users/kszucs/gists{/gist_id}
https://github.com/kszucs
kszucs
MDQ6VXNlcjk2MTc0Nw==
https://api.github.com/users/kszucs/orgs
https://api.github.com/users/kszucs/received_events
https://api.github.com/users/kszucs/repos
https://api.github.com/users/kszucs/starred{/owner}{/repo}
https://api.github.com/users/kszucs/subscriptions
User
https://api.github.com/users/kszucs
public
MEMBER
We should consider enabling page indexes by default when writing parquet files to enable page pruning readers like the next dataset viewer https://github.com/huggingface/dataset-viewer/pull/3199
2025-07-24T09:21:54Z
https://github.com/huggingface/datasets/pull/7589#issuecomment-3112719123
https://api.github.com/repos/huggingface/datasets/issues/7589
IC_kwDODunzps65iFMT
https://api.github.com/repos/huggingface/datasets/issues/comments/3112719123/reactions
2025-07-24T09:21:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3112719123
https://avatars.githubusercontent.com/u/961747?v=4
https://api.github.com/users/kszucs/events{/privacy}
https://api.github.com/users/kszucs/followers
https://api.github.com/users/kszucs/following{/other_user}
https://api.github.com/users/kszucs/gists{/gist_id}
https://github.com/kszucs
kszucs
MDQ6VXNlcjk2MTc0Nw==
https://api.github.com/users/kszucs/orgs
https://api.github.com/users/kszucs/received_events
https://api.github.com/users/kszucs/repos
https://api.github.com/users/kszucs/starred{/owner}{/repo}
https://api.github.com/users/kszucs/subscriptions
User
https://api.github.com/users/kszucs
public
Use content defined chunking by default when writing parquet files.
- [x] set the parameters in `io.parquet.ParquetDatasetReader`
- [x] set the parameters in `arrow_writer.ParquetWriter`

It requires a new pyarrow pin ">=21.0.0" which is released now.