Fix export to JSON when dataset larger than batch size
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7039). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2024-07-11T06:54:44Z
https://github.com/huggingface/datasets/pull/7039#issuecomment-2222174661
https://api.github.com/repos/huggingface/datasets/issues/7039
IC_kwDODunzps6Ec7HF
https://api.github.com/repos/huggingface/datasets/issues/comments/2222174661/reactions
2024-07-11T06:54:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222174661
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
The test before confirms the bug.

There are different possible solutions to this issue:
- the easiest would be to write multiple JSON files, one for each batch; this solution can be done in parallel if `num_proc` is passed
- alternatively, we could tweak the writing and remove the extra `[` and `]` characters; this solution will only be valid if `orient="records"`
- others?
2024-07-11T07:05:29Z
https://github.com/huggingface/datasets/pull/7039#issuecomment-2222191211
https://api.github.com/repos/huggingface/datasets/issues/7039
IC_kwDODunzps6Ec_Jr
https://api.github.com/repos/huggingface/datasets/issues/comments/2222191211/reactions
2024-07-11T07:27:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222191211
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
@albertvillanova I was planning to approach it in the second way for `orient="records"` , `orient="values"` and `orient="index"`. For `orient="split"`, the columns and index can be written in one go and the data can be written in streaming manner. For `orient="columns"`, each column can be written in a streaming way.  LMK if I should go ahead with this.

> The test before confirms the bug.
> 
> There are different possible solutions to this issue:
> 
>     * the easiest would be to write multiple JSON files, one for each batch; this solution can be done in parallel if `num_proc` is passed
> 
>     * alternatively, we could tweak the writing and remove the extra `[` and `]` characters; this solution will only be valid if `orient="records"`
> 
>     * others?


2024-09-26T17:39:30Z
https://github.com/huggingface/datasets/pull/7039#issuecomment-2377566082
https://api.github.com/repos/huggingface/datasets/issues/7039
IC_kwDODunzps6NtseC
https://api.github.com/repos/huggingface/datasets/issues/comments/2377566082/reactions
2024-09-26T17:39:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2377566082
https://avatars.githubusercontent.com/u/20443618?v=4
https://api.github.com/users/varadhbhatnagar/events{/privacy}
https://api.github.com/users/varadhbhatnagar/followers
https://api.github.com/users/varadhbhatnagar/following{/other_user}
https://api.github.com/users/varadhbhatnagar/gists{/gist_id}
https://github.com/varadhbhatnagar
varadhbhatnagar
MDQ6VXNlcjIwNDQzNjE4
https://api.github.com/users/varadhbhatnagar/orgs
https://api.github.com/users/varadhbhatnagar/received_events
https://api.github.com/users/varadhbhatnagar/repos
https://api.github.com/users/varadhbhatnagar/starred{/owner}{/repo}
https://api.github.com/users/varadhbhatnagar/subscriptions
User
https://api.github.com/users/varadhbhatnagar
public
Fix export to JSON (`lines=False`) when dataset larger than batch size.

Fix #7037.