A bug of Dataset.to_json() function
MEMBER
Thanks for reporting, @LinglingGreat.

I confirm this is a bug.
2024-07-10T13:07:39Z
https://github.com/huggingface/datasets/issues/7037#issuecomment-2220467466
https://api.github.com/repos/huggingface/datasets/issues/7037
IC_kwDODunzps6EWaUK
https://api.github.com/repos/huggingface/datasets/issues/comments/2220467466/reactions
2024-07-10T13:07:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2220467466
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
@albertvillanova I would like to take a shot at this if you aren't working on it currently. Let me know!
2024-09-22T13:16:06Z
https://github.com/huggingface/datasets/issues/7037#issuecomment-2366784805
https://api.github.com/repos/huggingface/datasets/issues/7037
IC_kwDODunzps6NEkUl
https://api.github.com/repos/huggingface/datasets/issues/comments/2366784805/reactions
2024-09-22T13:16:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2366784805
https://avatars.githubusercontent.com/u/20443618?v=4
https://api.github.com/users/varadhbhatnagar/events{/privacy}
https://api.github.com/users/varadhbhatnagar/followers
https://api.github.com/users/varadhbhatnagar/following{/other_user}
https://api.github.com/users/varadhbhatnagar/gists{/gist_id}
https://github.com/varadhbhatnagar
varadhbhatnagar
MDQ6VXNlcjIwNDQzNjE4
https://api.github.com/users/varadhbhatnagar/orgs
https://api.github.com/users/varadhbhatnagar/received_events
https://api.github.com/users/varadhbhatnagar/repos
https://api.github.com/users/varadhbhatnagar/starred{/owner}{/repo}
https://api.github.com/users/varadhbhatnagar/subscriptions
User
https://api.github.com/users/varadhbhatnagar
public
### Describe the bug

When using the Dataset.to_json() function, an unexpected error occurs if the parameter is set to lines=False. The stored data should be in the form of a list, but it actually turns into multiple lists, which causes an error when reading the data again.
The reason is that to_json() writes to the file in several segments based on the batch size. This is not a problem when lines=True, but it is incorrect when lines=False, because writing in several times will produce multiple lists(when len(dataset) > batch_size).

### Steps to reproduce the bug

try this code:
```python
from datasets import load_dataset
import json

train_dataset = load_dataset("Anthropic/hh-rlhf", data_dir="harmless-base")["train"]
output_path = "./harmless-base_hftojs.json"
print(len(train_dataset))
train_dataset.to_json(output_path, lines=False, force_ascii=False, indent=2)

with open(output_path, encoding="utf-8") as f:
    data = json.loads(f.read())

```
it raise error: json.decoder.JSONDecodeError: Extra data: line 4003 column 1 (char 1373709)

Extra square brackets have appeared here:
<img width="265" alt="image" src="https://github.com/huggingface/datasets/assets/26499566/81492332-386d-42e8-88d1-b6d4ae3682cc">


### Expected behavior

The code runs normally.

### Environment info

datasets=2.20.0