Unable to load JSON saved using `to_json`
MEMBER
Hi @DarshanDeshpande,

Please note that the default format of the method `Dataset.to_json` is [JSON-Lines](https://jsonlines.org/): it passes `orient="records", lines=True` to `pandas.DataFrame.to_json`. This format is specially useful for large datasets, since unlike regular JSON files, it does not require loading all the data into memory at once, but can be done iteratively by batches.

In order to read this file using the `json` library, you should parse line by line:
```python
with open("full_dataset.json", "r") as f:
    data = [json.loads(line) for line in f]
len(data)
```
Maybe we should explain this better in our docs.
2024-05-12T06:39:48Z
https://github.com/huggingface/datasets/issues/6891#issuecomment-2106139176
https://api.github.com/repos/huggingface/datasets/issues/6891
IC_kwDODunzps59iSIo
https://api.github.com/repos/huggingface/datasets/issues/comments/2106139176/reactions
2024-05-12T06:39:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2106139176
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Now we explain this better in out docs:
- #6895
2024-05-16T14:32:54Z
https://github.com/huggingface/datasets/issues/6891#issuecomment-2115420098
https://api.github.com/repos/huggingface/datasets/issues/6891
IC_kwDODunzps5-Fr_C
https://api.github.com/repos/huggingface/datasets/issues/comments/2115420098/reactions
2024-05-16T14:32:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2115420098
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
### Describe the bug

Datasets stored in the JSON format cannot be loaded using `json.load()`

### Steps to reproduce the bug

```
import json
from datasets import load_dataset

dataset = load_dataset("squad")
train_dataset, test_dataset = dataset["train"], dataset["validation"]
test_dataset.to_json("full_dataset.json")

# This works
loaded_test = load_dataset("json", data_files="full_dataset.json")

# This fails
loaded_test = json.load(open("full_dataset.json", "r"))
```

### Expected behavior

The JSON should be correctly formatted when writing so that it can be loaded using `json.load()`.

### Environment info

Colab: https://colab.research.google.com/drive/1st1iStFUVgu9ZPvnzSzL4vDeYWDwYpUm?usp=sharing