Fix datasets export to JSON
CONTRIBUTOR
Linked Issue: #7037
Ideas: #7039 
2024-09-29T15:56:14Z
https://github.com/huggingface/datasets/pull/7181#issuecomment-2381407717
https://api.github.com/repos/huggingface/datasets/issues/7181
IC_kwDODunzps6N8WXl
https://api.github.com/repos/huggingface/datasets/issues/comments/2381407717/reactions
2024-09-29T15:56:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2381407717
https://avatars.githubusercontent.com/u/20443618?v=4
https://api.github.com/users/varadhbhatnagar/events{/privacy}
https://api.github.com/users/varadhbhatnagar/followers
https://api.github.com/users/varadhbhatnagar/following{/other_user}
https://api.github.com/users/varadhbhatnagar/gists{/gist_id}
https://github.com/varadhbhatnagar
varadhbhatnagar
MDQ6VXNlcjIwNDQzNjE4
https://api.github.com/users/varadhbhatnagar/orgs
https://api.github.com/users/varadhbhatnagar/received_events
https://api.github.com/users/varadhbhatnagar/repos
https://api.github.com/users/varadhbhatnagar/starred{/owner}{/repo}
https://api.github.com/users/varadhbhatnagar/subscriptions
User
https://api.github.com/users/varadhbhatnagar
public
CONTRIBUTOR
@albertvillanova / @lhoestq any early feedback?

AFAIK there is no param `orient` in `load_dataset()`. So for orientations other than "records", the loading isn't very accurate. Any thoughts?
2024-09-29T15:56:26Z
https://github.com/huggingface/datasets/pull/7181#issuecomment-2381407779
https://api.github.com/repos/huggingface/datasets/issues/7181
IC_kwDODunzps6N8WYj
https://api.github.com/repos/huggingface/datasets/issues/comments/2381407779/reactions
2024-09-30T06:36:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2381407779
https://avatars.githubusercontent.com/u/20443618?v=4
https://api.github.com/users/varadhbhatnagar/events{/privacy}
https://api.github.com/users/varadhbhatnagar/followers
https://api.github.com/users/varadhbhatnagar/following{/other_user}
https://api.github.com/users/varadhbhatnagar/gists{/gist_id}
https://github.com/varadhbhatnagar
varadhbhatnagar
MDQ6VXNlcjIwNDQzNjE4
https://api.github.com/users/varadhbhatnagar/orgs
https://api.github.com/users/varadhbhatnagar/received_events
https://api.github.com/users/varadhbhatnagar/repos
https://api.github.com/users/varadhbhatnagar/starred{/owner}{/repo}
https://api.github.com/users/varadhbhatnagar/subscriptions
User
https://api.github.com/users/varadhbhatnagar
public
CONTRIBUTOR
`orient = "split"` can also be handled. I will add the changes soon
2024-10-09T18:20:46Z
https://github.com/huggingface/datasets/pull/7181#issuecomment-2402991455
https://api.github.com/repos/huggingface/datasets/issues/7181
IC_kwDODunzps6POr1f
https://api.github.com/repos/huggingface/datasets/issues/comments/2402991455/reactions
2024-10-09T18:21:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2402991455
https://avatars.githubusercontent.com/u/20443618?v=4
https://api.github.com/users/varadhbhatnagar/events{/privacy}
https://api.github.com/users/varadhbhatnagar/followers
https://api.github.com/users/varadhbhatnagar/following{/other_user}
https://api.github.com/users/varadhbhatnagar/gists{/gist_id}
https://github.com/varadhbhatnagar
varadhbhatnagar
MDQ6VXNlcjIwNDQzNjE4
https://api.github.com/users/varadhbhatnagar/orgs
https://api.github.com/users/varadhbhatnagar/received_events
https://api.github.com/users/varadhbhatnagar/repos
https://api.github.com/users/varadhbhatnagar/starred{/owner}{/repo}
https://api.github.com/users/varadhbhatnagar/subscriptions
User
https://api.github.com/users/varadhbhatnagar
public
MEMBER
Thanks for diving into this ! I don't think we want the JSON export to be that complex though, especially if people can do `ds.to_pandas().to_json(orient=...)`. Maybe we can just raise an error and suggest users to use pandas ? And also note that it loads the full dataset in memory so it's mainly for small scale datasets. The only acceptable option for large scale datasets is probably just JSON Lines anyway since it enables streaming.
2024-10-11T13:53:16Z
https://github.com/huggingface/datasets/pull/7181#issuecomment-2407465916
https://api.github.com/repos/huggingface/datasets/issues/7181
IC_kwDODunzps6PfwO8
https://api.github.com/repos/huggingface/datasets/issues/comments/2407465916/reactions
2024-10-11T13:53:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2407465916
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq Simply doing `ds.to_pandas().to_json(orient=...)` is not going to give any batching or multiprocessing benefits right? Also, which function are you referring to - when you say that its meant for small scale datasets only?
2024-10-11T17:18:15Z
https://github.com/huggingface/datasets/pull/7181#issuecomment-2407830282
https://api.github.com/repos/huggingface/datasets/issues/7181
IC_kwDODunzps6PhJMK
https://api.github.com/repos/huggingface/datasets/issues/comments/2407830282/reactions
2024-10-11T17:18:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2407830282
https://avatars.githubusercontent.com/u/20443618?v=4
https://api.github.com/users/varadhbhatnagar/events{/privacy}
https://api.github.com/users/varadhbhatnagar/followers
https://api.github.com/users/varadhbhatnagar/following{/other_user}
https://api.github.com/users/varadhbhatnagar/gists{/gist_id}
https://github.com/varadhbhatnagar
varadhbhatnagar
MDQ6VXNlcjIwNDQzNjE4
https://api.github.com/users/varadhbhatnagar/orgs
https://api.github.com/users/varadhbhatnagar/received_events
https://api.github.com/users/varadhbhatnagar/repos
https://api.github.com/users/varadhbhatnagar/starred{/owner}{/repo}
https://api.github.com/users/varadhbhatnagar/subscriptions
User
https://api.github.com/users/varadhbhatnagar
public
MEMBER
Yes indeed. Though I think it's fine since using something else than orient="lines" is only suitable/useful for small datasets. Or you know a case where a big dataset need to be in a format that is not orient="lines" ?
2024-10-14T12:10:35Z
https://github.com/huggingface/datasets/pull/7181#issuecomment-2411048066
https://api.github.com/repos/huggingface/datasets/issues/7181
IC_kwDODunzps6PtayC
https://api.github.com/repos/huggingface/datasets/issues/comments/2411048066/reactions
2024-10-14T12:10:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2411048066
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq Let me close this PR and open another one where I will add an error message, as suggested here.

> Thanks for diving into this ! I don't think we want the JSON export to be that complex though, especially if people can do `ds.to_pandas().to_json(orient=...)`. Maybe we can just raise an error and suggest users to use pandas ? And also note that it loads the full dataset in memory so it's mainly for small scale datasets. The only acceptable option for large scale datasets is probably just JSON Lines anyway since it enables streaming.


2024-10-27T13:28:16Z
https://github.com/huggingface/datasets/pull/7181#issuecomment-2440024598
https://api.github.com/repos/huggingface/datasets/issues/7181
IC_kwDODunzps6Rb9IW
https://api.github.com/repos/huggingface/datasets/issues/comments/2440024598/reactions
2024-10-27T13:28:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2440024598
https://avatars.githubusercontent.com/u/20443618?v=4
https://api.github.com/users/varadhbhatnagar/events{/privacy}
https://api.github.com/users/varadhbhatnagar/followers
https://api.github.com/users/varadhbhatnagar/following{/other_user}
https://api.github.com/users/varadhbhatnagar/gists{/gist_id}
https://github.com/varadhbhatnagar
varadhbhatnagar
MDQ6VXNlcjIwNDQzNjE4
https://api.github.com/users/varadhbhatnagar/orgs
https://api.github.com/users/varadhbhatnagar/received_events
https://api.github.com/users/varadhbhatnagar/repos
https://api.github.com/users/varadhbhatnagar/starred{/owner}{/repo}
https://api.github.com/users/varadhbhatnagar/subscriptions
User
https://api.github.com/users/varadhbhatnagar
public
CONTRIBUTOR
Addressed here: #7273 
@lhoestq 
2024-11-01T11:55:36Z
https://github.com/huggingface/datasets/pull/7181#issuecomment-2451755796
https://api.github.com/repos/huggingface/datasets/issues/7181
IC_kwDODunzps6SItMU
https://api.github.com/repos/huggingface/datasets/issues/comments/2451755796/reactions
2024-11-01T11:55:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2451755796
https://avatars.githubusercontent.com/u/20443618?v=4
https://api.github.com/users/varadhbhatnagar/events{/privacy}
https://api.github.com/users/varadhbhatnagar/followers
https://api.github.com/users/varadhbhatnagar/following{/other_user}
https://api.github.com/users/varadhbhatnagar/gists{/gist_id}
https://github.com/varadhbhatnagar
varadhbhatnagar
MDQ6VXNlcjIwNDQzNjE4
https://api.github.com/users/varadhbhatnagar/orgs
https://api.github.com/users/varadhbhatnagar/received_events
https://api.github.com/users/varadhbhatnagar/repos
https://api.github.com/users/varadhbhatnagar/starred{/owner}{/repo}
https://api.github.com/users/varadhbhatnagar/subscriptions
User
https://api.github.com/users/varadhbhatnagar
public