Allow 'to_json' to run in unordered fashion in order to lower memory footprint
CONTRIBUTOR
Hi @thomasw21, I remember suggesting `imap_unordered` to @lhoestq at that time to speed up `to_json` further but after trying `pool_imap` on multiple datasets (>9GB) , memory utilisation was almost constant and we decided to go ahead with that only. 

1. Did you try this without `gzip`? Because `gzip`  feature was introduced recently and I didn't check multi_proc thing with `gzip`. One thing I know is that `gzip` is slow in our implementation than `zip` (it's a WIP #3551) 
2. You can try reducing your batch size, this can also help in avoiding OOM errors!
2022-01-31T17:59:33Z
https://github.com/huggingface/datasets/pull/3650#issuecomment-1026053579
https://api.github.com/repos/huggingface/datasets/issues/3650
IC_kwDODunzps49KFXL
https://api.github.com/repos/huggingface/datasets/issues/comments/1026053579/reactions
2022-01-31T18:00:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1026053579
https://avatars.githubusercontent.com/u/19718818?v=4
https://api.github.com/users/bhavitvyamalik/events{/privacy}
https://api.github.com/users/bhavitvyamalik/followers
https://api.github.com/users/bhavitvyamalik/following{/other_user}
https://api.github.com/users/bhavitvyamalik/gists{/gist_id}
https://github.com/bhavitvyamalik
bhavitvyamalik
MDQ6VXNlcjE5NzE4ODE4
https://api.github.com/users/bhavitvyamalik/orgs
https://api.github.com/users/bhavitvyamalik/received_events
https://api.github.com/users/bhavitvyamalik/repos
https://api.github.com/users/bhavitvyamalik/starred{/owner}{/repo}
https://api.github.com/users/bhavitvyamalik/subscriptions
User
https://api.github.com/users/bhavitvyamalik
public
CONTRIBUTOR
Thanks @bhavitvyamalik ! I see. I'm not sure this PR actually fixes things for me either (I ended up reducing the num_proc/batch_size to lower it). It does allow the process to run for longer, but I think the reason why it was waiting is that one of the process crashes .... Unfortunately I was working on a setup with a low RAM/cpu core ratio. I'm actually very surprised that it doesn't change memory utilization, otherwise I don't see the purpose of `imap_unordered` existing. I think it's main purpose are when you have high variance in samples (in terms of bytes), which causes unecessary accumulation in `imap`
  1. Did not try without `gzip`
  2. Yeah or `num_proc`
2022-02-01T10:39:09Z
https://github.com/huggingface/datasets/pull/3650#issuecomment-1026701077
https://api.github.com/repos/huggingface/datasets/issues/3650
IC_kwDODunzps49MjcV
https://api.github.com/repos/huggingface/datasets/issues/comments/1026701077/reactions
2022-02-01T10:39:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1026701077
https://avatars.githubusercontent.com/u/24695242?v=4
https://api.github.com/users/thomasw21/events{/privacy}
https://api.github.com/users/thomasw21/followers
https://api.github.com/users/thomasw21/following{/other_user}
https://api.github.com/users/thomasw21/gists{/gist_id}
https://github.com/thomasw21
thomasw21
MDQ6VXNlcjI0Njk1MjQy
https://api.github.com/users/thomasw21/orgs
https://api.github.com/users/thomasw21/received_events
https://api.github.com/users/thomasw21/repos
https://api.github.com/users/thomasw21/starred{/owner}{/repo}
https://api.github.com/users/thomasw21/subscriptions
User
https://api.github.com/users/thomasw21
public
CONTRIBUTOR
Can you please try without `gzip` to see how it performs? If it works fine then we can improve `gzip` from our side (I'm already working on it)
2022-02-01T17:34:52Z
https://github.com/huggingface/datasets/pull/3650#issuecomment-1027107590
https://api.github.com/repos/huggingface/datasets/issues/3650
IC_kwDODunzps49OGsG
https://api.github.com/repos/huggingface/datasets/issues/comments/1027107590/reactions
2022-02-01T17:34:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1027107590
https://avatars.githubusercontent.com/u/19718818?v=4
https://api.github.com/users/bhavitvyamalik/events{/privacy}
https://api.github.com/users/bhavitvyamalik/followers
https://api.github.com/users/bhavitvyamalik/following{/other_user}
https://api.github.com/users/bhavitvyamalik/gists{/gist_id}
https://github.com/bhavitvyamalik
bhavitvyamalik
MDQ6VXNlcjE5NzE4ODE4
https://api.github.com/users/bhavitvyamalik/orgs
https://api.github.com/users/bhavitvyamalik/received_events
https://api.github.com/users/bhavitvyamalik/repos
https://api.github.com/users/bhavitvyamalik/starred{/owner}{/repo}
https://api.github.com/users/bhavitvyamalik/subscriptions
User
https://api.github.com/users/bhavitvyamalik
public
CONTRIBUTOR
I'll be busy for next few weeks on another project, will do as soon as I have some bandwidth.

2022-02-01T17:49:21Z
https://github.com/huggingface/datasets/pull/3650#issuecomment-1027120457
https://api.github.com/repos/huggingface/datasets/issues/3650
IC_kwDODunzps49OJ1J
https://api.github.com/repos/huggingface/datasets/issues/comments/1027120457/reactions
2022-02-01T17:49:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1027120457
https://avatars.githubusercontent.com/u/24695242?v=4
https://api.github.com/users/thomasw21/events{/privacy}
https://api.github.com/users/thomasw21/followers
https://api.github.com/users/thomasw21/following{/other_user}
https://api.github.com/users/thomasw21/gists{/gist_id}
https://github.com/thomasw21
thomasw21
MDQ6VXNlcjI0Njk1MjQy
https://api.github.com/users/thomasw21/orgs
https://api.github.com/users/thomasw21/received_events
https://api.github.com/users/thomasw21/repos
https://api.github.com/users/thomasw21/starred{/owner}{/repo}
https://api.github.com/users/thomasw21/subscriptions
User
https://api.github.com/users/thomasw21
public
MEMBER
Should we close this PR?
2023-09-24T09:59:22Z
https://github.com/huggingface/datasets/pull/3650#issuecomment-1732535238
https://api.github.com/repos/huggingface/datasets/issues/3650
IC_kwDODunzps5nRGPG
https://api.github.com/repos/huggingface/datasets/issues/comments/1732535238/reactions
2023-09-24T09:59:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1732535238
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
Yes we can close this PR if considered unneeded.
2023-09-24T16:45:48Z
https://github.com/huggingface/datasets/pull/3650#issuecomment-1732617423
https://api.github.com/repos/huggingface/datasets/issues/3650
IC_kwDODunzps5nRaTP
https://api.github.com/repos/huggingface/datasets/issues/comments/1732617423/reactions
2023-09-24T16:45:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1732617423
https://avatars.githubusercontent.com/u/24695242?v=4
https://api.github.com/users/thomasw21/events{/privacy}
https://api.github.com/users/thomasw21/followers
https://api.github.com/users/thomasw21/following{/other_user}
https://api.github.com/users/thomasw21/gists{/gist_id}
https://github.com/thomasw21
thomasw21
MDQ6VXNlcjI0Njk1MjQy
https://api.github.com/users/thomasw21/orgs
https://api.github.com/users/thomasw21/received_events
https://api.github.com/users/thomasw21/repos
https://api.github.com/users/thomasw21/starred{/owner}{/repo}
https://api.github.com/users/thomasw21/subscriptions
User
https://api.github.com/users/thomasw21
public
I'm using `to_json(..., num_proc=num_proc, compressiong='gzip')` with `num_proc>1`. I'm having an issue where things seem to deadlock at some point. Eventually I see OOM. I'm guessing it's an issue where one process starts to take a long time for a specific batch, and so other process keep accumulating their results in memory.

In order to flush memory, I propose we use optional `imap_unordered`. This will prevent one process to block the other ones. The logical thinking is that index are rarily relevant, and in one wants to keep an index, one can still create another column and reconstruct from there.