Jsonlines export error
MEMBER
Thanks for reporting @TevenLeScao! I'm having a look...
2021-07-09T14:12:57Z
https://github.com/huggingface/datasets/issues/2615#issuecomment-877219535
https://api.github.com/repos/huggingface/datasets/issues/2615
MDEyOklzc3VlQ29tbWVudDg3NzIxOTUzNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/877219535/reactions
2021-07-09T14:12:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/877219535
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
(not sure what just happened on the assignations sorry)
2021-07-09T14:14:24Z
https://github.com/huggingface/datasets/issues/2615#issuecomment-877220531
https://api.github.com/repos/huggingface/datasets/issues/2615
MDEyOklzc3VlQ29tbWVudDg3NzIyMDUzMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/877220531/reactions
2021-07-09T14:14:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/877220531
https://avatars.githubusercontent.com/u/26709476?v=4
https://api.github.com/users/TevenLeScao/events{/privacy}
https://api.github.com/users/TevenLeScao/followers
https://api.github.com/users/TevenLeScao/following{/other_user}
https://api.github.com/users/TevenLeScao/gists{/gist_id}
https://github.com/TevenLeScao
TevenLeScao
MDQ6VXNlcjI2NzA5NDc2
https://api.github.com/users/TevenLeScao/orgs
https://api.github.com/users/TevenLeScao/received_events
https://api.github.com/users/TevenLeScao/repos
https://api.github.com/users/TevenLeScao/starred{/owner}{/repo}
https://api.github.com/users/TevenLeScao/subscriptions
User
https://api.github.com/users/TevenLeScao
public
CONTRIBUTOR
For some reason this happens (both `datasets` version are on master) only on Python 3.6 and not Python 3.8.
2021-07-09T14:16:46Z
https://github.com/huggingface/datasets/issues/2615#issuecomment-877222165
https://api.github.com/repos/huggingface/datasets/issues/2615
MDEyOklzc3VlQ29tbWVudDg3NzIyMjE2NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/877222165/reactions
2021-07-09T14:16:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/877222165
https://avatars.githubusercontent.com/u/26709476?v=4
https://api.github.com/users/TevenLeScao/events{/privacy}
https://api.github.com/users/TevenLeScao/followers
https://api.github.com/users/TevenLeScao/following{/other_user}
https://api.github.com/users/TevenLeScao/gists{/gist_id}
https://github.com/TevenLeScao
TevenLeScao
MDQ6VXNlcjI2NzA5NDc2
https://api.github.com/users/TevenLeScao/orgs
https://api.github.com/users/TevenLeScao/received_events
https://api.github.com/users/TevenLeScao/repos
https://api.github.com/users/TevenLeScao/starred{/owner}{/repo}
https://api.github.com/users/TevenLeScao/subscriptions
User
https://api.github.com/users/TevenLeScao
public
MEMBER
@TevenLeScao we are using `pandas` to serialize the dataset to JSON Lines. So it must be due to pandas. Could you please check the pandas version causing the issue?
2021-07-09T14:51:25Z
https://github.com/huggingface/datasets/issues/2615#issuecomment-877245263
https://api.github.com/repos/huggingface/datasets/issues/2615
MDEyOklzc3VlQ29tbWVudDg3NzI0NTI2Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/877245263/reactions
2021-07-09T14:51:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/877245263
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
@TevenLeScao I have just checked it: this was a bug in `pandas` and it was fixed in version 1.2: https://github.com/pandas-dev/pandas/pull/36898
2021-07-09T15:01:58Z
https://github.com/huggingface/datasets/issues/2615#issuecomment-877252038
https://api.github.com/repos/huggingface/datasets/issues/2615
MDEyOklzc3VlQ29tbWVudDg3NzI1MjAzOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/877252038/reactions
2021-07-09T15:01:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/877252038
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Thanks ! I'm creating a PR
2021-07-09T15:02:46Z
https://github.com/huggingface/datasets/issues/2615#issuecomment-877252622
https://api.github.com/repos/huggingface/datasets/issues/2615
MDEyOklzc3VlQ29tbWVudDg3NzI1MjYyMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/877252622/reactions
2021-07-09T15:02:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/877252622
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Well I though it was me who has taken on this issue... 😅 
2021-07-09T15:06:59Z
https://github.com/huggingface/datasets/issues/2615#issuecomment-877255517
https://api.github.com/repos/huggingface/datasets/issues/2615
MDEyOklzc3VlQ29tbWVudDg3NzI1NTUxNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/877255517/reactions
2021-07-09T15:06:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/877255517
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Sorry, I was also talking to teven offline so I already had the PR ready before noticing x)
2021-07-09T15:13:55Z
https://github.com/huggingface/datasets/issues/2615#issuecomment-877260144
https://api.github.com/repos/huggingface/datasets/issues/2615
MDEyOklzc3VlQ29tbWVudDg3NzI2MDE0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/877260144/reactions
2021-07-09T15:13:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/877260144
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I was also already working in my PR... Nevermind. Next time we should pay attention if there is somebody (self-)assigned to an issue and if he/she is still working on it before overtaking it... 😄 
2021-07-09T15:18:33Z
https://github.com/huggingface/datasets/issues/2615#issuecomment-877263128
https://api.github.com/repos/huggingface/datasets/issues/2615
MDEyOklzc3VlQ29tbWVudDg3NzI2MzEyOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/877263128/reactions
2021-07-09T15:18:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/877263128
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
The fix is available on `master` @TevenLeScao , thanks for reporting
2021-07-09T15:29:07Z
https://github.com/huggingface/datasets/issues/2615#issuecomment-877269892
https://api.github.com/repos/huggingface/datasets/issues/2615
MDEyOklzc3VlQ29tbWVudDg3NzI2OTg5Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/877269892/reactions
2021-07-09T15:29:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/877269892
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Describe the bug
When exporting large datasets in jsonlines (c4 in my case) the created file has an error every 9999 lines: the 9999th and 10000th are concatenated, thus breaking the jsonlines format. This sounds like it is related to batching, which is by 10000 by default

## Steps to reproduce the bug
This what I'm running:

in python:

```
from datasets import load_dataset
ptb = load_dataset("ptb_text_only")
ptb["train"].to_json("ptb.jsonl")
```

then out of python:

```
head -10000 ptb.jsonl
```

## Expected results
Properly separated lines

## Actual results
The last line is a concatenation of two lines

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.1.dev0
- Platform: Linux-5.4.0-1046-gcp-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyArrow version: 4.0.1