Error when loading a HUGE json file (pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries)
MEMBER
Hi ! Can you try to increase the block size ? For example
```python
block_size_10MB = 10<<20
load_dataset("json", ..., block_size=block_size_10MB)
```
The block size corresponds to how much bytes to process at a time from the input stream.
This will determine multi-threading granularity as well as the size of individual chunks in the dataset.

You can also try with bigger block sizes if needed
2021-04-07T10:37:08Z
https://github.com/huggingface/datasets/issues/2181#issuecomment-814805348
https://api.github.com/repos/huggingface/datasets/issues/2181
MDEyOklzc3VlQ29tbWVudDgxNDgwNTM0OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/814805348/reactions
2021-04-07T10:37:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814805348
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi @lhoestq! Thank you for your prompt reply.
I have experimented with (10<<20, 10<<28, 10<<30, 10<<33, 10<<34), since my machine has 192G of memory, but it's either the above-mentioned error or processed killed because of OOM.

Could you give me a bit of background on why block size needs to be exactly calibrated?
To my understanding, small block sized should run just fine despite its slowness..



2021-04-07T11:49:25Z
https://github.com/huggingface/datasets/issues/2181#issuecomment-814849982
https://api.github.com/repos/huggingface/datasets/issues/2181
MDEyOklzc3VlQ29tbWVudDgxNDg0OTk4Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/814849982/reactions
2021-04-07T11:49:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814849982
https://avatars.githubusercontent.com/u/29157715?v=4
https://api.github.com/users/hwijeen/events{/privacy}
https://api.github.com/users/hwijeen/followers
https://api.github.com/users/hwijeen/following{/other_user}
https://api.github.com/users/hwijeen/gists{/gist_id}
https://github.com/hwijeen
hwijeen
MDQ6VXNlcjI5MTU3NzE1
https://api.github.com/users/hwijeen/orgs
https://api.github.com/users/hwijeen/received_events
https://api.github.com/users/hwijeen/repos
https://api.github.com/users/hwijeen/starred{/owner}{/repo}
https://api.github.com/users/hwijeen/subscriptions
User
https://api.github.com/users/hwijeen
public
MEMBER
We're using the JSON loader of pyarrow. It parses the file chunk by chunk to load the dataset.
This issue happens when there's no delimiter in one chunk of data. For json line, the delimiter is the end of line.
So with a big value for chunk_size this should have worked unless you have one extremely long line in your file.

Also what version of pyarrow are you using ?

FInally I wonder if it could be an issue on pyarrow's side when using big json files. (I haven't tested big json files like yours)
2021-04-07T12:58:47Z
https://github.com/huggingface/datasets/issues/2181#issuecomment-814892619
https://api.github.com/repos/huggingface/datasets/issues/2181
MDEyOklzc3VlQ29tbWVudDgxNDg5MjYxOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/814892619/reactions
2021-04-07T12:58:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814892619
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I'm using `pyarrow==3.0.0` with `datasets==1.5.0`.

Your point totally makes sense. I will check if my jsonl file contains an extremely long file and let you know. 

Here are some different error messages that I got when tweaking `block_size`. I also suspect that this is related to the pyarrow... but I guess it would be wonderful if datasesets could give a clear guide on how to play with large datasets! (I am suddenly experiencing various issue when working with large datasets.. e.g. #1992 )
```python
    return paj.ReadOptions(use_threads=self.use_threads, block_size=self.block_size)
  File "pyarrow/_json.pyx", line 56, in pyarrow._json.ReadOptions.__init__
  File "pyarrow/_json.pyx", line 81, in pyarrow._json.ReadOptions.block_size.__set__
OverflowError: value too large to convert to int32_t
```

```python

line 83, in _generate_tables
    parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 247, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Exceeded maximum rows
```
2021-04-07T15:32:23Z
https://github.com/huggingface/datasets/issues/2181#issuecomment-815010985
https://api.github.com/repos/huggingface/datasets/issues/2181
MDEyOklzc3VlQ29tbWVudDgxNTAxMDk4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/815010985/reactions
2021-04-07T15:32:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/815010985
https://avatars.githubusercontent.com/u/29157715?v=4
https://api.github.com/users/hwijeen/events{/privacy}
https://api.github.com/users/hwijeen/followers
https://api.github.com/users/hwijeen/following{/other_user}
https://api.github.com/users/hwijeen/gists{/gist_id}
https://github.com/hwijeen
hwijeen
MDQ6VXNlcjI5MTU3NzE1
https://api.github.com/users/hwijeen/orgs
https://api.github.com/users/hwijeen/received_events
https://api.github.com/users/hwijeen/repos
https://api.github.com/users/hwijeen/starred{/owner}{/repo}
https://api.github.com/users/hwijeen/subscriptions
User
https://api.github.com/users/hwijeen
public
NONE
I am getting the same error. When I tweak the block_size, I also find:
`OverflowError: value too large to convert to int32_t`
and 
`pyarrow.lib.ArrowInvalid: Exceeded maximum rows`

2021-04-08T15:36:13Z
https://github.com/huggingface/datasets/issues/2181#issuecomment-815922723
https://api.github.com/repos/huggingface/datasets/issues/2181
MDEyOklzc3VlQ29tbWVudDgxNTkyMjcyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/815922723/reactions
2021-04-08T15:36:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/815922723
https://avatars.githubusercontent.com/u/614861?v=4
https://api.github.com/users/jpilaul/events{/privacy}
https://api.github.com/users/jpilaul/followers
https://api.github.com/users/jpilaul/following{/other_user}
https://api.github.com/users/jpilaul/gists{/gist_id}
https://github.com/jpilaul
jpilaul
MDQ6VXNlcjYxNDg2MQ==
https://api.github.com/users/jpilaul/orgs
https://api.github.com/users/jpilaul/received_events
https://api.github.com/users/jpilaul/repos
https://api.github.com/users/jpilaul/starred{/owner}{/repo}
https://api.github.com/users/jpilaul/subscriptions
User
https://api.github.com/users/jpilaul
public
NONE
I made more tests. I used a smaller dataset and I was getting the same error, which means that it was not necessarily linked to the dataset size. To make both my smaller and larger datasets work, I got rid of lists with the json file. I had the following data format:
```python
[
  {'key': "a", 'value': ['one', 'two', 'three']},
  {'key': "b", 'value': ['four', 'five', 'six']}
]
```
I changed to:

```python
  {'key': "a", 'value': 'one\ntwo\nthree'},
  {'key': "b", 'value': 'four\nfive\nsix']}
```
and that worked!

I used the following to reformat my json file:
```python
with open(file_name, "w", encoding="utf-8") as f:
    for item in list_:
        f.write(json.dumps(item) + "\n")
```
This works with `block_size_10MB = 10 << 20` or without specifying `block_size`.
2021-04-08T16:42:37Z
https://github.com/huggingface/datasets/issues/2181#issuecomment-815975287
https://api.github.com/repos/huggingface/datasets/issues/2181
MDEyOklzc3VlQ29tbWVudDgxNTk3NTI4Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/815975287/reactions
2021-04-08T16:51:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/815975287
https://avatars.githubusercontent.com/u/614861?v=4
https://api.github.com/users/jpilaul/events{/privacy}
https://api.github.com/users/jpilaul/followers
https://api.github.com/users/jpilaul/following{/other_user}
https://api.github.com/users/jpilaul/gists{/gist_id}
https://github.com/jpilaul
jpilaul
MDQ6VXNlcjYxNDg2MQ==
https://api.github.com/users/jpilaul/orgs
https://api.github.com/users/jpilaul/received_events
https://api.github.com/users/jpilaul/repos
https://api.github.com/users/jpilaul/starred{/owner}{/repo}
https://api.github.com/users/jpilaul/subscriptions
User
https://api.github.com/users/jpilaul
public
MEMBER
Thanks @hwijeen for reporting and thanks @jpilaul for pointing this out.

Indeed, those are different JSON-like formats:
- the first one is the **standard JSON** format: all the file content is JSON-valid, thus all content is either a JSON object (between curly brackets `{...}`) or a JSON array (between square brackets `[...]`)
- the second one is called **JSON Lines**: the entire file content is not JSON-valid, but only every line (newline-delimited) is JSON-valid

Currently PyArrow only supports **JSON Lines** format: 
- https://arrow.apache.org/docs/python/generated/pyarrow.json.read_json.html
  > Currently only the line-delimited JSON format is supported.
- https://arrow.apache.org/docs/python/json.html
  > Arrow supports reading columnar data from line-delimited JSON files.
2021-04-09T09:47:31Z
https://github.com/huggingface/datasets/issues/2181#issuecomment-816564110
https://api.github.com/repos/huggingface/datasets/issues/2181
MDEyOklzc3VlQ29tbWVudDgxNjU2NDExMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/816564110/reactions
2021-04-09T09:48:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/816564110
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Thanks @albertvillanova for your explanation, it is helpful to know (maybe add to docs?)!
However, the problem I described above happened when I was dealing with jsonl files 😿
Although I did not thoroughly inspect, I suspect the cause was the one extremely long document in my case.
2021-04-09T09:55:18Z
https://github.com/huggingface/datasets/issues/2181#issuecomment-816568662
https://api.github.com/repos/huggingface/datasets/issues/2181
MDEyOklzc3VlQ29tbWVudDgxNjU2ODY2Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/816568662/reactions
2021-04-09T09:55:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/816568662
https://avatars.githubusercontent.com/u/29157715?v=4
https://api.github.com/users/hwijeen/events{/privacy}
https://api.github.com/users/hwijeen/followers
https://api.github.com/users/hwijeen/following{/other_user}
https://api.github.com/users/hwijeen/gists{/gist_id}
https://github.com/hwijeen
hwijeen
MDQ6VXNlcjI5MTU3NzE1
https://api.github.com/users/hwijeen/orgs
https://api.github.com/users/hwijeen/received_events
https://api.github.com/users/hwijeen/repos
https://api.github.com/users/hwijeen/starred{/owner}{/repo}
https://api.github.com/users/hwijeen/subscriptions
User
https://api.github.com/users/hwijeen
public
MEMBER
I see... I guess there is another problem going one then, related to the size.
2021-04-09T10:05:55Z
https://github.com/huggingface/datasets/issues/2181#issuecomment-816574966
https://api.github.com/repos/huggingface/datasets/issues/2181
MDEyOklzc3VlQ29tbWVudDgxNjU3NDk2Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/816574966/reactions
2021-04-09T10:05:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/816574966
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Hi, thanks for the great library. I have used the brilliant library for a couple of small projects, and now using it for a fairly big project.
When loading a huge json file of 500GB, pyarrow complains as follows:
```
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/datasets/builder.py", line 531, in incomplete_dir
    yield tmp_dir
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/datasets/builder.py", line 573, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/datasets/builder.py", line 650, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/datasets/builder.py", line 1027, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False, disable=not_verbose):
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/tqdm/std.py", line 1133, in __iter__
    for obj in iterable:
  File "/app/.cache/huggingface/modules/datasets_modules/datasets/json/9498524fd296a6cca99c66d6c5be507d1c0991f5a814e535b507f4a66096a641/json.py", line 83, in _generate_tables
    parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 247, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```
When using only a small portion of the sample file, say first 100 lines, it works perfectly well..

I see that it is the error from pyarrow, but could you give me a hint or possible solutions?
#369 describes the same error and #372 claims to have fixed the issue, but I have no clue why I am still getting this one. Thanks in advance!