Add error_bad_chunk to the JSON loader
MEMBER
Somebody reported the following error message which I think this is related to the goal of this PR:
```Python
03/24/2022 02:19:45 - INFO - __main__ - Step 5637: {'lr': 0.00018773333333333333, 'samples': 360768, 'batch_offset': 5637, 'completed_steps': 704, 'loss/train': 4.473083972930908, 'tokens/s': 6692.6176452714235}
03/24/2022 02:19:46 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [1/20]
03/24/2022 02:20:01 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [2/20]
03/24/2022 02:20:09 - ERROR - datasets.packaged_modules.json.json - Failed to read file 'gzip://file-000000000007.json::https://huggingface.co/datasets/lvwerra/codeparrot-clean-train/resolve/1d740acb9d09cf7a3307553323e2c677a6535407/file-000000000007.json.gz' with error <class 'pyarrow.lib.ArrowInvalid'>: JSON parse error: Invalid value. in row 0
03/24/2022 02:20:24 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [1/20]
03/24/2022 02:20:37 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [2/20]
03/24/2022 02:20:44 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [3/20]
03/24/2022 02:20:49 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [4/20]
03/24/2022 02:20:54 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [5/20]
03/24/2022 02:20:59 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [6/20]
03/24/2022 02:21:12 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [7/20]
03/24/2022 02:21:20 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [8/20]
03/24/2022 02:21:25 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [9/20]
03/24/2022 02:21:30 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [10/20]
03/24/2022 02:21:36 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [11/20]
03/24/2022 02:21:41 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [12/20]
03/24/2022 02:21:46 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [13/20]
03/24/2022 02:21:51 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [14/20]
03/24/2022 02:21:56 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [15/20]
03/24/2022 02:22:01 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [16/20]
03/24/2022 02:22:12 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [17/20]
03/24/2022 02:22:21 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 5sec [18/20]
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/datasets/packaged_modules/json/json.py", line 119, in _generate_tables
    io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
  File "pyarrow/_json.pyx", line 246, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: JSON parse error: Invalid value. in row 0
```
This comes from the CodeParrot training script where streaming is used. When the connection fails it can happen that the JSON cannot be read anymore and then an error is thrown.


2022-03-24T13:56:21Z
https://github.com/huggingface/datasets/pull/2838#issuecomment-1077656192
https://api.github.com/repos/huggingface/datasets/issues/2838
IC_kwDODunzps5AO7qA
https://api.github.com/repos/huggingface/datasets/issues/comments/1077656192/reactions
2022-03-24T13:56:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1077656192
https://avatars.githubusercontent.com/u/8264887?v=4
https://api.github.com/users/lvwerra/events{/privacy}
https://api.github.com/users/lvwerra/followers
https://api.github.com/users/lvwerra/following{/other_user}
https://api.github.com/users/lvwerra/gists{/gist_id}
https://github.com/lvwerra
lvwerra
MDQ6VXNlcjgyNjQ4ODc=
https://api.github.com/users/lvwerra/orgs
https://api.github.com/users/lvwerra/received_events
https://api.github.com/users/lvwerra/repos
https://api.github.com/users/lvwerra/starred{/owner}{/repo}
https://api.github.com/users/lvwerra/subscriptions
User
https://api.github.com/users/lvwerra
public
MEMBER
Yea if streaming makes a JSON unreadable then `error_bad_chunk` would help by skipping all the bad JSON data
2022-03-24T14:00:07Z
https://github.com/huggingface/datasets/pull/2838#issuecomment-1077660060
https://api.github.com/repos/huggingface/datasets/issues/2838
IC_kwDODunzps5AO8mc
https://api.github.com/repos/huggingface/datasets/issues/comments/1077660060/reactions
2022-03-24T14:00:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1077660060
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Should we close this PR?
2023-09-24T10:08:39Z
https://github.com/huggingface/datasets/pull/2838#issuecomment-1732536948
https://api.github.com/repos/huggingface/datasets/issues/2838
IC_kwDODunzps5nRGp0
https://api.github.com/repos/huggingface/datasets/issues/comments/1732536948/reactions
2023-09-24T10:08:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1732536948
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I didn't continue this PR but I think it's valuable (though now I think it would be better to have multiple options: raise, warn or ignore errors). I'll continue it at one point
2023-09-25T09:06:42Z
https://github.com/huggingface/datasets/pull/2838#issuecomment-1733229347
https://api.github.com/repos/huggingface/datasets/issues/2838
IC_kwDODunzps5nTvsj
https://api.github.com/repos/huggingface/datasets/issues/comments/1733229347/reactions
2023-09-25T09:06:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1733229347
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Add the `error_bad_chunk` parameter to the JSON loader.

Setting `error_bad_chunk=False` allows to skip an unparsable chunk of JSON data without raising an error.

Additional note:

In case of an unparsable JSON chunk, the JSON loader no longer tries to load the full JSON (which could take a lot of time in streaming mode) to get the JSON fields that the user may have forgotten to pass. Ex : for squad-like data, the user has to pass `field="data"` to tell the loader to get the list of examples from this field.

TODO: update docs

cc @lvwerra 