I tried to load a custom dataset using the following statement: dataset = load_dataset('json', data_files=data_files). The dataset contains 50 million text-image pairs, but an error occurred.
MEMBER
Thanks for reporting, @cjt222.

What is the structure of your JSON files. Please note that it is normally simpler if the data file format is JSON-Lines instead. 
2023-05-30T06:00:23Z
https://github.com/huggingface/datasets/issues/5913#issuecomment-1567797327
https://api.github.com/repos/huggingface/datasets/issues/5913
IC_kwDODunzps5dcrBP
https://api.github.com/repos/huggingface/datasets/issues/comments/1567797327/reactions
2023-05-30T06:00:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1567797327
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
> Thanks for reporting, @cjt222.
> 
> What is the structure of your JSON files. Please note that it is normally simpler if the data file format is JSON-Lines instead.

Thanks! I have encountered similar problems. I modify the json format from list to line and works!
2023-06-10T08:25:11Z
https://github.com/huggingface/datasets/issues/5913#issuecomment-1585560673
https://api.github.com/repos/huggingface/datasets/issues/5913
IC_kwDODunzps5egbxh
https://api.github.com/repos/huggingface/datasets/issues/comments/1585560673/reactions
2023-06-10T08:25:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1585560673
https://avatars.githubusercontent.com/u/64362896?v=4
https://api.github.com/users/BUAADreamer/events{/privacy}
https://api.github.com/users/BUAADreamer/followers
https://api.github.com/users/BUAADreamer/following{/other_user}
https://api.github.com/users/BUAADreamer/gists{/gist_id}
https://github.com/BUAADreamer
BUAADreamer
MDQ6VXNlcjY0MzYyODk2
https://api.github.com/users/BUAADreamer/orgs
https://api.github.com/users/BUAADreamer/received_events
https://api.github.com/users/BUAADreamer/repos
https://api.github.com/users/BUAADreamer/starred{/owner}{/repo}
https://api.github.com/users/BUAADreamer/subscriptions
User
https://api.github.com/users/BUAADreamer
public
### Describe the bug

File "/home/kas/.conda/envs/diffusers/lib/python3.7/site-packages/datasets/builder.py", line 1858, in _prepare_split_single
Downloading and preparing dataset json/default to /home/kas/diffusers/examples/dreambooth/cache_data/datasets/json/default-acf423d8c6ef99d0/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s] Downloading data files: 100%|██████████| 1/1 [00:00<00:00, 84.35it/s]
Extracting data files: 0%| | 0/1 [00:00<?, ?it/s] for _, table in generator:
File "/home/kas/.conda/envs/diffusers/lib/python3.7/site-packages/datasets/packaged_modules/json/json.py", line 114, in _generate_tables
io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
File "pyarrow/_json.pyx", line 258, in pyarrow._json.read_json
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 27.72it/s]
Generating train split: 0 examples [00:00, ? examples/s] File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 125, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2390448764

### Steps to reproduce the bug

1、data_files = ["1.json", "2.json", "3.json"]
2、dataset = load_dataset('json', data_files=data_files)

### Expected behavior

Read the dataset normally.



### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-4.15.0-29-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 1.3.5