Loading local datasets got strangely stuck
NONE
Yesterday I waited for more than 12 hours to make sure it was really **stuck** instead of proceeding too slow.
2023-08-01T02:30:43Z
https://github.com/huggingface/datasets/issues/6108#issuecomment-1659471581
https://api.github.com/repos/huggingface/datasets/issues/6108
IC_kwDODunzps5i6Ybd
https://api.github.com/repos/huggingface/datasets/issues/comments/1659471581/reactions
2023-08-01T02:30:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1659471581
https://avatars.githubusercontent.com/u/48412571?v=4
https://api.github.com/users/LoveCatc/events{/privacy}
https://api.github.com/users/LoveCatc/followers
https://api.github.com/users/LoveCatc/following{/other_user}
https://api.github.com/users/LoveCatc/gists{/gist_id}
https://github.com/LoveCatc
LoveCatc
MDQ6VXNlcjQ4NDEyNTcx
https://api.github.com/users/LoveCatc/orgs
https://api.github.com/users/LoveCatc/received_events
https://api.github.com/users/LoveCatc/repos
https://api.github.com/users/LoveCatc/starred{/owner}{/repo}
https://api.github.com/users/LoveCatc/subscriptions
User
https://api.github.com/users/LoveCatc
public
NONE
I've had similar weird issues with `load_dataset` as well. Not multiple files, but dataset is quite big, about 50G.
2023-08-03T12:03:30Z
https://github.com/huggingface/datasets/issues/6108#issuecomment-1663859321
https://api.github.com/repos/huggingface/datasets/issues/6108
IC_kwDODunzps5jLHp5
https://api.github.com/repos/huggingface/datasets/issues/comments/1663859321/reactions
2023-08-03T12:03:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1663859321
https://avatars.githubusercontent.com/u/5112840?v=4
https://api.github.com/users/harpone/events{/privacy}
https://api.github.com/users/harpone/followers
https://api.github.com/users/harpone/following{/other_user}
https://api.github.com/users/harpone/gists{/gist_id}
https://github.com/harpone
harpone
MDQ6VXNlcjUxMTI4NDA=
https://api.github.com/users/harpone/orgs
https://api.github.com/users/harpone/received_events
https://api.github.com/users/harpone/repos
https://api.github.com/users/harpone/starred{/owner}{/repo}
https://api.github.com/users/harpone/subscriptions
User
https://api.github.com/users/harpone
public
COLLABORATOR
We use a generic multiprocessing code, so there is little we can do about this - unfortunately, turning off multiprocessing seems to be the only solution. Multithreading would make our code easier to maintain and (most likely) avoid issues such as this one, but we cannot use it until the GIL is dropped (no-GIL Python should be released in 2024, so we can start exploring this then)
2023-08-17T17:36:45Z
https://github.com/huggingface/datasets/issues/6108#issuecomment-1682699051
https://api.github.com/repos/huggingface/datasets/issues/6108
IC_kwDODunzps5kS_Mr
https://api.github.com/repos/huggingface/datasets/issues/comments/1682699051/reactions
2023-08-17T17:36:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1682699051
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
The problem seems to be the `Generating train split`. Is it possible to avoid that? I have a dataset saved, just want to load it but somehow running into issues with that again.
2023-09-22T09:13:03Z
https://github.com/huggingface/datasets/issues/6108#issuecomment-1731085834
https://api.github.com/repos/huggingface/datasets/issues/6108
IC_kwDODunzps5nLkYK
https://api.github.com/repos/huggingface/datasets/issues/comments/1731085834/reactions
2023-09-22T09:13:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1731085834
https://avatars.githubusercontent.com/u/5112840?v=4
https://api.github.com/users/harpone/events{/privacy}
https://api.github.com/users/harpone/followers
https://api.github.com/users/harpone/following{/other_user}
https://api.github.com/users/harpone/gists{/gist_id}
https://github.com/harpone
harpone
MDQ6VXNlcjUxMTI4NDA=
https://api.github.com/users/harpone/orgs
https://api.github.com/users/harpone/received_events
https://api.github.com/users/harpone/repos
https://api.github.com/users/harpone/starred{/owner}{/repo}
https://api.github.com/users/harpone/subscriptions
User
https://api.github.com/users/harpone
public
NONE
Hey guys, recently I ran into this problem again and I spent one whole day trying to locate the problem. Finally I found the problem seems to be with `pyarrow`'s json parser, and it seems a long-existing problem. Similar issue can be found in #2181. Anyway, my solution is to adjust the `load_dataset`'s parameter `chunksize`. You can inspect the parameter set in `datasets/packaged_modules/json/json.py`, now the actual chunksize should be very small, and you can increase the value. For me, `chunksize=10<<23` could solve the stuck problem. But I also find that too big `chunksize`, like `10 << 30`, would also cause a stuck, which is rather weird. I think I may explore this when I am free. And hope this can help those who also encounter the same problem. 
2023-09-25T06:29:50Z
https://github.com/huggingface/datasets/issues/6108#issuecomment-1733000920
https://api.github.com/repos/huggingface/datasets/issues/6108
IC_kwDODunzps5nS37Y
https://api.github.com/repos/huggingface/datasets/issues/comments/1733000920/reactions
2023-09-25T06:30:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1733000920
https://avatars.githubusercontent.com/u/48412571?v=4
https://api.github.com/users/LoveCatc/events{/privacy}
https://api.github.com/users/LoveCatc/followers
https://api.github.com/users/LoveCatc/following{/other_user}
https://api.github.com/users/LoveCatc/gists{/gist_id}
https://github.com/LoveCatc
LoveCatc
MDQ6VXNlcjQ4NDEyNTcx
https://api.github.com/users/LoveCatc/orgs
https://api.github.com/users/LoveCatc/received_events
https://api.github.com/users/LoveCatc/repos
https://api.github.com/users/LoveCatc/starred{/owner}{/repo}
https://api.github.com/users/LoveCatc/subscriptions
User
https://api.github.com/users/LoveCatc
public
NONE
Experiencing the same issue with the `kaist-ai/Feedback-Collection` dataset, which is comparatively small i.e. 100k rows.
Code to reproduce

```
from datasets import load_dataset
dataset = load_dataset("kaist-ai/Feedback-Collection")
```

I have tried setting `num_proc=1` as well as `chunksize=1024, 64` but problem persists. Any pointers?
2024-02-05T08:50:55Z
https://github.com/huggingface/datasets/issues/6108#issuecomment-1926486517
https://api.github.com/repos/huggingface/datasets/issues/6108
IC_kwDODunzps5y09n1
https://api.github.com/repos/huggingface/datasets/issues/comments/1926486517/reactions
2024-02-05T08:55:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1926486517
https://avatars.githubusercontent.com/u/43938520?v=4
https://api.github.com/users/RomanEngeler1805/events{/privacy}
https://api.github.com/users/RomanEngeler1805/followers
https://api.github.com/users/RomanEngeler1805/following{/other_user}
https://api.github.com/users/RomanEngeler1805/gists{/gist_id}
https://github.com/RomanEngeler1805
RomanEngeler1805
MDQ6VXNlcjQzOTM4NTIw
https://api.github.com/users/RomanEngeler1805/orgs
https://api.github.com/users/RomanEngeler1805/received_events
https://api.github.com/users/RomanEngeler1805/repos
https://api.github.com/users/RomanEngeler1805/starred{/owner}{/repo}
https://api.github.com/users/RomanEngeler1805/subscriptions
User
https://api.github.com/users/RomanEngeler1805
public
NONE
sorry to disturb, at datasets==2.21.0, I add `chunksize` parameter but got error "doesn't have a 'chunksize' key". Is it got removed?
2024-12-31T16:00:59Z
https://github.com/huggingface/datasets/issues/6108#issuecomment-2566557566
https://api.github.com/repos/huggingface/datasets/issues/6108
IC_kwDODunzps6Y-o9-
https://api.github.com/repos/huggingface/datasets/issues/comments/2566557566/reactions
2024-12-31T16:00:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2566557566
https://avatars.githubusercontent.com/u/115414113?v=4
https://api.github.com/users/zxrys/events{/privacy}
https://api.github.com/users/zxrys/followers
https://api.github.com/users/zxrys/following{/other_user}
https://api.github.com/users/zxrys/gists{/gist_id}
https://github.com/zxrys
zxrys
U_kgDOBuEUYQ
https://api.github.com/users/zxrys/orgs
https://api.github.com/users/zxrys/received_events
https://api.github.com/users/zxrys/repos
https://api.github.com/users/zxrys/starred{/owner}{/repo}
https://api.github.com/users/zxrys/subscriptions
User
https://api.github.com/users/zxrys
public
### Describe the bug

I try to use `load_dataset()` to load several local `.jsonl` files as a dataset. Every line of these files is a json structure only containing one key `text` (yeah it is a dataset for NLP model). The code snippet is as:
```python
ds = load_dataset("json", data_files=LIST_OF_FILE_PATHS, num_proc=16)['train']
```
However, I found that the loading process can get stuck -- the progress bar `Generating train split` no more proceed. When I was trying to find the cause and solution, I found a really strange behavior. If I load the dataset in this way:
```python
dlist = list()
for _ in LIST_OF_FILE_PATHS:
    dlist.append(load_dataset("json", data_files=_)['train'])
ds = concatenate_datasets(dlist)
```
I can actually successfully load all the files despite its slow speed. But if I load them in batch like above, things go wrong. I did try to use Control-C to trace the stuck point but the program cannot be terminated in this way when `num_proc` is set to `None`. The only thing I can do is use Control-Z to hang it up then kill it. If I use more than 2 cpus, a Control-C would simply cause the following error:
```bash
^C
Process ForkPoolWorker-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/pool.py", line 114, in worker
    task = get()
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/queues.py", line 368, in get
    res = self._reader.recv_bytes()
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/connection.py", line 224, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/connection.py", line 422, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/connection.py", line 387, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
Generating train split: 92431 examples [01:23, 1104.25 examples/s]  
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 1373, in iflatmap_unordered
    yield queue.get(timeout=0.05)
  File "<string>", line 2, in get
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/managers.py", line 818, in _callmethod
    kind, result = conn.recv()
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/connection.py", line 258, in recv
    buf = self._recv_bytes()
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/connection.py", line 422, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/connection.py", line 387, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data/liyongyuan/source/batch_load.py", line 11, in <module>
    a = load_dataset(
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2133, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 954, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1049, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1842, in _prepare_split
    for job_id, done, content in iflatmap_unordered(
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 1387, in iflatmap_unordered
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 1387, in <listcomp>
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/usr/local/lib/python3.10/dist-packages/multiprocess/pool.py", line 770, in get
    raise TimeoutError
multiprocess.context.TimeoutError
```
I have validated the basic correctness of these `.jsonl` files. They are correctly formatted (or they cannot be loaded singly by `load_dataset`) though some of the json may contain too long text (more than 1e7 characters). I do not know if this could be the problem. And there should not be any bottleneck in system's resource. The whole dataset is ~300GB, and I am using a cloud server with plenty of storage and 1TB ram. 
Thanks for your efforts and patience! Any suggestion or help would be appreciated.

### Steps to reproduce the bug

1. use load_dataset() with `data_files = LIST_OF_FILES`

### Expected behavior

All the files should be smoothly loaded. 

### Environment info

- Datasets: A private dataset. ~2500 `.jsonl` files. ~300GB in total. Each json structure only contains one key: `text`. Format checked.
- `datasets` version: 2.14.2
- Platform: Linux-4.19.91-014.kangaroo.alios7.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.15.1
- PyArrow version: 10.0.1.dev0+ga6eabc2b.d20230609
- Pandas version: 1.5.2