AssertionError: daemonic processes are not allowed to have children
COLLABORATOR
We could add a "threads" parallel backend to `datasets.parallel.parallel_backend` to support downloading with threads but note that `download_and_extract` also decompresses archives, and this is a CPU-intensive task, which is not ideal for (Python) threads (good for IO-intensive tasks).
2023-07-28T11:36:25Z
https://github.com/huggingface/datasets/issues/6089#issuecomment-1655535373
https://api.github.com/repos/huggingface/datasets/issues/6089
IC_kwDODunzps5irXcN
https://api.github.com/repos/huggingface/datasets/issues/comments/1655535373/reactions
2023-07-28T11:36:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1655535373
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
> We could add a "threads" parallel backend to `datasets.parallel.parallel_backend` to support downloading with threads but note that `download_and_extract` also decompresses archives, and this is a CPU-intensive task, which is not ideal for (Python) threads (good for IO-intensive tasks).

Great! Download takes more time than extract, multiple threads can download in parallel, which can speed up a lot.
2023-07-31T02:33:23Z
https://github.com/huggingface/datasets/issues/6089#issuecomment-1657406972
https://api.github.com/repos/huggingface/datasets/issues/6089
IC_kwDODunzps5iygX8
https://api.github.com/repos/huggingface/datasets/issues/comments/1657406972/reactions
2023-07-31T02:34:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1657406972
https://avatars.githubusercontent.com/u/138426806?v=4
https://api.github.com/users/codingl2k1/events{/privacy}
https://api.github.com/users/codingl2k1/followers
https://api.github.com/users/codingl2k1/following{/other_user}
https://api.github.com/users/codingl2k1/gists{/gist_id}
https://github.com/codingl2k1
codingl2k1
U_kgDOCEA5tg
https://api.github.com/users/codingl2k1/orgs
https://api.github.com/users/codingl2k1/received_events
https://api.github.com/users/codingl2k1/repos
https://api.github.com/users/codingl2k1/starred{/owner}{/repo}
https://api.github.com/users/codingl2k1/subscriptions
User
https://api.github.com/users/codingl2k1
public
### Describe the bug

When I load_dataset with num_proc > 0 in a deamon process, I got an error:

```python
  File "/Users/codingl2k1/Work/datasets/src/datasets/download/download_manager.py", line 564, in download_and_extract
    return self.extract(self.download(url_or_urls))
    ^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/download/download_manager.py", line 427, in download
    downloaded_path_or_paths = map_nested(
    ^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/utils/py_utils.py", line 468, in map_nested
    mapped = parallel_map(function, iterable, num_proc, types, disable_tqdm, desc, _single_map_nested)
    ^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/utils/experimental.py", line 40, in _inner_fn
    return fn(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/parallel/parallel.py", line 34, in parallel_map
    return _map_with_multiprocessing_pool(
    ^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/Work/datasets/src/datasets/parallel/parallel.py", line 64, in _map_with_multiprocessing_pool
    with Pool(num_proc, initargs=initargs, initializer=initializer) as pool:
      ^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
    ^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/multiprocessing/pool.py", line 215, in __init__
    self._repopulate_pool()
    ^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/multiprocessing/pool.py", line 306, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
    ^^^^^^^^^^^^^^^^^
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/multiprocessing/pool.py", line 329, in _repopulate_pool_static
    w.start()
  File "/Users/codingl2k1/.pyenv/versions/3.11.4/lib/python3.11/multiprocessing/process.py", line 118, in start
    assert not _current_process._config.get('daemon'),     ^^^^^^^^^^^^^^^^^
AssertionError: daemonic processes are not allowed to have children
```

The download is io-intensive computing, may be datasets can replece the multi processing pool by a multi threading pool if in a deamon process.

### Steps to reproduce the bug

1. start a deamon process
2. run load_dataset with num_proc > 0

### Expected behavior

No error.

### Environment info

Python 3.11.4
datasets latest master