`SystemError 15` thrown in `Dataset.__del__` when using `Dataset.map()` with `num_proc>1`
NONE
NB: even if the error is raised, the dataset is successfully cached. So restarting the script after every `map()` allows to ultimately run the whole preprocessing. But this prevents to realistically run the code over multiple nodes.
2021-10-28T11:11:36Z
https://github.com/huggingface/datasets/issues/3172#issuecomment-953744696
https://api.github.com/repos/huggingface/datasets/issues/3172
IC_kwDODunzps442P04
https://api.github.com/repos/huggingface/datasets/issues/comments/953744696/reactions
2021-10-28T11:11:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/953744696
https://avatars.githubusercontent.com/u/9859840?v=4
https://api.github.com/users/vlievin/events{/privacy}
https://api.github.com/users/vlievin/followers
https://api.github.com/users/vlievin/following{/other_user}
https://api.github.com/users/vlievin/gists{/gist_id}
https://github.com/vlievin
vlievin
MDQ6VXNlcjk4NTk4NDA=
https://api.github.com/users/vlievin/orgs
https://api.github.com/users/vlievin/received_events
https://api.github.com/users/vlievin/repos
https://api.github.com/users/vlievin/starred{/owner}{/repo}
https://api.github.com/users/vlievin/subscriptions
User
https://api.github.com/users/vlievin
public
COLLABORATOR
Hi,

It's not easy to debug the problem without the script. I may be wrong since I'm not very familiar with PyTorch Lightning, but shouldn't you preprocess the data in the `prepare_data` function of `LightningDataModule` and not in the `setup` function.
As you can't modify the module state in `prepare_data` (according to the docs), use the `cache_file_name` argument in `Dataset.map` there, and reload the processed data in `setup` with `Dataset.from_file(cache_file_name)`. If `num_proc>1`, check the docs on the `suffix_template` argument of `Dataset.map` to get an idea what the final `cache_file_names` are going to be.

Let me know if this helps.
2021-10-28T13:57:19Z
https://github.com/huggingface/datasets/issues/3172#issuecomment-953871095
https://api.github.com/repos/huggingface/datasets/issues/3172
IC_kwDODunzps442ur3
https://api.github.com/repos/huggingface/datasets/issues/comments/953871095/reactions
2021-10-28T15:21:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/953871095
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Hi @mariosasko, thank you for the hint, that helped me to move forward with that issue. 

I did a major refactoring of my project to disentangle my `LightningDataModule` and `Dataset`.  Just FYI, it looks like:

```python
class Builder():
   def __call__() -> DatasetDict:
      # load and preprocess the data
      return dataset

class DataModule(LightningDataModule):
    def prepare_data():
       self.builder()
    def setup():
       self.dataset = self.builder()
```

Unfortunately, the entanglement between `LightningDataModule` and `Dataset` was not the issue.

The culprit was `hydra` and a slight adjustment of the structure of my project solved this issue. The problematic project  structure was:

```
src/
 | - cli.py
 | - training/
      | -experiment.py

# code in experiment.py
def run_experiment(config):
     # preprocess data and run
     
# code in cli.py
@hydra.main(...)
def run(config):
    return run_experiment(config)
```

Moving `run()` from `clip.py` to `training.experiment.py` solved the issue with `SystemError 15`. No idea why. 

Even if the traceback was referring to `Dataset.__del__`, the problem does not seem to be primarily related to `datasets`, so I will close this issue. Thank you for your help!
2021-11-03T11:26:10Z
https://github.com/huggingface/datasets/issues/3172#issuecomment-958941454
https://api.github.com/repos/huggingface/datasets/issues/3172
IC_kwDODunzps45KEkO
https://api.github.com/repos/huggingface/datasets/issues/comments/958941454/reactions
2021-11-03T11:27:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/958941454
https://avatars.githubusercontent.com/u/9859840?v=4
https://api.github.com/users/vlievin/events{/privacy}
https://api.github.com/users/vlievin/followers
https://api.github.com/users/vlievin/following{/other_user}
https://api.github.com/users/vlievin/gists{/gist_id}
https://github.com/vlievin
vlievin
MDQ6VXNlcjk4NTk4NDA=
https://api.github.com/users/vlievin/orgs
https://api.github.com/users/vlievin/received_events
https://api.github.com/users/vlievin/repos
https://api.github.com/users/vlievin/starred{/owner}{/repo}
https://api.github.com/users/vlievin/subscriptions
User
https://api.github.com/users/vlievin
public
NONE
Please allow me to revive this discussion, as I have an extremely similar issue. Instead of an error, my datasets functions simply aren't caching properly. My setup is almost the same as yours, with hydra to configure my experiment parameters.

@vlievin Could you confirm if your code correctly loads the cache? If so, do you have any public code that I can reference for comparison?

I will post a full example with hydra that illustrates this problem in a little bit, probably on another thread.
2022-01-15T12:53:39Z
https://github.com/huggingface/datasets/issues/3172#issuecomment-1013677400
https://api.github.com/repos/huggingface/datasets/issues/3172
IC_kwDODunzps48a31Y
https://api.github.com/repos/huggingface/datasets/issues/comments/1013677400/reactions
2022-01-15T12:53:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1013677400
https://avatars.githubusercontent.com/u/22415175?v=4
https://api.github.com/users/mariomeissner/events{/privacy}
https://api.github.com/users/mariomeissner/followers
https://api.github.com/users/mariomeissner/following{/other_user}
https://api.github.com/users/mariomeissner/gists{/gist_id}
https://github.com/mariomeissner
mariomeissner
MDQ6VXNlcjIyNDE1MTc1
https://api.github.com/users/mariomeissner/orgs
https://api.github.com/users/mariomeissner/received_events
https://api.github.com/users/mariomeissner/repos
https://api.github.com/users/mariomeissner/starred{/owner}{/repo}
https://api.github.com/users/mariomeissner/subscriptions
User
https://api.github.com/users/mariomeissner
public
NONE
Hello @mariomeissner, very sorry for the late reply, I hope you have found a solution to your problem!

I don't have public code at the moment. I have not experienced any other issue with hydra, even if I don't understand why changing the location of the definition of `run()` fixed the problem. 

Overall, I don't have issue with caching anymore, even when 
1. using custom fingerprints using the argument `new_fingerprint 
2. when using `num_proc>1`
2022-02-10T16:00:34Z
https://github.com/huggingface/datasets/issues/3172#issuecomment-1035093648
https://api.github.com/repos/huggingface/datasets/issues/3172
IC_kwDODunzps49skaQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1035093648/reactions
2022-02-10T16:00:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1035093648
https://avatars.githubusercontent.com/u/9859840?v=4
https://api.github.com/users/vlievin/events{/privacy}
https://api.github.com/users/vlievin/followers
https://api.github.com/users/vlievin/following{/other_user}
https://api.github.com/users/vlievin/gists{/gist_id}
https://github.com/vlievin
vlievin
MDQ6VXNlcjk4NTk4NDA=
https://api.github.com/users/vlievin/orgs
https://api.github.com/users/vlievin/received_events
https://api.github.com/users/vlievin/repos
https://api.github.com/users/vlievin/starred{/owner}{/repo}
https://api.github.com/users/vlievin/subscriptions
User
https://api.github.com/users/vlievin
public
NONE
I solved my issue by turning the map callable into a class static method, like they do in `lightning-transformers`. Very strange...
2022-02-11T01:31:05Z
https://github.com/huggingface/datasets/issues/3172#issuecomment-1035703868
https://api.github.com/repos/huggingface/datasets/issues/3172
IC_kwDODunzps49u5Y8
https://api.github.com/repos/huggingface/datasets/issues/comments/1035703868/reactions
2022-02-11T01:31:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1035703868
https://avatars.githubusercontent.com/u/22415175?v=4
https://api.github.com/users/mariomeissner/events{/privacy}
https://api.github.com/users/mariomeissner/followers
https://api.github.com/users/mariomeissner/following{/other_user}
https://api.github.com/users/mariomeissner/gists{/gist_id}
https://github.com/mariomeissner
mariomeissner
MDQ6VXNlcjIyNDE1MTc1
https://api.github.com/users/mariomeissner/orgs
https://api.github.com/users/mariomeissner/received_events
https://api.github.com/users/mariomeissner/repos
https://api.github.com/users/mariomeissner/starred{/owner}{/repo}
https://api.github.com/users/mariomeissner/subscriptions
User
https://api.github.com/users/mariomeissner
public
NONE
I have this issue with datasets v2.5.2 with Python 3.8.10 on Ubuntu 20.04.4 LTS.  It does not occur when num_proc=1.  When num_proc>1, it intermittently occurs and will cause process to hang.  As previously mentioned, it occurs even when datasets have been previously cached.  I have tried wrapping logic in a static class as suggested with @mariomeissner with no improvement.
2022-11-10T19:40:09Z
https://github.com/huggingface/datasets/issues/3172#issuecomment-1310807905
https://api.github.com/repos/huggingface/datasets/issues/3172
IC_kwDODunzps5OIVdh
https://api.github.com/repos/huggingface/datasets/issues/comments/1310807905/reactions
2022-11-10T19:40:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1310807905
https://avatars.githubusercontent.com/u/1519103?v=4
https://api.github.com/users/philipchung/events{/privacy}
https://api.github.com/users/philipchung/followers
https://api.github.com/users/philipchung/following{/other_user}
https://api.github.com/users/philipchung/gists{/gist_id}
https://github.com/philipchung
philipchung
MDQ6VXNlcjE1MTkxMDM=
https://api.github.com/users/philipchung/orgs
https://api.github.com/users/philipchung/received_events
https://api.github.com/users/philipchung/repos
https://api.github.com/users/philipchung/starred{/owner}{/repo}
https://api.github.com/users/philipchung/subscriptions
User
https://api.github.com/users/philipchung
public
NONE
@philipchung hello ,i have the same issue like yours,did you solve it?
2023-01-26T02:23:33Z
https://github.com/huggingface/datasets/issues/3172#issuecomment-1404488121
https://api.github.com/repos/huggingface/datasets/issues/3172
IC_kwDODunzps5Ttsm5
https://api.github.com/repos/huggingface/datasets/issues/comments/1404488121/reactions
2023-01-26T02:23:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1404488121
https://avatars.githubusercontent.com/u/84292488?v=4
https://api.github.com/users/c976237222/events{/privacy}
https://api.github.com/users/c976237222/followers
https://api.github.com/users/c976237222/following{/other_user}
https://api.github.com/users/c976237222/gists{/gist_id}
https://github.com/c976237222
c976237222
MDQ6VXNlcjg0MjkyNDg4
https://api.github.com/users/c976237222/orgs
https://api.github.com/users/c976237222/received_events
https://api.github.com/users/c976237222/repos
https://api.github.com/users/c976237222/starred{/owner}{/repo}
https://api.github.com/users/c976237222/subscriptions
User
https://api.github.com/users/c976237222
public
NONE
No.  I was not able to get num_proc>1 to work.
2023-01-26T07:07:54Z
https://github.com/huggingface/datasets/issues/3172#issuecomment-1404627145
https://api.github.com/repos/huggingface/datasets/issues/3172
IC_kwDODunzps5TuOjJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1404627145/reactions
2023-01-26T07:07:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1404627145
https://avatars.githubusercontent.com/u/1519103?v=4
https://api.github.com/users/philipchung/events{/privacy}
https://api.github.com/users/philipchung/followers
https://api.github.com/users/philipchung/following{/other_user}
https://api.github.com/users/philipchung/gists{/gist_id}
https://github.com/philipchung
philipchung
MDQ6VXNlcjE1MTkxMDM=
https://api.github.com/users/philipchung/orgs
https://api.github.com/users/philipchung/received_events
https://api.github.com/users/philipchung/repos
https://api.github.com/users/philipchung/starred{/owner}{/repo}
https://api.github.com/users/philipchung/subscriptions
User
https://api.github.com/users/philipchung
public
NONE
same problem here. It randomly occurs...
2023-09-02T23:10:42Z
https://github.com/huggingface/datasets/issues/3172#issuecomment-1703960855
https://api.github.com/repos/huggingface/datasets/issues/3172
IC_kwDODunzps5lkGEX
https://api.github.com/repos/huggingface/datasets/issues/comments/1703960855/reactions
2023-09-02T23:10:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1703960855
https://avatars.githubusercontent.com/u/34365327?v=4
https://api.github.com/users/yoonniverse/events{/privacy}
https://api.github.com/users/yoonniverse/followers
https://api.github.com/users/yoonniverse/following{/other_user}
https://api.github.com/users/yoonniverse/gists{/gist_id}
https://github.com/yoonniverse
yoonniverse
MDQ6VXNlcjM0MzY1MzI3
https://api.github.com/users/yoonniverse/orgs
https://api.github.com/users/yoonniverse/received_events
https://api.github.com/users/yoonniverse/repos
https://api.github.com/users/yoonniverse/starred{/owner}{/repo}
https://api.github.com/users/yoonniverse/subscriptions
User
https://api.github.com/users/yoonniverse
public
COLLABORATOR
Can someone provide a reproducer to help us debug this (e.g., a `hydra` repo with dummy model and data)?
2023-09-04T14:20:43Z
https://github.com/huggingface/datasets/issues/3172#issuecomment-1705353317
https://api.github.com/repos/huggingface/datasets/issues/3172
IC_kwDODunzps5lpaBl
https://api.github.com/repos/huggingface/datasets/issues/comments/1705353317/reactions
2023-09-04T14:20:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1705353317
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Hi, similarly here, is there any update for this issue? Particularly having the exact same message as in #6393, not sure this is a datasets or llm-foundry issue
2024-04-02T18:13:20Z
https://github.com/huggingface/datasets/issues/3172#issuecomment-2032730675
https://api.github.com/repos/huggingface/datasets/issues/3172
IC_kwDODunzps55KQIz
https://api.github.com/repos/huggingface/datasets/issues/comments/2032730675/reactions
2024-04-02T18:13:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2032730675
https://avatars.githubusercontent.com/u/11674033?v=4
https://api.github.com/users/subercui/events{/privacy}
https://api.github.com/users/subercui/followers
https://api.github.com/users/subercui/following{/other_user}
https://api.github.com/users/subercui/gists{/gist_id}
https://github.com/subercui
subercui
MDQ6VXNlcjExNjc0MDMz
https://api.github.com/users/subercui/orgs
https://api.github.com/users/subercui/received_events
https://api.github.com/users/subercui/repos
https://api.github.com/users/subercui/starred{/owner}{/repo}
https://api.github.com/users/subercui/subscriptions
User
https://api.github.com/users/subercui
public
## Describe the bug
I use `datasets.map` to preprocess some data in my application. The error `SystemError 15` is thrown at the end of the execution of `Dataset.map()` (only with `num_proc>1`. Traceback included bellow. 
 
The exception is raised only when the code runs within a specific context. Despite ~10h spent investigating this issue, I have failed to isolate the bug, so let me describe my setup. 

In my project, `Dataset` is wrapped into a `LightningDataModule` and the data is preprocessed when calling `LightningDataModule.setup()`. Calling `.setup()` in an isolated script works fine (even when wrapped with `hydra.main()`). However, when calling `.setup()` within the experiment script (depends on `pytorch_lightning`), the script crashes and `SystemError 15`.

I could avoid throwing this error by modifying ` Dataset.__del__()` (see bellow), but I believe this only moves the problem somewhere else. I am completely stuck with this issue, any hint would be welcome. 

```python
class Dataset()
    ...
    def __del__(self):
        if hasattr(self, "_data"):
            _ = self._data # <- ugly trick that allows avoiding the issue.
            del self._data
        if hasattr(self, "_indices"):
            del self._indices
```

## Steps to reproduce the bug
```python
# Unfortunately I couldn't isolate the bug.
```

## Expected results
Calling `Dataset.map()` without throwing an exception. Or at least raising a more detailed exception/traceback.

## Actual results
```
Exception ignored in: <function Dataset.__del__ at 0x7f7cec179160>███████████████████████████████████████████████████| 5/5 [00:05<00:00,  1.17ba/s]
Traceback (most recent call last):
  File ".../python3.8/site-packages/datasets/arrow_dataset.py", line 906, in __del__
    del self._data
  File ".../python3.8/site-packages/ray/worker.py", line 1033, in sigterm_handler
    sys.exit(signum)
SystemExit: 15

```

## Environment info

Tested on 2 environments:

**Environment 1.**
- `datasets` version: 1.14.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 6.0.0

**Environment 2.**
- `datasets` version: 1.14.0
- Platform: Linux-4.18.0-305.19.1.el8_4.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.7
- PyArrow version: 6.0.0
