Filter occasionally hangs
CONTRIBUTOR
It looks like I may not be the first to encounter this: https://github.com/huggingface/datasets/issues/3172
2023-11-09T06:19:28Z
https://github.com/huggingface/datasets/issues/6393#issuecomment-1803228528
https://api.github.com/repos/huggingface/datasets/issues/6393
IC_kwDODunzps5rexVw
https://api.github.com/repos/huggingface/datasets/issues/comments/1803228528/reactions
2023-11-09T06:19:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1803228528
https://avatars.githubusercontent.com/u/43149077?v=4
https://api.github.com/users/dakinggg/events{/privacy}
https://api.github.com/users/dakinggg/followers
https://api.github.com/users/dakinggg/following{/other_user}
https://api.github.com/users/dakinggg/gists{/gist_id}
https://github.com/dakinggg
dakinggg
MDQ6VXNlcjQzMTQ5MDc3
https://api.github.com/users/dakinggg/orgs
https://api.github.com/users/dakinggg/received_events
https://api.github.com/users/dakinggg/repos
https://api.github.com/users/dakinggg/starred{/owner}{/repo}
https://api.github.com/users/dakinggg/subscriptions
User
https://api.github.com/users/dakinggg
public
CONTRIBUTOR
Adding some more information, it seems to occur more frequently with large (millions of samples) datasets.
2023-11-09T21:19:38Z
https://github.com/huggingface/datasets/issues/6393#issuecomment-1804701205
https://api.github.com/repos/huggingface/datasets/issues/6393
IC_kwDODunzps5rkY4V
https://api.github.com/repos/huggingface/datasets/issues/comments/1804701205/reactions
2023-11-09T21:19:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1804701205
https://avatars.githubusercontent.com/u/43149077?v=4
https://api.github.com/users/dakinggg/events{/privacy}
https://api.github.com/users/dakinggg/followers
https://api.github.com/users/dakinggg/following{/other_user}
https://api.github.com/users/dakinggg/gists{/gist_id}
https://github.com/dakinggg
dakinggg
MDQ6VXNlcjQzMTQ5MDc3
https://api.github.com/users/dakinggg/orgs
https://api.github.com/users/dakinggg/received_events
https://api.github.com/users/dakinggg/repos
https://api.github.com/users/dakinggg/starred{/owner}{/repo}
https://api.github.com/users/dakinggg/subscriptions
User
https://api.github.com/users/dakinggg
public
CONTRIBUTOR
More information. My code is structured as (1) load (2) map (3) filter (4) filter. It was always the second filter that failed. Combining the two filters into one seems to reliably work.
2023-11-09T23:36:28Z
https://github.com/huggingface/datasets/issues/6393#issuecomment-1804845652
https://api.github.com/repos/huggingface/datasets/issues/6393
IC_kwDODunzps5rk8JU
https://api.github.com/repos/huggingface/datasets/issues/comments/1804845652/reactions
2023-11-09T23:36:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1804845652
https://avatars.githubusercontent.com/u/43149077?v=4
https://api.github.com/users/dakinggg/events{/privacy}
https://api.github.com/users/dakinggg/followers
https://api.github.com/users/dakinggg/following{/other_user}
https://api.github.com/users/dakinggg/gists{/gist_id}
https://github.com/dakinggg
dakinggg
MDQ6VXNlcjQzMTQ5MDc3
https://api.github.com/users/dakinggg/orgs
https://api.github.com/users/dakinggg/received_events
https://api.github.com/users/dakinggg/repos
https://api.github.com/users/dakinggg/starred{/owner}{/repo}
https://api.github.com/users/dakinggg/subscriptions
User
https://api.github.com/users/dakinggg
public
CONTRIBUTOR
@lhoestq it'd be great if someone had a chance to look at this. I suspect it is impacting many users given the other issue that I linked.
2023-11-17T19:28:43Z
https://github.com/huggingface/datasets/issues/6393#issuecomment-1816970989
https://api.github.com/repos/huggingface/datasets/issues/6393
IC_kwDODunzps5sTMbt
https://api.github.com/repos/huggingface/datasets/issues/comments/1816970989/reactions
2023-11-17T19:28:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1816970989
https://avatars.githubusercontent.com/u/43149077?v=4
https://api.github.com/users/dakinggg/events{/privacy}
https://api.github.com/users/dakinggg/followers
https://api.github.com/users/dakinggg/following{/other_user}
https://api.github.com/users/dakinggg/gists{/gist_id}
https://github.com/dakinggg
dakinggg
MDQ6VXNlcjQzMTQ5MDc3
https://api.github.com/users/dakinggg/orgs
https://api.github.com/users/dakinggg/received_events
https://api.github.com/users/dakinggg/repos
https://api.github.com/users/dakinggg/starred{/owner}{/repo}
https://api.github.com/users/dakinggg/subscriptions
User
https://api.github.com/users/dakinggg
public
MEMBER
Hi ! Sorry for the late response. Was it happening after the first or the second filter ?

It looks like an issue with the garbage collector (which makes it random). Maybe datasets created with `filter` are not always handled properly ? cc @mariosasko
2023-11-20T18:25:09Z
https://github.com/huggingface/datasets/issues/6393#issuecomment-1819589491
https://api.github.com/repos/huggingface/datasets/issues/6393
IC_kwDODunzps5sdLtz
https://api.github.com/repos/huggingface/datasets/issues/comments/1819589491/reactions
2023-11-20T18:25:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1819589491
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
It was after the second filter (and combining the two filters into one seemingly resolved it). I obviously haven't tried all settings to know that these details are causal, but it did work for me.
2023-11-20T18:43:46Z
https://github.com/huggingface/datasets/issues/6393#issuecomment-1819614987
https://api.github.com/repos/huggingface/datasets/issues/6393
IC_kwDODunzps5sdR8L
https://api.github.com/repos/huggingface/datasets/issues/comments/1819614987/reactions
2023-11-20T18:43:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1819614987
https://avatars.githubusercontent.com/u/43149077?v=4
https://api.github.com/users/dakinggg/events{/privacy}
https://api.github.com/users/dakinggg/followers
https://api.github.com/users/dakinggg/following{/other_user}
https://api.github.com/users/dakinggg/gists{/gist_id}
https://github.com/dakinggg
dakinggg
MDQ6VXNlcjQzMTQ5MDc3
https://api.github.com/users/dakinggg/orgs
https://api.github.com/users/dakinggg/received_events
https://api.github.com/users/dakinggg/repos
https://api.github.com/users/dakinggg/starred{/owner}{/repo}
https://api.github.com/users/dakinggg/subscriptions
User
https://api.github.com/users/dakinggg
public
MEMBER
Thanks, that's good to know.

The stacktrace suggests an issue when `del self._indices` is called, which happens when a filtered dataset falls out of scope. The indices are a PyArrow table memory mapped from disk, so I'm not quite sure how calling `del` on it can cause this issue. We do `del self._indices` to make sure the file on disk is not used anymore by the current process and avoid e.g. permission errors.

Hopefully we can find a way to reproduce this error, otherwise it will be quite hard to understand what happened
2023-11-21T11:34:33Z
https://github.com/huggingface/datasets/issues/6393#issuecomment-1820747200
https://api.github.com/repos/huggingface/datasets/issues/6393
IC_kwDODunzps5shmXA
https://api.github.com/repos/huggingface/datasets/issues/comments/1820747200/reactions
2023-11-21T11:34:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1820747200
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Yeah, I have a reliable repro, but it is not even close to minimal and uses a dataset I can't share. Perhaps you could try getting close to my setting.

(1) make a large (~20GB) jsonl with prompt/response pairs
(2) load it on a linux machine (`dataset = load_dataset(...)`)
(3) map a tokenizer to it, with multiprocessing (`tokenized_dataset = dataset.map(...)`)
(4) filter it once based on something, with multiprocessing (`filtered_1 = tokenized_dataset.filter(...)`)
(5) filter it again based on something, with multiprocessing (`filtered_2 = filtered_1.filter(...)`)

I included the variable names just in case it is relevant that I was creating new datasets each time, not overwriting the same variable.
2023-11-21T17:39:17Z
https://github.com/huggingface/datasets/issues/6393#issuecomment-1821372107
https://api.github.com/repos/huggingface/datasets/issues/6393
IC_kwDODunzps5sj-7L
https://api.github.com/repos/huggingface/datasets/issues/comments/1821372107/reactions
2023-11-21T17:39:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1821372107
https://avatars.githubusercontent.com/u/43149077?v=4
https://api.github.com/users/dakinggg/events{/privacy}
https://api.github.com/users/dakinggg/followers
https://api.github.com/users/dakinggg/following{/other_user}
https://api.github.com/users/dakinggg/gists{/gist_id}
https://github.com/dakinggg
dakinggg
MDQ6VXNlcjQzMTQ5MDc3
https://api.github.com/users/dakinggg/orgs
https://api.github.com/users/dakinggg/received_events
https://api.github.com/users/dakinggg/repos
https://api.github.com/users/dakinggg/starred{/owner}{/repo}
https://api.github.com/users/dakinggg/subscriptions
User
https://api.github.com/users/dakinggg
public
CONTRIBUTOR
@lhoestq I have another version of the repro that seems fairly reliably. I have lots of jsonl files, and I iteratively load each one with `load_dataset('json', data_files='path/to/my/file.jsonl', streaming=False, split='train')` and then `dataset.map(..., num_proc=<int>)`. This iteration hangs in a random place each time. So seems like there is a bug that hits with _some_ frequency.
2024-03-05T06:08:44Z
https://github.com/huggingface/datasets/issues/6393#issuecomment-1978031361
https://api.github.com/repos/huggingface/datasets/issues/6393
IC_kwDODunzps515l0B
https://api.github.com/repos/huggingface/datasets/issues/comments/1978031361/reactions
2024-03-05T06:08:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1978031361
https://avatars.githubusercontent.com/u/43149077?v=4
https://api.github.com/users/dakinggg/events{/privacy}
https://api.github.com/users/dakinggg/followers
https://api.github.com/users/dakinggg/following{/other_user}
https://api.github.com/users/dakinggg/gists{/gist_id}
https://github.com/dakinggg
dakinggg
MDQ6VXNlcjQzMTQ5MDc3
https://api.github.com/users/dakinggg/orgs
https://api.github.com/users/dakinggg/received_events
https://api.github.com/users/dakinggg/repos
https://api.github.com/users/dakinggg/starred{/owner}{/repo}
https://api.github.com/users/dakinggg/subscriptions
User
https://api.github.com/users/dakinggg
public
CONTRIBUTOR
With `num_proc=None` it works fine.
2024-03-05T06:09:02Z
https://github.com/huggingface/datasets/issues/6393#issuecomment-1978031632
https://api.github.com/repos/huggingface/datasets/issues/6393
IC_kwDODunzps515l4Q
https://api.github.com/repos/huggingface/datasets/issues/comments/1978031632/reactions
2024-03-05T06:09:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1978031632
https://avatars.githubusercontent.com/u/43149077?v=4
https://api.github.com/users/dakinggg/events{/privacy}
https://api.github.com/users/dakinggg/followers
https://api.github.com/users/dakinggg/following{/other_user}
https://api.github.com/users/dakinggg/gists{/gist_id}
https://github.com/dakinggg
dakinggg
MDQ6VXNlcjQzMTQ5MDc3
https://api.github.com/users/dakinggg/orgs
https://api.github.com/users/dakinggg/received_events
https://api.github.com/users/dakinggg/repos
https://api.github.com/users/dakinggg/starred{/owner}{/repo}
https://api.github.com/users/dakinggg/subscriptions
User
https://api.github.com/users/dakinggg
public
NONE
I am also having similar issue to #3172 when trying to tokenize the data. My dataset contains 10M samples. Is there anything that could be done without having to split up the processing into multiple datasets?
2024-03-05T16:03:10Z
https://github.com/huggingface/datasets/issues/6393#issuecomment-1979110815
https://api.github.com/repos/huggingface/datasets/issues/6393
IC_kwDODunzps519tWf
https://api.github.com/repos/huggingface/datasets/issues/comments/1979110815/reactions
2024-03-05T16:03:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1979110815
https://avatars.githubusercontent.com/u/30440108?v=4
https://api.github.com/users/agokrani/events{/privacy}
https://api.github.com/users/agokrani/followers
https://api.github.com/users/agokrani/following{/other_user}
https://api.github.com/users/agokrani/gists{/gist_id}
https://github.com/agokrani
agokrani
MDQ6VXNlcjMwNDQwMTA4
https://api.github.com/users/agokrani/orgs
https://api.github.com/users/agokrani/received_events
https://api.github.com/users/agokrani/repos
https://api.github.com/users/agokrani/starred{/owner}{/repo}
https://api.github.com/users/agokrani/subscriptions
User
https://api.github.com/users/agokrani
public
CONTRIBUTOR
https://github.com/huggingface/datasets/pull/7411 seems to have fixed the issue for me, curious if it resolves others issues too.
2025-02-19T18:26:45Z
https://github.com/huggingface/datasets/issues/6393#issuecomment-2669442830
https://api.github.com/repos/huggingface/datasets/issues/6393
IC_kwDODunzps6fHHcO
https://api.github.com/repos/huggingface/datasets/issues/comments/2669442830/reactions
2025-02-19T18:26:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2669442830
https://avatars.githubusercontent.com/u/43149077?v=4
https://api.github.com/users/dakinggg/events{/privacy}
https://api.github.com/users/dakinggg/followers
https://api.github.com/users/dakinggg/following{/other_user}
https://api.github.com/users/dakinggg/gists{/gist_id}
https://github.com/dakinggg
dakinggg
MDQ6VXNlcjQzMTQ5MDc3
https://api.github.com/users/dakinggg/orgs
https://api.github.com/users/dakinggg/received_events
https://api.github.com/users/dakinggg/repos
https://api.github.com/users/dakinggg/starred{/owner}{/repo}
https://api.github.com/users/dakinggg/subscriptions
User
https://api.github.com/users/dakinggg
public
### Describe the bug

A call to `.filter` occasionally hangs (after the filter is complete, according to tqdm)

There is a trace produced
```
Exception ignored in: <function Dataset.__del__ at 0x7efb48130c10>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/datasets/arrow_dataset.py", line 1366, in __del__
    if hasattr(self, "_indices"):
  File "/usr/lib/python3/dist-packages/composer/core/engine.py", line 123, in sigterm_handler
    sys.exit(128 + signal)
SystemExit: 143
```

but I'm not sure if the trace is actually from `datasets`, or from surrounding code that is trying to clean up after datasets gets stuck.

Unfortunately I can't reproduce this issue anywhere close to reliably. It happens infrequently when using `num_procs > 1`. Anecdotally I started seeing it when using larger datasets (~10M samples).

### Steps to reproduce the bug

N/A see description

### Expected behavior

map/filter calls always complete sucessfully

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-5.4.0-137-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.17.3
- PyArrow version: 13.0.0
- Pandas version: 2.1.2