fix bug #6877
NONE
@loicmagne, @KennethEnevoldsen
2024-05-10T02:11:19Z
https://github.com/huggingface/datasets/pull/6889#issuecomment-2103722500
https://api.github.com/repos/huggingface/datasets/issues/6889
IC_kwDODunzps59ZEIE
https://api.github.com/repos/huggingface/datasets/issues/comments/2103722500/reactions
2024-05-10T04:08:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2103722500
https://avatars.githubusercontent.com/u/16257131?v=4
https://api.github.com/users/arthasking123/events{/privacy}
https://api.github.com/users/arthasking123/followers
https://api.github.com/users/arthasking123/following{/other_user}
https://api.github.com/users/arthasking123/gists{/gist_id}
https://github.com/arthasking123
arthasking123
MDQ6VXNlcjE2MjU3MTMx
https://api.github.com/users/arthasking123/orgs
https://api.github.com/users/arthasking123/received_events
https://api.github.com/users/arthasking123/repos
https://api.github.com/users/arthasking123/starred{/owner}{/repo}
https://api.github.com/users/arthasking123/subscriptions
User
https://api.github.com/users/arthasking123
public
MEMBER
Can you give more details on why this fix works ?
2024-05-10T13:52:28Z
https://github.com/huggingface/datasets/pull/6889#issuecomment-2104650817
https://api.github.com/repos/huggingface/datasets/issues/6889
IC_kwDODunzps59cmxB
https://api.github.com/repos/huggingface/datasets/issues/comments/2104650817/reactions
2024-05-10T13:52:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2104650817
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6889). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2024-05-10T13:54:21Z
https://github.com/huggingface/datasets/pull/6889#issuecomment-2104653897
https://api.github.com/repos/huggingface/datasets/issues/6889
IC_kwDODunzps59cnhJ
https://api.github.com/repos/huggingface/datasets/issues/comments/2104653897/reactions
2024-05-10T13:54:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2104653897
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
> Can you give more details on why this fix works ?

In order to locate this file handle problem, I defined a print_open_files_count() function using psutil library:
```python
def print_open_files_count(markstr):
    pid = os.getpid()
    p = psutil.Process(pid)
    open_files = p.open_files()
    print(f"{markstr}_Open files count: {len(open_files)}")


```

and added this function as below:
```python

with open(file, "rb") as f:
    print_open_files_count('Before')
...
...
      batch_idx += 1
print_open_files_count('After')
```
and the console output as below when loading the 'mteb/biblenlp-corpus-mmteb' dataset :
```shell
Before_Open files count: 1
After_Open files count: 1
Before_Open files count: 2
After_Open files count: 2
Before_Open files count: 3
After_Open files count: 3
...
```
which indicated there was a file handle leakage in the dataset loading process. So I tried to close the file handle manually using os library and found it works although  the core issue has not been found temporarily
2024-05-10T14:34:32Z
https://github.com/huggingface/datasets/pull/6889#issuecomment-2104721582
https://api.github.com/repos/huggingface/datasets/issues/6889
IC_kwDODunzps59c4Cu
https://api.github.com/repos/huggingface/datasets/issues/comments/2104721582/reactions
2024-05-10T14:47:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2104721582
https://avatars.githubusercontent.com/u/16257131?v=4
https://api.github.com/users/arthasking123/events{/privacy}
https://api.github.com/users/arthasking123/followers
https://api.github.com/users/arthasking123/following{/other_user}
https://api.github.com/users/arthasking123/gists{/gist_id}
https://github.com/arthasking123
arthasking123
MDQ6VXNlcjE2MjU3MTMx
https://api.github.com/users/arthasking123/orgs
https://api.github.com/users/arthasking123/received_events
https://api.github.com/users/arthasking123/repos
https://api.github.com/users/arthasking123/starred{/owner}{/repo}
https://api.github.com/users/arthasking123/subscriptions
User
https://api.github.com/users/arthasking123
public
MEMBER
I think it would be better to find the cause and have a cleaner fix, because while your suggested fix works for a simple case, it will lead to files that will stay open if there is an error during the dataset generation for example.


Btw I was not able to reproduce locally (macbook pro m2) or on colab, so it might be something related to your environment. Also `open()` should close the file at the end of the `with` block so I don't really get how you can get this issue :/
2024-05-10T15:20:00Z
https://github.com/huggingface/datasets/pull/6889#issuecomment-2104795222
https://api.github.com/repos/huggingface/datasets/issues/6889
IC_kwDODunzps59dKBW
https://api.github.com/repos/huggingface/datasets/issues/comments/2104795222/reactions
2024-05-10T15:21:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2104795222
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Btw I was not able to reproduce locally (macbook pro m2) or on colab, so it might be something related to your environment. Also `open()` should close the file at the end of the `with` block so I don't really get how you can get this issue :/

how about setting the limitation of open files  to 1024?
2024-05-11T00:13:14Z
https://github.com/huggingface/datasets/pull/6889#issuecomment-2105400177
https://api.github.com/repos/huggingface/datasets/issues/6889
IC_kwDODunzps59fdtx
https://api.github.com/repos/huggingface/datasets/issues/comments/2105400177/reactions
2024-05-11T00:13:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2105400177
https://avatars.githubusercontent.com/u/16257131?v=4
https://api.github.com/users/arthasking123/events{/privacy}
https://api.github.com/users/arthasking123/followers
https://api.github.com/users/arthasking123/following{/other_user}
https://api.github.com/users/arthasking123/gists{/gist_id}
https://github.com/arthasking123
arthasking123
MDQ6VXNlcjE2MjU3MTMx
https://api.github.com/users/arthasking123/orgs
https://api.github.com/users/arthasking123/received_events
https://api.github.com/users/arthasking123/repos
https://api.github.com/users/arthasking123/starred{/owner}{/repo}
https://api.github.com/users/arthasking123/subscriptions
User
https://api.github.com/users/arthasking123
public
MEMBER
I was able to reproduce on colab with

```
!ulimit -n 256 && python -c "from datasets import load_dataset; load_dataset('mteb/biblenlp-corpus-mmteb')"
```

(also needed to `!pip install -qq git+https://github.com/huggingface/huggingface_hub.git@less-paths-info-calls` to fix a rate limit for some reason)

which led to me find that the issue came from the `GzipFileSystem` that wasn't closing files.

to reproduce:

```python
import gzip
import os

import datasets
import fsspec

# os.mkdir("tmp")
# for i in range(300):
#     with gzip.open(f"tmp/{i}.txt.gz", "wt") as f:
#         f.write("yo")

for i in range(300):
    with fsspec.open(f"gzip://{i}.txt::tmp/{i}.txt.gz", "rb") as f:
        f.read()
```

I opened https://github.com/huggingface/datasets/pull/6893 to fix this, can you try if it works on your side ?
2024-05-13T12:27:59Z
https://github.com/huggingface/datasets/pull/6889#issuecomment-2107445094
https://api.github.com/repos/huggingface/datasets/issues/6889
IC_kwDODunzps59nQ9m
https://api.github.com/repos/huggingface/datasets/issues/comments/2107445094/reactions
2024-05-13T12:38:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2107445094
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
ok



---- Replied Message ----
| From | Quentin ***@***.***> |
| Date | 05/13/2024 20:28 |
| To | ***@***.***> |
| Cc | ***@***.***>***@***.***> |
| Subject | Re: [huggingface/datasets] fix bug #6877 (PR #6889) |

I was able to reproduce on colab with

!ulimit -n 256 && python -c "from datasets import load_dataset; load_dataset('mteb/biblenlp-corpus-mmteb')"


(also needed to !pip install -qq ***@***.*** to fix a rate limit for some reason)

which lead to me find that the issue came from the GzipFileSystem that wasn't closing files.

to reproduce:

importgzipimportosimportdatasetsimportfsspec# os.mkdir("tmp")# for i in range(300):#     with gzip.open(f"tmp/{i}.txt.gz", "wt") as f:#         f.write("yo")foriinrange(300):
    withfsspec.open(f"gzip://::tmp/{i}.txt.gz", "rb") asf:
        f.read()

I opened #6893 to fix this, can you try if it works on your side ?

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
2024-05-13T12:31:54Z
https://github.com/huggingface/datasets/pull/6889#issuecomment-2107452962
https://api.github.com/repos/huggingface/datasets/issues/6889
IC_kwDODunzps59nS4i
https://api.github.com/repos/huggingface/datasets/issues/comments/2107452962/reactions
2024-05-13T12:31:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2107452962
https://avatars.githubusercontent.com/u/16257131?v=4
https://api.github.com/users/arthasking123/events{/privacy}
https://api.github.com/users/arthasking123/followers
https://api.github.com/users/arthasking123/following{/other_user}
https://api.github.com/users/arthasking123/gists{/gist_id}
https://github.com/arthasking123
arthasking123
MDQ6VXNlcjE2MjU3MTMx
https://api.github.com/users/arthasking123/orgs
https://api.github.com/users/arthasking123/received_events
https://api.github.com/users/arthasking123/repos
https://api.github.com/users/arthasking123/starred{/owner}{/repo}
https://api.github.com/users/arthasking123/subscriptions
User
https://api.github.com/users/arthasking123
public
MEMBER
Superseded by:
- #6893
2024-05-13T13:35:32Z
https://github.com/huggingface/datasets/pull/6889#issuecomment-2107599122
https://api.github.com/repos/huggingface/datasets/issues/6889
IC_kwDODunzps59n2kS
https://api.github.com/repos/huggingface/datasets/issues/comments/2107599122/reactions
2024-05-13T13:35:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2107599122
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
fix bug #6877 due to maybe f becomes invaild after yield process
the results are below:


Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:01<00:00, 420.41it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 26148.48it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 409731.44it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 289720.84it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 26663.42it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 434056.21it/s]
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 13222.33files/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:04<00:00, 180.67files/s]
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [01:35<00:00,  8.70files/s]
Generating train split: 1571592 examples [00:08, 176736.09 examples/s]
Generating test split: 85533 examples [00:01, 48224.56 examples/s]
Generating validation split: 86246 examples [00:01, 50164.16 examples/s]

Fix https://github.com/huggingface/datasets/issues/6877.

CC: @natolambert
