`load_dataset` consumes too much memory for audio + tar archives
MEMBER
Hi ! Could it be because you need to free the memory used by `tarfile` by emptying the tar `members` by any chance ?
```python
        yield key, {"audio": {"path": audio_name, "bytes": audio_file_obj.read()}}
        audio_tarfile.members = []  # free memory
        key += 1
```

and then you can set `DEFAULT_WRITER_BATCH_SIZE` to whatever value makes more sense for your dataset.

Let me know if the issue persists (which could happen, given that you managed to run your generator without RAM issues and using os.walk didn't solve the issue)
2022-03-30T14:39:19Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1083224251
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5AkLC7
https://api.github.com/repos/huggingface/datasets/issues/comments/1083224251/reactions
2022-03-30T14:48:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1083224251
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for your reply! Tried it but the issue persists. 
2022-03-30T15:20:45Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1083275933
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5AkXqd
https://api.github.com/repos/huggingface/datasets/issues/comments/1083275933/reactions
2022-03-30T15:20:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1083275933
https://avatars.githubusercontent.com/u/50839826?v=4
https://api.github.com/users/JFCeron/events{/privacy}
https://api.github.com/users/JFCeron/followers
https://api.github.com/users/JFCeron/following{/other_user}
https://api.github.com/users/JFCeron/gists{/gist_id}
https://github.com/JFCeron
JFCeron
MDQ6VXNlcjUwODM5ODI2
https://api.github.com/users/JFCeron/orgs
https://api.github.com/users/JFCeron/received_events
https://api.github.com/users/JFCeron/repos
https://api.github.com/users/JFCeron/starred{/owner}{/repo}
https://api.github.com/users/JFCeron/subscriptions
User
https://api.github.com/users/JFCeron
public
MEMBER
I also run out of memory when loading `mozilla-foundation/common_voice_8_0` that also uses `tarfile` via `dl_manager.iter_archive`. There seems to be some data files that stay in memory somewhere

I don't have the issue with other compression formats like gzipped files
2022-04-07T16:40:00Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1091968008
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5BFhwI
https://api.github.com/repos/huggingface/datasets/issues/comments/1091968008/reactions
2022-04-07T16:49:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1091968008
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I'm facing a similar memory leak issue when loading cv8. As you said @lhoestq 

`load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token=True, writer_batch_size=1)`

This issue is happening on a 32GB RAM machine. 

Any updates on how to fix this?
2022-04-24T16:20:16Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1107872717
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5CCMvN
https://api.github.com/repos/huggingface/datasets/issues/comments/1107872717/reactions
2022-04-24T16:20:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1107872717
https://avatars.githubusercontent.com/u/5097052?v=4
https://api.github.com/users/jonatasgrosman/events{/privacy}
https://api.github.com/users/jonatasgrosman/followers
https://api.github.com/users/jonatasgrosman/following{/other_user}
https://api.github.com/users/jonatasgrosman/gists{/gist_id}
https://github.com/jonatasgrosman
jonatasgrosman
MDQ6VXNlcjUwOTcwNTI=
https://api.github.com/users/jonatasgrosman/orgs
https://api.github.com/users/jonatasgrosman/received_events
https://api.github.com/users/jonatasgrosman/repos
https://api.github.com/users/jonatasgrosman/starred{/owner}{/repo}
https://api.github.com/users/jonatasgrosman/subscriptions
User
https://api.github.com/users/jonatasgrosman
public
CONTRIBUTOR
I've run a memory profiler to see where's the leak comes from:

![image](https://user-images.githubusercontent.com/5097052/165101712-e7060ae5-77b2-4f6a-92bd-2996dbd60b36.png)

... it seems that it's related to the tarfile lib buffer reader. But I don't know why it's only happening on the huggingface script
2022-04-25T14:03:28Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1108620012
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5CFDLs
https://api.github.com/repos/huggingface/datasets/issues/comments/1108620012/reactions
2022-04-25T14:03:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1108620012
https://avatars.githubusercontent.com/u/5097052?v=4
https://api.github.com/users/jonatasgrosman/events{/privacy}
https://api.github.com/users/jonatasgrosman/followers
https://api.github.com/users/jonatasgrosman/following{/other_user}
https://api.github.com/users/jonatasgrosman/gists{/gist_id}
https://github.com/jonatasgrosman
jonatasgrosman
MDQ6VXNlcjUwOTcwNTI=
https://api.github.com/users/jonatasgrosman/orgs
https://api.github.com/users/jonatasgrosman/received_events
https://api.github.com/users/jonatasgrosman/repos
https://api.github.com/users/jonatasgrosman/starred{/owner}{/repo}
https://api.github.com/users/jonatasgrosman/subscriptions
User
https://api.github.com/users/jonatasgrosman
public
CONTRIBUTOR
I have the same problem when loading video into numpy. 
```
yield id,{ 
    "video": imageio.v3.imread(video_path),
    "label": int(label)
}
```
Since video files are heavy, it can only processes a dozen samples before OOM.
2022-05-13T04:34:15Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1125648155
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5DGAcb
https://api.github.com/repos/huggingface/datasets/issues/comments/1125648155/reactions
2022-05-13T04:34:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1125648155
https://avatars.githubusercontent.com/u/12573521?v=4
https://api.github.com/users/lkhphuc/events{/privacy}
https://api.github.com/users/lkhphuc/followers
https://api.github.com/users/lkhphuc/following{/other_user}
https://api.github.com/users/lkhphuc/gists{/gist_id}
https://github.com/lkhphuc
lkhphuc
MDQ6VXNlcjEyNTczNTIx
https://api.github.com/users/lkhphuc/orgs
https://api.github.com/users/lkhphuc/received_events
https://api.github.com/users/lkhphuc/repos
https://api.github.com/users/lkhphuc/starred{/owner}{/repo}
https://api.github.com/users/lkhphuc/subscriptions
User
https://api.github.com/users/lkhphuc
public
MEMBER
For video datasets I think you can just define the max number of video that can stay in memory by adding this class attribute to your dataset builer:
```py
DEFAULT_WRITER_BATCH_SIZE = 8  # only 8 videos at a time in memory before flushing the dataset writer
```
2022-05-13T16:50:23Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1126254724
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5DIUiE
https://api.github.com/repos/huggingface/datasets/issues/comments/1126254724/reactions
2022-05-13T16:50:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1126254724
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
same thing happens for me with `load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token=True, writer_batch_size=1)` on azure ml. seems to fill up `tmp` and not release that memory until OOM
2022-06-02T10:49:21Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1144720090
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5EOwra
https://api.github.com/repos/huggingface/datasets/issues/comments/1144720090/reactions
2022-06-02T10:49:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1144720090
https://avatars.githubusercontent.com/u/10137?v=4
https://api.github.com/users/ghost/events{/privacy}
https://api.github.com/users/ghost/followers
https://api.github.com/users/ghost/following{/other_user}
https://api.github.com/users/ghost/gists{/gist_id}
https://github.com/ghost
ghost
MDQ6VXNlcjEwMTM3
https://api.github.com/users/ghost/orgs
https://api.github.com/users/ghost/received_events
https://api.github.com/users/ghost/repos
https://api.github.com/users/ghost/starred{/owner}{/repo}
https://api.github.com/users/ghost/subscriptions
User
https://api.github.com/users/ghost
public
NONE
I'll add that I'm encountering the same issue with
`load_dataset('wikipedia', 'ceb', runner='DirectRunner', split='train')`.
Same for `'es'` in place of `'ceb'`.
2022-06-23T02:30:38Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1163854807
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5FXwPX
https://api.github.com/repos/huggingface/datasets/issues/comments/1163854807/reactions
2022-06-23T02:30:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1163854807
https://avatars.githubusercontent.com/u/45244059?v=4
https://api.github.com/users/ddegenaro/events{/privacy}
https://api.github.com/users/ddegenaro/followers
https://api.github.com/users/ddegenaro/following{/other_user}
https://api.github.com/users/ddegenaro/gists{/gist_id}
https://github.com/ddegenaro
ddegenaro
MDQ6VXNlcjQ1MjQ0MDU5
https://api.github.com/users/ddegenaro/orgs
https://api.github.com/users/ddegenaro/received_events
https://api.github.com/users/ddegenaro/repos
https://api.github.com/users/ddegenaro/starred{/owner}{/repo}
https://api.github.com/users/ddegenaro/subscriptions
User
https://api.github.com/users/ddegenaro
public
MEMBER
> I'll add that I'm encountering the same issue with
> load_dataset('wikipedia', 'ceb', runner='DirectRunner', split='train').
> Same for 'es' in place of 'ceb'.

This is because the Apache Beam `DirectRunner` runs with the full data in memory unfortunately. Optimizing the `DirectRunner` is not in the scope of the `datasets` library, but rather in the Apache Beam project I believe. If you have memory issues with the `DirectRunner`, please consider switching to a machine with more RAM, or to distributed processing runtimes like Spark, Flink or DataFlow. There is a bit of documentation here: https://huggingface.co/docs/datasets/beam
2022-06-23T13:16:32Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1164396465
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5FZ0ex
https://api.github.com/repos/huggingface/datasets/issues/comments/1164396465/reactions
2022-06-23T13:16:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1164396465
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> > I'll add that I'm encountering the same issue with
> > `load_dataset('wikipedia', 'ceb', runner='DirectRunner', split='train')`.
> > Same for `'es'` in place of `'ceb'`.
> 
> This is because the Apache Beam `DirectRunner` runs with the full data in memory unfortunately. Optimizing the `DirectRunner` is not in the scope of the `datasets` library, but rather in the Apache Beam project I believe. If you have memory issues with the `DirectRunner`, please consider switching to a machine with more RAM, or to distributed processing runtimes like Spark, Flink or DataFlow. There is a bit of documentation here: https://huggingface.co/docs/datasets/beam

Fair enough, but this line of code crashed an AWS instance with 1024GB of RAM! I have also tried with `Runner='Flink'` on an environment with 51GB of RAM, which also failed.

Apache Beam has tons of open tickets already - is it worth submitting one to them over this?
2022-06-23T21:02:53Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1164870360
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5FboLY
https://api.github.com/repos/huggingface/datasets/issues/comments/1164870360/reactions
2022-06-23T21:04:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1164870360
https://avatars.githubusercontent.com/u/45244059?v=4
https://api.github.com/users/ddegenaro/events{/privacy}
https://api.github.com/users/ddegenaro/followers
https://api.github.com/users/ddegenaro/following{/other_user}
https://api.github.com/users/ddegenaro/gists{/gist_id}
https://github.com/ddegenaro
ddegenaro
MDQ6VXNlcjQ1MjQ0MDU5
https://api.github.com/users/ddegenaro/orgs
https://api.github.com/users/ddegenaro/received_events
https://api.github.com/users/ddegenaro/repos
https://api.github.com/users/ddegenaro/starred{/owner}{/repo}
https://api.github.com/users/ddegenaro/subscriptions
User
https://api.github.com/users/ddegenaro
public
MEMBER
> Fair enough, but this line of code crashed an AWS instance with 1024GB of RAM!

What, wikipedia is not even bigger than 20GB

cc @albertvillanova
2022-06-28T17:42:24Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1169032524
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5FrgVM
https://api.github.com/repos/huggingface/datasets/issues/comments/1169032524/reactions
2022-06-28T17:43:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1169032524
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> > Fair enough, but this line of code crashed an AWS instance with 1024GB of RAM!
> 
> What, wikipedia is not even bigger than 20GB
> 
> cc @albertvillanova

Luckily, on Colab you can watch the call stack at the bottom of the screen - much of the time and space complexity seems to come from `_parse_and_clean_wikicode()` rather than the actual download process. As far as I can tell, the script is loading the full dataset and then cleaning it all at once, which is consuming a lot of memory.
2022-06-30T03:26:23Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1170708490
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5Fx5gK
https://api.github.com/repos/huggingface/datasets/issues/comments/1170708490/reactions
2022-06-30T03:26:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1170708490
https://avatars.githubusercontent.com/u/45244059?v=4
https://api.github.com/users/ddegenaro/events{/privacy}
https://api.github.com/users/ddegenaro/followers
https://api.github.com/users/ddegenaro/following{/other_user}
https://api.github.com/users/ddegenaro/gists{/gist_id}
https://github.com/ddegenaro
ddegenaro
MDQ6VXNlcjQ1MjQ0MDU5
https://api.github.com/users/ddegenaro/orgs
https://api.github.com/users/ddegenaro/received_events
https://api.github.com/users/ddegenaro/repos
https://api.github.com/users/ddegenaro/starred{/owner}{/repo}
https://api.github.com/users/ddegenaro/subscriptions
User
https://api.github.com/users/ddegenaro
public
MEMBER
I think we are mixing many different bugs in this Issue page:
- TAR archive with audio files
- video file
- distributed parsing of Wikipedia using Apache Beam

@dan-the-meme-man may I ask you to open a separate Issue for your problem? Then I will address it. It is important to fix it because we are currently working on a Datasets enhancement to be able to provide all Wikipedias already preprocessed.

On the other hand, I think we could keep this Issue page for the original problem: TAR archive with audio files. That is not fixed yet either.
2022-06-30T09:20:21Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1170976609
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5Fy69h
https://api.github.com/repos/huggingface/datasets/issues/comments/1170976609/reactions
2022-06-30T09:22:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1170976609
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
Is there an update on the TAR archive issue with audio files? Happy to lend a hand in fixing this :)
2022-07-21T06:30:29Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1191093644
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5G_qWM
https://api.github.com/repos/huggingface/datasets/issues/comments/1191093644/reactions
2022-07-21T06:30:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1191093644
https://avatars.githubusercontent.com/u/93869735?v=4
https://api.github.com/users/sanchit-gandhi/events{/privacy}
https://api.github.com/users/sanchit-gandhi/followers
https://api.github.com/users/sanchit-gandhi/following{/other_user}
https://api.github.com/users/sanchit-gandhi/gists{/gist_id}
https://github.com/sanchit-gandhi
sanchit-gandhi
U_kgDOBZhWpw
https://api.github.com/users/sanchit-gandhi/orgs
https://api.github.com/users/sanchit-gandhi/received_events
https://api.github.com/users/sanchit-gandhi/repos
https://api.github.com/users/sanchit-gandhi/starred{/owner}{/repo}
https://api.github.com/users/sanchit-gandhi/subscriptions
User
https://api.github.com/users/sanchit-gandhi
public
MEMBER
I found the issue with Common Voice 8 and opened a PR to fix it: https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0/discussions/2

Basically the `metadata` dict that contains the transcripts per audio file was continuously getting filled with bytes from `f.read()` because of this code:
```python
result = metadata[path]
result["audio"] = {"path": path, "bytes": f.read()}
```
copying the result with `result = dict(metadata[path])` fixes it: the bytes are no longer added to `metadata`

I also opened PRs to the other CV datasets
2022-07-28T14:55:33Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1198259643
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5Ha_27
https://api.github.com/repos/huggingface/datasets/issues/comments/1198259643/reactions
2022-07-28T14:55:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1198259643
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Amazing, that's a great find! Thanks @lhoestq!
2022-07-28T15:57:28Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1198341506
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5HbT2C
https://api.github.com/repos/huggingface/datasets/issues/comments/1198341506/reactions
2022-07-28T15:57:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1198341506
https://avatars.githubusercontent.com/u/93869735?v=4
https://api.github.com/users/sanchit-gandhi/events{/privacy}
https://api.github.com/users/sanchit-gandhi/followers
https://api.github.com/users/sanchit-gandhi/following{/other_user}
https://api.github.com/users/sanchit-gandhi/gists{/gist_id}
https://github.com/sanchit-gandhi
sanchit-gandhi
U_kgDOBZhWpw
https://api.github.com/users/sanchit-gandhi/orgs
https://api.github.com/users/sanchit-gandhi/received_events
https://api.github.com/users/sanchit-gandhi/repos
https://api.github.com/users/sanchit-gandhi/starred{/owner}{/repo}
https://api.github.com/users/sanchit-gandhi/subscriptions
User
https://api.github.com/users/sanchit-gandhi
public
MEMBER
I'm closing this one for now, but feel free to reopen if you encounter other memory issues with audio datasets
2022-08-16T10:22:55Z
https://github.com/huggingface/datasets/issues/4057#issuecomment-1216448878
https://api.github.com/repos/huggingface/datasets/issues/4057
IC_kwDODunzps5IgYlu
https://api.github.com/repos/huggingface/datasets/issues/comments/1216448878/reactions
2022-08-16T10:22:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1216448878
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public


## Description
`load_dataset` consumes more and more memory until it's killed, even though it's made with a generator. I'm adding a loading script for a new dataset, made up of ~15s audio coming from a tar file. Tried setting `DEFAULT_WRITER_BATCH_SIZE = 1` as per the discussion in #741 but the problem persists.

## Steps to reproduce the bug
Here's my implementation of `_generate_examples`:

```python
class MyDatasetBuilder(datasets.GeneratorBasedBuilder):
    DEFAULT_WRITER_BATCH_SIZE = 1
    ...

    def _split_generators(self, dl_manager):
        archive_path = dl_manager.download(_DL_URLS[self.config.name])
            return [
                datasets.SplitGenerator(
                    name=datasets.Split.TRAIN,
                    gen_kwargs={
                        "audio_tarfile_path": archive_path["audio_tarfile"]
                    },
                ),
            ]
    
    def _generate_examples(self, audio_tarfile_path):
        key = 0
        with tarfile.open(audio_tarfile_path, mode="r|") as audio_tarfile:
            for audio_tarinfo in audio_tarfile:
                audio_name = audio_tarinfo.name
                audio_file_obj = audio_tarfile.extractfile(audio_tarinfo)
                yield key, {"audio": {"path": audio_name, "bytes": audio_file_obj.read()}}
                key += 1
```
I then try to load via `ds = load_dataset('./datasets/my_new_dataset', writer_batch_size=1)`, and memory usage grows until all 8GB of my machine are taken and process is killed (`Killed`). Also tried an untarred version of this using `os.walk` but the same happened.

I created a script to confirm that one can safely go through such a generator, which runs just fine with memory <500MB at all times.

```python
import tarfile

def generate_examples():
    audio_tarfile = tarfile.open("audios.tar", mode="r|")
    key = 0
    for audio_tarinfo in audio_tarfile:
        audio_name = audio_tarinfo.name
        audio_file_obj = audio_tarfile.extractfile(audio_tarinfo)
        yield key, {"audio": {"path": audio_name, "bytes": audio_file_obj.read()}}
        key += 1

if __name__ == "__main__":
    examples = generate_examples()
    for example in examples:
        pass
```
## Expected results
Memory consumption should be similar to the non-huggingface script.

## Actual results
Process is killed after consuming too much memory.

## Environment info
- `datasets` version: 2.0.1.dev0
- Platform: Linux-4.19.0-20-cloud-amd64-x86_64-with-debian-10.12
- Python version: 3.7.12
- PyArrow version: 6.0.1
- Pandas version: 1.3.5