Creating dataset consumes too much memory
MEMBER
Thanks for reporting.
In theory since the dataset script is just made to yield examples to write them into an arrow file, it's not supposed to create memory issues.

Could you please try to run this exact same loop in a separate script to see if it's not an issue with `PIL` ?
You can just copy paste what's inside `_generate_examples` and remove all the code for `datasets` (remove yield).

If the RAM usage stays low after 600 examples it means that it comes from some sort of memory leak in the library, or with pyarrow.
2020-10-20T08:59:49Z
https://github.com/huggingface/datasets/issues/741#issuecomment-712703410
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDcxMjcwMzQxMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/712703410/reactions
2020-10-20T08:59:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/712703410
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Here's an equivalent loading code:
```python
images_path = "PHOENIX-2014-T-release-v3/PHOENIX-2014-T/features/fullFrame-210x260px/train"

for dir_path in tqdm(os.listdir(images_path)):
    frames_path = os.path.join(images_path, dir_path)
    np_frames = []
    for frame_name in os.listdir(frames_path):
        frame_path = os.path.join(frames_path, frame_name)
        im = Image.open(frame_path)
        np_frames.append(np.asarray(im))
        im.close()
```

The process takes 0.3% of memory, even after 1000 examples on the small machine with 120GB RAM.

I guess something in the datasets library doesn't release the reference to the objects I'm yielding, but no idea how to test for this
2020-10-20T14:14:29Z
https://github.com/huggingface/datasets/issues/741#issuecomment-712884043
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDcxMjg4NDA0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/712884043/reactions
2020-10-20T14:39:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/712884043
https://avatars.githubusercontent.com/u/5757359?v=4
https://api.github.com/users/AmitMY/events{/privacy}
https://api.github.com/users/AmitMY/followers
https://api.github.com/users/AmitMY/following{/other_user}
https://api.github.com/users/AmitMY/gists{/gist_id}
https://github.com/AmitMY
AmitMY
MDQ6VXNlcjU3NTczNTk=
https://api.github.com/users/AmitMY/orgs
https://api.github.com/users/AmitMY/received_events
https://api.github.com/users/AmitMY/repos
https://api.github.com/users/AmitMY/starred{/owner}{/repo}
https://api.github.com/users/AmitMY/subscriptions
User
https://api.github.com/users/AmitMY
public
MEMBER
I've had similar issues with Arrow once. I'll investigate...

For now maybe we can simply use the images paths in the dataset you want to add. I don't expect to fix this memory issue until 1-2 weeks unfortunately. Then we can just update the dataset with the images. What do you think ?
2020-10-21T09:43:22Z
https://github.com/huggingface/datasets/issues/741#issuecomment-713446499
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDcxMzQ0NjQ5OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/713446499/reactions
2020-10-21T09:43:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/713446499
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
If it's just 1-2 weeks, I think it's best if we wait. I don't think it is very urgent to add it, and it will be much more useful with the images loaded rather than not (the images are low resolution, and thus papers using this dataset actually fit the entire video into memory anyway)

I'll keep working on other datasets in the meanwhile :) 
2020-10-21T14:12:53Z
https://github.com/huggingface/datasets/issues/741#issuecomment-713607621
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDcxMzYwNzYyMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/713607621/reactions
2020-10-21T14:13:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/713607621
https://avatars.githubusercontent.com/u/5757359?v=4
https://api.github.com/users/AmitMY/events{/privacy}
https://api.github.com/users/AmitMY/followers
https://api.github.com/users/AmitMY/following{/other_user}
https://api.github.com/users/AmitMY/gists{/gist_id}
https://github.com/AmitMY
AmitMY
MDQ6VXNlcjU3NTczNTk=
https://api.github.com/users/AmitMY/orgs
https://api.github.com/users/AmitMY/received_events
https://api.github.com/users/AmitMY/repos
https://api.github.com/users/AmitMY/starred{/owner}{/repo}
https://api.github.com/users/AmitMY/subscriptions
User
https://api.github.com/users/AmitMY
public
MEMBER
Ok found the issue. This is because the batch size used by the writer is set to 10 000 elements by default so it would load your full dataset in memory (the writer has a buffer that flushes only after each batch). Moreover to write in Apache Arrow we have to use python objects so what's stored inside the ArrowWriter's buffer is actually python integers (32 bits).

Lowering the batch size to 10 should do the job.

I will add a flag to the DatasetBuilder class of dataset scripts, so that we can customize the batch size.
2020-10-29T16:39:15Z
https://github.com/huggingface/datasets/issues/741#issuecomment-718876278
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDcxODg3NjI3OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/718876278/reactions
2020-10-29T16:39:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/718876278
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks, that's awesome you managed to find the problem.

About the 32 bits - really? there isn't a way to serialize the numpy array somehow? 32 bits would take 4 times the memory / disk space needed to store these videos.

Please let me know when the batch size is customizable and I'll try again!
2020-10-30T10:36:04Z
https://github.com/huggingface/datasets/issues/741#issuecomment-719476127
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDcxOTQ3NjEyNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/719476127/reactions
2020-10-30T10:36:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/719476127
https://avatars.githubusercontent.com/u/5757359?v=4
https://api.github.com/users/AmitMY/events{/privacy}
https://api.github.com/users/AmitMY/followers
https://api.github.com/users/AmitMY/following{/other_user}
https://api.github.com/users/AmitMY/gists{/gist_id}
https://github.com/AmitMY
AmitMY
MDQ6VXNlcjU3NTczNTk=
https://api.github.com/users/AmitMY/orgs
https://api.github.com/users/AmitMY/received_events
https://api.github.com/users/AmitMY/repos
https://api.github.com/users/AmitMY/starred{/owner}{/repo}
https://api.github.com/users/AmitMY/subscriptions
User
https://api.github.com/users/AmitMY
public
MEMBER
The 32 bit integrers are only used in the writer's buffer because Arrow doesn't take numpy arrays correctly as input. On disk it's stored as uint8 in arrow format ;)
2020-10-30T11:48:09Z
https://github.com/huggingface/datasets/issues/741#issuecomment-719507328
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDcxOTUwNzMyOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/719507328/reactions
2020-10-30T11:48:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/719507328
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> I don't expect to fix this memory issue until 1-2 weeks unfortunately.

Hi @lhoestq 
not to rush of course, but I was wondering if you have a new timeline so I know how to plan my work around this :) 
2020-11-08T06:46:00Z
https://github.com/huggingface/datasets/issues/741#issuecomment-723538019
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDcyMzUzODAxOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/723538019/reactions
2020-11-08T06:46:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/723538019
https://avatars.githubusercontent.com/u/5757359?v=4
https://api.github.com/users/AmitMY/events{/privacy}
https://api.github.com/users/AmitMY/followers
https://api.github.com/users/AmitMY/following{/other_user}
https://api.github.com/users/AmitMY/gists{/gist_id}
https://github.com/AmitMY
AmitMY
MDQ6VXNlcjU3NTczNTk=
https://api.github.com/users/AmitMY/orgs
https://api.github.com/users/AmitMY/received_events
https://api.github.com/users/AmitMY/repos
https://api.github.com/users/AmitMY/starred{/owner}{/repo}
https://api.github.com/users/AmitMY/subscriptions
User
https://api.github.com/users/AmitMY
public
MEMBER
Hi ! Next week for sure :) 
2020-11-08T12:36:19Z
https://github.com/huggingface/datasets/issues/741#issuecomment-723571659
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDcyMzU3MTY1OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/723571659/reactions
2020-11-08T12:36:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/723571659
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Alright it should be good now.
You just have to specify `_writer_batch_size = 10` for example as a class attribute of the dataset builder class.
2020-11-10T16:30:36Z
https://github.com/huggingface/datasets/issues/741#issuecomment-724815801
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDcyNDgxNTgwMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/724815801/reactions
2020-11-10T16:30:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/724815801
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I added it, but still it consumes as much memory

https://github.com/huggingface/datasets/pull/722/files#diff-2e0d865dd4a60dedd1861d6f8c5ed281ded71508467908e1e0b1dbe7d2d420b1R66

Did I not do it correctly?
2020-11-17T10:25:17Z
https://github.com/huggingface/datasets/issues/741#issuecomment-728835782
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDcyODgzNTc4Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/728835782/reactions
2020-11-17T10:25:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/728835782
https://avatars.githubusercontent.com/u/5757359?v=4
https://api.github.com/users/AmitMY/events{/privacy}
https://api.github.com/users/AmitMY/followers
https://api.github.com/users/AmitMY/following{/other_user}
https://api.github.com/users/AmitMY/gists{/gist_id}
https://github.com/AmitMY
AmitMY
MDQ6VXNlcjU3NTczNTk=
https://api.github.com/users/AmitMY/orgs
https://api.github.com/users/AmitMY/received_events
https://api.github.com/users/AmitMY/repos
https://api.github.com/users/AmitMY/starred{/owner}{/repo}
https://api.github.com/users/AmitMY/subscriptions
User
https://api.github.com/users/AmitMY
public
MEMBER
Yes you did it right.
Did you rebase to include the changes of #828 ?

EDIT: looks like you merged from master in the PR. Not sure why you still have an issue then, I will investigate
2020-11-17T10:31:02Z
https://github.com/huggingface/datasets/issues/741#issuecomment-728838751
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDcyODgzODc1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/728838751/reactions
2020-11-17T10:31:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/728838751
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi @lhoestq, any update on this?
Perhaps even a direction I could try myself?
2020-12-19T05:52:16Z
https://github.com/huggingface/datasets/issues/741#issuecomment-748424462
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDc0ODQyNDQ2Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/748424462/reactions
2020-12-19T05:52:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/748424462
https://avatars.githubusercontent.com/u/5757359?v=4
https://api.github.com/users/AmitMY/events{/privacy}
https://api.github.com/users/AmitMY/followers
https://api.github.com/users/AmitMY/following{/other_user}
https://api.github.com/users/AmitMY/gists{/gist_id}
https://github.com/AmitMY
AmitMY
MDQ6VXNlcjU3NTczNTk=
https://api.github.com/users/AmitMY/orgs
https://api.github.com/users/AmitMY/received_events
https://api.github.com/users/AmitMY/repos
https://api.github.com/users/AmitMY/starred{/owner}{/repo}
https://api.github.com/users/AmitMY/subscriptions
User
https://api.github.com/users/AmitMY
public
MEMBER
Sorry for the delay, I was busy with the dataset sprint and the incredible amount of contributions to the library ^^'

What you can try to do to find what's wrong is check at which frequency the arrow writer writes all the examples from its in-memory buffer on disk. This happens [here](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_writer.py#L257-L258) in the code.

The idea is that `write_on_file` writes the examples every `writer_batch_size` examples and clear the buffer `self. current_rows`. As soon as `writer_batch_size` is small enough you shouldn't have memory issues in theory.

Let me know if you have questions or if I can help.

Since the dataset sprint is over and I will also be done with all the PRs soon I will be able to go back at it and take a look.
2020-12-21T10:25:31Z
https://github.com/huggingface/datasets/issues/741#issuecomment-748897819
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDc0ODg5NzgxOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/748897819/reactions
2020-12-21T10:25:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/748897819
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks. I gave it a try and no success. I'm not sure what's happening there
2020-12-30T19:14:08Z
https://github.com/huggingface/datasets/issues/741#issuecomment-752728561
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDc1MjcyODU2MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/752728561/reactions
2020-12-30T19:14:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/752728561
https://avatars.githubusercontent.com/u/5757359?v=4
https://api.github.com/users/AmitMY/events{/privacy}
https://api.github.com/users/AmitMY/followers
https://api.github.com/users/AmitMY/following{/other_user}
https://api.github.com/users/AmitMY/gists{/gist_id}
https://github.com/AmitMY
AmitMY
MDQ6VXNlcjU3NTczNTk=
https://api.github.com/users/AmitMY/orgs
https://api.github.com/users/AmitMY/received_events
https://api.github.com/users/AmitMY/repos
https://api.github.com/users/AmitMY/starred{/owner}{/repo}
https://api.github.com/users/AmitMY/subscriptions
User
https://api.github.com/users/AmitMY
public
NONE
I had the same issue. It works for me by setting `DEFAULT_WRITER_BATCH_SIZE = 10` of my dataset builder class. (And not `_writer_batch_size` as previously mentioned).  I guess this is because `_writer_batch_size` is overwritten in `__init__` (see [here](https://github.com/huggingface/datasets/blob/0e2563e5d5c2fc193ea27d7c24607bb35607f2d5/src/datasets/builder.py#L934))
2021-03-26T15:37:15Z
https://github.com/huggingface/datasets/issues/741#issuecomment-808316975
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDgwODMxNjk3NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/808316975/reactions
2021-03-26T15:38:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/808316975
https://avatars.githubusercontent.com/u/22685854?v=4
https://api.github.com/users/jblemoine/events{/privacy}
https://api.github.com/users/jblemoine/followers
https://api.github.com/users/jblemoine/following{/other_user}
https://api.github.com/users/jblemoine/gists{/gist_id}
https://github.com/jblemoine
jblemoine
MDQ6VXNlcjIyNjg1ODU0
https://api.github.com/users/jblemoine/orgs
https://api.github.com/users/jblemoine/received_events
https://api.github.com/users/jblemoine/repos
https://api.github.com/users/jblemoine/starred{/owner}{/repo}
https://api.github.com/users/jblemoine/subscriptions
User
https://api.github.com/users/jblemoine
public
MEMBER
Yes the class attribute you can change is `DEFAULT_WRITER_BATCH_SIZE`.
Otherwise in `load_dataset` you can specify `writer_batch_size=`
2021-03-26T15:41:34Z
https://github.com/huggingface/datasets/issues/741#issuecomment-808319837
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDgwODMxOTgzNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/808319837/reactions
2021-03-26T15:41:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/808319837
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Ok thanks for the tips. Maybe the documentation should be updated accordingly https://huggingface.co/docs/datasets/add_dataset.html.
2021-03-26T16:05:27Z
https://github.com/huggingface/datasets/issues/741#issuecomment-808335506
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDgwODMzNTUwNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/808335506/reactions
2021-03-26T16:05:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/808335506
https://avatars.githubusercontent.com/u/22685854?v=4
https://api.github.com/users/jblemoine/events{/privacy}
https://api.github.com/users/jblemoine/followers
https://api.github.com/users/jblemoine/following{/other_user}
https://api.github.com/users/jblemoine/gists{/gist_id}
https://github.com/jblemoine
jblemoine
MDQ6VXNlcjIyNjg1ODU0
https://api.github.com/users/jblemoine/orgs
https://api.github.com/users/jblemoine/received_events
https://api.github.com/users/jblemoine/repos
https://api.github.com/users/jblemoine/starred{/owner}{/repo}
https://api.github.com/users/jblemoine/subscriptions
User
https://api.github.com/users/jblemoine
public
MEMBER
Thanks for reporting this mistake in the docs.
I just fixed it at https://github.com/huggingface/datasets/commit/85cf7ff920c90ca2e12bedca12b36d2a043c3da2
2021-03-30T09:47:08Z
https://github.com/huggingface/datasets/issues/741#issuecomment-810080635
https://api.github.com/repos/huggingface/datasets/issues/741
MDEyOklzc3VlQ29tbWVudDgxMDA4MDYzNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/810080635/reactions
2021-03-30T09:47:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/810080635
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
May I close this issue, @AmitMY?
2022-02-15T10:45:34Z
https://github.com/huggingface/datasets/issues/741#issuecomment-1040122484
https://api.github.com/repos/huggingface/datasets/issues/741
IC_kwDODunzps49_wJ0
https://api.github.com/repos/huggingface/datasets/issues/comments/1040122484/reactions
2022-02-15T10:45:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1040122484
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Moving this issue from https://github.com/huggingface/datasets/pull/722 here, because it seems like a general issue.

Given the following dataset example, where each example saves a sequence of 260x210x3 images (max length 400):
```python
    def _generate_examples(self, base_path, split):
        """ Yields examples. """

        filepath = os.path.join(base_path, "annotations", "manual", "PHOENIX-2014-T." + split + ".corpus.csv")
        images_path = os.path.join(base_path, "features", "fullFrame-210x260px", split)

        with open(filepath, "r", encoding="utf-8") as f:
            data = csv.DictReader(f, delimiter="|", quoting=csv.QUOTE_NONE)
            for row in data:
                frames_path = os.path.join(images_path, row["video"])[:-7]
                np_frames = []
                for frame_name in os.listdir(frames_path):
                    frame_path = os.path.join(frames_path, frame_name)
                    im = Image.open(frame_path)
                    np_frames.append(np.asarray(im))
                    im.close()

                yield row["name"], {"video": np_frames}
```

The dataset creation process goes out of memory on a machine with 500GB RAM.
I was under the impression that the "generator" here is exactly for that, to avoid memory constraints.


However, even if you want the entire dataset in memory, it would be in the worst case
`260x210x3 x 400 max length x 7000 samples` in bytes (uint8) = 458.64 gigabytes
So I'm not sure why it's taking more than 500GB.

And the dataset creation fails after 170 examples on a machine with 120gb RAM, and after 672 examples on a machine with 500GB RAM.


---

## Info that might help:
Iterating over examples is extremely slow.
![image](https://user-images.githubusercontent.com/5757359/96359590-3c666780-111d-11eb-9347-1f833ad982a9.png)
If I perform this iteration in my own, custom loop (Without saving to file), it runs at 8-9 examples/sec

And you can see at this state it is using 94% of the memory:
![image](https://user-images.githubusercontent.com/5757359/96359606-7afc2200-111d-11eb-8c11-0afbdba1a6a3.png)

And it is only using one CPU core, which is probably why it's so slow:
![image](https://user-images.githubusercontent.com/5757359/96359630-a3841c00-111d-11eb-9ba0-7fd3cdf51d26.png)
