Regarding On-the-fly Data Loading
NONE
Possible duplicate

#1776 https://github.com/huggingface/datasets/issues/

really looking PR for this feature
2021-02-06T01:43:56Z
https://github.com/huggingface/datasets/issues/1827#issuecomment-774374760
https://api.github.com/repos/huggingface/datasets/issues/1827
MDEyOklzc3VlQ29tbWVudDc3NDM3NDc2MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/774374760/reactions
2021-02-06T01:43:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774374760
https://avatars.githubusercontent.com/u/56231298?v=4
https://api.github.com/users/acul3/events{/privacy}
https://api.github.com/users/acul3/followers
https://api.github.com/users/acul3/following{/other_user}
https://api.github.com/users/acul3/gists{/gist_id}
https://github.com/acul3
acul3
MDQ6VXNlcjU2MjMxMjk4
https://api.github.com/users/acul3/orgs
https://api.github.com/users/acul3/received_events
https://api.github.com/users/acul3/repos
https://api.github.com/users/acul3/starred{/owner}{/repo}
https://api.github.com/users/acul3/subscriptions
User
https://api.github.com/users/acul3
public
CONTRIBUTOR
Hi @acul3 

Issue #1776 talks about  doing on-the-fly data pre-processing, which I think is solved in the next release as mentioned in the issue #1825. I also look forward to using this feature, though :)

I wanted to ask about on-the-fly data loading from the cache (before pre-processing).
2021-02-06T05:35:28Z
https://github.com/huggingface/datasets/issues/1827#issuecomment-774403932
https://api.github.com/repos/huggingface/datasets/issues/1827
MDEyOklzc3VlQ29tbWVudDc3NDQwMzkzMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/774403932/reactions
2021-02-06T05:35:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/774403932
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
MEMBER
Hi ! Currently when you load a dataset via `load_dataset` for example, then the dataset is memory-mapped from an Arrow file on disk. Therefore there's almost no RAM usage even if your dataset contains TB of data.
Usually at training time only one batch of data at a time is loaded in memory.

Does that answer your question or were you thinking about something else ?
2021-02-08T11:38:43Z
https://github.com/huggingface/datasets/issues/1827#issuecomment-775083557
https://api.github.com/repos/huggingface/datasets/issues/1827
MDEyOklzc3VlQ29tbWVudDc3NTA4MzU1Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/775083557/reactions
2021-02-08T11:38:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/775083557
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi @lhoestq,

I apologize for the late response. This answers my question. Thanks a lot.
2021-02-18T13:55:16Z
https://github.com/huggingface/datasets/issues/1827#issuecomment-781359793
https://api.github.com/repos/huggingface/datasets/issues/1827
MDEyOklzc3VlQ29tbWVudDc4MTM1OTc5Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/781359793/reactions
2021-02-18T13:55:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/781359793
https://avatars.githubusercontent.com/u/29076344?v=4
https://api.github.com/users/gchhablani/events{/privacy}
https://api.github.com/users/gchhablani/followers
https://api.github.com/users/gchhablani/following{/other_user}
https://api.github.com/users/gchhablani/gists{/gist_id}
https://github.com/gchhablani
gchhablani
MDQ6VXNlcjI5MDc2MzQ0
https://api.github.com/users/gchhablani/orgs
https://api.github.com/users/gchhablani/received_events
https://api.github.com/users/gchhablani/repos
https://api.github.com/users/gchhablani/starred{/owner}{/repo}
https://api.github.com/users/gchhablani/subscriptions
User
https://api.github.com/users/gchhablani
public
Hi,

I was wondering if it is possible to load images/texts as a batch during the training process, without loading the entire dataset on the RAM at any given point.

Thanks,
Gunjan