cache_dir parameter for load_from_disk ?
MEMBER
Hi ! `load_from_disk` doesn't move the data. If you specify a local path to your mounted drive, then the dataset is going to be loaded directly from the arrow file in this directory. The cache files that result from `map` operations are also stored in the same directory by default.

However note than writing data to your google drive actually fills the VM's disk (see https://github.com/huggingface/datasets/issues/643)

Given that, I don't think that changing the cache directory changes anything.

Let me know what you think
2021-06-11T15:29:17Z
https://github.com/huggingface/datasets/issues/2474#issuecomment-859661439
https://api.github.com/repos/huggingface/datasets/issues/2474
MDEyOklzc3VlQ29tbWVudDg1OTY2MTQzOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/859661439/reactions
2021-06-11T15:29:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/859661439
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for your answer! I am a little surprised since I just want to read the dataset.

After debugging a bit, I noticed that the VM’s disk fills up when the tables (generator) are converted to a list:

https://github.com/huggingface/datasets/blob/5ba149773d23369617563d752aca922081277ec2/src/datasets/table.py#L850

If I try to iterate through the table’s generator e.g.: 

`length = sum(1 for x in tables)`

the VM’s disk fills up as well.

I’m running out of Ideas 😄 
2021-06-13T22:56:16Z
https://github.com/huggingface/datasets/issues/2474#issuecomment-860281554
https://api.github.com/repos/huggingface/datasets/issues/2474
MDEyOklzc3VlQ29tbWVudDg2MDI4MTU1NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/860281554/reactions
2021-06-13T22:56:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/860281554
https://avatars.githubusercontent.com/u/7063207?v=4
https://api.github.com/users/chbensch/events{/privacy}
https://api.github.com/users/chbensch/followers
https://api.github.com/users/chbensch/following{/other_user}
https://api.github.com/users/chbensch/gists{/gist_id}
https://github.com/chbensch
chbensch
MDQ6VXNlcjcwNjMyMDc=
https://api.github.com/users/chbensch/orgs
https://api.github.com/users/chbensch/received_events
https://api.github.com/users/chbensch/repos
https://api.github.com/users/chbensch/starred{/owner}{/repo}
https://api.github.com/users/chbensch/subscriptions
User
https://api.github.com/users/chbensch
public
MEMBER
Indeed reading the data shouldn't increase the VM's disk. Not sure what google colab does under the hood for that to happen
2021-06-14T08:41:09Z
https://github.com/huggingface/datasets/issues/2474#issuecomment-860505947
https://api.github.com/repos/huggingface/datasets/issues/2474
MDEyOklzc3VlQ29tbWVudDg2MDUwNTk0Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/860505947/reactions
2021-06-14T08:41:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/860505947
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Apparently, Colab uses a local cache of the data files read/written from Google Drive. See:
- https://github.com/googlecolab/colabtools/issues/2087#issuecomment-860818457
- https://github.com/googlecolab/colabtools/issues/1915#issuecomment-804234540
- https://github.com/googlecolab/colabtools/issues/2147#issuecomment-885052636
2022-02-16T14:55:00Z
https://github.com/huggingface/datasets/issues/2474#issuecomment-1041576327
https://api.github.com/repos/huggingface/datasets/issues/2474
IC_kwDODunzps4-FTGH
https://api.github.com/repos/huggingface/datasets/issues/comments/1041576327/reactions
2022-02-16T14:55:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1041576327
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
**Is your feature request related to a problem? Please describe.**
When using Google Colab big datasets can be an issue, as they won't fit on the VM's disk. Therefore mounting google drive could be a possible solution. Unfortunatly when loading my own dataset by using the _load_from_disk_ function, the data gets cached to the VM's disk:

`
from datasets import load_from_disk

myPreprocessedData = load_from_disk("/content/gdrive/MyDrive/ASR_data/myPreprocessedData")

`
I know that chaching on google drive could slow down learning. But at least it would run.

**Describe the solution you'd like**
Add cache_Dir parameter to the load_from_disk function.

**Describe alternatives you've considered**
It looks like you could write a custom loading script for the load_dataset function. But this seems to be much too complex for my use case. Is there perhaps a template here that uses the load_from_disk function?
