Question: what gets stored in the datasets cache and why is it so huge?
CONTRIBUTOR
Echo'ing this observation: I have a few datasets in the neighborhood of 2GB CSVs uncompressed, and when I use something like `Dataset.save_to_disk()` it's ~18GB on disk.

If this is unexpected behavior, would be happy to help run debugging as needed.
2021-03-02T18:29:02Z
https://github.com/huggingface/datasets/issues/1973#issuecomment-789118304
https://api.github.com/repos/huggingface/datasets/issues/1973
MDEyOklzc3VlQ29tbWVudDc4OTExODMwNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/789118304/reactions
2021-03-02T18:29:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/789118304
https://avatars.githubusercontent.com/u/7731709?v=4
https://api.github.com/users/justin-yan/events{/privacy}
https://api.github.com/users/justin-yan/followers
https://api.github.com/users/justin-yan/following{/other_user}
https://api.github.com/users/justin-yan/gists{/gist_id}
https://github.com/justin-yan
justin-yan
MDQ6VXNlcjc3MzE3MDk=
https://api.github.com/users/justin-yan/orgs
https://api.github.com/users/justin-yan/received_events
https://api.github.com/users/justin-yan/repos
https://api.github.com/users/justin-yan/starred{/owner}{/repo}
https://api.github.com/users/justin-yan/subscriptions
User
https://api.github.com/users/justin-yan
public
MEMBER
Thanks @ioana-blue for pointing out this problem (and thanks also @justin-yan). You are right that current implementation of the datasets caching files take too much memory. We are definitely changing this and optimizing the defaults, so that the file sizes are considerably reduced. I will come back to you as soon as this is fixed.
2021-03-03T12:56:39Z
https://github.com/huggingface/datasets/issues/1973#issuecomment-789695372
https://api.github.com/repos/huggingface/datasets/issues/1973
MDEyOklzc3VlQ29tbWVudDc4OTY5NTM3Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/789695372/reactions
2021-03-03T12:56:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/789695372
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Thank you! Also I noticed that the files don't seem to be cleaned after the jobs finish. Last night I had only 3 jobs running, but the cache was still at 180GB. 
2021-03-03T15:21:14Z
https://github.com/huggingface/datasets/issues/1973#issuecomment-789793011
https://api.github.com/repos/huggingface/datasets/issues/1973
MDEyOklzc3VlQ29tbWVudDc4OTc5MzAxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/789793011/reactions
2021-03-03T15:21:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/789793011
https://avatars.githubusercontent.com/u/17202292?v=4
https://api.github.com/users/ioana-blue/events{/privacy}
https://api.github.com/users/ioana-blue/followers
https://api.github.com/users/ioana-blue/following{/other_user}
https://api.github.com/users/ioana-blue/gists{/gist_id}
https://github.com/ioana-blue
ioana-blue
MDQ6VXNlcjE3MjAyMjky
https://api.github.com/users/ioana-blue/orgs
https://api.github.com/users/ioana-blue/received_events
https://api.github.com/users/ioana-blue/repos
https://api.github.com/users/ioana-blue/starred{/owner}{/repo}
https://api.github.com/users/ioana-blue/subscriptions
User
https://api.github.com/users/ioana-blue
public
NONE
And to clarify, it's not memory, it's disk space. Thank you!
2021-03-03T15:21:45Z
https://github.com/huggingface/datasets/issues/1973#issuecomment-789793363
https://api.github.com/repos/huggingface/datasets/issues/1973
MDEyOklzc3VlQ29tbWVudDc4OTc5MzM2Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/789793363/reactions
2021-03-03T15:21:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/789793363
https://avatars.githubusercontent.com/u/17202292?v=4
https://api.github.com/users/ioana-blue/events{/privacy}
https://api.github.com/users/ioana-blue/followers
https://api.github.com/users/ioana-blue/following{/other_user}
https://api.github.com/users/ioana-blue/gists{/gist_id}
https://github.com/ioana-blue
ioana-blue
MDQ6VXNlcjE3MjAyMjky
https://api.github.com/users/ioana-blue/orgs
https://api.github.com/users/ioana-blue/received_events
https://api.github.com/users/ioana-blue/repos
https://api.github.com/users/ioana-blue/starred{/owner}{/repo}
https://api.github.com/users/ioana-blue/subscriptions
User
https://api.github.com/users/ioana-blue
public
MEMBER
Hi ! As Albert said they can sometimes take more space that expected but we'll fix that soon.

Also, to give more details about caching: computations on a dataset are cached by default so that you don't have to recompute them the next time you run them.

So by default the cache files stay on your disk when you job is finished (so that if you re-execute it, it will be reloaded from the cache).
Feel free to clear your cache after your job has finished, or disable caching using
```python
import datasets

datasets.set_caching_enabled(False)
```
2021-03-03T16:25:11Z
https://github.com/huggingface/datasets/issues/1973#issuecomment-789842110
https://api.github.com/repos/huggingface/datasets/issues/1973
MDEyOklzc3VlQ29tbWVudDc4OTg0MjExMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/789842110/reactions
2021-03-03T16:25:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/789842110
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for the tip, this is useful. 
2021-03-03T17:28:16Z
https://github.com/huggingface/datasets/issues/1973#issuecomment-789909461
https://api.github.com/repos/huggingface/datasets/issues/1973
MDEyOklzc3VlQ29tbWVudDc4OTkwOTQ2MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/789909461/reactions
2021-03-03T17:28:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/789909461
https://avatars.githubusercontent.com/u/17202292?v=4
https://api.github.com/users/ioana-blue/events{/privacy}
https://api.github.com/users/ioana-blue/followers
https://api.github.com/users/ioana-blue/following{/other_user}
https://api.github.com/users/ioana-blue/gists{/gist_id}
https://github.com/ioana-blue
ioana-blue
MDQ6VXNlcjE3MjAyMjky
https://api.github.com/users/ioana-blue/orgs
https://api.github.com/users/ioana-blue/received_events
https://api.github.com/users/ioana-blue/repos
https://api.github.com/users/ioana-blue/starred{/owner}{/repo}
https://api.github.com/users/ioana-blue/subscriptions
User
https://api.github.com/users/ioana-blue
public
MEMBER
Hi @ioana-blue, we have optimized Datasets' disk usage in the latest release v1.5.

Feel free to update your Datasets version
```shell
pip install -U datasets
```
and see if it better suits your needs.
2021-03-22T11:59:50Z
https://github.com/huggingface/datasets/issues/1973#issuecomment-804008018
https://api.github.com/repos/huggingface/datasets/issues/1973
MDEyOklzc3VlQ29tbWVudDgwNDAwODAxOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/804008018/reactions
2021-03-22T11:59:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/804008018
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Thank you!
2021-03-22T20:29:00Z
https://github.com/huggingface/datasets/issues/1973#issuecomment-804372085
https://api.github.com/repos/huggingface/datasets/issues/1973
MDEyOklzc3VlQ29tbWVudDgwNDM3MjA4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/804372085/reactions
2021-03-22T20:29:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/804372085
https://avatars.githubusercontent.com/u/17202292?v=4
https://api.github.com/users/ioana-blue/events{/privacy}
https://api.github.com/users/ioana-blue/followers
https://api.github.com/users/ioana-blue/following{/other_user}
https://api.github.com/users/ioana-blue/gists{/gist_id}
https://github.com/ioana-blue
ioana-blue
MDQ6VXNlcjE3MjAyMjky
https://api.github.com/users/ioana-blue/orgs
https://api.github.com/users/ioana-blue/received_events
https://api.github.com/users/ioana-blue/repos
https://api.github.com/users/ioana-blue/starred{/owner}{/repo}
https://api.github.com/users/ioana-blue/subscriptions
User
https://api.github.com/users/ioana-blue
public
I'm running several training jobs (around 10) with a relatively large dataset (3M samples). The datasets cache reached 178G and it seems really large. What is it stored in there and why is it so large? I don't think I noticed this problem before and seems to be related to the new version of the datasets library. Any insight? Thank you!