datasets generate large arrow file
MEMBER
Hi ! The cache files are the results of all the transforms you applied to the dataset using `map` for example.
Did you run a transform that could potentially blow up the size of the dataset ?
2022-09-06T09:25:50Z
https://github.com/huggingface/datasets/issues/4897#issuecomment-1237895669
https://api.github.com/repos/huggingface/datasets/issues/4897
IC_kwDODunzps5JyMn1
https://api.github.com/repos/huggingface/datasets/issues/comments/1237895669/reactions
2022-09-06T09:25:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1237895669
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq,
I don't remember, but I can't imagine what kind of transform may generate data that grow over 200 times in size. 
I think maybe it doesn' matter, it's just cache after all.
2022-09-18T05:07:52Z
https://github.com/huggingface/datasets/issues/4897#issuecomment-1250192851
https://api.github.com/repos/huggingface/datasets/issues/4897
IC_kwDODunzps5KhG3T
https://api.github.com/repos/huggingface/datasets/issues/comments/1250192851/reactions
2022-09-18T05:07:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1250192851
https://avatars.githubusercontent.com/u/18533904?v=4
https://api.github.com/users/jax11235/events{/privacy}
https://api.github.com/users/jax11235/followers
https://api.github.com/users/jax11235/following{/other_user}
https://api.github.com/users/jax11235/gists{/gist_id}
https://github.com/jax11235
jax11235
MDQ6VXNlcjE4NTMzOTA0
https://api.github.com/users/jax11235/orgs
https://api.github.com/users/jax11235/received_events
https://api.github.com/users/jax11235/repos
https://api.github.com/users/jax11235/starred{/owner}{/repo}
https://api.github.com/users/jax11235/subscriptions
User
https://api.github.com/users/jax11235
public
Checking the large file in disk, and found the large cache file in the cifar10 data directory:

![image](https://user-images.githubusercontent.com/18533904/186830449-ba96cdeb-0fe8-4543-994d-2abe7145933f.png)

As we know, the size of cifar10 dataset is ~130MB, but the cache file has almost 30GB size, there may be some problems here.