Add option to delete temporary files (e.g. extracted files) when loading dataset
MEMBER
Hi !
If we want something more general, we could either
1. delete the extracted files after the arrow data generation automatically, or 
2. delete each extracted file during the arrow generation right after it has been closed.

Solution 2 is better to save disk space during the arrow generation. Is it what you had in mind ?

The API could look like
```python
load_dataset(..., delete_extracted_files_after_usage=True)
```

In terms of implementation, here are some directions we could take for each solution:
1. get the list of the extracted files from the DownloadManager and then delete them after the dataset is processed. This can be implemented in `download_and_prepare` I guess
2. maybe wrap and mock `open` in the builder to make it delete the file when the file is closed.
2021-07-07T08:18:29Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-875395020
https://api.github.com/repos/huggingface/datasets/issues/2604
MDEyOklzc3VlQ29tbWVudDg3NTM5NTAyMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/875395020/reactions
2021-07-07T08:18:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/875395020
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Also, if I delete the extracted files they need to be re-extracted again instead of loading from the Arrow cache files
2021-07-07T08:20:11Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-875396341
https://api.github.com/repos/huggingface/datasets/issues/2604
MDEyOklzc3VlQ29tbWVudDg3NTM5NjM0MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/875396341/reactions
2021-07-07T08:20:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/875396341
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
I think we already opened an issue about this topic (suggested by @stas00): duplicated of #2481?

This is in our TODO list... 😅 
2021-07-07T09:06:33Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-875430175
https://api.github.com/repos/huggingface/datasets/issues/2604
MDEyOklzc3VlQ29tbWVudDg3NTQzMDE3NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/875430175/reactions
2021-07-07T09:06:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/875430175
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I think the deletion of each extracted file could be implemented in our CacheManager and ExtractManager (once merged to master: #2295, #2277). 😉 
2021-07-07T09:09:57Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-875433038
https://api.github.com/repos/huggingface/datasets/issues/2604
MDEyOklzc3VlQ29tbWVudDg3NTQzMzAzOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/875433038/reactions
2021-07-07T09:16:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/875433038
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Oh yes sorry, I didn't check if this was a duplicate
2021-07-07T10:15:12Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-875479743
https://api.github.com/repos/huggingface/datasets/issues/2604
MDEyOklzc3VlQ29tbWVudDg3NTQ3OTc0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/875479743/reactions
2021-07-07T10:15:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/875479743
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Nevermind @thomwolf, I just mentioned the other issue so that both appear linked in GitHub and we do not forget to close both once we make the corresponding Pull Request... That was the main reason! 😄 
2021-07-07T10:53:28Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-875504467
https://api.github.com/repos/huggingface/datasets/issues/2604
MDEyOklzc3VlQ29tbWVudDg3NTUwNDQ2Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/875504467/reactions
2021-07-07T10:53:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/875504467
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Ok yes. I think this is an important feature to be able to use large datasets which are pretty much always compressed files.

In particular now this requires to keep the extracted file on the drive if you want to avoid reprocessing the dataset so in my case, this require using always ~400GB of drive instead of just 200GB (which is already significant). 

Two nice features would be to:
- allow to delete the extracted files without loosing the ability to load the dataset from the cached arrow-file
- streamlined decompression when only the currently read file is extracted - this might require to read the list of files from the extracted archives before processing them?
2021-07-07T11:10:09Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-875514721
https://api.github.com/repos/huggingface/datasets/issues/2604
MDEyOklzc3VlQ29tbWVudDg3NTUxNDcyMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/875514721/reactions
2021-07-07T11:10:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/875514721
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Here is a sample dataset with 2 such large compressed JSON files for debugging: https://huggingface.co/datasets/thomwolf/github-python
2021-07-07T11:57:35Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-875542618
https://api.github.com/repos/huggingface/datasets/issues/2604
MDEyOklzc3VlQ29tbWVudDg3NTU0MjYxOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/875542618/reactions
2021-07-07T11:57:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/875542618
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Note that I'm confirming that with the current master branch of dataset, deleting extracted files (without deleting the arrow cache file) lead to **re-extracting** these files when reloading the dataset instead of directly loading the arrow cache file.
2021-07-14T16:45:33Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-880047800
https://api.github.com/repos/huggingface/datasets/issues/2604
MDEyOklzc3VlQ29tbWVudDg4MDA0NzgwMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/880047800/reactions
2021-07-14T16:46:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/880047800
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Hi ! That's weird, it doesn't do that on my side (tested on master on my laptop by deleting the `extracted` folder in the download cache directory). You tested with one of the files at https://huggingface.co/datasets/thomwolf/github-python that you have locally ?
2021-07-15T15:58:48Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-880814970
https://api.github.com/repos/huggingface/datasets/issues/2604
MDEyOklzc3VlQ29tbWVudDg4MDgxNDk3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/880814970/reactions
2021-07-15T15:58:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/880814970
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Yes it’s when I load local compressed JSON line files with load_dataset(‘json’, data_files=…) 
2021-07-15T17:23:51Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-880879263
https://api.github.com/repos/huggingface/datasets/issues/2604
MDEyOklzc3VlQ29tbWVudDg4MDg3OTI2Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/880879263/reactions
2021-07-15T17:23:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/880879263
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
@thomwolf I'm sorry but I can't reproduce this problem. I'm also using: 
```python
ds = load_dataset("json", split="train", data_files=data_files, cache_dir=cache_dir)
```
after having removed the extracted files:
```python
assert sorted((cache_dir / "downloads" / "extracted").iterdir()) == []
```

I get the logging message:
```shell
WARNING  datasets.builder:builder.py:531 Reusing dataset json ...
```
2021-07-16T15:02:36Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-881515280
https://api.github.com/repos/huggingface/datasets/issues/2604
IC_kwDODunzps40itsQ
https://api.github.com/repos/huggingface/datasets/issues/comments/881515280/reactions
2021-07-16T15:02:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/881515280
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Do you confirm the extracted folder stays empty after reloading?
2021-07-16T15:50:49Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-881546755
https://api.github.com/repos/huggingface/datasets/issues/2604
IC_kwDODunzps40i1YD
https://api.github.com/repos/huggingface/datasets/issues/comments/881546755/reactions
2021-07-16T15:50:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/881546755
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
> 
> 
> Do you confirm the extracted folder stays empty after reloading?

Yes, I have the above mentioned assertion on the emptiness of the extracted folder:
```python
assert sorted((cache_dir / "downloads" / "extracted").iterdir()) == []
```

2021-07-16T17:16:06Z
https://github.com/huggingface/datasets/issues/2604#issuecomment-881597915
https://api.github.com/repos/huggingface/datasets/issues/2604
IC_kwDODunzps40jB3b
https://api.github.com/repos/huggingface/datasets/issues/comments/881597915/reactions
2021-07-16T17:16:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/881597915
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
I'm loading a dataset constituted of 44 GB of compressed JSON files.

When loading the dataset with the JSON script, extracting the files create about 200 GB of uncompressed files before creating the 180GB of arrow cache tables

Having a simple way to delete the extracted files after usage (or even better, to stream extraction/delete) would be nice to avoid disk cluter.

I can maybe tackle this one in the JSON script unless you want a more general solution.