Bug: Can't download TriviaQA with `load_dataset` - custom `cache_dir`
CONTRIBUTOR
Hi @SapirWeissbuch,
When you are saying it freezes, at that time it is unzipping the file from the zip file it downloaded. Since it's a very heavy file it'll take some time. It was taking ~11GB after unzipping when it started reading examples for me. Hope that helps!
![Screenshot 2020-12-21 at 23 40 52](https://user-images.githubusercontent.com/19718818/102808355-3b380c00-43e6-11eb-81ab-c31019ae6322.png)

2020-12-21T18:20:37Z
https://github.com/huggingface/datasets/issues/1615#issuecomment-749122572
https://api.github.com/repos/huggingface/datasets/issues/1615
MDEyOklzc3VlQ29tbWVudDc0OTEyMjU3Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/749122572/reactions
2020-12-21T18:20:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749122572
https://avatars.githubusercontent.com/u/19718818?v=4
https://api.github.com/users/bhavitvyamalik/events{/privacy}
https://api.github.com/users/bhavitvyamalik/followers
https://api.github.com/users/bhavitvyamalik/following{/other_user}
https://api.github.com/users/bhavitvyamalik/gists{/gist_id}
https://github.com/bhavitvyamalik
bhavitvyamalik
MDQ6VXNlcjE5NzE4ODE4
https://api.github.com/users/bhavitvyamalik/orgs
https://api.github.com/users/bhavitvyamalik/received_events
https://api.github.com/users/bhavitvyamalik/repos
https://api.github.com/users/bhavitvyamalik/starred{/owner}{/repo}
https://api.github.com/users/bhavitvyamalik/subscriptions
User
https://api.github.com/users/bhavitvyamalik
public
NONE
Hi @bhavitvyamalik 
Thanks for the reply!
Actually I let it run for 30 minutes before I killed the process. In this time, 30GB were extracted (much more than 11GB), I checked the size of the destination directory.

What version of Datasets are you using?

2020-12-21T18:28:13Z
https://github.com/huggingface/datasets/issues/1615#issuecomment-749125751
https://api.github.com/repos/huggingface/datasets/issues/1615
MDEyOklzc3VlQ29tbWVudDc0OTEyNTc1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/749125751/reactions
2020-12-21T18:28:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749125751
https://avatars.githubusercontent.com/u/44585792?v=4
https://api.github.com/users/SapirWeissbuch/events{/privacy}
https://api.github.com/users/SapirWeissbuch/followers
https://api.github.com/users/SapirWeissbuch/following{/other_user}
https://api.github.com/users/SapirWeissbuch/gists{/gist_id}
https://github.com/SapirWeissbuch
SapirWeissbuch
MDQ6VXNlcjQ0NTg1Nzky
https://api.github.com/users/SapirWeissbuch/orgs
https://api.github.com/users/SapirWeissbuch/received_events
https://api.github.com/users/SapirWeissbuch/repos
https://api.github.com/users/SapirWeissbuch/starred{/owner}{/repo}
https://api.github.com/users/SapirWeissbuch/subscriptions
User
https://api.github.com/users/SapirWeissbuch
public
CONTRIBUTOR
I'm using datasets version: 1.1.3. I think you should drop `cache_dir` and use only
`dataset = datasets.load_dataset("trivia_qa", "rc")`

Tried that on colab and it's working there too
![image](https://user-images.githubusercontent.com/19718818/102814269-4db74300-43f0-11eb-8f26-ecfcf4632002.png)

2020-12-21T19:25:46Z
https://github.com/huggingface/datasets/issues/1615#issuecomment-749154303
https://api.github.com/repos/huggingface/datasets/issues/1615
MDEyOklzc3VlQ29tbWVudDc0OTE1NDMwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/749154303/reactions
2020-12-21T19:25:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749154303
https://avatars.githubusercontent.com/u/19718818?v=4
https://api.github.com/users/bhavitvyamalik/events{/privacy}
https://api.github.com/users/bhavitvyamalik/followers
https://api.github.com/users/bhavitvyamalik/following{/other_user}
https://api.github.com/users/bhavitvyamalik/gists{/gist_id}
https://github.com/bhavitvyamalik
bhavitvyamalik
MDQ6VXNlcjE5NzE4ODE4
https://api.github.com/users/bhavitvyamalik/orgs
https://api.github.com/users/bhavitvyamalik/received_events
https://api.github.com/users/bhavitvyamalik/repos
https://api.github.com/users/bhavitvyamalik/starred{/owner}{/repo}
https://api.github.com/users/bhavitvyamalik/subscriptions
User
https://api.github.com/users/bhavitvyamalik
public
CONTRIBUTOR
Train, Validation, and Test splits contain 138384, 18669, and 17210 samples respectively. It takes some time to read the samples. Even in your colab notebook it was reading the samples before you killed the process. Let me know if it works now!
2020-12-21T19:48:33Z
https://github.com/huggingface/datasets/issues/1615#issuecomment-749165276
https://api.github.com/repos/huggingface/datasets/issues/1615
MDEyOklzc3VlQ29tbWVudDc0OTE2NTI3Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/749165276/reactions
2020-12-21T19:48:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749165276
https://avatars.githubusercontent.com/u/19718818?v=4
https://api.github.com/users/bhavitvyamalik/events{/privacy}
https://api.github.com/users/bhavitvyamalik/followers
https://api.github.com/users/bhavitvyamalik/following{/other_user}
https://api.github.com/users/bhavitvyamalik/gists{/gist_id}
https://github.com/bhavitvyamalik
bhavitvyamalik
MDQ6VXNlcjE5NzE4ODE4
https://api.github.com/users/bhavitvyamalik/orgs
https://api.github.com/users/bhavitvyamalik/received_events
https://api.github.com/users/bhavitvyamalik/repos
https://api.github.com/users/bhavitvyamalik/starred{/owner}{/repo}
https://api.github.com/users/bhavitvyamalik/subscriptions
User
https://api.github.com/users/bhavitvyamalik
public
NONE
Hi, it works on colab but it still doesn't work on my computer, same problem as before - overly large and long extraction process.
I have to use a custom 'cache_dir' because I don't have any space left in my home directory where it is defaulted, maybe this could be the issue?
2020-12-21T21:44:31Z
https://github.com/huggingface/datasets/issues/1615#issuecomment-749213150
https://api.github.com/repos/huggingface/datasets/issues/1615
MDEyOklzc3VlQ29tbWVudDc0OTIxMzE1MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/749213150/reactions
2020-12-21T21:46:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749213150
https://avatars.githubusercontent.com/u/44585792?v=4
https://api.github.com/users/SapirWeissbuch/events{/privacy}
https://api.github.com/users/SapirWeissbuch/followers
https://api.github.com/users/SapirWeissbuch/following{/other_user}
https://api.github.com/users/SapirWeissbuch/gists{/gist_id}
https://github.com/SapirWeissbuch
SapirWeissbuch
MDQ6VXNlcjQ0NTg1Nzky
https://api.github.com/users/SapirWeissbuch/orgs
https://api.github.com/users/SapirWeissbuch/received_events
https://api.github.com/users/SapirWeissbuch/repos
https://api.github.com/users/SapirWeissbuch/starred{/owner}{/repo}
https://api.github.com/users/SapirWeissbuch/subscriptions
User
https://api.github.com/users/SapirWeissbuch
public
NONE
I tried running this again - More details of the problem:
Code:
```
datasets.load_dataset("trivia_qa", "rc", cache_dir="/path/to/cache")
```

The output:
```
Downloading and preparing dataset trivia_qa/rc (download: 2.48 GiB, generated: 14.92 GiB, post-processed: Unknown size, total: 17.40 GiB) to path/to/cache/trivia_qa/rc/1.1.0/e734e28133f4d9a353af322aa52b9f266f6f27cbf2f072690a1694e577546b0d...                                                                          
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.67G/2.67G [03:38<00:00, 12.2MB/s]

```
The process continues (no progress bar is visible).
I tried `du -sh .` in `path/to/cache`, and the size keeps increasing, reached 35G before I killed the process.

Google Colab with custom `cache_dir` has same issue.
https://colab.research.google.com/drive/1nn1Lw02GhfGFylzbS2j6yksGjPo7kkN-?usp=sharing#scrollTo=2G2O0AeNIXan
2020-12-21T22:52:13Z
https://github.com/huggingface/datasets/issues/1615#issuecomment-749241097
https://api.github.com/repos/huggingface/datasets/issues/1615
MDEyOklzc3VlQ29tbWVudDc0OTI0MTA5Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/749241097/reactions
2020-12-21T23:42:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749241097
https://avatars.githubusercontent.com/u/44585792?v=4
https://api.github.com/users/SapirWeissbuch/events{/privacy}
https://api.github.com/users/SapirWeissbuch/followers
https://api.github.com/users/SapirWeissbuch/following{/other_user}
https://api.github.com/users/SapirWeissbuch/gists{/gist_id}
https://github.com/SapirWeissbuch
SapirWeissbuch
MDQ6VXNlcjQ0NTg1Nzky
https://api.github.com/users/SapirWeissbuch/orgs
https://api.github.com/users/SapirWeissbuch/received_events
https://api.github.com/users/SapirWeissbuch/repos
https://api.github.com/users/SapirWeissbuch/starred{/owner}{/repo}
https://api.github.com/users/SapirWeissbuch/subscriptions
User
https://api.github.com/users/SapirWeissbuch
public
CONTRIBUTOR
1) You can clear the huggingface folder in your `.cache` directory to use default directory for datasets. Speed of extraction and loading of samples depends a lot on your machine's configurations too.

2) I tried on colab `dataset = datasets.load_dataset("trivia_qa", "rc", cache_dir = "./datasets")`. After memory usage reached around 42GB (starting from 32GB used already), the dataset was loaded in the memory. Even Your colab notebook shows 
![image](https://user-images.githubusercontent.com/19718818/102852229-c7c4e780-4443-11eb-91d6-bf21024358a3.png)
which means it's loaded now.
2020-12-22T05:24:05Z
https://github.com/huggingface/datasets/issues/1615#issuecomment-749345832
https://api.github.com/repos/huggingface/datasets/issues/1615
MDEyOklzc3VlQ29tbWVudDc0OTM0NTgzMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/749345832/reactions
2020-12-22T05:24:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749345832
https://avatars.githubusercontent.com/u/19718818?v=4
https://api.github.com/users/bhavitvyamalik/events{/privacy}
https://api.github.com/users/bhavitvyamalik/followers
https://api.github.com/users/bhavitvyamalik/following{/other_user}
https://api.github.com/users/bhavitvyamalik/gists{/gist_id}
https://github.com/bhavitvyamalik
bhavitvyamalik
MDQ6VXNlcjE5NzE4ODE4
https://api.github.com/users/bhavitvyamalik/orgs
https://api.github.com/users/bhavitvyamalik/received_events
https://api.github.com/users/bhavitvyamalik/repos
https://api.github.com/users/bhavitvyamalik/starred{/owner}{/repo}
https://api.github.com/users/bhavitvyamalik/subscriptions
User
https://api.github.com/users/bhavitvyamalik
public
NONE
Facing the same issue.
I am able to download datasets without `cache_dir`, however, when I specify the `cache_dir`, the process hangs indefinitely after partial download. 
Tried for `data = load_dataset("cnn_dailymail", "3.0.0")`
2021-06-25T11:46:40Z
https://github.com/huggingface/datasets/issues/1615#issuecomment-868441888
https://api.github.com/repos/huggingface/datasets/issues/1615
MDEyOklzc3VlQ29tbWVudDg2ODQ0MTg4OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/868441888/reactions
2021-06-25T11:46:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/868441888
https://avatars.githubusercontent.com/u/2375919?v=4
https://api.github.com/users/ashutoshml/events{/privacy}
https://api.github.com/users/ashutoshml/followers
https://api.github.com/users/ashutoshml/following{/other_user}
https://api.github.com/users/ashutoshml/gists{/gist_id}
https://github.com/ashutoshml
ashutoshml
MDQ6VXNlcjIzNzU5MTk=
https://api.github.com/users/ashutoshml/orgs
https://api.github.com/users/ashutoshml/received_events
https://api.github.com/users/ashutoshml/repos
https://api.github.com/users/ashutoshml/starred{/owner}{/repo}
https://api.github.com/users/ashutoshml/subscriptions
User
https://api.github.com/users/ashutoshml
public
CONTRIBUTOR
Hi @ashutoshml,
I tried this and it worked for me:
`data = load_dataset("cnn_dailymail", "3.0.0", cache_dir="./dummy")`

I'm using datasets==1.8.0. It took around 3-4 mins for dataset to unpack and start loading examples.
2021-06-25T12:16:06Z
https://github.com/huggingface/datasets/issues/1615#issuecomment-868457855
https://api.github.com/repos/huggingface/datasets/issues/1615
MDEyOklzc3VlQ29tbWVudDg2ODQ1Nzg1NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/868457855/reactions
2021-06-25T12:16:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/868457855
https://avatars.githubusercontent.com/u/19718818?v=4
https://api.github.com/users/bhavitvyamalik/events{/privacy}
https://api.github.com/users/bhavitvyamalik/followers
https://api.github.com/users/bhavitvyamalik/following{/other_user}
https://api.github.com/users/bhavitvyamalik/gists{/gist_id}
https://github.com/bhavitvyamalik
bhavitvyamalik
MDQ6VXNlcjE5NzE4ODE4
https://api.github.com/users/bhavitvyamalik/orgs
https://api.github.com/users/bhavitvyamalik/received_events
https://api.github.com/users/bhavitvyamalik/repos
https://api.github.com/users/bhavitvyamalik/starred{/owner}{/repo}
https://api.github.com/users/bhavitvyamalik/subscriptions
User
https://api.github.com/users/bhavitvyamalik
public
NONE
Ok. I waited for 20-30 mins, and it still is stuck.
I am using datasets==1.8.0.

Is there anyway to check what is happening? like a` --verbose` flag?

![Screenshot 2021-06-25 at 6 37 43 PM](https://user-images.githubusercontent.com/2375919/123429653-cdfb7280-d5e4-11eb-9fa7-ff295800cc86.png)

2021-06-25T13:10:38Z
https://github.com/huggingface/datasets/issues/1615#issuecomment-868488917
https://api.github.com/repos/huggingface/datasets/issues/1615
MDEyOklzc3VlQ29tbWVudDg2ODQ4ODkxNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/868488917/reactions
2021-06-25T13:11:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/868488917
https://avatars.githubusercontent.com/u/2375919?v=4
https://api.github.com/users/ashutoshml/events{/privacy}
https://api.github.com/users/ashutoshml/followers
https://api.github.com/users/ashutoshml/following{/other_user}
https://api.github.com/users/ashutoshml/gists{/gist_id}
https://github.com/ashutoshml
ashutoshml
MDQ6VXNlcjIzNzU5MTk=
https://api.github.com/users/ashutoshml/orgs
https://api.github.com/users/ashutoshml/received_events
https://api.github.com/users/ashutoshml/repos
https://api.github.com/users/ashutoshml/starred{/owner}{/repo}
https://api.github.com/users/ashutoshml/subscriptions
User
https://api.github.com/users/ashutoshml
public
Hello,
I'm having issue downloading TriviaQA dataset with `load_dataset`.

## Environment info
- `datasets` version: 1.1.3
- Platform: Linux-4.19.129-aufs-1-x86_64-with-debian-10.1
- Python version: 3.7.3

## The code I'm running:
```python
import datasets
dataset = datasets.load_dataset("trivia_qa", "rc", cache_dir = "./datasets")
```

## The output:
1. Download begins:
```
Downloading and preparing dataset trivia_qa/rc (download: 2.48 GiB, generated: 14.92 GiB, post-processed: Unknown size, total: 17.40 GiB) to /cs/labs/gabis/sapirweissbuch/tr
ivia_qa/rc/1.1.0/e734e28133f4d9a353af322aa52b9f266f6f27cbf2f072690a1694e577546b0d...                                                                                         
Downloading:  17%|███████████████████▉                                                                                                   | 446M/2.67G [00:37<04:45, 7.77MB/s]
```
2. 100% is reached
3. It got stuck here for about an hour, and added additional 30G of data to "./datasets" directory. I killed the process eventually.

A similar issue can be observed in Google Colab:

https://colab.research.google.com/drive/1nn1Lw02GhfGFylzbS2j6yksGjPo7kkN-?usp=sharing

## Expected behaviour:
The dataset "TriviaQA" should be successfully downloaded.
