Caching processed dataset at wrong folder
MEMBER
Thanks for reporting !
It uses a temporary file to write the data.
However it looks like the temporary file is not placed in the right directory during the processing
2020-09-19T16:49:40Z
https://github.com/huggingface/datasets/issues/643#issuecomment-695330945
https://api.github.com/repos/huggingface/datasets/issues/643
MDEyOklzc3VlQ29tbWVudDY5NTMzMDk0NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/695330945/reactions
2020-09-19T16:49:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/695330945
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Well actually I just tested and the temporary file is placed in the same directory, so it should work as expected.
Which version of `datasets` are you using ?
2020-09-21T15:03:19Z
https://github.com/huggingface/datasets/issues/643#issuecomment-696173941
https://api.github.com/repos/huggingface/datasets/issues/643
MDEyOklzc3VlQ29tbWVudDY5NjE3Mzk0MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/696173941/reactions
2020-09-21T15:03:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696173941
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
`datasets-1.0.1`
Here you can reproduce it here:
https://colab.research.google.com/drive/1O0KcepTFsmpkBbrbLLMq42iwTKmQh8d5?usp=sharing

2020-09-21T17:28:21Z
https://github.com/huggingface/datasets/issues/643#issuecomment-696258584
https://api.github.com/repos/huggingface/datasets/issues/643
MDEyOklzc3VlQ29tbWVudDY5NjI1ODU4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/696258584/reactions
2020-09-21T17:28:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696258584
https://avatars.githubusercontent.com/u/3653789?v=4
https://api.github.com/users/mrm8488/events{/privacy}
https://api.github.com/users/mrm8488/followers
https://api.github.com/users/mrm8488/following{/other_user}
https://api.github.com/users/mrm8488/gists{/gist_id}
https://github.com/mrm8488
mrm8488
MDQ6VXNlcjM2NTM3ODk=
https://api.github.com/users/mrm8488/orgs
https://api.github.com/users/mrm8488/received_events
https://api.github.com/users/mrm8488/repos
https://api.github.com/users/mrm8488/starred{/owner}{/repo}
https://api.github.com/users/mrm8488/subscriptions
User
https://api.github.com/users/mrm8488
public
MEMBER
It looks like a pyarrow issue with google colab.
For some reason this code increases the disk usage of google colab while it actually writes into google drive:

```python
import pyarrow as pa

stream = pa.OSFile("/content/drive/My Drive/path/to/file.arrow", "wb")
writer = pa.RecordBatchStreamWriter(stream, schema=pa.schema({"text": pa.string()}))
writer.write_table(pa.Table.from_pydict({"text": ["a"*511 + "\n"] * ((1 << 30) // 512)}))  # 1GiB
writer.close()
stream.close()
```

Moreover if I `rm` the file on google drive, it frees disk space on google colab.
2020-09-22T14:41:47Z
https://github.com/huggingface/datasets/issues/643#issuecomment-696767016
https://api.github.com/repos/huggingface/datasets/issues/643
MDEyOklzc3VlQ29tbWVudDY5Njc2NzAxNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/696767016/reactions
2020-09-22T14:41:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696767016
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
It looks like replacing `pa.OSFile` by `open` fixes it, I'm going to open a PR
2020-09-22T14:57:48Z
https://github.com/huggingface/datasets/issues/643#issuecomment-696777167
https://api.github.com/repos/huggingface/datasets/issues/643
MDEyOklzc3VlQ29tbWVudDY5Njc3NzE2Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/696777167/reactions
2020-09-22T14:57:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696777167
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Ok. Thank you so much!
2020-09-22T14:59:35Z
https://github.com/huggingface/datasets/issues/643#issuecomment-696778317
https://api.github.com/repos/huggingface/datasets/issues/643
MDEyOklzc3VlQ29tbWVudDY5Njc3ODMxNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/696778317/reactions
2020-09-22T14:59:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696778317
https://avatars.githubusercontent.com/u/3653789?v=4
https://api.github.com/users/mrm8488/events{/privacy}
https://api.github.com/users/mrm8488/followers
https://api.github.com/users/mrm8488/following{/other_user}
https://api.github.com/users/mrm8488/gists{/gist_id}
https://github.com/mrm8488
mrm8488
MDQ6VXNlcjM2NTM3ODk=
https://api.github.com/users/mrm8488/orgs
https://api.github.com/users/mrm8488/received_events
https://api.github.com/users/mrm8488/repos
https://api.github.com/users/mrm8488/starred{/owner}{/repo}
https://api.github.com/users/mrm8488/subscriptions
User
https://api.github.com/users/mrm8488
public
MEMBER
Actually I did more tests it doesn't >.<
I'll let you know if I find a way to fix that
2020-09-22T15:16:41Z
https://github.com/huggingface/datasets/issues/643#issuecomment-696789087
https://api.github.com/repos/huggingface/datasets/issues/643
MDEyOklzc3VlQ29tbWVudDY5Njc4OTA4Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/696789087/reactions
2020-09-22T15:16:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696789087
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Actually I also have the issue when writing a regular text file

```python
f = open("/content/drive/My Drive/path/to/file", "w")
f.write(("a"*511 + "\n") * ((1 << 30) // 512))  # 1GiB
f.close()
```

Is that supposed to happen ?
2020-09-22T15:26:42Z
https://github.com/huggingface/datasets/issues/643#issuecomment-696795157
https://api.github.com/repos/huggingface/datasets/issues/643
MDEyOklzc3VlQ29tbWVudDY5Njc5NTE1Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/696795157/reactions
2020-09-22T15:28:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/696795157
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
The code you wrote should write a 1GB file in the Google Drive folder. Doesn't it? 
2020-09-24T01:11:54Z
https://github.com/huggingface/datasets/issues/643#issuecomment-698053143
https://api.github.com/repos/huggingface/datasets/issues/643
MDEyOklzc3VlQ29tbWVudDY5ODA1MzE0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/698053143/reactions
2020-09-24T01:11:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/698053143
https://avatars.githubusercontent.com/u/3653789?v=4
https://api.github.com/users/mrm8488/events{/privacy}
https://api.github.com/users/mrm8488/followers
https://api.github.com/users/mrm8488/following{/other_user}
https://api.github.com/users/mrm8488/gists{/gist_id}
https://github.com/mrm8488
mrm8488
MDQ6VXNlcjM2NTM3ODk=
https://api.github.com/users/mrm8488/orgs
https://api.github.com/users/mrm8488/received_events
https://api.github.com/users/mrm8488/repos
https://api.github.com/users/mrm8488/starred{/owner}{/repo}
https://api.github.com/users/mrm8488/subscriptions
User
https://api.github.com/users/mrm8488
public
MEMBER
Yes it does, but the disk usage of google colab also increases by 1GB
2020-09-24T12:53:14Z
https://github.com/huggingface/datasets/issues/643#issuecomment-698324658
https://api.github.com/repos/huggingface/datasets/issues/643
MDEyOklzc3VlQ29tbWVudDY5ODMyNDY1OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/698324658/reactions
2020-09-24T12:53:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/698324658
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I could check it and as you say as I write to te Drive disk the colab disk also increases...
2020-09-28T16:01:59Z
https://github.com/huggingface/datasets/issues/643#issuecomment-700128252
https://api.github.com/repos/huggingface/datasets/issues/643
MDEyOklzc3VlQ29tbWVudDcwMDEyODI1Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/700128252/reactions
2020-09-28T16:01:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/700128252
https://avatars.githubusercontent.com/u/3653789?v=4
https://api.github.com/users/mrm8488/events{/privacy}
https://api.github.com/users/mrm8488/followers
https://api.github.com/users/mrm8488/following{/other_user}
https://api.github.com/users/mrm8488/gists{/gist_id}
https://github.com/mrm8488
mrm8488
MDQ6VXNlcjM2NTM3ODk=
https://api.github.com/users/mrm8488/orgs
https://api.github.com/users/mrm8488/received_events
https://api.github.com/users/mrm8488/repos
https://api.github.com/users/mrm8488/starred{/owner}{/repo}
https://api.github.com/users/mrm8488/subscriptions
User
https://api.github.com/users/mrm8488
public
CONTRIBUTOR
To reproduce it: 
```bash
!df -h | grep sda1
```
```python
f = open("/content/drive/My Drive/test_to_remove.txt", "w")
f.write(("a"*511 + "\n") * ((1 << 30) // 512))  # 1GiB
f.write(("a"*511 + "\n") * ((1 << 30) // 512))  # 1GiB
f.close()
```
```bash
!ls -lh /content/drive/My\ Drive/test_to_remove.txt

!df -h | grep sda1

!rm -rf /content/drive/My\ Drive/test_to_remove.txt

```
[Colab](https://colab.research.google.com/drive/1D0UiweCYQwwWZ65EEhuqqbaDDbhJYXfm?usp=sharing)



2020-09-28T16:14:40Z
https://github.com/huggingface/datasets/issues/643#issuecomment-700135570
https://api.github.com/repos/huggingface/datasets/issues/643
MDEyOklzc3VlQ29tbWVudDcwMDEzNTU3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/700135570/reactions
2020-09-28T16:14:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/700135570
https://avatars.githubusercontent.com/u/3653789?v=4
https://api.github.com/users/mrm8488/events{/privacy}
https://api.github.com/users/mrm8488/followers
https://api.github.com/users/mrm8488/following{/other_user}
https://api.github.com/users/mrm8488/gists{/gist_id}
https://github.com/mrm8488
mrm8488
MDQ6VXNlcjM2NTM3ODk=
https://api.github.com/users/mrm8488/orgs
https://api.github.com/users/mrm8488/received_events
https://api.github.com/users/mrm8488/repos
https://api.github.com/users/mrm8488/starred{/owner}{/repo}
https://api.github.com/users/mrm8488/subscriptions
User
https://api.github.com/users/mrm8488
public
MEMBER
Apparently, Colab uses a local cache of the data files read/written from Google Drive. See:
- https://github.com/googlecolab/colabtools/issues/2087#issuecomment-860818457
- https://github.com/googlecolab/colabtools/issues/1915#issuecomment-804234540
- https://github.com/googlecolab/colabtools/issues/2147#issuecomment-885052636
2022-02-16T14:53:28Z
https://github.com/huggingface/datasets/issues/643#issuecomment-1041574740
https://api.github.com/repos/huggingface/datasets/issues/643
IC_kwDODunzps4-FStU
https://api.github.com/repos/huggingface/datasets/issues/comments/1041574740/reactions
2022-02-16T14:53:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1041574740
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Hi guys, I run this on my Colab (PRO):

```python
from datasets import load_dataset
dataset = load_dataset('text', data_files='/content/corpus.txt', cache_dir='/content/drive/My Drive', split='train')

def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')

dataset = dataset.map(encode, batched=True)
```
The file is about 4 GB, so I cannot process it on the Colab HD because there is no enough space. So I decided to mount my Google Drive fs and do it on it.
The dataset is cached in the right place but by processing it (applying `encode` function) seems to use a different folder because Colab HD starts to grow and it crashes when it should be done in the Drive fs.

What gets me crazy, it prints it is processing/encoding the dataset in the right folder:
```
Testing the mapped function outputs
Testing finished, running the mapping function on the dataset
Caching processed dataset at /content/drive/My Drive/text/default-ad3e69d6242ee916/0.0.0/7e13bc0fa76783d4ef197f079dc8acfe54c3efda980f2c9adfab046ede2f0ff7/cache-b16341780a59747d.arrow
```