cannot save the dataset to disk after rename_column
MEMBER
Hi ! That's because you are trying to overwrite a file that is already open and being used.
Indeed `foo/dataset.arrow` is open and used by your `dataset` object.

When you do `rename_column`, the resulting dataset reads the data from the same arrow file.
In other cases like when using `map` on the other hand, the resulting dataset reads the data from another arrow file that is the result of the map transform.

Therefore overwriting a dataset after `rename_column` is not possible, but it is possible after `map`, since `rename_column` doesn't switch to using another arrow file (the actual data stay the same).
2021-07-21T09:19:51Z
https://github.com/huggingface/datasets/issues/2689#issuecomment-884033171
https://api.github.com/repos/huggingface/datasets/issues/2689
IC_kwDODunzps40sUaT
https://api.github.com/repos/huggingface/datasets/issues/comments/884033171/reactions
2021-07-21T09:19:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/884033171
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Ok, thanks for clearing it up :)
2021-07-21T13:11:04Z
https://github.com/huggingface/datasets/issues/2689#issuecomment-884177858
https://api.github.com/repos/huggingface/datasets/issues/2689
IC_kwDODunzps40s3vC
https://api.github.com/repos/huggingface/datasets/issues/comments/884177858/reactions
2021-07-21T13:11:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/884177858
https://avatars.githubusercontent.com/u/25532159?v=4
https://api.github.com/users/PaulLerner/events{/privacy}
https://api.github.com/users/PaulLerner/followers
https://api.github.com/users/PaulLerner/following{/other_user}
https://api.github.com/users/PaulLerner/gists{/gist_id}
https://github.com/PaulLerner
PaulLerner
MDQ6VXNlcjI1NTMyMTU5
https://api.github.com/users/PaulLerner/orgs
https://api.github.com/users/PaulLerner/received_events
https://api.github.com/users/PaulLerner/repos
https://api.github.com/users/PaulLerner/starred{/owner}{/repo}
https://api.github.com/users/PaulLerner/subscriptions
User
https://api.github.com/users/PaulLerner
public
NONE
so what would be the right way to read a dataset, then change something and then overwrite it with the new version?
2023-10-22T13:43:32Z
https://github.com/huggingface/datasets/issues/2689#issuecomment-1774099529
https://api.github.com/repos/huggingface/datasets/issues/2689
IC_kwDODunzps5pvpxJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1774099529/reactions
2023-10-22T13:43:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1774099529
https://avatars.githubusercontent.com/u/11448010?v=4
https://api.github.com/users/sd3ntato/events{/privacy}
https://api.github.com/users/sd3ntato/followers
https://api.github.com/users/sd3ntato/following{/other_user}
https://api.github.com/users/sd3ntato/gists{/gist_id}
https://github.com/sd3ntato
sd3ntato
MDQ6VXNlcjExNDQ4MDEw
https://api.github.com/users/sd3ntato/orgs
https://api.github.com/users/sd3ntato/received_events
https://api.github.com/users/sd3ntato/repos
https://api.github.com/users/sd3ntato/starred{/owner}{/repo}
https://api.github.com/users/sd3ntato/subscriptions
User
https://api.github.com/users/sd3ntato
public
MEMBER
You have to write to a new directory, then delete the former directory, and finally rename the new directory.
2023-11-02T14:53:59Z
https://github.com/huggingface/datasets/issues/2689#issuecomment-1790889981
https://api.github.com/repos/huggingface/datasets/issues/2689
IC_kwDODunzps5qvs_9
https://api.github.com/repos/huggingface/datasets/issues/comments/1790889981/reactions
2023-11-02T14:53:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1790889981
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
This worked for me; I know it's hacky but it works:

```
ds.add_column...
ds = Dataset.from_dict(ds.to_dict())
ds.save_to_disk...
```
2025-02-11T23:23:16Z
https://github.com/huggingface/datasets/issues/2689#issuecomment-2652284167
https://api.github.com/repos/huggingface/datasets/issues/2689
IC_kwDODunzps6eFqUH
https://api.github.com/repos/huggingface/datasets/issues/comments/2652284167/reactions
2025-02-11T23:23:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2652284167
https://avatars.githubusercontent.com/u/13894030?v=4
https://api.github.com/users/dipta007/events{/privacy}
https://api.github.com/users/dipta007/followers
https://api.github.com/users/dipta007/following{/other_user}
https://api.github.com/users/dipta007/gists{/gist_id}
https://github.com/dipta007
dipta007
MDQ6VXNlcjEzODk0MDMw
https://api.github.com/users/dipta007/orgs
https://api.github.com/users/dipta007/received_events
https://api.github.com/users/dipta007/repos
https://api.github.com/users/dipta007/starred{/owner}{/repo}
https://api.github.com/users/dipta007/subscriptions
User
https://api.github.com/users/dipta007
public
## Describe the bug
If you use `rename_column` and do no other modification, you will be unable to save the dataset using `save_to_disk`

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
In [1]: from datasets import Dataset, load_from_disk
In [5]: dataset=Dataset.from_dict({'foo': [0]})
In [7]: dataset.save_to_disk('foo')
In [8]: dataset=load_from_disk('foo')
In [10]: dataset=dataset.rename_column('foo', 'bar')
In [11]: dataset.save_to_disk('foo')
---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
<ipython-input-11-a3bc0d4fc339> in <module>
----> 1 dataset.save_to_disk('foo')

/mnt/beegfs/projects/meerqat/anaconda3/envs/meerqat/lib/python3.7/site-packages/datasets/arrow_dataset.py in save_to_disk(self, dataset_path
, fs)
    597             if Path(dataset_path, config.DATASET_ARROW_FILENAME) in cache_files_paths:
    598                 raise PermissionError(
--> 599                     f"Tried to overwrite {Path(dataset_path, config.DATASET_ARROW_FILENAME)} but a dataset can't overwrite itself."
    600                 )
    601             if Path(dataset_path, config.DATASET_INDICES_FILENAME) in cache_files_paths:

PermissionError: Tried to overwrite foo/dataset.arrow but a dataset can't overwrite itself.
```

N. B. I created the dataset from dict to enable easy reproduction but the same happens if you load an existing dataset (e.g. starting from `In [8]`)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-3.10.0-1160.11.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
- Python version: 3.7.10
- PyArrow version: 3.0.0

