Ability to update a dataset
MEMBER
This [reply](https://discuss.huggingface.co/t/how-do-i-add-things-rows-to-an-already-saved-dataset/27423) from @mariosasko on the forums may be useful :)
2023-05-04T18:14:24Z
https://github.com/huggingface/datasets/issues/5818#issuecomment-1535206992
https://api.github.com/repos/huggingface/datasets/issues/5818
IC_kwDODunzps5bgWZQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1535206992/reactions
2023-05-04T18:14:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1535206992
https://avatars.githubusercontent.com/u/59462357?v=4
https://api.github.com/users/stevhliu/events{/privacy}
https://api.github.com/users/stevhliu/followers
https://api.github.com/users/stevhliu/following{/other_user}
https://api.github.com/users/stevhliu/gists{/gist_id}
https://github.com/stevhliu
stevhliu
MDQ6VXNlcjU5NDYyMzU3
https://api.github.com/users/stevhliu/orgs
https://api.github.com/users/stevhliu/received_events
https://api.github.com/users/stevhliu/repos
https://api.github.com/users/stevhliu/starred{/owner}{/repo}
https://api.github.com/users/stevhliu/subscriptions
User
https://api.github.com/users/stevhliu
public
COLLABORATOR
In this case, I think we can avoid the `PermissionError` by unpacking the underlying `ConcatenationTable` and saving only the newly added data blocks (in new files).
2023-05-04T18:43:12Z
https://github.com/huggingface/datasets/issues/5818#issuecomment-1535239556
https://api.github.com/repos/huggingface/datasets/issues/5818
IC_kwDODunzps5bgeWE
https://api.github.com/repos/huggingface/datasets/issues/comments/1535239556/reactions
2023-05-04T18:43:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1535239556
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Thanks @stevhliu and @mariosasko , so saving to individual files then loading them later, concatenating again and saving again is the recommended way. Good to know.

Question that I hope doesn't sound rude: is this sort of thing (processing a dataset that doesn't fit in memory) outside of `datasets`'s core area of focus? Are there other tools you would recommend to do this sort of thing that play nice with `datasets`? Or is it just that I've found myself in a niche situation that hasn't specifically been catered for?
2023-05-04T20:43:38Z
https://github.com/huggingface/datasets/issues/5818#issuecomment-1535387057
https://api.github.com/repos/huggingface/datasets/issues/5818
IC_kwDODunzps5bhCWx
https://api.github.com/repos/huggingface/datasets/issues/comments/1535387057/reactions
2023-05-04T20:43:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1535387057
https://avatars.githubusercontent.com/u/4443482?v=4
https://api.github.com/users/davidgilbertson/events{/privacy}
https://api.github.com/users/davidgilbertson/followers
https://api.github.com/users/davidgilbertson/following{/other_user}
https://api.github.com/users/davidgilbertson/gists{/gist_id}
https://github.com/davidgilbertson
davidgilbertson
MDQ6VXNlcjQ0NDM0ODI=
https://api.github.com/users/davidgilbertson/orgs
https://api.github.com/users/davidgilbertson/received_events
https://api.github.com/users/davidgilbertson/repos
https://api.github.com/users/davidgilbertson/starred{/owner}{/repo}
https://api.github.com/users/davidgilbertson/subscriptions
User
https://api.github.com/users/davidgilbertson
public
### Feature request

The ability to load a dataset, add or change something, and save it back to disk.

Maybe it's possible, but I can't work out how to do it, e.g. this fails:

```py
import datasets

dataset = datasets.load_from_disk("data/test1")
dataset = dataset.add_item({"text": "A new item"})
dataset.save_to_disk("data/test1")
```

With the error:
```
PermissionError: Tried to overwrite /mnt/c/Users/david/py/learning/mini_projects/data_sorting_and_filtering/data/test1 but a dataset can't overwrite itself.
```


### Motivation

My use case is that I want to process a dataset in a particular way but it doesn't fit in memory if I do it in one go. So I want to perform a loop and at each step in the loop, process one shard and append it to an ever-growing dataset. The code in the loop will load a dataset, add some rows, then save it again.

Maybe I'm just thinking about things incorrectly and there's a better approach. FWIW I can't use `dataset.map()` to do the task because that doesn't work with `num_proc` when adding rows, so is confined to a single process which is too slow.

The only other way I can think of is to create a new file each time, but surely that's not how people do this sort of thing.

### Your contribution

na