in-place dataset update
MEMBER
We won't support in-place modifications since `datasets` is based on the Apache Arrow format which doesn't support in-place modifications.

In your case the old dataset is garbage collected pretty quickly so you won't have memory issues.

Note that datasets loaded from disk (memory mapped) are not loaded in memory, and therefore the new dataset actually use the same buffers as the old one.
2023-03-01T17:54:47Z
https://github.com/huggingface/datasets/issues/5597#issuecomment-1450599019
https://api.github.com/repos/huggingface/datasets/issues/5597
IC_kwDODunzps5WdmJr
https://api.github.com/repos/huggingface/datasets/issues/comments/1450599019/reactions
2023-03-02T13:29:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1450599019
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thank you for your detailed reply.

> In your case the old dataset is garbage collected pretty quickly so you won't have memory issues.

I understand this, but it still copies the old dataset to create the new one, is this correct? So maybe it is not memory-consuming, but time-consuming?
2023-03-02T03:47:00Z
https://github.com/huggingface/datasets/issues/5597#issuecomment-1451255774
https://api.github.com/repos/huggingface/datasets/issues/5597
IC_kwDODunzps5WgGfe
https://api.github.com/repos/huggingface/datasets/issues/comments/1451255774/reactions
2023-03-02T03:47:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1451255774
https://avatars.githubusercontent.com/u/3585459?v=4
https://api.github.com/users/speedcell4/events{/privacy}
https://api.github.com/users/speedcell4/followers
https://api.github.com/users/speedcell4/following{/other_user}
https://api.github.com/users/speedcell4/gists{/gist_id}
https://github.com/speedcell4
speedcell4
MDQ6VXNlcjM1ODU0NTk=
https://api.github.com/users/speedcell4/orgs
https://api.github.com/users/speedcell4/received_events
https://api.github.com/users/speedcell4/repos
https://api.github.com/users/speedcell4/starred{/owner}{/repo}
https://api.github.com/users/speedcell4/subscriptions
User
https://api.github.com/users/speedcell4
public
MEMBER
Indeed, and because of that it is more efficient to add multiple rows at once instead of one by one, using `concatenate_datasets` for example.
2023-03-02T13:30:41Z
https://github.com/huggingface/datasets/issues/5597#issuecomment-1451869769
https://api.github.com/repos/huggingface/datasets/issues/5597
IC_kwDODunzps5WicZJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1451869769/reactions
2023-03-02T13:30:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1451869769
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Motivation

For the circumstance that I creat an empty `Dataset` and keep appending new rows into it, I found that it leads to creating a new dataset at each call. It looks quite memory-consuming. I just wonder if there is any more efficient way to do this.

```python
from datasets import Dataset

ds = Dataset.from_list([])

ds.add_item({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: [],
>>>     num_rows: 0
>>> })

ds = ds.add_item({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: ['a', 'b'],
>>>     num_rows: 1
>>> })
```

### Feature request

Call for in-place dataset update functions, that update the existing `Dataset` in place without creating a new copy. The interface is supposed to keep the same style as PyTorch, such as the in-place version of a `function` is named `function_`. For example, the in-pace version of `add_item`, i.e., `add_item_`, immediately updates the `Dataset`.

```python
from datasets import Dataset

ds = Dataset.from_list([])

ds.add_item({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: [],
>>>     num_rows: 0
>>> })

ds.add_item_({'a': [1, 2, 3], 'b': 4})
print(ds)
>>> Dataset({
>>>     features: ['a', 'b'],
>>>     num_rows: 1
>>> })
```

### Related Functions

* `.map`
* `.filter`
* `.add_item`