Add an entry to an arrow dataset
CONTRIBUTOR
Hi @ameet-1997,
I think what you are looking for is the `concatenate_datasets` function: https://huggingface.co/docs/datasets/processing.html?highlight=concatenate#concatenate-several-datasets

For your use case, I would use the [`map` method](https://huggingface.co/docs/datasets/processing.html?highlight=concatenate#processing-data-with-map) to transform the SQuAD sentences and the `concatenate` the original and mapped dataset.

Let me know If this helps!
2021-01-13T09:29:15Z
https://github.com/huggingface/datasets/issues/1728#issuecomment-759323249
https://api.github.com/repos/huggingface/datasets/issues/1728
MDEyOklzc3VlQ29tbWVudDc1OTMyMzI0OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/759323249/reactions
2021-01-13T09:29:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/759323249
https://avatars.githubusercontent.com/u/33657802?v=4
https://api.github.com/users/SBrandeis/events{/privacy}
https://api.github.com/users/SBrandeis/followers
https://api.github.com/users/SBrandeis/following{/other_user}
https://api.github.com/users/SBrandeis/gists{/gist_id}
https://github.com/SBrandeis
SBrandeis
MDQ6VXNlcjMzNjU3ODAy
https://api.github.com/users/SBrandeis/orgs
https://api.github.com/users/SBrandeis/received_events
https://api.github.com/users/SBrandeis/repos
https://api.github.com/users/SBrandeis/starred{/owner}{/repo}
https://api.github.com/users/SBrandeis/subscriptions
User
https://api.github.com/users/SBrandeis
public
NONE
That's a great idea! Thank you so much!

When I try that solution, I get the following error when I try to concatenate `datasets` and `modified_dataset`. I have also attached the output I get when I print out those two variables. Am I missing something?

Code:
``` python
combined_dataset = concatenate_datasets([datasets, modified_dataset])
```

Error:
```
AttributeError: 'DatasetDict' object has no attribute 'features'
```

Output:
```
(Pdb) datasets
DatasetDict({
    train: Dataset({
        features: ['attention_mask', 'input_ids', 'special_tokens_mask'],
        num_rows: 493
    })
})
(Pdb) modified_dataset
DatasetDict({
    train: Dataset({
        features: ['attention_mask', 'input_ids', 'special_tokens_mask'],
        num_rows: 493
    })
})
```

The error is stemming from the fact that the attribute `datasets.features` does not exist. Would it not be possible to use `concatenate_datasets` in such a case? Is there an alternate solution?
2021-01-13T20:00:15Z
https://github.com/huggingface/datasets/issues/1728#issuecomment-759706044
https://api.github.com/repos/huggingface/datasets/issues/1728
MDEyOklzc3VlQ29tbWVudDc1OTcwNjA0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/759706044/reactions
2021-01-13T20:00:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/759706044
https://avatars.githubusercontent.com/u/18645407?v=4
https://api.github.com/users/ameet-1997/events{/privacy}
https://api.github.com/users/ameet-1997/followers
https://api.github.com/users/ameet-1997/following{/other_user}
https://api.github.com/users/ameet-1997/gists{/gist_id}
https://github.com/ameet-1997
ameet-1997
MDQ6VXNlcjE4NjQ1NDA3
https://api.github.com/users/ameet-1997/orgs
https://api.github.com/users/ameet-1997/received_events
https://api.github.com/users/ameet-1997/repos
https://api.github.com/users/ameet-1997/starred{/owner}{/repo}
https://api.github.com/users/ameet-1997/subscriptions
User
https://api.github.com/users/ameet-1997
public
MEMBER
You should do `combined_dataset = concatenate_datasets([datasets['train'], modified_dataset['train']])`

Didn't we talk about returning a Dataset instead of a DatasetDict with load_dataset and no split provided @lhoestq? Not sure it's the way to go but I'm wondering if it's not simpler for some use-cases.
2021-01-13T21:25:05Z
https://github.com/huggingface/datasets/issues/1728#issuecomment-759748913
https://api.github.com/repos/huggingface/datasets/issues/1728
MDEyOklzc3VlQ29tbWVudDc1OTc0ODkxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/759748913/reactions
2021-01-13T21:25:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/759748913
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
> Didn't we talk about returning a Dataset instead of a DatasetDict with load_dataset and no split provided @lhoestq? Not sure it's the way to go but I'm wondering if it's not simpler for some use-cases.

My opinion is that users should always know in advance what type of objects they're going to get. Otherwise the development workflow on their side is going to be pretty chaotic with sometimes unexpected behaviors.
For instance is `split=` is not specified it's currently always returning a DatasetDict. And if `split="train"` is given for example it's always returning a Dataset.
2021-01-14T09:52:40Z
https://github.com/huggingface/datasets/issues/1728#issuecomment-760086171
https://api.github.com/repos/huggingface/datasets/issues/1728
MDEyOklzc3VlQ29tbWVudDc2MDA4NjE3MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/760086171/reactions
2021-01-14T09:52:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/760086171
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks @thomwolf. Your solution worked!
2021-01-18T19:15:32Z
https://github.com/huggingface/datasets/issues/1728#issuecomment-762428665
https://api.github.com/repos/huggingface/datasets/issues/1728
MDEyOklzc3VlQ29tbWVudDc2MjQyODY2NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/762428665/reactions
2021-01-18T19:15:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/762428665
https://avatars.githubusercontent.com/u/18645407?v=4
https://api.github.com/users/ameet-1997/events{/privacy}
https://api.github.com/users/ameet-1997/followers
https://api.github.com/users/ameet-1997/following{/other_user}
https://api.github.com/users/ameet-1997/gists{/gist_id}
https://github.com/ameet-1997
ameet-1997
MDQ6VXNlcjE4NjQ1NDA3
https://api.github.com/users/ameet-1997/orgs
https://api.github.com/users/ameet-1997/received_events
https://api.github.com/users/ameet-1997/repos
https://api.github.com/users/ameet-1997/starred{/owner}{/repo}
https://api.github.com/users/ameet-1997/subscriptions
User
https://api.github.com/users/ameet-1997
public
Is it possible to add an entry to a dataset object?

**Motivation: I want to transform the sentences in the dataset and add them to the original dataset**

For example, say we have the following code:

``` python
from datasets import load_dataset

# Load a dataset and print the first examples in the training set
squad_dataset = load_dataset('squad')
print(squad_dataset['train'][0])
```

Is it possible to add an entry to `squad_dataset`? Something like the following?

``` python
squad_dataset.append({'text': "This is a new sentence"})
```

The motivation for doing this is that I want to transform the sentences in the squad dataset and add them to the original dataset.

If the above doesn't work, is there any other way of achieving the motivation mentioned above? Perhaps by creating a new arrow dataset by using the older one and the transformer sentences?
