How to augment data ?
MEMBER
Using batched map is probably the easiest way at the moment.
What kind of augmentation would you like to do ?
2020-07-09T08:17:10Z
https://github.com/huggingface/datasets/issues/365#issuecomment-655981373
https://api.github.com/repos/huggingface/datasets/issues/365
MDEyOklzc3VlQ29tbWVudDY1NTk4MTM3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/655981373/reactions
2020-07-09T08:17:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/655981373
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Some samples in the dataset are too long, I want to divide them in several samples.
2020-07-09T23:36:49Z
https://github.com/huggingface/datasets/issues/365#issuecomment-656401535
https://api.github.com/repos/huggingface/datasets/issues/365
MDEyOklzc3VlQ29tbWVudDY1NjQwMTUzNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/656401535/reactions
2020-07-09T23:36:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656401535
https://avatars.githubusercontent.com/u/43774355?v=4
https://api.github.com/users/astariul/events{/privacy}
https://api.github.com/users/astariul/followers
https://api.github.com/users/astariul/following{/other_user}
https://api.github.com/users/astariul/gists{/gist_id}
https://github.com/astariul
astariul
MDQ6VXNlcjQzNzc0MzU1
https://api.github.com/users/astariul/orgs
https://api.github.com/users/astariul/received_events
https://api.github.com/users/astariul/repos
https://api.github.com/users/astariul/starred{/owner}{/repo}
https://api.github.com/users/astariul/subscriptions
User
https://api.github.com/users/astariul
public
MEMBER
Using batched map is the way to go then.
We'll make it clearer in the docs that map could be used for augmentation.

Let me know if you think there should be another way to do it. Or feel free to close the issue otherwise.
2020-07-10T07:56:28Z
https://github.com/huggingface/datasets/issues/365#issuecomment-656540813
https://api.github.com/repos/huggingface/datasets/issues/365
MDEyOklzc3VlQ29tbWVudDY1NjU0MDgxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/656540813/reactions
2020-07-10T07:56:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656540813
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
It just feels awkward to use map to augment data. Also it means it's not possible to augment data in a non-batched way.

But to be honest I have no idea of a good API...
2020-07-10T08:22:15Z
https://github.com/huggingface/datasets/issues/365#issuecomment-656552042
https://api.github.com/repos/huggingface/datasets/issues/365
MDEyOklzc3VlQ29tbWVudDY1NjU1MjA0Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/656552042/reactions
2020-07-10T08:22:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656552042
https://avatars.githubusercontent.com/u/43774355?v=4
https://api.github.com/users/astariul/events{/privacy}
https://api.github.com/users/astariul/followers
https://api.github.com/users/astariul/following{/other_user}
https://api.github.com/users/astariul/gists{/gist_id}
https://github.com/astariul
astariul
MDQ6VXNlcjQzNzc0MzU1
https://api.github.com/users/astariul/orgs
https://api.github.com/users/astariul/received_events
https://api.github.com/users/astariul/repos
https://api.github.com/users/astariul/starred{/owner}{/repo}
https://api.github.com/users/astariul/subscriptions
User
https://api.github.com/users/astariul
public
NONE
Or for non-batched samples, how about returning a tuple ?

```python
def aug(sample):
    # Simply copy the existing data to have x2 amount of data
    return sample, sample

dataset = dataset.map(aug)
```

It feels really natural and easy, but :

* it means the behavior with batched data is different
* I don't know how doable it is backend-wise

@lhoestq 
2020-07-10T08:24:25Z
https://github.com/huggingface/datasets/issues/365#issuecomment-656552977
https://api.github.com/repos/huggingface/datasets/issues/365
MDEyOklzc3VlQ29tbWVudDY1NjU1Mjk3Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/656552977/reactions
2020-07-10T08:24:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656552977
https://avatars.githubusercontent.com/u/43774355?v=4
https://api.github.com/users/astariul/events{/privacy}
https://api.github.com/users/astariul/followers
https://api.github.com/users/astariul/following{/other_user}
https://api.github.com/users/astariul/gists{/gist_id}
https://github.com/astariul
astariul
MDQ6VXNlcjQzNzc0MzU1
https://api.github.com/users/astariul/orgs
https://api.github.com/users/astariul/received_events
https://api.github.com/users/astariul/repos
https://api.github.com/users/astariul/starred{/owner}{/repo}
https://api.github.com/users/astariul/subscriptions
User
https://api.github.com/users/astariul
public
MEMBER
As we're working with arrow's columnar format we prefer to play with batches that are dictionaries instead of tuples.
If we have tuple it implies to re-format the data each time we want to write to arrow, which can lower the speed of map for example.

It's also a matter of coherence, as we don't want users to be confused whether they have to return dictionaries for some functions and tuples for others when they're doing batches.
2020-07-10T09:12:07Z
https://github.com/huggingface/datasets/issues/365#issuecomment-656573998
https://api.github.com/repos/huggingface/datasets/issues/365
MDEyOklzc3VlQ29tbWVudDY1NjU3Mzk5OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/656573998/reactions
2020-07-10T09:12:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656573998
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Is there any clean way to augment data ?

For now my work-around is to use batched map, like this :

```python
def aug(samples):
    # Simply copy the existing data to have x2 amount of data
    for k, v in samples.items():
        samples[k].extend(v)
    return samples

dataset = dataset.map(aug, batched=True)
```