[Question] Combine 2 datasets which have the same columns
MEMBER
We are thinking about ways to combine datasets for T5 in #217, feel free to share your thoughts about this.
2020-06-08T09:28:03Z
https://github.com/huggingface/datasets/issues/206#issuecomment-640486324
https://api.github.com/repos/huggingface/datasets/issues/206
MDEyOklzc3VlQ29tbWVudDY0MDQ4NjMyNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/640486324/reactions
2020-06-08T09:28:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/640486324
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Ok great! I will look at it. Thanks
2020-06-10T09:11:14Z
https://github.com/huggingface/datasets/issues/206#issuecomment-641865248
https://api.github.com/repos/huggingface/datasets/issues/206
MDEyOklzc3VlQ29tbWVudDY0MTg2NTI0OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/641865248/reactions
2020-06-10T09:11:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/641865248
https://avatars.githubusercontent.com/u/25703835?v=4
https://api.github.com/users/airKlizz/events{/privacy}
https://api.github.com/users/airKlizz/followers
https://api.github.com/users/airKlizz/following{/other_user}
https://api.github.com/users/airKlizz/gists{/gist_id}
https://github.com/airKlizz
airKlizz
MDQ6VXNlcjI1NzAzODM1
https://api.github.com/users/airKlizz/orgs
https://api.github.com/users/airKlizz/received_events
https://api.github.com/users/airKlizz/repos
https://api.github.com/users/airKlizz/starred{/owner}{/repo}
https://api.github.com/users/airKlizz/subscriptions
User
https://api.github.com/users/airKlizz
public
Hi,

I am using ``nlp`` to load personal datasets. I created summarization datasets in multi-languages based on wikinews. I have one dataset for english and one for german (french is getting to be ready as well). I want to keep these datasets independent because they need different pre-processing (add different task-specific prefixes for T5 : *summarize:* for english and *zusammenfassen:* for german)

My issue is that I want to train T5 on the combined english and german datasets to see if it improves results. So I would like to combine 2 datasets (which have the same columns) to make one and train T5 on it. I was wondering if there is a proper way to do it? I assume that it can be done by combining all examples of each dataset but maybe you have a better solution.

Hoping this is clear enough,

Thanks a lot 😊
Best