Multi-task dataset mixing
CONTRIBUTOR
I like this feature! I think the first question we should decide on is how to convert all datasets into the same format. In T5, the authors decided to format every dataset into a text-to-text format. If the dataset had "multiple" inputs like MNLI, the inputs were concatenated. So in MNLI the input:

> - **Hypothesis**: The St. Louis Cardinals have always won.
> 
> - **Premise**: yeah well losing is i mean i’m i’m originally from Saint Louis and Saint Louis Cardinals when they were there were uh a mostly a losing team but 

was flattened to a single input:

> mnli hypothesis: The St. Louis Cardinals have always won. premise:
> yeah well losing is i mean i’m i’m originally from Saint Louis and Saint Louis Cardinals
> when they were there were uh a mostly a losing team but.

This flattening is actually a very simple operation in `nlp` already. You would just need to do the following:

```python 
def flatten_inputs(example):
    return {"input": "mnli hypothesis: " + example['hypothesis'] + " premise: " + example['premise']}

t5_ready_mnli_ds = mnli_ds.map(flatten_inputs, remove_columns=[<all columns except output>])
```

So I guess converting the datasets into the same format can be left to the user for now. 
Then the question is how we can merge the datasets. I would probably be in favor of a simple 

```python 
dataset.add()
```

function that checks if the dataset is of the same format and if yes merges the two datasets. Finally, how should the sampling be implemented? **Examples-proportional mixing** corresponds to just merging the datasets and shuffling. For the other two sampling approaches we would need some higher-level features, maybe even a `dataset.sample()` function for merged datasets. 

What are your thoughts on this @thomwolf @lhoestq  @ghomasHudson @enzoampil ?
2020-06-03T08:26:19Z
https://github.com/huggingface/datasets/issues/217#issuecomment-638044136
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDYzODA0NDEzNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/638044136/reactions
2020-06-03T08:26:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/638044136
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
I agree that we should leave the flattening of the dataset to the user for now. Especially because although the T5 framing seems obvious, there are slight variations on how the T5 authors do it in comparison to other approaches such as gpt-3 and decaNLP.

In terms of sampling, Examples-proportional mixing does seem the simplest to implement so would probably be a good starting point.

Temperature-scaled mixing would probably most useful, offering flexibility as it can simulate the other 2 methods by setting the temperature parameter. There is a [relevant part of the T5 repo](https://github.com/google-research/text-to-text-transfer-transformer/blob/03c94165a7d52e4f7230e5944a0541d8c5710788/t5/data/utils.py#L889-L1118) which should help with implementation.

According to the T5 authors, equal-mixing performs worst. Among the other two methods, tuning the K value (the artificial dataset size limit) has a large impact.

2020-06-03T10:01:08Z
https://github.com/huggingface/datasets/issues/217#issuecomment-638094641
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDYzODA5NDY0MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/638094641/reactions
2020-06-03T10:01:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/638094641
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
NONE
I agree with going with temperature-scaled mixing for its flexibility!

For the function that combines the datasets, I also find `dataset.add()` okay while also considering that users may want it to be easy to combine a list of say 10 data sources in one go.

`dataset.sample()` should also be good. By the looks of it, we're planning to have as main parameters: `temperature`, and `K`.

On converting the datasets to the same format, I agree that we can leave these to the users for now. But, I do imagine it'd be an awesome feature for the future to have this automatically handled, based on a chosen *approach* to formatting :smile: 

E.g. T5, GPT-3, decaNLP, original raw formatting, or a contributed way of formatting in text-to-text. 
2020-06-03T23:23:37Z
https://github.com/huggingface/datasets/issues/217#issuecomment-638511640
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDYzODUxMTY0MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/638511640/reactions
2020-06-03T23:30:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/638511640
https://avatars.githubusercontent.com/u/39557688?v=4
https://api.github.com/users/enzoampil/events{/privacy}
https://api.github.com/users/enzoampil/followers
https://api.github.com/users/enzoampil/following{/other_user}
https://api.github.com/users/enzoampil/gists{/gist_id}
https://github.com/enzoampil
enzoampil
MDQ6VXNlcjM5NTU3Njg4
https://api.github.com/users/enzoampil/orgs
https://api.github.com/users/enzoampil/received_events
https://api.github.com/users/enzoampil/repos
https://api.github.com/users/enzoampil/starred{/owner}{/repo}
https://api.github.com/users/enzoampil/subscriptions
User
https://api.github.com/users/enzoampil
public
MEMBER
This is an interesting discussion indeed and it would be nice to make multi-task easier.

Probably the best would be to have a new type of dataset especially designed for that in order to easily combine and sample from the multiple datasets.

This way we could probably handle the combination of datasets with differing schemas as well (unlike T5).
2020-06-08T10:11:45Z
https://github.com/huggingface/datasets/issues/217#issuecomment-640509585
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY0MDUwOTU4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/640509585/reactions
2020-06-08T10:11:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/640509585
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
@thomwolf Are you suggesting making a wrapper class which can take existing datasets as arguments and do all the required sampling/combining, to present the same interface as a normal dataset?

That doesn't seem too complicated to implement.

2020-06-08T10:36:44Z
https://github.com/huggingface/datasets/issues/217#issuecomment-640521699
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY0MDUyMTY5OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/640521699/reactions
2020-06-08T11:03:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/640521699
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
CONTRIBUTOR
I guess we're looking at the end user writing something like:
``` python
ds = nlp.load_dataset('multitask-t5',datasets=["squad","cnn_dm",...], k=1000, t=2.0)
```
Using the t5 method of combining here (or this could be a function passed in as an arg) 

Passing kwargs to each 'sub-dataset' might become tricky.
2020-06-08T11:09:00Z
https://github.com/huggingface/datasets/issues/217#issuecomment-640536499
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY0MDUzNjQ5OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/640536499/reactions
2020-06-08T11:10:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/640536499
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
CONTRIBUTOR
From thinking upon @thomwolf 's suggestion, I've started experimenting:
```python
class MultitaskDataset(DatasetBuilder):
    def __init__(self, *args, **kwargs):
        super(MultitaskDataset, self).__init__(*args, **kwargs)
        self._datasets = kwargs.get("datasets")

    def _info(self):
        return nlp.DatasetInfo(
            description=_DESCRIPTION,
            features=nlp.Features({
                    "source": nlp.Value("string"),
                    "target": nlp.Sequence(nlp.Value("string"))
                })
        )

    def _get_common_splits(self):
        '''Finds the common splits present in all self._datasets'''
        min_set = None
        for dataset in self._datasets:
            if min_set != None:
                min_set.intersection(set(dataset.keys()))
            else:
                min_set = set(dataset.keys())
        return min_set

....

# Maybe this?:
squad = nlp.load_dataset("squad")
cnn_dm = nlp.load_dataset("cnn_dailymail","3.0.0")
multitask_dataset = nlp.load_dataset(
    'multitask_dataset',
    datasets=[squad,cnn_dailymail], 
    k=1000, 
    t=2.0
)

```

Does anyone know what methods of `MultitaskDataset` I would need to implement? Maybe `as_dataset` and `download_and_prepare`? Most of these should be just calling the methods of the sub-datasets. 

I'm assuming DatasetBuilder is better than the more specific `GeneratorBasedBuilder`, `BeamBasedBuilder`, etc....

One of the other problems is that the dataset size is unknown till you construct it (as you can pick the sub-datasets). Am hoping not to need to make changes to `nlp.load_dataset` just for this class.

I'd appreciate it if anyone more familiar with nlp's internal workings could tell me if I'm on the right track!
2020-06-15T09:10:49Z
https://github.com/huggingface/datasets/issues/217#issuecomment-644006136
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY0NDAwNjEzNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/644006136/reactions
2020-06-15T09:11:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/644006136
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
MEMBER
I think I would probably go for a `MultiDataset` wrapper around a list of `Dataset`.

I'm not sure we need to give it `k` and `t` parameters at creation, it can maybe be something along the lines of:
```python
squad = nlp.load_dataset("squad")
cnn_dm = nlp.load_dataset("cnn_dailymail","3.0.0")

multitask_dataset = nlp.MultiDataset(squad, cnn_dm)

batch = multitask_dataset.sample(10, temperature=2.0, k=1000)
```

The first proof-of-concept for multi-task datasets could definitely require that the provided datasets have the same name/type for columns (if needed you easily rename/cast a column prior to instantiating the `MultiDataset`).

It's good to think about it for some time though and don't overfit too much on the T5 examples (in particular for the ways/kwargs for sampling among datasets).
2020-06-20T21:22:25Z
https://github.com/huggingface/datasets/issues/217#issuecomment-647046814
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY0NzA0NjgxNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/647046814/reactions
2020-06-20T21:22:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/647046814
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
The problem with changing `k` and `t` per sampling is that you'd have to somehow remember which examples you'd already returned while re-weighting the remaining examples based on the new `k` and `t`values. It seems possible but complicated (I can't really see a reason why you'd want to change the weighting of datasets after you constructed the multidataset).

Wouldn't it be convenient if it implemented the dataset interface? Then if someone has code using a single nlp dataset, they can replace it with a multitask combination of more datasets without having to change other code. We would at least need to be able to pass it into a `DataLoader`.


2020-06-23T12:28:37Z
https://github.com/huggingface/datasets/issues/217#issuecomment-648113351
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY0ODExMzM1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/648113351/reactions
2020-06-23T12:28:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/648113351
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
CONTRIBUTOR
A very janky (but working) implementation of  `multitask_dataset.sample()` could be something like this:
```python
import nlp
import torch

class MultiDataset():
    def __init__(self, *args, temperature=2.0, k=1000, maximum=None, scale=1):
        self.datasets = args
        self._dataloaders = {}
        for split in self._get_common_splits():
            split_datasets = [ds[split] for ds in self.datasets]
            mixing_rates = self._calc_mixing_rates(split_datasets,temperature, k, maximum, scale)
            weights = []
            for i in range(len(self.datasets)):
                weights += [mixing_rates[i]]*len(self.datasets[i][split])
            self._dataloaders[split] = torch.utils.data.DataLoader(torch.utils.data.ConcatDataset(split_datasets),
                                                        sampler=torch.utils.data.sampler.WeightedRandomSampler(
                                                            num_samples=len(weights),
                                                            weights = weights,
                                                            replacement=True),
                                                        shuffle=False)

    def _get_common_splits(self):
        '''Finds the common splits present in all self.datasets'''
        min_set = None
        for dataset in self.datasets:
            if min_set != None:
                min_set.intersection(set(dataset.keys()))
            else:
                min_set = set(dataset.keys())
        return min_set


    def _calc_mixing_rates(self,datasets, temperature=2.0, k=1000, maximum=None, scale=1):
       '''Work out the weighting of each dataset based on t and k'''
        mixing_rates = []
        for dataset in datasets:
            rate = len(dataset)
            rate *= scale
            if maximum:
                rate = min(rate, maximum)
            if temperature != 1.0:
                rate = rate ** (1.0/temperature)
            mixing_rates.append(rate)
        return mixing_rates

    def sample(self,n,split):
        batch = []
        for example in self._dataloaders[split]:
            batch.append(example)
            n -= 1
            if n == 0:
                return batch


def flatten(dataset,flatten_fn):
    for k in dataset.keys():
        if isinstance(dataset[k],nlp.Dataset):
            dataset[k] = dataset[k].map(flatten_fn,remove_columns=dataset[k].column_names)

# Squad
def flatten_squad(example):
    return {"source": "squad context: " + example['context'] + " question: " + example['question'],"target":example["answers"]["text"]}
squad = nlp.load_dataset("squad")
flatten(squad,flatten_squad)

# CNN_DM
def flatten_cnn_dm(example):
    return {"source": "cnn_dm: " + example['article'],"target":[example["highlights"]]}
cnn_dm = nlp.load_dataset("cnn_dailymail", "3.0.0")
flatten(cnn_dm,flatten_cnn_dm)

multitask_dataset = MultiDataset(squad, cnn_dm)
batch = multitask_dataset.sample(100,"train")
```

There's definitely a more sensible way than embedding `DataLoader`s inside. 
2020-06-23T12:33:22Z
https://github.com/huggingface/datasets/issues/217#issuecomment-648115586
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY0ODExNTU4Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/648115586/reactions
2020-06-23T12:33:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/648115586
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
MEMBER
There is an interesting related investigation by @zphang here https://colab.research.google.com/github/zphang/zphang.github.io/blob/master/files/notebooks/Multi_task_Training_with_Transformers_NLP.ipynb
2020-06-26T10:17:45Z
https://github.com/huggingface/datasets/issues/217#issuecomment-650104333
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY1MDEwNDMzMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/650104333/reactions
2020-06-26T10:17:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/650104333
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
Good spot! Here are my thoughts:

- Aside: Adding `MultitaskModel` to transformers might be a thing to raise - even though having task-specific heads has become unfashionable in recent times in favour of text-to-text type models.
- Adding the task name as an extra field also seems useful for these kind of models which have task-specific heads
- There is some validation of our approach that the user should be expected to `map` datasets into a common form.
- The size-proportional sampling (also called "Examples-proportional mixing") used here doesn't perform too badly in the T5 paper (it's comparable to temperature-scaled mixing in many cases but less flexible. This is only reasonable with a `K` maximum size parameter to prevent very large datasets dominating). This might be good for a first prototype using:
    ```python
    def __iter__(self):
        """
        For each batch, sample a task, and yield a batch from the respective
        task Dataloader.

        We use size-proportional sampling, but you could easily modify this
        to sample from some-other distribution.
        """
        task_choice_list = []
        for i, task_name in enumerate(self.task_name_list):
            task_choice_list += [i] * self.num_batches_dict[task_name]
        task_choice_list = np.array(task_choice_list)
        np.random.shuffle(task_choice_list)

        dataloader_iter_dict = {
            task_name: iter(dataloader) 
            for task_name, dataloader in self.dataloader_dict.items()
        }
        for task_choice in task_choice_list:
            task_name = self.task_name_list[task_choice]
            yield next(dataloader_iter_dict[task_name])    
    ```
    We'd just need to pull samples from the raw datasets and not from `DataLoader`s for each task. We can assume the user has done `dataset.shuffle()` if they want to.

    Other sampling methods can later be implemented by changing how the `task_choice_list` is generated. This should allow more flexibility and not tie us to specific methods for sampling among datasets.

2020-06-26T11:11:16Z
https://github.com/huggingface/datasets/issues/217#issuecomment-650123812
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY1MDEyMzgxMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/650123812/reactions
2020-06-26T11:15:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/650123812
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
CONTRIBUTOR
Another thought: Multitasking over benchmarks (represented as Meta-datasets in nlp) is probably a common use case. Would be nice to pass an entire benchmark to our `MultiDataset` wrapper rather than having to pass individual components.
2020-06-26T11:19:49Z
https://github.com/huggingface/datasets/issues/217#issuecomment-650126957
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY1MDEyNjk1Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/650126957/reactions
2020-06-26T11:19:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/650126957
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
CONTRIBUTOR
Here's a fully working implementation based on the `__iter__` function of @zphang.

- I've generated the task choice list in the constructor as it allows us to index into the MultiDataset just like a normal dataset. I'm changing `task_choice_list` into a list of `(dataset_idx, example_idx)` so each entry references a unique dataset example. The shuffling has to be done before this as we don't want to shuffle within each task (we assume this is done by the user if this is what they intend).
-  I'm slightly concerned this list could become very large if many large datasets were used. Can't see a way round it at the moment though.
- I've used `task.info.builder_name` as the dataset name. Not sure if this is correct.
- I'd love to add some of the other `Dataset` methods (map, slicing by column, etc...). Would be great to implement the whole interface so a single dataset can be simply replaced by this.
- This does everything on the individual example-level. If some application required batches all from a single task in turn we can't really do that.

```python
import nlp
import numpy as np

class MultiDataset:
    def __init__(self,tasks):
        self.tasks = tasks

        # Create random order of tasks
        # Using size-proportional sampling
        task_choice_list = []
        for i, task in enumerate(self.tasks):
            task_choice_list += [i] * len(task)
        task_choice_list = np.array(task_choice_list)
        np.random.shuffle(task_choice_list)

        # Add index into each dataset
        # - We don't want to shuffle within each task
        counters = {}
        self.task_choice_list = []
        for i in range(len(task_choice_list)):
            idx = counters.get(task_choice_list[i],0)
            self.task_choice_list.append((task_choice_list[i],idx))
            counters[task_choice_list[i]] = idx + 1


    def __len__(self):
        return np.sum([len(t) for t in self.tasks])

    def __repr__(self):
        task_str = ", ".join([str(t) for t in self.tasks])
        return f"MultiDataset(tasks: {task_str})"

    def __getitem__(self,key):
        if isinstance(key, int):
            task_idx, example_idx = self.task_choice_list[key]
            task = self.tasks[task_idx]
            example = task[example_idx]
            example["task_name"] = task.info.builder_name
            return example
        elif isinstance(key, slice):
            raise NotImplementedError()

    def __iter__(self):
        for i in range(len(self)):
            yield self[i]


def load_multitask(*datasets):
    '''Create multitask datasets per split'''

    def _get_common_splits(datasets):
        '''Finds the common splits present in all self.datasets'''
        min_set = None
        for dataset in datasets:
            if min_set != None:
                min_set.intersection(set(dataset.keys()))
            else:
                min_set = set(dataset.keys())
        return min_set

    common_splits = _get_common_splits(datasets)
    out = {}
    for split in common_splits:
        out[split] = MultiDataset([d[split] for d in datasets])
    return out


##########################################
# Dataset Flattening

def flatten(dataset,flatten_fn):
    for k in dataset.keys():
        if isinstance(dataset[k],nlp.Dataset):
            dataset[k] = dataset[k].map(flatten_fn,remove_columns=dataset[k].column_names)

# Squad
def flatten_squad(example):
    return {"source": "squad context: " + example['context'] + " question: " + example['question'],
          "target":example["answers"]["text"]}
squad = nlp.load_dataset("squad")
flatten(squad,flatten_squad)

# CNN_DM
def flatten_cnn_dm(example):
    return {"source": "cnn_dm: " + example['article'],"target":[example["highlights"]]}
cnn_dm = nlp.load_dataset("cnn_dailymail", "3.0.0")
flatten(cnn_dm,flatten_cnn_dm)

#############################################

mtds = load_multitask(squad,cnn_dm)

for example in mtds["train"]:
    print(example["task_name"],example["target"])
```
Let me know if you have any thoughts. I've started using this in some of my projects and it seems to work. If people are happy with the general approach for a first version, I can make a pull request.
2020-06-26T19:04:44Z
https://github.com/huggingface/datasets/issues/217#issuecomment-650346849
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY1MDM0Njg0OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/650346849/reactions
2020-06-26T19:11:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/650346849
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
NONE
Hey! Happy to jump into the discussion here. I'm still getting familiar with bits of this code, but the reasons I sampled over data loaders rather than datasets is 1) ensuring that each sampled batch corresponds to only 1 task (in case of different inputs formats/downstream models) and 2) potentially having different batch sizes per task (e.g. some tasks have very long/short inputs). How are you currently dealing with these in your PR?
2020-06-29T08:18:33Z
https://github.com/huggingface/datasets/issues/217#issuecomment-651008026
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY1MTAwODAyNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/651008026/reactions
2020-06-29T08:18:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/651008026
https://avatars.githubusercontent.com/u/1668462?v=4
https://api.github.com/users/zphang/events{/privacy}
https://api.github.com/users/zphang/followers
https://api.github.com/users/zphang/following{/other_user}
https://api.github.com/users/zphang/gists{/gist_id}
https://github.com/zphang
zphang
MDQ6VXNlcjE2Njg0NjI=
https://api.github.com/users/zphang/orgs
https://api.github.com/users/zphang/received_events
https://api.github.com/users/zphang/repos
https://api.github.com/users/zphang/starred{/owner}{/repo}
https://api.github.com/users/zphang/subscriptions
User
https://api.github.com/users/zphang
public
CONTRIBUTOR
The short answer is - I'm not! Everything is currently on a per-example basis. It would be fairly simple to add a `batch_size` argument which would ensure that every `batch_size` examples come from the same task. That should suit most use-cases (unless you wanted to ensure batches all came from the same task and apply something like `SortishSampler` on each task first)

Your notebook was really inspiring by the way - thanks!
2020-06-29T08:34:54Z
https://github.com/huggingface/datasets/issues/217#issuecomment-651016744
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY1MTAxNjc0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/651016744/reactions
2020-06-29T08:34:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/651016744
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
CONTRIBUTOR
@zphang is having different batch sizes per task actually helpful? Would be interesting to know as it's not something I've come across as a technique used by any MTL papers.
2020-07-04T18:29:53Z
https://github.com/huggingface/datasets/issues/217#issuecomment-653797671
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY1Mzc5NzY3MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/653797671/reactions
2020-07-04T18:29:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/653797671
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
CONTRIBUTOR
mt-dnn's [batcher.py](https://github.com/namisan/mt-dnn/blob/master/mt_dnn/batcher.py) might be worth looking at.
2020-07-04T19:23:48Z
https://github.com/huggingface/datasets/issues/217#issuecomment-653802151
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDY1MzgwMjE1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/653802151/reactions
2020-07-04T19:23:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/653802151
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
NONE
> @zphang is having different batch sizes per task actually helpful? Would be interesting to know as it's not something I've come across as a technique used by any MTL papers.

I think having different batch sizes per task is particularly helpful in some scenarios where each task has different amount of data. For example, the problem I'm currently facing is one task has tens of thousands of samples while one task has a couple hundreds. I think in this case different batch size could help. But if using the same batch size is a lot simpler to implement, I guess it makes sense to go with that.
2020-09-30T17:34:34Z
https://github.com/huggingface/datasets/issues/217#issuecomment-701536813
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDcwMTUzNjgxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/701536813/reactions
2020-09-30T17:34:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/701536813
https://avatars.githubusercontent.com/u/8862053?v=4
https://api.github.com/users/ldong87/events{/privacy}
https://api.github.com/users/ldong87/followers
https://api.github.com/users/ldong87/following{/other_user}
https://api.github.com/users/ldong87/gists{/gist_id}
https://github.com/ldong87
ldong87
MDQ6VXNlcjg4NjIwNTM=
https://api.github.com/users/ldong87/orgs
https://api.github.com/users/ldong87/received_events
https://api.github.com/users/ldong87/repos
https://api.github.com/users/ldong87/starred{/owner}{/repo}
https://api.github.com/users/ldong87/subscriptions
User
https://api.github.com/users/ldong87
public
NONE
I think that instead of proportional to size sampling you should specify weights or probabilities for drawing a batch from each dataset. We should also ensure that the smaller datasets are repeated so that the encoder layer doesn't overtrain on the largest dataset.
2020-10-01T02:33:46Z
https://github.com/huggingface/datasets/issues/217#issuecomment-701767114
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDcwMTc2NzExNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/701767114/reactions
2020-10-01T02:33:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/701767114
https://avatars.githubusercontent.com/u/2000204?v=4
https://api.github.com/users/timothyjlaurent/events{/privacy}
https://api.github.com/users/timothyjlaurent/followers
https://api.github.com/users/timothyjlaurent/following{/other_user}
https://api.github.com/users/timothyjlaurent/gists{/gist_id}
https://github.com/timothyjlaurent
timothyjlaurent
MDQ6VXNlcjIwMDAyMDQ=
https://api.github.com/users/timothyjlaurent/orgs
https://api.github.com/users/timothyjlaurent/received_events
https://api.github.com/users/timothyjlaurent/repos
https://api.github.com/users/timothyjlaurent/starred{/owner}{/repo}
https://api.github.com/users/timothyjlaurent/subscriptions
User
https://api.github.com/users/timothyjlaurent
public
CONTRIBUTOR
Are there any references for people doing different batch sizes per task in the literature? I've only seen constant batch sizes with differing numbers of batches for each task which seems sufficient to prevent the impact of large datasets (Read 3.5.3 of the [T5 paper](https://arxiv.org/pdf/1910.10683.pdf) for example).


2020-10-01T10:55:21Z
https://github.com/huggingface/datasets/issues/217#issuecomment-702055076
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDcwMjA1NTA3Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/702055076/reactions
2020-10-01T10:57:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/702055076
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
CONTRIBUTOR
Hi,
regarding building T5 dataset , I think we can use datasets https://github.com/huggingface/datasets and then need something similar to tf.data.experimental.sample_from_datasets, do you know if similar functionality exist in pytorch? Which can sample multiple datasets with the given rates. thanks. 
2020-10-26T08:46:33Z
https://github.com/huggingface/datasets/issues/217#issuecomment-716402475
https://api.github.com/repos/huggingface/datasets/issues/217
MDEyOklzc3VlQ29tbWVudDcxNjQwMjQ3NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/716402475/reactions
2020-10-26T08:46:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/716402475
https://avatars.githubusercontent.com/u/6278280?v=4
https://api.github.com/users/rabeehk/events{/privacy}
https://api.github.com/users/rabeehk/followers
https://api.github.com/users/rabeehk/following{/other_user}
https://api.github.com/users/rabeehk/gists{/gist_id}
https://github.com/rabeehk
rabeehk
MDQ6VXNlcjYyNzgyODA=
https://api.github.com/users/rabeehk/orgs
https://api.github.com/users/rabeehk/received_events
https://api.github.com/users/rabeehk/repos
https://api.github.com/users/rabeehk/starred{/owner}{/repo}
https://api.github.com/users/rabeehk/subscriptions
User
https://api.github.com/users/rabeehk
public
NONE
Is this feature part of a `datasets` release yet? 
2022-10-13T17:19:06Z
https://github.com/huggingface/datasets/issues/217#issuecomment-1277937841
https://api.github.com/repos/huggingface/datasets/issues/217
IC_kwDODunzps5MK8ix
https://api.github.com/repos/huggingface/datasets/issues/comments/1277937841/reactions
2022-10-13T17:19:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1277937841
https://avatars.githubusercontent.com/u/43276957?v=4
https://api.github.com/users/StefanHeng/events{/privacy}
https://api.github.com/users/StefanHeng/followers
https://api.github.com/users/StefanHeng/following{/other_user}
https://api.github.com/users/StefanHeng/gists{/gist_id}
https://github.com/StefanHeng
StefanHeng
MDQ6VXNlcjQzMjc2OTU3
https://api.github.com/users/StefanHeng/orgs
https://api.github.com/users/StefanHeng/received_events
https://api.github.com/users/StefanHeng/repos
https://api.github.com/users/StefanHeng/starred{/owner}{/repo}
https://api.github.com/users/StefanHeng/subscriptions
User
https://api.github.com/users/StefanHeng
public
NONE
> Here's a fully working implementation based on the `__iter__` function of @zphang.
> 
> * I've generated the task choice list in the constructor as it allows us to index into the MultiDataset just like a normal dataset. I'm changing `task_choice_list` into a list of `(dataset_idx, example_idx)` so each entry references a unique dataset example. The shuffling has to be done before this as we don't want to shuffle within each task (we assume this is done by the user if this is what they intend).
> * I'm slightly concerned this list could become very large if many large datasets were used. Can't see a way round it at the moment though.
> * I've used `task.info.builder_name` as the dataset name. Not sure if this is correct.
> * I'd love to add some of the other `Dataset` methods (map, slicing by column, etc...). Would be great to implement the whole interface so a single dataset can be simply replaced by this.
> * This does everything on the individual example-level. If some application required batches all from a single task in turn we can't really do that.
> 
> ```python
> import nlp
> import numpy as np
> 
> class MultiDataset:
>     def __init__(self,tasks):
>         self.tasks = tasks
> 
>         # Create random order of tasks
>         # Using size-proportional sampling
>         task_choice_list = []
>         for i, task in enumerate(self.tasks):
>             task_choice_list += [i] * len(task)
>         task_choice_list = np.array(task_choice_list)
>         np.random.shuffle(task_choice_list)
> 
>         # Add index into each dataset
>         # - We don't want to shuffle within each task
>         counters = {}
>         self.task_choice_list = []
>         for i in range(len(task_choice_list)):
>             idx = counters.get(task_choice_list[i],0)
>             self.task_choice_list.append((task_choice_list[i],idx))
>             counters[task_choice_list[i]] = idx + 1
> 
> 
>     def __len__(self):
>         return np.sum([len(t) for t in self.tasks])
> 
>     def __repr__(self):
>         task_str = ", ".join([str(t) for t in self.tasks])
>         return f"MultiDataset(tasks: {task_str})"
> 
>     def __getitem__(self,key):
>         if isinstance(key, int):
>             task_idx, example_idx = self.task_choice_list[key]
>             task = self.tasks[task_idx]
>             example = task[example_idx]
>             example["task_name"] = task.info.builder_name
>             return example
>         elif isinstance(key, slice):
>             raise NotImplementedError()
> 
>     def __iter__(self):
>         for i in range(len(self)):
>             yield self[i]
> 
> 
> def load_multitask(*datasets):
>     '''Create multitask datasets per split'''
> 
>     def _get_common_splits(datasets):
>         '''Finds the common splits present in all self.datasets'''
>         min_set = None
>         for dataset in datasets:
>             if min_set != None:
>                 min_set.intersection(set(dataset.keys()))
>             else:
>                 min_set = set(dataset.keys())
>         return min_set
> 
>     common_splits = _get_common_splits(datasets)
>     out = {}
>     for split in common_splits:
>         out[split] = MultiDataset([d[split] for d in datasets])
>     return out
> 
> 
> ##########################################
> # Dataset Flattening
> 
> def flatten(dataset,flatten_fn):
>     for k in dataset.keys():
>         if isinstance(dataset[k],nlp.Dataset):
>             dataset[k] = dataset[k].map(flatten_fn,remove_columns=dataset[k].column_names)
> 
> # Squad
> def flatten_squad(example):
>     return {"source": "squad context: " + example['context'] + " question: " + example['question'],
>           "target":example["answers"]["text"]}
> squad = nlp.load_dataset("squad")
> flatten(squad,flatten_squad)
> 
> # CNN_DM
> def flatten_cnn_dm(example):
>     return {"source": "cnn_dm: " + example['article'],"target":[example["highlights"]]}
> cnn_dm = nlp.load_dataset("cnn_dailymail", "3.0.0")
> flatten(cnn_dm,flatten_cnn_dm)
> 
> #############################################
> 
> mtds = load_multitask(squad,cnn_dm)
> 
> for example in mtds["train"]:
>     print(example["task_name"],example["target"])
> ```
> 
> Let me know if you have any thoughts. I've started using this in some of my projects and it seems to work. If people are happy with the general approach for a first version, I can make a pull request.

Not sure if this is what I'm looking for, but I implemented a version of Examples-Proportional mixing supporting only the basic feature [here](https://stackoverflow.com/a/74070116/10732321), seems to work in my project. 
2022-10-20T11:35:08Z
https://github.com/huggingface/datasets/issues/217#issuecomment-1285374843
https://api.github.com/repos/huggingface/datasets/issues/217
IC_kwDODunzps5MnUN7
https://api.github.com/repos/huggingface/datasets/issues/comments/1285374843/reactions
2022-10-20T11:35:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1285374843
https://avatars.githubusercontent.com/u/43276957?v=4
https://api.github.com/users/StefanHeng/events{/privacy}
https://api.github.com/users/StefanHeng/followers
https://api.github.com/users/StefanHeng/following{/other_user}
https://api.github.com/users/StefanHeng/gists{/gist_id}
https://github.com/StefanHeng
StefanHeng
MDQ6VXNlcjQzMjc2OTU3
https://api.github.com/users/StefanHeng/orgs
https://api.github.com/users/StefanHeng/received_events
https://api.github.com/users/StefanHeng/repos
https://api.github.com/users/StefanHeng/starred{/owner}{/repo}
https://api.github.com/users/StefanHeng/subscriptions
User
https://api.github.com/users/StefanHeng
public
MEMBER
You can use `interleave_datasets` to mix several datasets together. By default it alternates between all the datasets, but you can also provide sampling probabilities if you want to oversample from one of the datasets

```python
from datasets import load_dataset, interleave_datasets

squad = load_dataset("squad", split="train")
cnn_dm = load_dataset("cnn_dailymail", "3.0.0", split="train")
ds = interleave_datasets([squad, cnn_dm])

print(ds[0])
# {'id': '5733be284776f41900661182',
#  'title': 'University_of_Notre_Dame',
#  'context': 'Architecturally, the school has a Catholic character...',
#  'question': 'To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?',
#  'answers': {'text': ['Saint Bernadette Soubirous'], 'answer_start': [515]},
#  'article': None,
#  'highlights': None}
print(ds[1])
# {'id': '42c027e4ff9730fbb3de84c1af0d2c506e41c3e4',
#  'title': None,
#  'context': None,
#  'question': None,
#  'answers': None,
#  'article': 'LONDON, England (Reuters) -- Harry Potter star Daniel Radcliffe...',
#  'highlights': "Harry Potter star Daniel Radcliffe..."}
```

see docs at https://huggingface.co/docs/datasets/v2.6.1/en/package_reference/main_classes#datasets.interleave_datasets
2022-10-20T12:30:14Z
https://github.com/huggingface/datasets/issues/217#issuecomment-1285462637
https://api.github.com/repos/huggingface/datasets/issues/217
IC_kwDODunzps5Mnppt
https://api.github.com/repos/huggingface/datasets/issues/comments/1285462637/reactions
2022-10-20T12:32:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1285462637
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I also have this implementation of multi-task sampler here which I used it to tune T5: https://github.com/rabeehk/hyperformer/blob/main/hyperformer/data/multitask_sampler.py 
2022-10-22T00:43:39Z
https://github.com/huggingface/datasets/issues/217#issuecomment-1287561635
https://api.github.com/repos/huggingface/datasets/issues/217
IC_kwDODunzps5MvqGj
https://api.github.com/repos/huggingface/datasets/issues/comments/1287561635/reactions
2022-10-22T00:45:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1287561635
https://avatars.githubusercontent.com/u/6278280?v=4
https://api.github.com/users/rabeehk/events{/privacy}
https://api.github.com/users/rabeehk/followers
https://api.github.com/users/rabeehk/following{/other_user}
https://api.github.com/users/rabeehk/gists{/gist_id}
https://github.com/rabeehk
rabeehk
MDQ6VXNlcjYyNzgyODA=
https://api.github.com/users/rabeehk/orgs
https://api.github.com/users/rabeehk/received_events
https://api.github.com/users/rabeehk/repos
https://api.github.com/users/rabeehk/starred{/owner}{/repo}
https://api.github.com/users/rabeehk/subscriptions
User
https://api.github.com/users/rabeehk
public
It seems like many of the best performing models on the GLUE benchmark make some use of multitask learning (simultaneous training on multiple tasks).

The [T5 paper](https://arxiv.org/pdf/1910.10683.pdf) highlights multiple ways of mixing the tasks together during finetuning:
- **Examples-proportional mixing** - sample from tasks proportionally to their dataset size
- **Equal mixing** - sample uniformly from each task
- **Temperature-scaled mixing** - The generalized approach used by multilingual BERT which uses a temperature T, where the mixing rate of each task is raised to the power 1/T and renormalized. When T=1 this is equivalent to equal mixing, and becomes closer to equal mixing with increasing T.

Following this discussion https://github.com/huggingface/transformers/issues/4340 in [transformers](https://github.com/huggingface/transformers), @enzoampil suggested that the `nlp` library might be a better place for this functionality.

Some method for combining datasets could be implemented ,e.g.
```
dataset = nlp.load_multitask(['squad','imdb','cnn_dm'], temperature=2.0, ...)
```

We would need a few additions:
- Method of identifying the tasks - how can we support adding a string to each task as an identifier: e.g. 'summarisation: '?
- Method of combining the metrics - a standard approach is to use the specific metric for each task and add them together for a combined score.

It would be great to support common use cases such as pretraining on the GLUE benchmark before fine-tuning on each GLUE task in turn. 

I'm willing to write bits/most of this I just need some guidance on the interface and other library details so I can integrate it properly.

