Task casting for text classification & question answering
CONTRIBUTOR
cc @abhi1thakur 
2021-04-23T16:09:05Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-825762119
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDgyNTc2MjExOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/825762119/reactions
2021-04-23T16:09:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/825762119
https://avatars.githubusercontent.com/u/33657802?v=4
https://api.github.com/users/SBrandeis/events{/privacy}
https://api.github.com/users/SBrandeis/followers
https://api.github.com/users/SBrandeis/following{/other_user}
https://api.github.com/users/SBrandeis/gists{/gist_id}
https://github.com/SBrandeis
SBrandeis
MDQ6VXNlcjMzNjU3ODAy
https://api.github.com/users/SBrandeis/orgs
https://api.github.com/users/SBrandeis/received_events
https://api.github.com/users/SBrandeis/repos
https://api.github.com/users/SBrandeis/starred{/owner}{/repo}
https://api.github.com/users/SBrandeis/subscriptions
User
https://api.github.com/users/SBrandeis
public
MEMBER
Looks really nice so far, thanks !
Maybe if a dataset doesn't have a template for a specific task we could try the default template of this task ?
2021-04-23T18:38:56Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-825847802
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDgyNTg0NzgwMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/825847802/reactions
2021-04-23T18:38:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/825847802
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
hey @SBrandeis @lhoestq,

i now have a better idea about what you guys are trying to achieve with the task templates and have a few follow-up questions:

1. how did you envision using `DatasetInfo` for running evaluation? my understanding is that all `dataset_infos.json` files are stored in the `datasets` repo (unlike `transformers` where each model's weights etc are stored in a dedicated repo). 
this suggests the following workflow:

```
- git clone datasets
- load target dataset to evaluate
- load `dataset_infos.json` for target dataset
- run eval for each task template in `task_templates`
- store metrics as evaluation cards (similar to what is done in `autonlp`)
```
2. assuming the above workflow, i see that the current `TaskTemplate` attributes of `task`, `input_schema`, and `label_schema` still require some wrangling from `dataset_infos.json` to reproduce additional mappings like `label2id` that we'd need for e.g. text classification. an alternative would be to instantiate the task template class directly from the JSON with something like
```python
from datasets.tasks import TextClassification
from transformers import AutoModelForSequenceClassification, AutoConfig

tc = TextClassification.from_json("path/to/dataset_infos.json")
# load a model with the desired config
model_ckpt = ...
config = AutoConfig.from_pretrained(model_ckpt, label2id=tc.label2id, id2label=tc.id2label)
model = AutoModelForSequenceClassification.from_pretrained(model_ckpt, config=config)
# run eval ...
```
perhaps this is what @SBrandeis had in mind with the `TaskTemplate.from_dict` method?

3. i personally prefer using `task_templates` over `supervised_keys` because it encourages the contributor to think in terms of 1 or more tasks. my question here is do we currently use `supervised_keys` for anything important in the `datasets` library?
2021-05-05T07:49:33Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-832486915
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDgzMjQ4NjkxNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/832486915/reactions
2021-05-05T07:49:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/832486915
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
CONTRIBUTOR
1. How do you envision using DatasetInfo for running evaluation?

The initial idea was to be able to do something like this:
```python
from datasets import load_dataset
dset = load_dataset("name", task="binary_classification")
# OR
dset = load_dataset("name")
dset = dset.prepare_for_task("binary_classification")
```

2. I don't think that's needed if we proceed as mentioned above

3. `supervised_keys` are mostly a legacy compatibility thing with TF datasets, not sure it's used for anything right now. I'll let @lhoestq give more details on that

[Edit 1] Typo
2021-05-05T09:28:24Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-832548115
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDgzMjU0ODExNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/832548115/reactions
2021-05-05T09:29:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/832548115
https://avatars.githubusercontent.com/u/33657802?v=4
https://api.github.com/users/SBrandeis/events{/privacy}
https://api.github.com/users/SBrandeis/followers
https://api.github.com/users/SBrandeis/following{/other_user}
https://api.github.com/users/SBrandeis/gists{/gist_id}
https://github.com/SBrandeis
SBrandeis
MDQ6VXNlcjMzNjU3ODAy
https://api.github.com/users/SBrandeis/orgs
https://api.github.com/users/SBrandeis/received_events
https://api.github.com/users/SBrandeis/repos
https://api.github.com/users/SBrandeis/starred{/owner}{/repo}
https://api.github.com/users/SBrandeis/subscriptions
User
https://api.github.com/users/SBrandeis
public
MEMBER
> The initial idea was to be able to do something like this:
> 
> ```python
> from datasets import load_dataset
> dset = load_dataset("name", task="binary_classification")
> # OR
> dset = load_dataset("name")
> dset = dset.prepare_for_task("binary_classification")
> ```

ah that's very elegant! just so i've completely understood, the result would be that the relevant column names of `dset` would be mapped to e.g. `text` and `label` and thus we'd have a uniform schema for the evaluation of all `binary_classification` tasks?
2021-05-05T09:42:00Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-832555462
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDgzMjU1NTQ2Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/832555462/reactions
2021-05-05T09:42:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/832555462
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
CONTRIBUTOR
That's correct! Also, the features need to be appropriately casted
For a classification task for example, we would need to cast the datasets features to something like this:
```python
datasets.Features({
    "text": datasets.Value("string"),
    "label": datasets.ClassLabel(names=[...]),
})
```

2021-05-05T10:06:21Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-832569777
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDgzMjU2OTc3Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/832569777/reactions
2021-05-05T10:06:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/832569777
https://avatars.githubusercontent.com/u/33657802?v=4
https://api.github.com/users/SBrandeis/events{/privacy}
https://api.github.com/users/SBrandeis/followers
https://api.github.com/users/SBrandeis/following{/other_user}
https://api.github.com/users/SBrandeis/gists{/gist_id}
https://github.com/SBrandeis
SBrandeis
MDQ6VXNlcjMzNjU3ODAy
https://api.github.com/users/SBrandeis/orgs
https://api.github.com/users/SBrandeis/received_events
https://api.github.com/users/SBrandeis/repos
https://api.github.com/users/SBrandeis/starred{/owner}{/repo}
https://api.github.com/users/SBrandeis/subscriptions
User
https://api.github.com/users/SBrandeis
public
MEMBER
3. We can ignore `supervised_keys` (it came from TFDS and we're not using it) and use `task_templates`
2021-05-05T10:27:58Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-832583012
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDgzMjU4MzAxMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/832583012/reactions
2021-05-05T10:27:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/832583012
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
great, thanks a lot for your answers! now it's much clearer what i need to do next 😃 
2021-05-05T11:43:08Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-832622193
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDgzMjYyMjE5Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/832622193/reactions
2021-05-05T11:43:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/832622193
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
hey @lhoestq @SBrandeis, 

i've made some small tweaks to @SBrandeis's code so that `Dataset.prepare_for_task` is called in `DatasetBuilder`. using the `emotion` dataset as a test case, the following now works:

 ```python
# DatasetDict with default columns
ds = load_dataset("./datasets/emotion/")
# DatasetDict({
#     train: Dataset({
#         features: ['tweet', 'emotion'],
#         num_rows: 16000
#     })
#     validation: Dataset({
#         features: ['tweet', 'emotion'],
#         num_rows: 2000
#     })
#     test: Dataset({
#         features: ['tweet', 'emotion'],
#         num_rows: 2000
#     })
# })

# DatasetDict with remapped columns
ds = load_dataset("./datasets/emotion/", task="text_classification")
DatasetDict({
#     train: Dataset({
#         features: ['text', 'label'],
#         num_rows: 16000
#     })
#     validation: Dataset({
#         features: ['text', 'label'],
#         num_rows: 2000
#     })
#     test: Dataset({
#         features: ['text', 'label'],
#         num_rows: 2000
#     })
# })

# Dataset with default columns
ds = load_dataset("./datasets/emotion/", split="train")
# Map/cast features
ds = ds.prepare_for_task("text_classification")
# Dataset({
#     features: ['text', 'label'],
#     num_rows: 16000
# })
```

i have a few follow-up questions / remarks:

1. i'm working under the assumption that contributors / users only provide a unique set of task types. in particular, the current implementation does not support something like:
```python
task_templates=[TextClassification(labels=class_names, text_column="tweet", label_column="emotion"), TextClassification(labels=class_names, text_column="some_other_column", label_column="some_other_column")]
```
since we use `TaskTemplate.task` and the filter for compatible templates in `Dataset.prepare_for_task`. should we support these scenarios? my hunch is that this is rare in practice, but please correct me if i'm wrong.

2. when we eventually run evaluation for `transformers` models, i expect we'll be using the `Trainer` for which we can pass the standard label names to `TrainingArguments.label_names`. if that's the case, it might be prudent to heed the warning from the [docs](https://huggingface.co/transformers/main_classes/trainer.html?highlight=trainer#trainer) and use `labels` instead of `label` in the schema:
> your model can accept multiple label arguments (use the label_names in your TrainingArguments to indicate their name to the Trainer) but none of them should be named "label".

3. i plan to forge ahead on the rest of the pipeline taxonomy. please let me know if you'd prefer smaller, self-contained pull requests (e.g. one per task)
2021-05-07T13:29:41Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-834389673
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDgzNDM4OTY3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/834389673/reactions
2021-05-07T13:29:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/834389673
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
hey @lhoestq @SBrandeis, i think this is ready for another review 😃 

in addition to a few comments / questions i've left in the pr, here's a few remarks:

1. after some experimentation, i decided against allowing the user to specify nested column names for question-answering. i couldn't find a simple solution with the current api and suspect that i'd have to touch many areas of `datasets` to "unflatten" columns in a generic fashion.
2. in the current implementation, the user can specify the outer column name for question-answering, but is expected to follow the inner schema for e.g. `answers.text` and `answers.answer_start`. we can decide later how much flexibility we want to give users
3. i added a few unit tests
4. as discussed, let's keep this pr focused on text classification / question answering and i'll add the other tasks in separate prs
5. i renamed the tasks e.g. `text_classification` -> `text-classification` for consistency with the `Trainer` model cards [here](https://github.com/huggingface/transformers/pull/11599#pullrequestreview-656371007).
2021-05-10T20:32:20Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-837293390
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDgzNzI5MzM5MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/837293390/reactions
2021-05-11T08:21:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/837293390
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
i'm not sure why the benchmarks are getting cancelled - is this expected?
2021-05-14T14:30:46Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-841280869
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDg0MTI4MDg2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/841280869/reactions
2021-05-14T14:30:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/841280869
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
> i'm not sure why the benchmarks are getting cancelled - is this expected?

Hmm I don't know. It's certainly unrelated to this PR though. Maybe github has some issues
2021-05-14T14:52:38Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-841293557
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDg0MTI5MzU1Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/841293557/reactions
2021-05-14T14:52:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/841293557
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Something is happening with actions: https://www.githubstatus.com/
2021-05-14T14:57:13Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-841296270
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDg0MTI5NjI3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/841296270/reactions
2021-05-14T14:57:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/841296270
https://avatars.githubusercontent.com/u/33657802?v=4
https://api.github.com/users/SBrandeis/events{/privacy}
https://api.github.com/users/SBrandeis/followers
https://api.github.com/users/SBrandeis/following{/other_user}
https://api.github.com/users/SBrandeis/gists{/gist_id}
https://github.com/SBrandeis
SBrandeis
MDQ6VXNlcjMzNjU3ODAy
https://api.github.com/users/SBrandeis/orgs
https://api.github.com/users/SBrandeis/received_events
https://api.github.com/users/SBrandeis/repos
https://api.github.com/users/SBrandeis/starred{/owner}{/repo}
https://api.github.com/users/SBrandeis/subscriptions
User
https://api.github.com/users/SBrandeis
public
MEMBER
hey @lhoestq and @SBrandeis, i've: 

* extended the `prepare_for_task` API along the lines that @lhoestq suggested. i wasn't entirely sure what the `datasets` convention is for docstrings with mixed types, so please see if my proposal makes sense
* added a few new tests to check that we trigger the value errors on incorrect input

i think this is ready for another review :)
2021-05-17T16:47:22Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-842476218
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDg0MjQ3NjIxOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/842476218/reactions
2021-05-17T17:12:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/842476218
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
> Looks all good thank you :)
> 
> Can you also add `prepare_for_task` in the `main_classes.rst` file of the documentation ?

Done! I also remembered that I needed to do the same for `DatasetDict`, so included this as well :)
2021-05-17T19:53:26Z
https://github.com/huggingface/datasets/pull/2255#issuecomment-842588824
https://api.github.com/repos/huggingface/datasets/issues/2255
MDEyOklzc3VlQ29tbWVudDg0MjU4ODgyNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/842588824/reactions
2021-05-17T19:53:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/842588824
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
This PR implements task preparation for a given task, in the continuation of #2143 

Task taxonomy follows 🤗 Transformers's pipelines taxonomy: https://github.com/huggingface/transformers/tree/master/src/transformers/pipelines

Edit by @lewtun:

This PR implements support for the following tasks:

* `text-classification`
* `question-answering`

The intended usage is as follows:

```python
# Load a dataset with default column names / features
ds = load_dataset("dataset_name")
# Cast column names / features to schema. Casting is defined in the dataset's `DatasetInfo`
ds = ds.prepare_for_task(task="text-classification")
# Casting can also be realised during load
ds = load_dataset("dataset_name", task="text-classification")
# We can also combine shared tasks across dataset concatenation
ds1 = load_dataset("dataset_name_1", task="text-classification")
ds2 = load_dataset("dataset_name_2", task="text-classification")
# If the tasks have the same schema, so will `ds_concat`
ds_concat = concatenate_datasets([ds1, ds2])
```

Note that the current implementation assumes that `DatasetInfo.task_templates` has been pre-defined by the user / contributor when overriding the `MyDataset(GeneratorBasedBuilder)._info` function.

As pointed out by @SBrandeis, for evaluation we'll need a way to detect which datasets are already have a compatible schema so we don't have to edit hundreds of dataset scripts. One possibility is to check if the schema features are a subset of the dataset ones, e.g.

```python
squad = load_dataset("./datasets/squad", split="train")
qa = QuestionAnswering()
schema = Features({**qa.input_schema, **qa.label_schema})
assert all(item in squad.features.items() for item in schema.items())
```