Dataset configuration
MEMBER
Originally we also though about adding it to the YAML part of the README.md:

```yaml
builder_config:
  data_dir: data
  data_files:
  - split: train
    pattern: "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
```

Having it in the README.md could make it easier to modify it in the UI on HF, and for validation on commit
2023-04-03T13:14:41Z
https://github.com/huggingface/datasets/issues/5694#issuecomment-1494306377
https://api.github.com/repos/huggingface/datasets/issues/5694
IC_kwDODunzps5ZEU5J
https://api.github.com/repos/huggingface/datasets/issues/comments/1494306377/reactions
2023-04-03T13:20:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1494306377
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
From internal discussions we agreed to go with the YAML approach, since it's the one that seems more appropriate to be modified by a human on the Hub or locally (while JSON e.g. for models are usually created programmatically).
2023-04-03T14:47:53Z
https://github.com/huggingface/datasets/issues/5694#issuecomment-1494466780
https://api.github.com/repos/huggingface/datasets/issues/5694
IC_kwDODunzps5ZE8Dc
https://api.github.com/repos/huggingface/datasets/issues/comments/1494466780/reactions
2023-04-03T14:47:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1494466780
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Current format:
```yaml
builder_config:
  data_files:
  - split: train
    pattern: data/train-*
```
2023-04-04T14:54:37Z
https://github.com/huggingface/datasets/issues/5694#issuecomment-1496116897
https://api.github.com/repos/huggingface/datasets/issues/5694
IC_kwDODunzps5ZLO6h
https://api.github.com/repos/huggingface/datasets/issues/comments/1496116897/reactions
2023-04-04T14:54:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1496116897
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Following discussions from https://github.com/huggingface/datasets/pull/5331

We could have something like `config.json` to define the configuration of a dataset.

```json
{
    "data_dir": "data"
    "data_files":  {
        "train": "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
    }
}
```

we could also support a list for several configs with a 'config_name' field.

The alternative was to use YAML in the README.md.

I think it could also support a `dataset_type` field to specify which dataset builder class to use, and the other parameters would be the builder's parameters. Some parameters exist for all builders like `data_files` and `data_dir`, but some parameters are builder specific like `sep` for csv.

This format would be used in `push_to_hub` to be able to push multiple configs.

cc @huggingface/datasets 

EDIT: actually we're going for the YAML approach in README.md