Support for multiple configs in packaged modules via metadata yaml info
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-12-02T16:48:06Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1335523916
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5PmnpM
https://api.github.com/repos/huggingface/datasets/issues/comments/1335523916/reactions
2023-07-13T13:28:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1335523916
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
feel free to merge `main` into your PR to fix the CI :)
2023-03-10T14:07:35Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1463855229
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5XQKh9
https://api.github.com/repos/huggingface/datasets/issues/comments/1463855229/reactions
2023-03-10T14:07:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1463855229
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Let me see if I can fix the pattern thing ^^'
2023-03-30T15:05:41Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1490469736
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5Y1sNo
https://api.github.com/repos/huggingface/datasets/issues/comments/1490469736/reactions
2023-03-30T15:05:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1490469736
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Hmm I think it would be easier to specify the `data_files` in the end, because having a split pattern like `{split}-...` at the root of the repository can lead to unexpected behaviors IMO, and we probably don't want to have a different behavior for `data_files` depending if it's inside a `data_dir` or not

Maybe something like
```yaml
builder_config:
  data_dir: data_dir
  data_files:
  - split: train
    pattern: train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*
```
2023-03-31T18:04:32Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1492392310
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5Y9Bl2
https://api.github.com/repos/huggingface/datasets/issues/comments/1492392310/reactions
2023-03-31T18:04:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1492392310
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
 > Also, I'm not sure if it's a good idea to have this field in the YAML metadata - Transformers use this part of the card only for Hub-related stuff (widgets, tags, CO2 emission, etc.), and I think we should aim to do the same in Datasets. We could achieve this by having these kwargs in a special file (they can be seen as a faster way of defining a builder (builder script) that subclasses a packaged builder) and removing the dataset_info field (the only useful info there seem to be features and we can fetch those directly from a dataset script/Parquet files).

Something like `config.json`?

```json
{
    "data_dir": "data"
    "data_files":  {
        "train": "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
    }
}
```

we could also support lists for several configs
2023-04-01T13:02:24Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1492964964
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5Y_NZk
https://api.github.com/repos/huggingface/datasets/issues/comments/1492964964/reactions
2023-04-01T13:02:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1492964964
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
opened https://github.com/huggingface/datasets/issues/5694
2023-04-01T13:08:24Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1492966702
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5Y_N0u
https://api.github.com/repos/huggingface/datasets/issues/comments/1492966702/reactions
2023-04-01T13:08:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1492966702
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I opened a PR to this PR to add data_files in YAML: https://github.com/polinaeterna/datasets/pull/1

```yaml
builder_config:
  data_files:
  - split: train
    pattern: data/train-*
```
2023-04-04T14:45:01Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1496101140
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5ZLLEU
https://api.github.com/repos/huggingface/datasets/issues/comments/1496101140/reactions
2023-04-04T14:45:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1496101140
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Let me open a PR to see if I can move the data files resolution outside of the MetadataConfigs to not modify it in-place
2023-04-28T15:40:57Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1527751103
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5bD6G_
https://api.github.com/repos/huggingface/datasets/issues/comments/1527751103/reactions
2023-04-28T15:40:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1527751103
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I wonder if we can make the cache backward compatible: we could just check if the cache directory with the old path exists. It will be useful for the research team which has a big datasets cache
2023-05-04T16:40:57Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1535061977
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5bfy_Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1535061977/reactions
2023-05-04T16:40:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1535061977
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> I wonder if we can make the cache backward compatible: we could just check if the cache directory with the old path exists. It will be useful for the research team which has a big datasets cache

![image](https://github.com/huggingface/datasets/assets/16348744/90a96e79-2a0d-4d37-95bd-b75fa962c094)

In the next PR maybe? :D 
It's possible but requires some additional logic to correctly pass old `config_kwargs` (which used to include `data_files` but now it's `None` for builders from metadata) to generate the hash which is used to create the path.
2023-05-10T12:15:41Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1542102394
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5b6p16
https://api.github.com/repos/huggingface/datasets/issues/comments/1542102394/reactions
2023-05-10T12:15:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1542102394
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
MEMBER
If we only consider datasets that were pushed to hub, it's just a matter of using `"{username}__parquet"` instead of `"{username}__{dataset_name}"` in the cache directory name. The hashes stay the same :)

EDIT: and the config name
EDIT2: and the arrow file names
2023-05-10T12:42:44Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1542144960
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5b60PA
https://api.github.com/repos/huggingface/datasets/issues/comments/1542144960/reactions
2023-05-10T12:56:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1542144960
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Did a small PR for backward compatibility, it was easy to add in the end: https://github.com/polinaeterna/datasets/pull/3
2023-05-10T13:24:36Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1542207876
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5b7DmE
https://api.github.com/repos/huggingface/datasets/issues/comments/1542207876/reactions
2023-05-10T13:24:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1542207876
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Just created a branch [dev-3.0](https://github.com/huggingface/datasets/tree/dev-3.0) in which we can merge this one and the other datasets 3.0 related PRs
2023-05-15T14:22:57Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1547972474
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5cRC96
https://api.github.com/repos/huggingface/datasets/issues/comments/1547972474/reactions
2023-05-15T14:22:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1547972474
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq why can't we merge it in main?
2023-05-15T18:13:27Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1548333565
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5cSbH9
https://api.github.com/repos/huggingface/datasets/issues/comments/1548333565/reactions
2023-05-15T18:13:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548333565
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
MEMBER
We can, it was just in case we had other things to merge after @mariosasko or @albertvillanova 's reviews
2023-05-15T23:31:22Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1548752054
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5cUBS2
https://api.github.com/repos/huggingface/datasets/issues/comments/1548752054/reactions
2023-05-15T23:31:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548752054
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq @albertvillanova @mariosasko we agreed on having `configs` (in plural) as a metadata field in readme but apparently Hub's yaml validation doesn't allow it to be not a list :D 
![image](https://github.com/huggingface/datasets/assets/16348744/52131ee8-80e0-4f6e-90cd-8ff83caf4625)
(with `config` (in singular) it works)

edit: and now the tests for hub datasets with metadata configs are failing because I cannot change the yaml there...
2023-05-31T18:53:19Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1570751476
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5dn8P0
https://api.github.com/repos/huggingface/datasets/issues/comments/1570751476/reactions
2023-05-31T18:55:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1570751476
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
COLLABORATOR
> we agreed on having configs (in plural) as a metadata field in readme but apparently Hub's yaml validation doesn't allow it to be not a list :D

If the `configs` field is specified in the YAML, the Hub can use it to [improve](https://github.com/huggingface/moon-landing/blob/97aca4cac32fbb7d84ce5eba9b18afad87968c4a/server/views/components/DatasetLibraryModal/datasetLibrarySnippets.ts#L11) the `Use in dataset library` snippet by listing the possible config values in `load_dataset`. So I think this needs to be fixed on the Hub side.

PS: I couldn't find an instance of someone using this field on the Hub, so I think using it for this feature is OK.
2023-06-01T19:26:57Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1572653548
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5dvMns
https://api.github.com/repos/huggingface/datasets/issues/comments/1572653548/reactions
2023-06-01T19:26:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1572653548
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
> I couldn't find an instance of someone using this field on the Hub, so I think using it for this feature is OK.

@mariosasko I think it's because @lhoestq renamed `configs` to `config_names` in all canonical datasets :D so yes, `configs` field is now supposed to include custom configuration parameters introduced in this PR, and `config_names` is used (not really used lol) for list of strings of config names. It's being fixed on the Hub's side https://github.com/huggingface/moon-landing/pull/6490
2023-06-02T09:43:21Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1573452176
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5dyPmQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1573452176/reactions
2023-06-02T09:43:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1573452176
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
MEMBER
after more thought I agree it's maybe overkill to do a major release for this one, since we have a good backward compatibility
2023-06-09T17:35:14Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1584927964
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5eeBTc
https://api.github.com/repos/huggingface/datasets/issues/comments/1584927964/reactions
2023-06-09T17:35:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1584927964
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
There is one edge case I forgot to mention in the reviews - I think it's a good idea to support passing config params that are functions (Pandas uses them a lot) using this API (e.g. `converters` in the CSV config for converting a string column into a sequence). I see two solutions: string blocks with Python code in YAML or PyYAML [tags](https://pyyaml.org/wiki/PyYAMLDocumentation#yaml-tags-and-python-types). 

But I think this can be addressed later.
2023-06-13T17:15:22Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1589719305
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5ewTEJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1589719305/reactions
2023-06-13T17:15:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1589719305
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
I'm resolving the conflicts and writing some docs :) let's merge this soon !
2023-07-07T12:55:58Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1625376763
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5g4Uf7
https://api.github.com/repos/huggingface/datasets/issues/comments/1625376763/reactions
2023-07-07T12:55:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1625376763
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005868 / 0.011353 (-0.005485) | 0.003544 / 0.011008 (-0.007464) | 0.080329 / 0.038508 (0.041821) | 0.061072 / 0.023109 (0.037963) | 0.307802 / 0.275898 (0.031904) | 0.340353 / 0.323480 (0.016873) | 0.004665 / 0.007986 (-0.003321) | 0.002779 / 0.004328 (-0.001550) | 0.062065 / 0.004250 (0.057815) | 0.046350 / 0.037052 (0.009297) | 0.312045 / 0.258489 (0.053556) | 0.353524 / 0.293841 (0.059683) | 0.026965 / 0.128546 (-0.101581) | 0.007906 / 0.075646 (-0.067740) | 0.260678 / 0.419271 (-0.158593) | 0.044167 / 0.043533 (0.000634) | 0.309757 / 0.255139 (0.054618) | 0.340188 / 0.283200 (0.056988) | 0.020440 / 0.141683 (-0.121243) | 1.486886 / 1.452155 (0.034732) | 1.548330 / 1.492716 (0.055614) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.188658 / 0.018006 (0.170652) | 0.422204 / 0.000490 (0.421715) | 0.003508 / 0.000200 (0.003308) | 0.000068 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025173 / 0.037411 (-0.012238) | 0.072868 / 0.014526 (0.058343) | 0.084817 / 0.176557 (-0.091739) | 0.151667 / 0.737135 (-0.585468) | 0.085632 / 0.296338 (-0.210706) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400998 / 0.215209 (0.185789) | 4.022274 / 2.077655 (1.944619) | 2.025768 / 1.504120 (0.521648) | 1.874193 / 1.541195 (0.332998) | 2.006537 / 1.468490 (0.538047) | 0.501799 / 4.584777 (-4.082978) | 2.987487 / 3.745712 (-0.758225) | 4.552295 / 5.269862 (-0.717566) | 2.775859 / 4.565676 (-1.789817) | 0.057596 / 0.424275 (-0.366679) | 0.006449 / 0.007607 (-0.001158) | 0.470776 / 0.226044 (0.244732) | 4.725933 / 2.268929 (2.457005) | 2.480130 / 55.444624 (-52.964494) | 2.183919 / 6.876477 (-4.692558) | 2.408052 / 2.142072 (0.265979) | 0.584038 / 4.805227 (-4.221190) | 0.124964 / 6.500664 (-6.375701) | 0.060939 / 0.075469 (-0.014530) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.221263 / 1.841788 (-0.620524) | 18.326372 / 8.074308 (10.252064) | 13.398937 / 10.191392 (3.207545) | 0.149153 / 0.680424 (-0.531271) | 0.016941 / 0.534201 (-0.517260) | 0.332106 / 0.579283 (-0.247177) | 0.339958 / 0.434364 (-0.094406) | 0.378125 / 0.540337 (-0.162212) | 0.517787 / 1.386936 (-0.869149) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005927 / 0.011353 (-0.005426) | 0.003607 / 0.011008 (-0.007402) | 0.062925 / 0.038508 (0.024417) | 0.058676 / 0.023109 (0.035566) | 0.362129 / 0.275898 (0.086231) | 0.395864 / 0.323480 (0.072384) | 0.004652 / 0.007986 (-0.003334) | 0.002893 / 0.004328 (-0.001435) | 0.062696 / 0.004250 (0.058445) | 0.049988 / 0.037052 (0.012935) | 0.365366 / 0.258489 (0.106877) | 0.412326 / 0.293841 (0.118485) | 0.027118 / 0.128546 (-0.101429) | 0.008179 / 0.075646 (-0.067467) | 0.068048 / 0.419271 (-0.351223) | 0.041065 / 0.043533 (-0.002468) | 0.359858 / 0.255139 (0.104719) | 0.386589 / 0.283200 (0.103390) | 0.020467 / 0.141683 (-0.121216) | 1.438070 / 1.452155 (-0.014084) | 1.479617 / 1.492716 (-0.013099) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231516 / 0.018006 (0.213510) | 0.413407 / 0.000490 (0.412917) | 0.000358 / 0.000200 (0.000158) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026071 / 0.037411 (-0.011340) | 0.076486 / 0.014526 (0.061960) | 0.085943 / 0.176557 (-0.090613) | 0.138087 / 0.737135 (-0.599048) | 0.087466 / 0.296338 (-0.208872) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417711 / 0.215209 (0.202502) | 4.171915 / 2.077655 (2.094260) | 2.140677 / 1.504120 (0.636557) | 1.960164 / 1.541195 (0.418969) | 2.002134 / 1.468490 (0.533644) | 0.499699 / 4.584777 (-4.085078) | 2.991814 / 3.745712 (-0.753898) | 2.906589 / 5.269862 (-2.363272) | 1.842305 / 4.565676 (-2.723372) | 0.057633 / 0.424275 (-0.366642) | 0.006465 / 0.007607 (-0.001142) | 0.492874 / 0.226044 (0.266830) | 4.931613 / 2.268929 (2.662684) | 2.623161 / 55.444624 (-52.821463) | 2.310624 / 6.876477 (-4.565853) | 2.483146 / 2.142072 (0.341074) | 0.586910 / 4.805227 (-4.218317) | 0.124681 / 6.500664 (-6.375983) | 0.061561 / 0.075469 (-0.013908) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.319111 / 1.841788 (-0.522677) | 18.637326 / 8.074308 (10.563018) | 13.803912 / 10.191392 (3.612520) | 0.143989 / 0.680424 (-0.536435) | 0.017025 / 0.534201 (-0.517176) | 0.333156 / 0.579283 (-0.246127) | 0.342163 / 0.434364 (-0.092201) | 0.380357 / 0.540337 (-0.159981) | 0.512261 / 1.386936 (-0.874675) |

</details>
</details>

![](https://cml.dev/watermark.png#f49a16346dc35e5eabeec39778d0f2e4e850dfd7 "CML watermark")

2023-07-13T13:36:24Z
https://github.com/huggingface/datasets/pull/5331#issuecomment-1634261078
https://api.github.com/repos/huggingface/datasets/issues/5331
IC_kwDODunzps5haNhW
https://api.github.com/repos/huggingface/datasets/issues/comments/1634261078/reactions
2023-07-13T13:36:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1634261078
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
will solve https://github.com/huggingface/datasets/issues/5209 and https://github.com/huggingface/datasets/issues/5151 and many other...

Config parameters for packaged builders are parsed from `“builder_config”` field in README.md file (separate firs-level field, not part of “dataset_info”), example:
```yaml
---
dataset_info:
...
configs:
  - config_name: v1
    data_dir: v1
    drop_labels: true
  - config_name: v2
    data_dir: v2
    drop_labels: false
```

I tried to align packaged builders with custom configs parsed from metadata with scripts dataset builder as much as possible. Their builders are created dynamically (see `configure_builder_class()` in load.py`) and have `BUILDER_CONFIGS` attribute filled with `BuilderConfig` objects in the same way as for datasets with script.

## load_dataset
1. If there is single config in meta and it doesn’t have a name, the name becomes “default” (as we do for “dataset_info”), [example](https://huggingface.co/datasets/polinaeterna/audiofolder_one_default_config_in_metadata/blob/main/README.md):
```python
load_dataset("ds") == load_dataset("ds", "default")   # load with the params provided in metadata

load_dataset("ds", "random name")  # ValueError: BuilderConfig 'random_name' not found. Available: ['default']
```
2. If there is single config in metadata with `config_name` provided, it becomes a default one (loaded when no `config_name` is specified, [example](https://huggingface.co/datasets/polinaeterna/audiofolder_one_nondefault_config_in_metadata)
```python

load_dataset("ds") == load_dataset("ds", "custom")   # load with the params provided in meta

load_dataset("ds", "random name")  # ValueError: BuilderConfig 'random_name' not found. Available: ['custom']
```
3. If there are several configs in metadata with names [example](https://huggingface.co/datasets/polinaeterna/audiofolder_two_configs_in_metadata/blob/main/README.md)
```python
load_dataset("ds", "v1")  # load with "v1" params

load_dataset("ds", "v2")  # load with "v2" params

load_dataset("ds")  # ValueError: BuilderConfig 'default' not found. Available: ['v1', 'v2']

```
Thanks to @lhoestq and [this change](https://github.com/polinaeterna/datasets/pull/1), it's possible to add `"default"` field in yaml and set it to True, to make the config a default one (loaded when no config is specified):
```yaml
configs:
  - config_name: v1
    drop_labels: true
    default: true
  - config_name: v2
...
```
then `load_dataset("ds") == load_dataset("ds", "v1")`.

## dataset_name and cache
I decided that it’s reasonable to add a `dataset_name` attribute to `DatasetBuilder` class which would be equal to `name` for scripts dataset but reflect a real dataset name for packaged builders (last part of path/name from hub). This is mostly to reorganize cache structure (I believe we can do this in the major release?) because otherwise, with custom configs for packaged builders which were all stored in the same directory, i it was becoming a mess. And in general it makes much more sense like this, from datasets server perspective too, though it’s a breaking change 

So the cache dir has the following structure: `"{namespace__}<dataset_name>/<config_name>/<version>/<hash>/"` and arrow/parquet filenames are also `"<dataset_name>-<split>.arrow"`. 
For example `polinaeterna___audiofolder_two_configs_in_metadata/v1-5532fac9443ea252/0.0.0/6cbdd16f8688354c63b4e2a36e1585d05de285023ee6443ffd71c4182055c0fc/` for `polinaeterna/audiofolder_two_configs_in_metadata` Hub dataset, train arrow file is `audiofolder_two_configs_in_metadata-train.arrow`.

For script datasets it remains unchanged.

## push_to_hub
To support custom configs with `push_to_hub`, the data is put under directory named either as `<config_name>` if `config_name` is **not** "default" or "data" if `config_name` is omitted or "default" (for backward compatibility). `"builder_config"` field is added to README.md, with `config_name` (optional) and `data_files` fields. for `"data_files"`, `"pattern"` parameter is introduced, to resolve data files correctly, see https://github.com/polinaeterna/datasets/pull/1.
 - `ds.push_to_hub("ds")` --> one config ("default"), put under "data" directory, [example](https://huggingface.co/datasets/polinaeterna/push_to_hub_single_config/blob/main/README.md)
```yaml
dataset_info:
...
configs:
  data_files:
  - split: train
    pattern: data/train-*
...
``` 
- `ds.push_to_hub("ds", "custom")` --> put under "custom" directory, [example](https://huggingface.co/datasets/polinaeterna/push_to_hub_singe_nondefault_config/blob/main/README.md)

```yaml
configs:
  config_name: custom
  data_files:
    - split: train
      path: custom/train-*
...
``` 

 - for many configs, [example](https://huggingface.co/datasets/polinaeterna/push_to_hub_many_configs/blob/main/README.md):
```yaml
configs:
  - config_name: v1
    data_files:
      - split: train
        path: v1/train-*
...
  - config_name: v2
    data_files:
      - split: train
        path: v2/train-*
...
```
Thanks to @lhoestq and https://github.com/polinaeterna/datasets/pull/1, when pushing to datasets created **before** this change, README.md is updated accordingly (config for old data is added along with the one that is being pushed).  

`"dataset_info"` yaml field is updated accordingly (new configs are added).
This shouldn't break anything!

TODO in separate PRs:
- [x] docs
- [ ] probably update test cli util (make --save_info not rewrite `builder_config` in readme)