Remove hacking license tags when mirroring datasets on the Hub
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-05-10T06:04:57Z
https://github.com/huggingface/datasets/pull/4302#issuecomment-1121966017
https://api.github.com/repos/huggingface/datasets/issues/4302
IC_kwDODunzps5C39fB
https://api.github.com/repos/huggingface/datasets/issues/comments/1121966017/reactions
2022-05-20T09:48:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1121966017
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
The Hub doesn't allow these characters in the YAML tags, and git push fails if you want to push a dataset card containing these characters.
2022-05-10T09:19:10Z
https://github.com/huggingface/datasets/pull/4302#issuecomment-1122141517
https://api.github.com/repos/huggingface/datasets/issues/4302
IC_kwDODunzps5C4oVN
https://api.github.com/repos/huggingface/datasets/issues/comments/1122141517/reactions
2022-05-10T09:19:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1122141517
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Ok, let me rename the bad config names :) I think I can also keep backward compatibility with a warning
2022-05-11T08:54:04Z
https://github.com/huggingface/datasets/pull/4302#issuecomment-1123386541
https://api.github.com/repos/huggingface/datasets/issues/4302
IC_kwDODunzps5C9YSt
https://api.github.com/repos/huggingface/datasets/issues/comments/1123386541/reactions
2022-05-11T08:54:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1123386541
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Almost done with it btw, will submit a PR that shows all the configuration name changes (from a bit more than 20 datasets)
2022-05-12T13:29:17Z
https://github.com/huggingface/datasets/pull/4302#issuecomment-1124996818
https://api.github.com/repos/huggingface/datasets/issues/4302
IC_kwDODunzps5DDhbS
https://api.github.com/repos/huggingface/datasets/issues/comments/1124996818/reactions
2022-05-12T13:29:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1124996818
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Please, let me know when the renaming of configs is done. If not enough bandwidth, I can take care of it...
2022-05-17T06:37:01Z
https://github.com/huggingface/datasets/pull/4302#issuecomment-1128471339
https://api.github.com/repos/huggingface/datasets/issues/4302
IC_kwDODunzps5DQxsr
https://api.github.com/repos/huggingface/datasets/issues/comments/1128471339/reactions
2022-05-17T06:37:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1128471339
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Will focus on this this afternoon ;)
2022-05-17T10:17:52Z
https://github.com/huggingface/datasets/pull/4302#issuecomment-1128687054
https://api.github.com/repos/huggingface/datasets/issues/4302
IC_kwDODunzps5DRmXO
https://api.github.com/repos/huggingface/datasets/issues/comments/1128687054/reactions
2022-05-17T10:17:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1128687054
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I realized when renaming all the configurations with dots in https://github.com/huggingface/datasets/pull/4365 that it's not ideal for certain cases. For example:
- many configurations have a version like "1.0.0" in their names
- to avoid breaking changes we need to replace dots with underscores in the user input and show a warning, which hurts the experience
- our second most downloaded dataset at the moment is affected: `newsgroup`
- if we disallow dots, then we'll never be able to make the [allenai/c4](https://huggingface.co/datasets/allenai/c4) work with its different configurations since they contain dots, and we can't rename them because they are the official download links

I was thinking of other alternatives:
1. just stop separating tags per config name completely, and have a single flat YAML for all configurations. Dataset search doesn't use this info anyway
2. use another YAML structure to avoid having config names as keys, such as
```yaml
languages:
- config: 20220301_en
  values:
  - en
```

I'm down for 1, to keep things simple
2022-05-18T10:24:27Z
https://github.com/huggingface/datasets/pull/4302#issuecomment-1129837528
https://api.github.com/repos/huggingface/datasets/issues/4302
IC_kwDODunzps5DV_PY
https://api.github.com/repos/huggingface/datasets/issues/comments/1129837528/reactions
2022-05-18T10:29:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1129837528
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq I agree:
- better not changing config names (so that we do not introduce any braking change)
- therefore, we should not use them as keys

In relation with the proposed solutions, I have no strong opinion:
- option 1 is simpler and aligns better with current usage on the Hub (configs are ignored)
- however:
  - we will lose all the information per config we already have (for those datasets containing config keys; contributors made an effort to put that information per config)
  - and this information might be useful on the Hub in the future, in case we would like to enrich the search feature with more granularity; this is only applicable if this feature could eventually make sense

So, no strong opinion...
2022-05-18T15:54:14Z
https://github.com/huggingface/datasets/pull/4302#issuecomment-1130197234
https://api.github.com/repos/huggingface/datasets/issues/4302
IC_kwDODunzps5DXXDy
https://api.github.com/repos/huggingface/datasets/issues/comments/1130197234/reactions
2022-05-18T15:54:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1130197234
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Closing in favor of https://github.com/huggingface/datasets/pull/4367
2022-05-20T09:40:20Z
https://github.com/huggingface/datasets/pull/4302#issuecomment-1132695140
https://api.github.com/repos/huggingface/datasets/issues/4302
IC_kwDODunzps5Dg45k
https://api.github.com/repos/huggingface/datasets/issues/comments/1132695140/reactions
2022-05-20T09:40:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1132695140
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Currently, when mirroring datasets on the Hub, the license tags are hacked: removed of characters "." and "$". On the contrary, this hacking is not applied to community datasets on the Hub. This generates multiple variants of the same tag on the Hub. 

I guess this hacking is no longer necessary:
- it is not applied to community datasets
- all canonical datasets are validated by maintainers before being merged: CI + maintainers make sure license tags are the right ones

Fix #4298.