Metric card template
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-03-14T20:14:26Z
https://github.com/huggingface/datasets/pull/3915#issuecomment-1067243474
https://api.github.com/repos/huggingface/datasets/issues/3915
IC_kwDODunzps4_nNfS
https://api.github.com/repos/huggingface/datasets/issues/comments/1067243474/reactions
2022-05-04T10:44:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1067243474
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
CONTRIBUTOR
Looks like a great start! I have a general comment and a few specific comments.

My general comment is I wonder if we need a post for this template and the data and model card templates (or a combined one?) explaining why this documentation is needed and how it serves both the writer and the audience.

Specific comments:
- Maybe we can add some more desiderata to the overview instructions like: what task was the metric originally developed for, what tasks is it used for now, what is the range of possible outputs?
- In the data card, we call the data instances inputs `fields`. It might be good to synchronize on that across the templates and change `input_name` to `input_field`? Also are the instructions for the `input_name` complete? It ends with 'In the *' and I'm not sure what that refers to.
- 'Values' seems ambiguous to me, maybe 'scores' would be more explicit? Also could add a request for the range of possible outputs.
- We could add a reference in the examples section to the overview section if that's where further explanation should go. Suggestion to add: 'Provide a range of examples that show both typical and atypical results' or something similar.
- I'm not sure if we'd want to add this to the example section or make a new section, but it would be good to prompt somewhere for links to specific use cases in HF
- In the limitations and bias section, add 'with links'

2022-03-15T19:51:20Z
https://github.com/huggingface/datasets/pull/3915#issuecomment-1068403968
https://api.github.com/repos/huggingface/datasets/issues/3915
IC_kwDODunzps4_ro0A
https://api.github.com/repos/huggingface/datasets/issues/comments/1068403968/reactions
2022-03-15T19:52:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1068403968
https://avatars.githubusercontent.com/u/26722925?v=4
https://api.github.com/users/mcmillanmajora/events{/privacy}
https://api.github.com/users/mcmillanmajora/followers
https://api.github.com/users/mcmillanmajora/following{/other_user}
https://api.github.com/users/mcmillanmajora/gists{/gist_id}
https://github.com/mcmillanmajora
mcmillanmajora
MDQ6VXNlcjI2NzIyOTI1
https://api.github.com/users/mcmillanmajora/orgs
https://api.github.com/users/mcmillanmajora/received_events
https://api.github.com/users/mcmillanmajora/repos
https://api.github.com/users/mcmillanmajora/starred{/owner}{/repo}
https://api.github.com/users/mcmillanmajora/subscriptions
User
https://api.github.com/users/mcmillanmajora
public
CONTRIBUTOR
Looks like a great start! I have a general comment and a few specific comments.

My general comment is I wonder if we need a post for this template and the data and model card templates (or a combined one?) explaining why this documentation is needed and how it serves both the writer and the audience.

Specific comments:
- Maybe we can add some more desiderata to the overview instructions like: what task was the metric originally developed for, what tasks is it used for now, what is the range of possible outputs?
- In the data card, we call the data instances `fields`. It might be good to synchronize on that across the templates and change `input_name` to `input_field`? Also are the instructions for the `input_name` complete? It ends with 'In the *' and I'm not sure what that refers to.
- 'Values' seems ambiguous to me, maybe 'scores' would be more explicit? Also could add a request for the range of possible outputs.
- We could add a reference to the examples section to the overview section if that's where further explanation should go. Suggestion to add: 'Provide a range of examples that show both typical and atypical results' or something similar.
- I'm not sure if we'd want to add this to the example section or make a new section, but it would be good to prompt somewhere for links to specific use cases in HF
- In the limitations and bias section, add 'with links'

2022-03-15T19:51:20Z
https://github.com/huggingface/datasets/pull/3915#issuecomment-1068403969
https://api.github.com/repos/huggingface/datasets/issues/3915
IC_kwDODunzps4_ro0B
https://api.github.com/repos/huggingface/datasets/issues/comments/1068403969/reactions
2022-03-15T19:51:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1068403969
https://avatars.githubusercontent.com/u/26722925?v=4
https://api.github.com/users/mcmillanmajora/events{/privacy}
https://api.github.com/users/mcmillanmajora/followers
https://api.github.com/users/mcmillanmajora/following{/other_user}
https://api.github.com/users/mcmillanmajora/gists{/gist_id}
https://github.com/mcmillanmajora
mcmillanmajora
MDQ6VXNlcjI2NzIyOTI1
https://api.github.com/users/mcmillanmajora/orgs
https://api.github.com/users/mcmillanmajora/received_events
https://api.github.com/users/mcmillanmajora/repos
https://api.github.com/users/mcmillanmajora/starred{/owner}{/repo}
https://api.github.com/users/mcmillanmajora/subscriptions
User
https://api.github.com/users/mcmillanmajora
public
CONTRIBUTOR
Thanks for your feedback, @mcmillanmajora ! I totally agree that we should write a post -- we were going to write one up when we are done with a good chunk of the metric cards, but we can also do that earlier :smile: 

With regards to your more specific comments:

- It is our intention to put what the metric was developed for (whether it is a specific task or dataset, for example). You can see the [WER](https://github.com/huggingface/datasets/tree/master/metrics/wer) metric card for that.
- `input_field` works for me!
- the values aren't always scores, it's more like the values the metric can take. And it does include the range of possible values, including the max and min, that are outputted.
- I like the suggestion to add: 'Provide a range of examples that show both typical and atypical results' :hugs: 
- I have been putting specific use cases in 'Further references', just because there isn't always something to put there, especially for less popular metrics
2022-03-15T19:57:02Z
https://github.com/huggingface/datasets/pull/3915#issuecomment-1068409485
https://api.github.com/repos/huggingface/datasets/issues/3915
IC_kwDODunzps4_rqKN
https://api.github.com/repos/huggingface/datasets/issues/comments/1068409485/reactions
2022-03-15T19:57:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1068409485
https://avatars.githubusercontent.com/u/14205986?v=4
https://api.github.com/users/sashavor/events{/privacy}
https://api.github.com/users/sashavor/followers
https://api.github.com/users/sashavor/following{/other_user}
https://api.github.com/users/sashavor/gists{/gist_id}
https://github.com/sashavor
sashavor
MDQ6VXNlcjE0MjA1OTg2
https://api.github.com/users/sashavor/orgs
https://api.github.com/users/sashavor/received_events
https://api.github.com/users/sashavor/repos
https://api.github.com/users/sashavor/starred{/owner}{/repo}
https://api.github.com/users/sashavor/subscriptions
User
https://api.github.com/users/sashavor
public
CONTRIBUTOR
Oh cool! I was just looking at the template, it definitely helps seeing an example metric card. Based on just the instructions, I had assumed that examples meant research papers where the metric was used to evaluate a model, but I like the explicit coding examples! 
2022-03-15T20:06:11Z
https://github.com/huggingface/datasets/pull/3915#issuecomment-1068418469
https://api.github.com/repos/huggingface/datasets/issues/3915
IC_kwDODunzps4_rsWl
https://api.github.com/repos/huggingface/datasets/issues/comments/1068418469/reactions
2022-03-15T20:06:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1068418469
https://avatars.githubusercontent.com/u/26722925?v=4
https://api.github.com/users/mcmillanmajora/events{/privacy}
https://api.github.com/users/mcmillanmajora/followers
https://api.github.com/users/mcmillanmajora/following{/other_user}
https://api.github.com/users/mcmillanmajora/gists{/gist_id}
https://github.com/mcmillanmajora
mcmillanmajora
MDQ6VXNlcjI2NzIyOTI1
https://api.github.com/users/mcmillanmajora/orgs
https://api.github.com/users/mcmillanmajora/received_events
https://api.github.com/users/mcmillanmajora/repos
https://api.github.com/users/mcmillanmajora/starred{/owner}{/repo}
https://api.github.com/users/mcmillanmajora/subscriptions
User
https://api.github.com/users/mcmillanmajora
public
CONTRIBUTOR
Oh cool! I was just looking at the template, it definitely helps seeing an example metric card. Based on just the instructions, I had assumed that examples meant research papers where the metric was used to evaluate a model, but I like the explicit coding examples! 
2022-03-15T20:06:11Z
https://github.com/huggingface/datasets/pull/3915#issuecomment-1068418473
https://api.github.com/repos/huggingface/datasets/issues/3915
IC_kwDODunzps4_rsWp
https://api.github.com/repos/huggingface/datasets/issues/comments/1068418473/reactions
2022-03-15T20:06:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1068418473
https://avatars.githubusercontent.com/u/26722925?v=4
https://api.github.com/users/mcmillanmajora/events{/privacy}
https://api.github.com/users/mcmillanmajora/followers
https://api.github.com/users/mcmillanmajora/following{/other_user}
https://api.github.com/users/mcmillanmajora/gists{/gist_id}
https://github.com/mcmillanmajora
mcmillanmajora
MDQ6VXNlcjI2NzIyOTI1
https://api.github.com/users/mcmillanmajora/orgs
https://api.github.com/users/mcmillanmajora/received_events
https://api.github.com/users/mcmillanmajora/repos
https://api.github.com/users/mcmillanmajora/starred{/owner}{/repo}
https://api.github.com/users/mcmillanmajora/subscriptions
User
https://api.github.com/users/mcmillanmajora
public
Adding a metric card template, based on ideas and edits from @sashavor and I, as well as from comments from @lhoestq and others (thank you!).

All feedback is welcome, but am especially curious about feedback in terms of:
- things that should be included but aren't
- things that are included but should be changed or removed
- the instructions I included, and whether they should be added to, clarified, or deleted altogether