Add support for continuous metrics (RMSE, MAE)
NONE
Hey @ck37 

You can always use a custom metric as explained [in this guide from HF](https://huggingface.co/docs/datasets/master/loading_metrics.html#using-a-custom-metric-script).

If this issue needs to be contributed to (for enhancing the metric API) I think [this link](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html) would be helpful for the `MAE` metric.
2022-01-20T15:28:31Z
https://github.com/huggingface/datasets/issues/3608#issuecomment-1017624184
https://api.github.com/repos/huggingface/datasets/issues/3608
IC_kwDODunzps48p7Z4
https://api.github.com/repos/huggingface/datasets/issues/comments/1017624184/reactions
2022-01-20T15:28:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1017624184
https://avatars.githubusercontent.com/u/36856589?v=4
https://api.github.com/users/ariG23498/events{/privacy}
https://api.github.com/users/ariG23498/followers
https://api.github.com/users/ariG23498/following{/other_user}
https://api.github.com/users/ariG23498/gists{/gist_id}
https://github.com/ariG23498
ariG23498
MDQ6VXNlcjM2ODU2NTg5
https://api.github.com/users/ariG23498/orgs
https://api.github.com/users/ariG23498/received_events
https://api.github.com/users/ariG23498/repos
https://api.github.com/users/ariG23498/starred{/owner}{/repo}
https://api.github.com/users/ariG23498/subscriptions
User
https://api.github.com/users/ariG23498
public
NONE
You can use a local metric script just by providing its path instead of the usual shortcut name 
2022-01-24T17:16:32Z
https://github.com/huggingface/datasets/issues/3608#issuecomment-1020342669
https://api.github.com/repos/huggingface/datasets/issues/3608
IC_kwDODunzps480TGN
https://api.github.com/repos/huggingface/datasets/issues/comments/1020342669/reactions
2022-01-24T17:16:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1020342669
https://avatars.githubusercontent.com/u/39478293?v=4
https://api.github.com/users/callmekofi/events{/privacy}
https://api.github.com/users/callmekofi/followers
https://api.github.com/users/callmekofi/following{/other_user}
https://api.github.com/users/callmekofi/gists{/gist_id}
https://github.com/callmekofi
callmekofi
MDQ6VXNlcjM5NDc4Mjkz
https://api.github.com/users/callmekofi/orgs
https://api.github.com/users/callmekofi/received_events
https://api.github.com/users/callmekofi/repos
https://api.github.com/users/callmekofi/starred{/owner}{/repo}
https://api.github.com/users/callmekofi/subscriptions
User
https://api.github.com/users/callmekofi
public
CONTRIBUTOR
#self-assign I have starting working on this issue to enhance the metric API.
2022-03-07T14:47:15Z
https://github.com/huggingface/datasets/issues/3608#issuecomment-1060766577
https://api.github.com/repos/huggingface/datasets/issues/3608
IC_kwDODunzps4_OgNx
https://api.github.com/repos/huggingface/datasets/issues/comments/1060766577/reactions
2022-03-07T14:47:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1060766577
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
**Is your feature request related to a problem? Please describe.**

I am uploading our dataset and models for the "Constructing interval measures" method we've developed, which uses item response theory to convert multiple discrete labels into a continuous spectrum for hate speech. Once we have this outcome our NLP models conduct regression rather than classification, so binary metrics are not relevant. The only continuous metrics available at https://huggingface.co/metrics are pearson & spearman correlation, which don't ensure that the prediction is on the same scale as the outcome.

**Describe the solution you'd like**
I would like to be able to tag our models on the Hub with the following metrics:
- RMSE
- MAE

**Describe alternatives you've considered**

I don't know if there are any alternatives.

**Additional context**
Our preprint is available here: https://arxiv.org/abs/2009.10277 . We are making it available for use in Jigsaw's Toxic Severity Rating Kaggle competition: https://www.kaggle.com/c/jigsaw-toxic-severity-rating/overview . I have our first model uploaded to the Hub at https://huggingface.co/ucberkeley-dlab/hate-measure-roberta-large

Thanks,
Chris
