Feature/fidelity metrics from emnlp2020 evaluating and characterizing human rationales
MEMBER
Hi ! This looks interesting, thanks for adding it :) 

For metrics there should only be two features fields: references and predictions.
Both of them can be defined as you want using nested structures if you need to.
Also I'm not sure what goes into references and what goes into predictions, could you give more details please ?
All the other computations parameters (model etc.) are fine though. Maybe explain a bit more what they're used for
2020-10-30T14:34:41Z
https://github.com/huggingface/datasets/pull/779#issuecomment-719588665
https://api.github.com/repos/huggingface/datasets/issues/779
MDEyOklzc3VlQ29tbWVudDcxOTU4ODY2NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/719588665/reactions
2020-10-30T14:34:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/719588665
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Hi ! This looks interesting, thanks for adding it :)
> 
> For metrics there should only be two features fields: references and predictions.
> Both of them can be defined as you want using nested structures if you need to.
> Also I'm not sure what goes into references and what goes into predictions, could you give more details please ?
> All the other computations parameters (model etc.) are fine though. Maybe explain a bit more what they're used for

The `predictions` are the predicted labels by a model for a particular input. Do you mean making `prob_y_hat` - the probability of the prediction being the predicted label, `prob_y_hat_alpha` - the probability of the prediction being the predicted label when the input is reduced subject to alpha and the `null_difference` is the difference between the probability of the prediction being the predicted label in full information minus the probability in zero information a part of references? Also, I have added the description for other parameters in kwargs_description. I can expand it if that makes sense?
2020-10-30T15:32:58Z
https://github.com/huggingface/datasets/pull/779#issuecomment-719623591
https://api.github.com/repos/huggingface/datasets/issues/779
MDEyOklzc3VlQ29tbWVudDcxOTYyMzU5MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/719623591/reactions
2020-10-30T15:46:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/719623591
https://avatars.githubusercontent.com/u/11327413?v=4
https://api.github.com/users/rathoreanirudh/events{/privacy}
https://api.github.com/users/rathoreanirudh/followers
https://api.github.com/users/rathoreanirudh/following{/other_user}
https://api.github.com/users/rathoreanirudh/gists{/gist_id}
https://github.com/rathoreanirudh
rathoreanirudh
MDQ6VXNlcjExMzI3NDEz
https://api.github.com/users/rathoreanirudh/orgs
https://api.github.com/users/rathoreanirudh/received_events
https://api.github.com/users/rathoreanirudh/repos
https://api.github.com/users/rathoreanirudh/starred{/owner}{/repo}
https://api.github.com/users/rathoreanirudh/subscriptions
User
https://api.github.com/users/rathoreanirudh
public
MEMBER
I think every value that is generated by the model (so label, prob_y_hat, prob_y_hat_alpha etc.) should be in `predictions`.
Feel free to add more details in the kwargs_description, this is very useful for the end user.
2020-10-30T16:54:24Z
https://github.com/huggingface/datasets/pull/779#issuecomment-719670936
https://api.github.com/repos/huggingface/datasets/issues/779
MDEyOklzc3VlQ29tbWVudDcxOTY3MDkzNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/719670936/reactions
2020-10-30T16:54:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/719670936
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi @lhoestq , I have updated the code according to your feedback. Please, let me know if it looks good and can be merged now.
2020-11-13T18:11:22Z
https://github.com/huggingface/datasets/pull/779#issuecomment-726931086
https://api.github.com/repos/huggingface/datasets/issues/779
MDEyOklzc3VlQ29tbWVudDcyNjkzMTA4Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/726931086/reactions
2020-11-13T18:11:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/726931086
https://avatars.githubusercontent.com/u/11327413?v=4
https://api.github.com/users/rathoreanirudh/events{/privacy}
https://api.github.com/users/rathoreanirudh/followers
https://api.github.com/users/rathoreanirudh/following{/other_user}
https://api.github.com/users/rathoreanirudh/gists{/gist_id}
https://github.com/rathoreanirudh
rathoreanirudh
MDQ6VXNlcjExMzI3NDEz
https://api.github.com/users/rathoreanirudh/orgs
https://api.github.com/users/rathoreanirudh/received_events
https://api.github.com/users/rathoreanirudh/repos
https://api.github.com/users/rathoreanirudh/starred{/owner}{/repo}
https://api.github.com/users/rathoreanirudh/subscriptions
User
https://api.github.com/users/rathoreanirudh
public
MEMBER
Metrics are deprecated in `datasets` and `evaluate` should be used instead: https://github.com/huggingface/evaluate
2023-07-11T09:36:30Z
https://github.com/huggingface/datasets/pull/779#issuecomment-1630488908
https://api.github.com/repos/huggingface/datasets/issues/779
IC_kwDODunzps5hL0lM
https://api.github.com/repos/huggingface/datasets/issues/comments/1630488908/reactions
2023-07-11T09:36:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1630488908
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
This metric computes fidelity (Yu et al. 2019, DeYoung et al. 2019) and normalized fidelity (Carton et al. 2020).