added metrics for CUAD
NONE
> For now I've added F1, AUPR, Precision at 80% recall, and Precision at 90%. Last 3 metrics were reported in the [paper](https://arxiv.org/pdf/2103.06268.pdf). Please let me know if we require `exact_match` metric too here

@bhavitvyamalik  I guess the mentioned metrics are enough but it would be better if exact match is also added since the standard SQUAD dataset also has it.
2021-04-24T21:20:04Z
https://github.com/huggingface/datasets/pull/2257#issuecomment-826155037
https://api.github.com/repos/huggingface/datasets/issues/2257
MDEyOklzc3VlQ29tbWVudDgyNjE1NTAzNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/826155037/reactions
2021-04-24T21:20:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826155037
https://avatars.githubusercontent.com/u/31034499?v=4
https://api.github.com/users/MohammedRakib/events{/privacy}
https://api.github.com/users/MohammedRakib/followers
https://api.github.com/users/MohammedRakib/following{/other_user}
https://api.github.com/users/MohammedRakib/gists{/gist_id}
https://github.com/MohammedRakib
MohammedRakib
MDQ6VXNlcjMxMDM0NDk5
https://api.github.com/users/MohammedRakib/orgs
https://api.github.com/users/MohammedRakib/received_events
https://api.github.com/users/MohammedRakib/repos
https://api.github.com/users/MohammedRakib/starred{/owner}{/repo}
https://api.github.com/users/MohammedRakib/subscriptions
User
https://api.github.com/users/MohammedRakib
public
NONE
I would like to quote it from the website that I am following to learn
these things.
Exact Match:
This metric is as simple as it sounds. For each question+answer pair, if
the characters of the model's prediction exactly match the characters of
*(one
of) the True Answer(s)*, EM = 1, otherwise EM = 0. This is a strict
all-or-nothing metric; being off by a single character results in a score
of 0. When assessing against a negative example, if the model predicts any
text at all, it automatically receives a 0 for that example.

So, I guess you need to ensure at least 1 predicted answer matches for EM
to be 1.
Source:
https://qa.fastforwardlabs.com/no%20answer/null%20threshold/bert/distilbert/exact%20match/f1/robust%20predictions/2020/06/09/Evaluating_BERT_on_SQuAD.html

You can go to their homepage and read the other links. They have detailed
explanations on evaluation metrics. You can also have a look at the
squad_v2 metric file for further clarification.

Regards,
Mohammed Rakib

On Sun, 25 Apr 2021 at 15:20, Bhavitvya Malik ***@***.***>
wrote:

> I'm a little confused when it comes to 2 ground truths which can be a
> possible answer. Like here for eg.
>
> predictions = [{'prediction_text': ['The seller:', 'The buyer/End-User:
> Shenzhen LOHAS Supply Chain Management Co., Ltd.'], 'id':
> 'LohaCompanyltd_20191209_F-1_EX-10.16_11917878_EX-10.16_Supply
> Agreement__Parties'}]
>
> references = [{'answers': {'answer_start': [143, 49], 'text': ['The
> seller:', 'The buyer/End-User: Shenzhen LOHAS Supply Chain Management Co.,
> Ltd.']}, 'id':
> 'LohaCompanyltd_20191209_F-1_EX-10.16_11917878_EX-10.16_Supply
> Agreement__Parties'}]
>
> Should I ensure at least 1 predicted answer matches or both predicted
> answers should match (like in this case) for EM to be 1?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/2257#issuecomment-826289753>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHMYZAZSAEZNFWEMVAPK6M3TKPNHLANCNFSM43QFZVPQ>
> .
>

2021-04-25T09:57:47Z
https://github.com/huggingface/datasets/pull/2257#issuecomment-826295063
https://api.github.com/repos/huggingface/datasets/issues/2257
MDEyOklzc3VlQ29tbWVudDgyNjI5NTA2Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/826295063/reactions
2021-04-25T09:57:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826295063
https://avatars.githubusercontent.com/u/31034499?v=4
https://api.github.com/users/MohammedRakib/events{/privacy}
https://api.github.com/users/MohammedRakib/followers
https://api.github.com/users/MohammedRakib/following{/other_user}
https://api.github.com/users/MohammedRakib/gists{/gist_id}
https://github.com/MohammedRakib
MohammedRakib
MDQ6VXNlcjMxMDM0NDk5
https://api.github.com/users/MohammedRakib/orgs
https://api.github.com/users/MohammedRakib/received_events
https://api.github.com/users/MohammedRakib/repos
https://api.github.com/users/MohammedRakib/starred{/owner}{/repo}
https://api.github.com/users/MohammedRakib/subscriptions
User
https://api.github.com/users/MohammedRakib
public
CONTRIBUTOR
Updated the same @MohammedRakib! Even if a single answer matches I'm returning 1 in that case for EM (not traversing all predictions once we have one `exact_match` from prediction)
2021-04-25T11:32:22Z
https://github.com/huggingface/datasets/pull/2257#issuecomment-826308803
https://api.github.com/repos/huggingface/datasets/issues/2257
MDEyOklzc3VlQ29tbWVudDgyNjMwODgwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/826308803/reactions
2021-04-25T11:32:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826308803
https://avatars.githubusercontent.com/u/19718818?v=4
https://api.github.com/users/bhavitvyamalik/events{/privacy}
https://api.github.com/users/bhavitvyamalik/followers
https://api.github.com/users/bhavitvyamalik/following{/other_user}
https://api.github.com/users/bhavitvyamalik/gists{/gist_id}
https://github.com/bhavitvyamalik
bhavitvyamalik
MDQ6VXNlcjE5NzE4ODE4
https://api.github.com/users/bhavitvyamalik/orgs
https://api.github.com/users/bhavitvyamalik/received_events
https://api.github.com/users/bhavitvyamalik/repos
https://api.github.com/users/bhavitvyamalik/starred{/owner}{/repo}
https://api.github.com/users/bhavitvyamalik/subscriptions
User
https://api.github.com/users/bhavitvyamalik
public
For now I've added F1, AUPR, Precision at 80% recall, and Precision at 90%. Last 3 metrics were reported in the [paper](https://arxiv.org/pdf/2103.06268.pdf). Please let me know if we require `exact_match` metric too here