Add configurable options to `seqeval` metric
MEMBER
Hi @marrodion. 

Thanks for pointing this out. It would be great to incorporate this metric-specific enhancement.

Another possibility would be to require the user to input the scheme as a string `mode="strict", scheme="IOB2"` and then dynamically import the corresponding module using Python `importlib`:
```python
if scheme:
    scheme = importlib.import_module(f"seqeval.scheme.{scheme}")
```

Feel free to create a Pull Request to make this contribution.
2021-04-07T06:29:55Z
https://github.com/huggingface/datasets/issues/2148#issuecomment-814641153
https://api.github.com/repos/huggingface/datasets/issues/2148
MDEyOklzc3VlQ29tbWVudDgxNDY0MTE1Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/814641153/reactions
2021-04-07T06:29:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814641153
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Right now `load_metric("seqeval")` only works in the default mode of evaluation (equivalent to conll evaluation).

However, seqeval library [supports](https://github.com/chakki-works/seqeval#support-features) different evaluation schemes (IOB1, IOB2, etc.), which can be plugged in just by supporting additional kwargs in `Seqeval._compute`
https://github.com/huggingface/datasets/blob/85cf7ff920c90ca2e12bedca12b36d2a043c3da2/metrics/seqeval/seqeval.py#L109

Things that would be relevant are, for example, supporting `mode="strict", scheme=IOB2` to count only full entity match as a true positive and omit partial matches.

The only problem I see is that the spirit of `metrics` seems to not require additional imports from user. `seqeval` only supports schemes as objects, without any string aliases. 

It can be solved naively with mapping like `{"IOB2": seqeval.scheme.IOB2}`. Or just left as is and require user to explicitly import scheme from `seqeval` if he wants to configure it past the default implementation.

If that makes sense, I am happy to implement the change.