Standardizing datasets dtypes
CONTRIBUTOR
@lhoestq - apologies for the multiple PRs, my previous one (#1905) got mangled due to some merge conflicts that I had trouble resolving so I just cherry-picked my changes onto a fresh branch here.
2021-02-20T22:05:35Z
https://github.com/huggingface/datasets/pull/1921#issuecomment-782756372
https://api.github.com/repos/huggingface/datasets/issues/1921
MDEyOklzc3VlQ29tbWVudDc4Mjc1NjM3Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/782756372/reactions
2021-02-20T22:05:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/782756372
https://avatars.githubusercontent.com/u/7731709?v=4
https://api.github.com/users/justin-yan/events{/privacy}
https://api.github.com/users/justin-yan/followers
https://api.github.com/users/justin-yan/following{/other_user}
https://api.github.com/users/justin-yan/gists{/gist_id}
https://github.com/justin-yan
justin-yan
MDQ6VXNlcjc3MzE3MDk=
https://api.github.com/users/justin-yan/orgs
https://api.github.com/users/justin-yan/received_events
https://api.github.com/users/justin-yan/repos
https://api.github.com/users/justin-yan/starred{/owner}{/repo}
https://api.github.com/users/justin-yan/subscriptions
User
https://api.github.com/users/justin-yan
public
This PR follows up on discussion in #1900 to have an explicit set of basic dtypes for datasets.

This moves away from str(pyarrow.DataType) as the method of choice for creating dtypes, favoring an explicit mapping to a list of supported Value dtypes.

I believe in practice this should be backward compatible, since anyone previously using Value() would only have been able to use dtypes that had an identically named pyarrow factory function, which are all explicitly supported here, with `float32` and `float64` acting as the official datasets dtypes, which resolves the tension between `double` being the pyarrow dtype and `float64` being the pyarrow type factory function.