Fix timestamp conversion from Pandas to Python datetime in streaming mode
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-06-22T13:46:21Z
https://github.com/huggingface/datasets/pull/4541#issuecomment-1163119268
https://api.github.com/repos/huggingface/datasets/issues/4541
IC_kwDODunzps5FU8qk
https://api.github.com/repos/huggingface/datasets/issues/comments/1163119268/reactions
2022-06-22T16:39:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1163119268
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
CI failures are unrelated to this PR, merging
2022-06-22T16:29:05Z
https://github.com/huggingface/datasets/pull/4541#issuecomment-1163342610
https://api.github.com/repos/huggingface/datasets/issues/4541
IC_kwDODunzps5FVzMS
https://api.github.com/repos/huggingface/datasets/issues/comments/1163342610/reactions
2022-06-22T16:29:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1163342610
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Arrow accepts both pd.Timestamp and datetime.datetime objects to create timestamp arrays.
However a timestamp array is always converted to datetime.datetime objects.

This created an inconsistency between streaming in non-streaming. e.g. the `ett` dataset outputs datetime.datetime objects in non-streaming but pd.timestamp in streaming.

I fixed this by always converting pd.Timestamp to datetime.datetime during the example encoding step.
I fixed the same issue for pd.Timedelta as well. Finally I added an extra step of conversion for Series and DataFrame to take this into account in case such data are passed as Series or DataFrame.

Fix https://github.com/huggingface/datasets/issues/4533
Related to https://github.com/huggingface/datasets-server/issues/397