`write_examples_on_file` and `write_batch` are broken in `ArrowWriter`
COLLABORATOR
Thanks for reporting! I've left some comments on the PR on how to fix this recent change rather than reverting it.
2024-02-15T18:41:36Z
https://github.com/huggingface/datasets/issues/6663#issuecomment-1946919978
https://api.github.com/repos/huggingface/datasets/issues/6663
IC_kwDODunzps50C6Qq
https://api.github.com/repos/huggingface/datasets/issues/comments/1946919978/reactions
2024-02-15T18:41:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1946919978
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
> Thanks for reporting! I've left some comments on the PR on how to fix this recent change rather than reverting it.

I feel that'd be good, but it'd be great to release a hotfix ASAP (a revert is a fast thing to do) so people can continue using this library and then focus on still applying the improvement.
2024-02-15T19:01:22Z
https://github.com/huggingface/datasets/issues/6663#issuecomment-1946983021
https://api.github.com/repos/huggingface/datasets/issues/6663
IC_kwDODunzps50DJpt
https://api.github.com/repos/huggingface/datasets/issues/comments/1946983021/reactions
2024-02-15T19:01:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1946983021
https://avatars.githubusercontent.com/u/3905501?v=4
https://api.github.com/users/bryant1410/events{/privacy}
https://api.github.com/users/bryant1410/followers
https://api.github.com/users/bryant1410/following{/other_user}
https://api.github.com/users/bryant1410/gists{/gist_id}
https://github.com/bryant1410
bryant1410
MDQ6VXNlcjM5MDU1MDE=
https://api.github.com/users/bryant1410/orgs
https://api.github.com/users/bryant1410/received_events
https://api.github.com/users/bryant1410/repos
https://api.github.com/users/bryant1410/starred{/owner}{/repo}
https://api.github.com/users/bryant1410/subscriptions
User
https://api.github.com/users/bryant1410
public
MEMBER
Fixed by #6664 
2024-02-16T09:25:00Z
https://github.com/huggingface/datasets/issues/6663#issuecomment-1948028159
https://api.github.com/repos/huggingface/datasets/issues/6663
IC_kwDODunzps50HIz_
https://api.github.com/repos/huggingface/datasets/issues/comments/1948028159/reactions
2024-02-16T09:25:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1948028159
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
### Describe the bug

`write_examples_on_file` and `write_batch` are broken in `ArrowWriter` since #6636. The order between the columns and the schema is not preserved anymore. So these functions don't work anymore unless the order happens to align well.

### Steps to reproduce the bug

Try to do `write_batch` with anything that has many columns, and it's likely to break.

### Expected behavior

I expect these functions to work, instead of it trying to cast a column to its incorrect type.

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-5.15.0-1040-aws-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0