Add regression test for `ArrowWriter.write_batch` when batch is empty
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-06-16T09:02:48Z
https://github.com/huggingface/datasets/pull/4510#issuecomment-1157414685
https://api.github.com/repos/huggingface/datasets/issues/4510
IC_kwDODunzps5E_L8d
https://api.github.com/repos/huggingface/datasets/issues/comments/1157414685/reactions
2022-06-16T12:38:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1157414685
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
As mentioned by @lhoestq, the current behavior is correct and we should not expect batches with different columns, in that case, the if should fail, as the values of the batch can be empty, but not the actual `batch_examples` value.
2022-06-16T10:16:53Z
https://github.com/huggingface/datasets/pull/4510#issuecomment-1157489306
https://api.github.com/repos/huggingface/datasets/issues/4510
IC_kwDODunzps5E_eKa
https://api.github.com/repos/huggingface/datasets/issues/comments/1157489306/reactions
2022-06-16T10:16:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1157489306
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
As spotted by @cccntu in #4502, there's a logic bug in `ArrowWriter.write_batch` as the if-statement to handle the empty batches as detailed in the docstrings of the function ("Ignores the batch if it appears to be empty, preventing a potential schema update of unknown types."), the current if-statement is not handling properly `writer.write_batch({})` as an error is triggered.

Also, if we add a regression test in `test_arrow_writer.py::test_write_batch` before applying the fix, the test will fail as when trying to write an empty batch as follows:

```
=================================================================================== short test summary info ===================================================================================
FAILED tests/test_arrow_writer.py::test_write_batch[None-None] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[None-1] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[None-10] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields1-None] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields1-1] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields1-10] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields2-None] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields2-1] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields2-10] - ValueError: Schema and number of arrays unequal
======================================================================== 9 failed, 73 deselected, 7 warnings in 0.81s =========================================================================
```

So the batch is not ignored when empty, as `batch_examples={}` won't match the condition `if batch_examples: ...`.