Support for decoding Image/Audio types in map when format type is not default one
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-11-16T15:08:43Z
https://github.com/huggingface/datasets/pull/5252#issuecomment-1317170704
https://api.github.com/repos/huggingface/datasets/issues/5252
IC_kwDODunzps5Ogm4Q
https://api.github.com/repos/huggingface/datasets/issues/comments/1317170704/reactions
2022-12-13T17:01:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1317170704
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5252). All of your documentation changes will be reflected on that endpoint.
2022-11-16T17:47:22Z
https://github.com/huggingface/datasets/pull/5252#issuecomment-1317416739
https://api.github.com/repos/huggingface/datasets/issues/5252
IC_kwDODunzps5Ohi8j
https://api.github.com/repos/huggingface/datasets/issues/comments/1317416739/reactions
2022-11-16T17:47:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1317416739
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
COLLABORATOR
Yes, if the image column is the first in the batch keys, it will decode the images because it reads the actual values. We could avoid this by checking the batch type, and if it's `LazyDict`, `num_examples` is equal to `len(batch.pa_table)`, which doesn't lead to decoding.
2022-12-07T16:07:36Z
https://github.com/huggingface/datasets/pull/5252#issuecomment-1341196576
https://api.github.com/repos/huggingface/datasets/issues/5252
IC_kwDODunzps5P8Qkg
https://api.github.com/repos/huggingface/datasets/issues/comments/1341196576/reactions
2022-12-07T16:09:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1341196576
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Good idea. This can be done in a subsequent PR btw, since it's out of scope of the original goal of this PR
2022-12-07T22:04:52Z
https://github.com/huggingface/datasets/pull/5252#issuecomment-1341652938
https://api.github.com/repos/huggingface/datasets/issues/5252
IC_kwDODunzps5P9__K
https://api.github.com/repos/huggingface/datasets/issues/comments/1341652938/reactions
2022-12-07T22:04:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1341652938
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Just fixed a small bug where it would show the pyarrow 10 warning about None -> empty lists conversions even with an Array2D with no nulls
2022-12-12T14:52:41Z
https://github.com/huggingface/datasets/pull/5252#issuecomment-1346641123
https://api.github.com/repos/huggingface/datasets/issues/5252
IC_kwDODunzps5QRBzj
https://api.github.com/repos/huggingface/datasets/issues/comments/1346641123/reactions
2022-12-12T14:52:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1346641123
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Fixed another bug when your map function returns a mix of LazyDict or regular dict and added some tests
2022-12-12T16:57:55Z
https://github.com/huggingface/datasets/pull/5252#issuecomment-1346877280
https://api.github.com/repos/huggingface/datasets/issues/5252
IC_kwDODunzps5QR7dg
https://api.github.com/repos/huggingface/datasets/issues/comments/1346877280/reactions
2022-12-12T16:58:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1346877280
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Add support for decoding the `Image`/`Audio` types in `map` for the formats (Numpy, TF, Jax, PyTorch) other than the default one (Python).

Additional improvements:
* make `Dataset`'s "iter" API  cleaner by removing `_iter` and replacing `_iter_batches` with `iter(batch_size)` (also implemented for `IterableDataset`)
* iterate over arrow tables in `map` to avoid `_getitem` calls, which are much slower than `__iter__`/`iter(batch_size)`, when the `format_type` is not Python
* fix `_iter_batches` (now named `iter`) when `drop_last_batch=True` and `pyarrow<=8.0.0` is installed
* lazily extract and decode arrow data in the default format 

TODO:
* [x] update the `iter` benchmark in the docs (the `BeamBuilder` cannot load the preprocessed datasets from our bucket, so wait for this to be fixed (cc @lhoestq))

Fix https://github.com/huggingface/datasets/issues/3992, fix https://github.com/huggingface/datasets/issues/3756