Add Arrow type casting to struct for Image and Audio + Support nested casting
MEMBER
Regarding the tests I'm just missing the FixedSizeListType type casting for ListArray objects, will to it tomorrow as well as adding new tests + docstrings

and also adding soundfile in the CI
2022-01-13T17:48:31Z
https://github.com/huggingface/datasets/pull/3575#issuecomment-1012367476
https://api.github.com/repos/huggingface/datasets/issues/3575
IC_kwDODunzps48V4B0
https://api.github.com/repos/huggingface/datasets/issues/comments/1012367476/reactions
2022-01-13T17:49:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1012367476
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
While writing some tests I noticed that the ExtensionArray can't be directly concatenated - maybe we can get rid of the extension types/arrays and only keep their storages in native arrow types.

In this case the `cast_storage` functions should be the responsibility of the Image and Audio classes directly. And therefore we would need to never cast to a pyarrow type again but to a HF feature - since they'd end up being the one able to tell what's castable or not. This is fine in my opinion but let me know what you think. I can take care of this on monday I think
2022-01-14T16:39:47Z
https://github.com/huggingface/datasets/pull/3575#issuecomment-1013281229
https://api.github.com/repos/huggingface/datasets/issues/3575
IC_kwDODunzps48ZXHN
https://api.github.com/repos/huggingface/datasets/issues/comments/1013281229/reactions
2022-01-19T18:46:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1013281229
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Alright I got rid of all the extension type stuff, I'm writing the new tests now :)
2022-01-18T11:38:41Z
https://github.com/huggingface/datasets/pull/3575#issuecomment-1015328592
https://api.github.com/repos/huggingface/datasets/issues/3575
IC_kwDODunzps48hK9Q
https://api.github.com/repos/huggingface/datasets/issues/comments/1015328592/reactions
2022-01-18T11:38:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1015328592
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Tests are done, I'll finish the comments and docstrings tomorrow and set the PR on ready for review once it's done !
2022-01-18T19:28:11Z
https://github.com/huggingface/datasets/pull/3575#issuecomment-1015759448
https://api.github.com/repos/huggingface/datasets/issues/3575
IC_kwDODunzps48i0JY
https://api.github.com/repos/huggingface/datasets/issues/comments/1015759448/reactions
2022-01-18T19:28:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1015759448
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
> While writing some tests I noticed that the ExtensionArray can't be directly concatenated - maybe we can get rid of the extension types/arrays and only keep their storages in native arrow types.
>
>In this case the cast_storage functions should be the responsibility of the Image and Audio classes directly. And therefore we would need two never cast to a pyarrow type again but to a HF feature - since they'd end up being the one able to tell what's castable or not. This is fine in my opinion but let me know what you think. I can take care of this on monday I think

Does this change affect performance?
2022-01-19T17:25:43Z
https://github.com/huggingface/datasets/pull/3575#issuecomment-1016695497
https://api.github.com/repos/huggingface/datasets/issues/3575
IC_kwDODunzps48mYrJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1016695497/reactions
2022-01-19T17:25:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1016695497
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
> Does this change affect performance?

In general it shouldn't have a significant impact on performance since the structure of the features is rarely complex (in general we have <20 features and <4 levels of nesting)

Regarding Audio and Image specifically, casting from a StringArray is a little bit more costly since it creates the "bytes" BinaryArray with `None` values with the same length as the "path" array. From the tests I did locally this is very fast though and shouldn't affect the user experience at the current scale of the audio/image datasets we have. It also requires a little bit of RAM though

2022-01-19T18:54:11Z
https://github.com/huggingface/datasets/pull/3575#issuecomment-1016769130
https://api.github.com/repos/huggingface/datasets/issues/3575
IC_kwDODunzps48mqpq
https://api.github.com/repos/huggingface/datasets/issues/comments/1016769130/reactions
2022-01-19T19:10:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1016769130
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Alright this is ready for review now ! Let me know if you have comments and/or improvements :)
2022-01-19T18:57:14Z
https://github.com/huggingface/datasets/pull/3575#issuecomment-1016771735
https://api.github.com/repos/huggingface/datasets/issues/3575
IC_kwDODunzps48mrSX
https://api.github.com/repos/huggingface/datasets/issues/comments/1016771735/reactions
2022-01-19T18:57:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1016771735
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I am facing the issue ArrowNotImplementedError  but no solution is working. Please help me
2022-11-29T07:07:33Z
https://github.com/huggingface/datasets/pull/3575#issuecomment-1330176785
https://api.github.com/repos/huggingface/datasets/issues/3575
IC_kwDODunzps5PSOMR
https://api.github.com/repos/huggingface/datasets/issues/comments/1330176785/reactions
2022-11-29T07:07:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1330176785
https://avatars.githubusercontent.com/u/45088131?v=4
https://api.github.com/users/m-aliabbas/events{/privacy}
https://api.github.com/users/m-aliabbas/followers
https://api.github.com/users/m-aliabbas/following{/other_user}
https://api.github.com/users/m-aliabbas/gists{/gist_id}
https://github.com/m-aliabbas
m-aliabbas
MDQ6VXNlcjQ1MDg4MTMx
https://api.github.com/users/m-aliabbas/orgs
https://api.github.com/users/m-aliabbas/received_events
https://api.github.com/users/m-aliabbas/repos
https://api.github.com/users/m-aliabbas/starred{/owner}{/repo}
https://api.github.com/users/m-aliabbas/subscriptions
User
https://api.github.com/users/m-aliabbas
public
MEMBER
Can you open an new issue and share the error message as well as the script you used ? We'd be happy to help :)
2022-11-29T11:14:16Z
https://github.com/huggingface/datasets/pull/3575#issuecomment-1330467427
https://api.github.com/repos/huggingface/datasets/issues/3575
IC_kwDODunzps5PTVJj
https://api.github.com/repos/huggingface/datasets/issues/comments/1330467427/reactions
2022-11-29T11:14:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1330467427
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Intro

1. Currently, it's not possible to have nested features containing Audio or Image. 

2. Moreover one can keep an Arrow array as a StringArray to store paths to images, but such arrays can't be directly concatenated to another image array if it's stored an another Arrow type (typically, a StructType).

3. Allowing several Arrow types for a single HF feature type also leads to bugs like this one #3497 

4. Issues like #3247 are quite frequent and happen when Arrow fails to reorder StructArrays. 

5. Casting Audio feature type is blocking preparation for the ASR task template: https://github.com/huggingface/datasets/pull/3364

All those issues are linked together by the fact that:
- we are limited by the Arrow type casting which is lacking features for nested types.
- and especially for Audio and Image: they are not robust enough for concatenation and feature inference.

## Proposed solution

To fix 1 and 4 I implemented nested array type casting (which is missing in PyArrow).

To fix 2, 3 and 5 while having a simple implementation for nested array type casting, I changed the storage type of Audio and Image to always be a StructType. Also casting from StringType is directly implemented via a new function `cast_storage` that is defined individually for Audio and Image. I also added nested decoding.

## Implementation details

### I. Better Arrow data type casting for nested data structures

I implemented new functions `array_cast` and `table_cast` that do the exact same as `pyarrow.Array.cast` or `pyarrow.Table.cast` but support nested struct casting and array re-ordering.

These functions can be used on PyArrow objects, and are already integrated in our own `datasets.table.Table.cast` functions. So one can do `my_dataset.data.cast(pyarrow_schema_with_custom_hf_types)` directly.

### II. New image and audio extension types with custom casting

I used PyArrow extension types to be able to define what casting is allowed or not. For example both StringType->ImageExtensionType and StructType->ImageExtensionType are allowed, via the `cast_storage` method.

I factorized all the PyArrow + Pandas extension stuff in the `base_extension.py` file. This aims at separating the front-facing API code of `datasets` from the Arrow back-end which requires advanced knowledge.

### III. Nested feature decoding

I added a new function `decode_nested_example` to decode image and audio data in nested data structures. For optimization's sake, this function is only called if a column has at least one feature that requires decoding.

## Alternative considered

The casting to struct type could have been done directly with python objects using some Audio and Image methods, but bringing arrow data to python objects is expensive. The Audio and Image types could also have been able to convert the arrow data directly, but this is not convenient to use when casting a full Arrow Table with nested fields. Therefore I decided to keep the Arrow data casting logic in Arrow extension types.

## Future work

This work can be used to allow the ArrayND feature types to be nested too (see issue #887)

## TODO

- [x] fix current tests
- [x] add new tests
- [x] docstrings/comments