ArrayND error in pyarrow 5
CONTRIBUTOR
Where do we bump the required pyarrow version? Any inputs on how I fix this issue? 
2022-04-28T15:50:43Z
https://github.com/huggingface/datasets/issues/4152#issuecomment-1112372941
https://api.github.com/repos/huggingface/datasets/issues/4152
IC_kwDODunzps5CTXbN
https://api.github.com/repos/huggingface/datasets/issues/comments/1112372941/reactions
2022-04-28T15:50:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1112372941
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
MEMBER
We need to bump it in `setup.py` as well as update some CI job to use pyarrow 6 instead of 5 in `.circleci/config.yaml` and `.github/workflows/benchmarks.yaml`
2022-04-28T15:53:41Z
https://github.com/huggingface/datasets/issues/4152#issuecomment-1112376184
https://api.github.com/repos/huggingface/datasets/issues/4152
IC_kwDODunzps5CTYN4
https://api.github.com/repos/huggingface/datasets/issues/comments/1112376184/reactions
2022-04-28T15:54:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1112376184
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
As found in https://github.com/huggingface/datasets/pull/3903, The ArrayND features fail on pyarrow 5:
```python
import pyarrow as pa
from datasets import Array2D
from datasets.table import cast_array_to_feature

arr = pa.array([[[0]]])
feature_type = Array2D(shape=(1, 1), dtype="int64")
cast_array_to_feature(arr, feature_type)
```
raises
```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-04610f9fa78c> in <module>
----> 1 cast_array_to_feature(pa.array([[[0]]]), Array2D(shape=(1, 1), dtype="int32"))

~/Desktop/hf/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1672             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1673         else:
-> 1674             return func(array, *args, **kwargs)
   1675 
   1676     return wrapper

~/Desktop/hf/datasets/src/datasets/table.py in cast_array_to_feature(array, feature, allow_number_to_str)
   1806         return array_cast(array, get_nested_type(feature), allow_number_to_str=allow_number_to_str)
   1807     elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 1808         return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
   1809     raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
   1810 

~/Desktop/hf/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1672             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1673         else:
-> 1674             return func(array, *args, **kwargs)
   1675 
   1676     return wrapper

~/Desktop/hf/datasets/src/datasets/table.py in array_cast(array, pa_type, allow_number_to_str)
   1705         array = array.storage
   1706     if isinstance(pa_type, pa.ExtensionType):
-> 1707         return pa_type.wrap_array(array)
   1708     elif pa.types.is_struct(array.type):
   1709         if pa.types.is_struct(pa_type) and (

AttributeError: 'Array2DExtensionType' object has no attribute 'wrap_array'
```

The thing is that `cast_array_to_feature` is called when writing an Arrow file, so creating an Arrow dataset using any ArrayND type currently fails.

`wrap_array` has been added in pyarrow 6, so we can either bump the required pyarrow version or fix this for pyarrow 5