IterableDataset: cannot resolve features from list of numpy arrays
NONE
Assign this issue to me under Hacktoberfest with hacktoberfest label inserted on the issue
2024-10-03T05:47:22Z
https://github.com/huggingface/datasets/issues/7100#issuecomment-2390575038
https://api.github.com/repos/huggingface/datasets/issues/7100
IC_kwDODunzps6OfUe-
https://api.github.com/repos/huggingface/datasets/issues/comments/2390575038/reactions
2024-10-03T05:47:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2390575038
https://avatars.githubusercontent.com/u/40622097?v=4
https://api.github.com/users/vishalmaurya850/events{/privacy}
https://api.github.com/users/vishalmaurya850/followers
https://api.github.com/users/vishalmaurya850/following{/other_user}
https://api.github.com/users/vishalmaurya850/gists{/gist_id}
https://github.com/vishalmaurya850
vishalmaurya850
MDQ6VXNlcjQwNjIyMDk3
https://api.github.com/users/vishalmaurya850/orgs
https://api.github.com/users/vishalmaurya850/received_events
https://api.github.com/users/vishalmaurya850/repos
https://api.github.com/users/vishalmaurya850/starred{/owner}{/repo}
https://api.github.com/users/vishalmaurya850/subscriptions
User
https://api.github.com/users/vishalmaurya850
public
### Describe the bug

when resolve features of `IterableDataset`, got `pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values` error.

```
Traceback (most recent call last):
  File "test.py", line 6
    iter_ds = iter_ds._resolve_features()
  File "lib/python3.10/site-packages/datasets/iterable_dataset.py", line 2876, in _resolve_features
    features = _infer_features_from_batch(self.with_format(None)._head())
  File "lib/python3.10/site-packages/datasets/iterable_dataset.py", line 63, in _infer_features_from_batch
    pa_table = pa.Table.from_pydict(batch)
  File "pyarrow/table.pxi", line 1813, in pyarrow.lib._Tabular.from_pydict
  File "pyarrow/table.pxi", line 5339, in pyarrow.lib._from_pydict
  File "pyarrow/array.pxi", line 374, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 344, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 42, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
```

### Steps to reproduce the bug

```python
from datasets import Dataset
import numpy as np

# create list of numpy
iter_ds = Dataset.from_dict({'a': [[[1, 2, 3], [1, 2, 3]]]}).to_iterable_dataset().map(lambda x: {'a': [np.array(x['a'])]})
iter_ds = iter_ds._resolve_features()  # errors here
```

### Expected behavior

features can be successfully resolved

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-5.15.0-94-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0