Allow dynamic first dimension for ArrayXD
CONTRIBUTOR
@lhoestq, thanks for your review.

I added test for `to_pylist`, I didn't do that for `to_numpy` because this method shouldn't be called for dynamic dimension ArrayXD - this method will try to make a single numpy array for the whole column which cannot be done for dynamic arrays.

I dig into `to_pandas()` functionality and I found it quite difficult to implement. `PandasArrayExtensionArray` takes single np.array as an argument. It might be a bit of changes to make it work with the list of arrays. Do you mind if we exclude this work from this PR. I added an error message for the case if somebody tries to use dynamic arrays with `to_pandas`
2021-09-23T10:33:21Z
https://github.com/huggingface/datasets/pull/2891#issuecomment-925692110
https://api.github.com/repos/huggingface/datasets/issues/2891
IC_kwDODunzps43LPDO
https://api.github.com/repos/huggingface/datasets/issues/comments/925692110/reactions
2021-09-23T10:33:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/925692110
https://avatars.githubusercontent.com/u/10357417?v=4
https://api.github.com/users/rpowalski/events{/privacy}
https://api.github.com/users/rpowalski/followers
https://api.github.com/users/rpowalski/following{/other_user}
https://api.github.com/users/rpowalski/gists{/gist_id}
https://github.com/rpowalski
rpowalski
MDQ6VXNlcjEwMzU3NDE3
https://api.github.com/users/rpowalski/orgs
https://api.github.com/users/rpowalski/received_events
https://api.github.com/users/rpowalski/repos
https://api.github.com/users/rpowalski/starred{/owner}{/repo}
https://api.github.com/users/rpowalski/subscriptions
User
https://api.github.com/users/rpowalski
public
CONTRIBUTOR
@lhoestq, I just fixed all the tests. Let me know if there is anything else to add.
2021-09-29T12:59:38Z
https://github.com/huggingface/datasets/pull/2891#issuecomment-930151691
https://api.github.com/repos/huggingface/datasets/issues/2891
IC_kwDODunzps43cP0L
https://api.github.com/repos/huggingface/datasets/issues/comments/930151691/reactions
2021-09-29T12:59:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/930151691
https://avatars.githubusercontent.com/u/10357417?v=4
https://api.github.com/users/rpowalski/events{/privacy}
https://api.github.com/users/rpowalski/followers
https://api.github.com/users/rpowalski/following{/other_user}
https://api.github.com/users/rpowalski/gists{/gist_id}
https://github.com/rpowalski
rpowalski
MDQ6VXNlcjEwMzU3NDE3
https://api.github.com/users/rpowalski/orgs
https://api.github.com/users/rpowalski/received_events
https://api.github.com/users/rpowalski/repos
https://api.github.com/users/rpowalski/starred{/owner}{/repo}
https://api.github.com/users/rpowalski/subscriptions
User
https://api.github.com/users/rpowalski
public
CONTRIBUTOR
@lhoestq, any chance you had some time to check out this PR?

2021-10-11T08:54:58Z
https://github.com/huggingface/datasets/pull/2891#issuecomment-939825158
https://api.github.com/repos/huggingface/datasets/issues/2891
IC_kwDODunzps44BJgG
https://api.github.com/repos/huggingface/datasets/issues/comments/939825158/reactions
2021-10-11T08:54:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/939825158
https://avatars.githubusercontent.com/u/10357417?v=4
https://api.github.com/users/rpowalski/events{/privacy}
https://api.github.com/users/rpowalski/followers
https://api.github.com/users/rpowalski/following{/other_user}
https://api.github.com/users/rpowalski/gists{/gist_id}
https://github.com/rpowalski
rpowalski
MDQ6VXNlcjEwMzU3NDE3
https://api.github.com/users/rpowalski/orgs
https://api.github.com/users/rpowalski/received_events
https://api.github.com/users/rpowalski/repos
https://api.github.com/users/rpowalski/starred{/owner}{/repo}
https://api.github.com/users/rpowalski/subscriptions
User
https://api.github.com/users/rpowalski
public
MEMBER
Hi ! Sorry for the delay

It looks good to me ! I think the only thing missing is the support for passing a list of numpy arrays to `map` when the first dimension is dynamic.

Currently it raises an error:
```python
from datasets import *
import numpy as np

features=  Features({"a": Array3D(shape=(None, 5, 2), dtype="int32")})
d = Dataset.from_dict({"a": [np.zeros((5,5,2)), np.zeros((2,5,2))]}, features=features)
d = d.map(lambda a: {"a": np.concatenate([a]*2)}, input_columns="a")
print(d[0])
```
raises
```python
Traceback (most recent call last):
  File "playground/ttest.py", line 6, in <module>
    d = d.map(lambda x: {"a": np.concatenate([x]*2)}, input_columns="a")
  File "/home/truent/hf/datasets/src/datasets/arrow_dataset.py", line 1932, in map
    return self._map_single(
  File "/home/truent/hf/datasets/src/datasets/arrow_dataset.py", line 426, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/truent/hf/datasets/src/datasets/fingerprint.py", line 406, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/truent/hf/datasets/src/datasets/arrow_dataset.py", line 2317, in _map_single
    writer.finalize()
  File "/home/truent/hf/datasets/src/datasets/arrow_writer.py", line 443, in finalize
    self.write_examples_on_file()
  File "/home/truent/hf/datasets/src/datasets/arrow_writer.py", line 312, in write_examples_on_file
    pa_array = pa.array(typed_sequence)
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/truent/hf/datasets/src/datasets/arrow_writer.py", line 108, in __arrow_array__
    storage = pa.array(self.data, type.storage_dtype)
  File "pyarrow/array.pxi", line 305, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
```

I think the issue is that here we don't cover the case where self.data is a list of numpy arrays:

https://github.com/huggingface/datasets/blob/55fd140a63b8f03a0e72985647e498f1fc799d3f/src/datasets/arrow_writer.py#L104-L109

We should remove the `isinstance(self.data[0], np.ndarray)` part and add these lines to cover this case:

https://github.com/huggingface/datasets/blob/55fd140a63b8f03a0e72985647e498f1fc799d3f/src/datasets/arrow_writer.py#L112-L113
2021-10-11T09:28:19Z
https://github.com/huggingface/datasets/pull/2891#issuecomment-939853678
https://api.github.com/repos/huggingface/datasets/issues/2891
IC_kwDODunzps44BQdu
https://api.github.com/repos/huggingface/datasets/issues/comments/939853678/reactions
2021-10-11T09:28:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/939853678
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq, thanks, good catch!
Are you able to run this check with fixed dimension ArrayXD?
for below example
```
import numpy as np
from datasets import *

features = Features({"a": Array3D(shape=(2, 5, 2), dtype="int32")})
d = Dataset.from_dict({"a": [np.zeros((2, 5, 2)), np.zeros((2, 5, 2))]}, features=features)
d = d.map(lambda a: {"a": np.array(a) + 1}, input_columns="a")
print(d[0])
```

I am getting:
```
  File "/home/ib/datasets/src/datasets/arrow_writer.py", line 116, in __arrow_array__
    if trying_type and out[0].as_py() != self.data[0]:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
```
2021-10-11T16:46:31Z
https://github.com/huggingface/datasets/pull/2891#issuecomment-940189401
https://api.github.com/repos/huggingface/datasets/issues/2891
IC_kwDODunzps44CibZ
https://api.github.com/repos/huggingface/datasets/issues/comments/940189401/reactions
2021-10-11T16:46:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/940189401
https://avatars.githubusercontent.com/u/10357417?v=4
https://api.github.com/users/rpowalski/events{/privacy}
https://api.github.com/users/rpowalski/followers
https://api.github.com/users/rpowalski/following{/other_user}
https://api.github.com/users/rpowalski/gists{/gist_id}
https://github.com/rpowalski
rpowalski
MDQ6VXNlcjEwMzU3NDE3
https://api.github.com/users/rpowalski/orgs
https://api.github.com/users/rpowalski/received_events
https://api.github.com/users/rpowalski/repos
https://api.github.com/users/rpowalski/starred{/owner}{/repo}
https://api.github.com/users/rpowalski/subscriptions
User
https://api.github.com/users/rpowalski
public
CONTRIBUTOR
Nevertheless, I tried to fix that. Let me know if that works.
2021-10-11T17:22:34Z
https://github.com/huggingface/datasets/pull/2891#issuecomment-940213585
https://api.github.com/repos/huggingface/datasets/issues/2891
IC_kwDODunzps44CoVR
https://api.github.com/repos/huggingface/datasets/issues/comments/940213585/reactions
2021-10-11T17:22:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/940213585
https://avatars.githubusercontent.com/u/10357417?v=4
https://api.github.com/users/rpowalski/events{/privacy}
https://api.github.com/users/rpowalski/followers
https://api.github.com/users/rpowalski/following{/other_user}
https://api.github.com/users/rpowalski/gists{/gist_id}
https://github.com/rpowalski
rpowalski
MDQ6VXNlcjEwMzU3NDE3
https://api.github.com/users/rpowalski/orgs
https://api.github.com/users/rpowalski/received_events
https://api.github.com/users/rpowalski/repos
https://api.github.com/users/rpowalski/starred{/owner}{/repo}
https://api.github.com/users/rpowalski/subscriptions
User
https://api.github.com/users/rpowalski
public
CONTRIBUTOR
@lhoestq, just resolved the conflicts. Let me know if there is anything left to do with this PR
2021-10-14T10:30:41Z
https://github.com/huggingface/datasets/pull/2891#issuecomment-943228109
https://api.github.com/repos/huggingface/datasets/issues/2891
IC_kwDODunzps44OITN
https://api.github.com/repos/huggingface/datasets/issues/comments/943228109/reactions
2021-10-14T10:30:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/943228109
https://avatars.githubusercontent.com/u/10357417?v=4
https://api.github.com/users/rpowalski/events{/privacy}
https://api.github.com/users/rpowalski/followers
https://api.github.com/users/rpowalski/following{/other_user}
https://api.github.com/users/rpowalski/gists{/gist_id}
https://github.com/rpowalski
rpowalski
MDQ6VXNlcjEwMzU3NDE3
https://api.github.com/users/rpowalski/orgs
https://api.github.com/users/rpowalski/received_events
https://api.github.com/users/rpowalski/repos
https://api.github.com/users/rpowalski/starred{/owner}{/repo}
https://api.github.com/users/rpowalski/subscriptions
User
https://api.github.com/users/rpowalski
public
CONTRIBUTOR
Hi, thanks a lot for your comments.
Agree, happy to contribute to this topic in future PRs
2021-10-28T07:59:23Z
https://github.com/huggingface/datasets/pull/2891#issuecomment-953595396
https://api.github.com/repos/huggingface/datasets/issues/2891
IC_kwDODunzps441rYE
https://api.github.com/repos/huggingface/datasets/issues/comments/953595396/reactions
2021-10-28T07:59:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/953595396
https://avatars.githubusercontent.com/u/10357417?v=4
https://api.github.com/users/rpowalski/events{/privacy}
https://api.github.com/users/rpowalski/followers
https://api.github.com/users/rpowalski/following{/other_user}
https://api.github.com/users/rpowalski/gists{/gist_id}
https://github.com/rpowalski
rpowalski
MDQ6VXNlcjEwMzU3NDE3
https://api.github.com/users/rpowalski/orgs
https://api.github.com/users/rpowalski/received_events
https://api.github.com/users/rpowalski/repos
https://api.github.com/users/rpowalski/starred{/owner}{/repo}
https://api.github.com/users/rpowalski/subscriptions
User
https://api.github.com/users/rpowalski
public
MEMBER
Hi @rpowalski, thanks for adding this feature! 

I wanted to check if you are still interested in documenting this, otherwise I'd be happy to help with it :)
2021-11-23T15:33:12Z
https://github.com/huggingface/datasets/pull/2891#issuecomment-976725948
https://api.github.com/repos/huggingface/datasets/issues/2891
IC_kwDODunzps46N6e8
https://api.github.com/repos/huggingface/datasets/issues/comments/976725948/reactions
2021-11-23T15:33:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/976725948
https://avatars.githubusercontent.com/u/59462357?v=4
https://api.github.com/users/stevhliu/events{/privacy}
https://api.github.com/users/stevhliu/followers
https://api.github.com/users/stevhliu/following{/other_user}
https://api.github.com/users/stevhliu/gists{/gist_id}
https://github.com/stevhliu
stevhliu
MDQ6VXNlcjU5NDYyMzU3
https://api.github.com/users/stevhliu/orgs
https://api.github.com/users/stevhliu/received_events
https://api.github.com/users/stevhliu/repos
https://api.github.com/users/stevhliu/starred{/owner}{/repo}
https://api.github.com/users/stevhliu/subscriptions
User
https://api.github.com/users/stevhliu
public
Add support for dynamic first dimension for ArrayXD features. See issue [#887](https://github.com/huggingface/datasets/issues/887).
Following changes allow for `to_pylist` method of `ArrayExtensionArray` to return a list of numpy arrays where fist dimension can vary.

@lhoestq Could you suggest how you want to extend test suit. For now I added only very limited testing.