support LargeListArray in pyarrow
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4800). All of your documentation changes will be reflected on that endpoint.
2022-08-08T04:06:13Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1207637695
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5H-xa_
https://api.github.com/repos/huggingface/datasets/issues/comments/1207637695/reactions
2022-08-08T04:06:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1207637695
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
COLLABORATOR
Hi, thanks for working on this! Can you run `make style` at the repo root to fix the code quality error in CI and add a test?
2022-08-10T12:12:03Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1210586688
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5IKBZA
https://api.github.com/repos/huggingface/datasets/issues/comments/1210586688/reactions
2022-08-10T12:12:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1210586688
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Hi, I have fixed the code quality error and added a test
2022-08-10T15:30:46Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1210874918
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5ILHwm
https://api.github.com/repos/huggingface/datasets/issues/comments/1210874918/reactions
2022-08-10T15:30:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1210874918
https://avatars.githubusercontent.com/u/48146603?v=4
https://api.github.com/users/Jiaxin-Wen/events{/privacy}
https://api.github.com/users/Jiaxin-Wen/followers
https://api.github.com/users/Jiaxin-Wen/following{/other_user}
https://api.github.com/users/Jiaxin-Wen/gists{/gist_id}
https://github.com/Jiaxin-Wen
Jiaxin-Wen
MDQ6VXNlcjQ4MTQ2NjAz
https://api.github.com/users/Jiaxin-Wen/orgs
https://api.github.com/users/Jiaxin-Wen/received_events
https://api.github.com/users/Jiaxin-Wen/repos
https://api.github.com/users/Jiaxin-Wen/starred{/owner}{/repo}
https://api.github.com/users/Jiaxin-Wen/subscriptions
User
https://api.github.com/users/Jiaxin-Wen
public
CONTRIBUTOR
It seems that CI fails due to the lack of memory for allocating a large array, while I pass the test locally.
2022-08-11T02:14:34Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1211485008
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5INctQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1211485008/reactions
2022-08-11T02:14:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1211485008
https://avatars.githubusercontent.com/u/48146603?v=4
https://api.github.com/users/Jiaxin-Wen/events{/privacy}
https://api.github.com/users/Jiaxin-Wen/followers
https://api.github.com/users/Jiaxin-Wen/following{/other_user}
https://api.github.com/users/Jiaxin-Wen/gists{/gist_id}
https://github.com/Jiaxin-Wen
Jiaxin-Wen
MDQ6VXNlcjQ4MTQ2NjAz
https://api.github.com/users/Jiaxin-Wen/orgs
https://api.github.com/users/Jiaxin-Wen/received_events
https://api.github.com/users/Jiaxin-Wen/repos
https://api.github.com/users/Jiaxin-Wen/starred{/owner}{/repo}
https://api.github.com/users/Jiaxin-Wen/subscriptions
User
https://api.github.com/users/Jiaxin-Wen
public
COLLABORATOR
Also, the current implementation of the NumPy-to-PyArrow conversion creates a lot of copies, which is not ideal for large arrays.

We can improve performance significantly if we rewrite this part:
https://github.com/huggingface/datasets/blob/83f695c14507a3a38e9f4d84612cf49e5f50c153/src/datasets/features/features.py#L1322-L1323

as
```python
 values = pa.array(arr.ravel(), type=type) 
```
2022-08-11T17:32:14Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1212280549
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5IQe7l
https://api.github.com/repos/huggingface/datasets/issues/comments/1212280549/reactions
2022-08-11T17:32:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1212280549
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
COLLABORATOR
@xwwwwww Feel free to ignore https://github.com/huggingface/datasets/pull/4800#issuecomment-1212280549 and revert the changes you've made to address it. 

Without copying the array, this would be possible:
```python
arr = np.array([
    [1, 2, 3],
    [4, 5, 6]
])

dset = Dataset.from_dict({"data": [arr]})

arr[0][0] = 100 # this change would be reflected in dset's PyArrow table -> a breaking change and also probably unexpected by the user 
```
2022-08-12T13:23:54Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1213106385
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5ITojR
https://api.github.com/repos/huggingface/datasets/issues/comments/1213106385/reactions
2022-08-12T13:23:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1213106385
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
> @xwwwwww Feel free to ignore [#4800 (comment)](https://github.com/huggingface/datasets/pull/4800#issuecomment-1212280549) and revert the changes you've made to address it.
> 
> Without copying the array, this would be possible:
> 
> ```python
> arr = np.array([
>     [1, 2, 3],
>     [4, 5, 6]
> ])
> 
> dset = Dataset.from_dict({"data": [arr]})
> 
> arr[0][0] = 100 # this change would be reflected in dset's PyArrow table -> a breaking change and also probably unexpected by the user 
> ```

Oh, that makes sense.
2022-08-12T14:32:39Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1213179963
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5IT6g7
https://api.github.com/repos/huggingface/datasets/issues/comments/1213179963/reactions
2022-08-12T14:32:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1213179963
https://avatars.githubusercontent.com/u/48146603?v=4
https://api.github.com/users/Jiaxin-Wen/events{/privacy}
https://api.github.com/users/Jiaxin-Wen/followers
https://api.github.com/users/Jiaxin-Wen/following{/other_user}
https://api.github.com/users/Jiaxin-Wen/gists{/gist_id}
https://github.com/Jiaxin-Wen
Jiaxin-Wen
MDQ6VXNlcjQ4MTQ2NjAz
https://api.github.com/users/Jiaxin-Wen/orgs
https://api.github.com/users/Jiaxin-Wen/received_events
https://api.github.com/users/Jiaxin-Wen/repos
https://api.github.com/users/Jiaxin-Wen/starred{/owner}{/repo}
https://api.github.com/users/Jiaxin-Wen/subscriptions
User
https://api.github.com/users/Jiaxin-Wen
public
CONTRIBUTOR
passed tests in ubuntu while failed in windows
2022-08-13T03:04:06Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1213639985
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5IVq0x
https://api.github.com/repos/huggingface/datasets/issues/comments/1213639985/reactions
2022-08-13T03:04:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1213639985
https://avatars.githubusercontent.com/u/48146603?v=4
https://api.github.com/users/Jiaxin-Wen/events{/privacy}
https://api.github.com/users/Jiaxin-Wen/followers
https://api.github.com/users/Jiaxin-Wen/following{/other_user}
https://api.github.com/users/Jiaxin-Wen/gists{/gist_id}
https://github.com/Jiaxin-Wen
Jiaxin-Wen
MDQ6VXNlcjQ4MTQ2NjAz
https://api.github.com/users/Jiaxin-Wen/orgs
https://api.github.com/users/Jiaxin-Wen/received_events
https://api.github.com/users/Jiaxin-Wen/repos
https://api.github.com/users/Jiaxin-Wen/starred{/owner}{/repo}
https://api.github.com/users/Jiaxin-Wen/subscriptions
User
https://api.github.com/users/Jiaxin-Wen
public
CONTRIBUTOR
@mariosasko Hi, do you have any clue about this failure in windows?
2022-08-14T11:19:17Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1214343742
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5IYWo-
https://api.github.com/repos/huggingface/datasets/issues/comments/1214343742/reactions
2022-08-14T11:19:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1214343742
https://avatars.githubusercontent.com/u/48146603?v=4
https://api.github.com/users/Jiaxin-Wen/events{/privacy}
https://api.github.com/users/Jiaxin-Wen/followers
https://api.github.com/users/Jiaxin-Wen/following{/other_user}
https://api.github.com/users/Jiaxin-Wen/gists{/gist_id}
https://github.com/Jiaxin-Wen
Jiaxin-Wen
MDQ6VXNlcjQ4MTQ2NjAz
https://api.github.com/users/Jiaxin-Wen/orgs
https://api.github.com/users/Jiaxin-Wen/received_events
https://api.github.com/users/Jiaxin-Wen/repos
https://api.github.com/users/Jiaxin-Wen/starred{/owner}{/repo}
https://api.github.com/users/Jiaxin-Wen/subscriptions
User
https://api.github.com/users/Jiaxin-Wen
public
COLLABORATOR
Perhaps we can skip the added test on Windows then.

Not sure if this can help, but the ERR tool available on Windows outputs the following for the returned error code `-1073741819`:
```
# for decimal -1073741819 / hex 0xc0000005
  ISCSI_ERR_SETUP_NETWORK_NODE                                   iscsilog.h
# Failed to setup initiator portal. Error status is given in
# the dump data.
  STATUS_ACCESS_VIOLATION                                        ntstatus.h
# The instruction at 0x%p referenced memory at 0x%p. The
# memory could not be %s.
  USBD_STATUS_DEV_NOT_RESPONDING                                 usb.h
# as an HRESULT: Severity: FAILURE (1), FACILITY_NONE (0x0), Code 0x5
# for decimal 5 / hex 0x5
  WINBIO_FP_TOO_FAST                                             winbio_err.h
# Move your finger more slowly on the fingerprint reader.
# as an HRESULT: Severity: FAILURE (1), FACILITY_NULL (0x0), Code 0x5
  ERROR_ACCESS_DENIED                                            winerror.h
# Access is denied.
# 5 matches found for "-1073741819"
```
2022-08-16T15:49:59Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1216824241
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5Ih0Ox
https://api.github.com/repos/huggingface/datasets/issues/comments/1216824241/reactions
2022-08-16T15:49:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1216824241
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
What's the proper way to skip the added test in windows?
I tried `if platform.system() == 'Linux'`, but the CI test seems stuck
2022-08-21T07:10:47Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1221484597
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5IzmA1
https://api.github.com/repos/huggingface/datasets/issues/comments/1221484597/reactions
2022-08-21T07:10:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1221484597
https://avatars.githubusercontent.com/u/48146603?v=4
https://api.github.com/users/Jiaxin-Wen/events{/privacy}
https://api.github.com/users/Jiaxin-Wen/followers
https://api.github.com/users/Jiaxin-Wen/following{/other_user}
https://api.github.com/users/Jiaxin-Wen/gists{/gist_id}
https://github.com/Jiaxin-Wen
Jiaxin-Wen
MDQ6VXNlcjQ4MTQ2NjAz
https://api.github.com/users/Jiaxin-Wen/orgs
https://api.github.com/users/Jiaxin-Wen/received_events
https://api.github.com/users/Jiaxin-Wen/repos
https://api.github.com/users/Jiaxin-Wen/starred{/owner}{/repo}
https://api.github.com/users/Jiaxin-Wen/subscriptions
User
https://api.github.com/users/Jiaxin-Wen
public
CONTRIBUTOR
@mariosasko Hi, any idea about this :)
2022-08-27T02:06:14Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1229099313
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5JQpEx
https://api.github.com/repos/huggingface/datasets/issues/comments/1229099313/reactions
2022-08-27T02:06:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1229099313
https://avatars.githubusercontent.com/u/48146603?v=4
https://api.github.com/users/Jiaxin-Wen/events{/privacy}
https://api.github.com/users/Jiaxin-Wen/followers
https://api.github.com/users/Jiaxin-Wen/following{/other_user}
https://api.github.com/users/Jiaxin-Wen/gists{/gist_id}
https://github.com/Jiaxin-Wen
Jiaxin-Wen
MDQ6VXNlcjQ4MTQ2NjAz
https://api.github.com/users/Jiaxin-Wen/orgs
https://api.github.com/users/Jiaxin-Wen/received_events
https://api.github.com/users/Jiaxin-Wen/repos
https://api.github.com/users/Jiaxin-Wen/starred{/owner}{/repo}
https://api.github.com/users/Jiaxin-Wen/subscriptions
User
https://api.github.com/users/Jiaxin-Wen
public
COLLABORATOR
Hi again! We want to skip the test on Windows but not on Linux. You can use this decorator to do so: 
```python
@pytest.mark.skipif(os.name == "nt" and (os.getenv("CIRCLECI") == "true" or os.getenv("GITHUB_ACTIONS") == "true"), reason="The Windows CI runner does not have enough RAM to run this test")
@pytest.mark.parametrize(...)
def test_large_array_xd_with_np(...):
    ...
```
2022-08-29T11:13:51Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1230141685
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5JUnj1
https://api.github.com/repos/huggingface/datasets/issues/comments/1230141685/reactions
2022-08-29T11:13:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1230141685
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
> Hi again! We want to skip the test on Windows but not on Linux. You can use this decorator to do so:
> 
> ```python
> @pytest.mark.skipif(os.name == "nt" and (os.getenv("CIRCLECI") == "true" or os.getenv("GITHUB_ACTIONS") == "true"), reason="The Windows CI runner does not have enough RAM to run this test")
> @pytest.mark.parametrize(...)
> def test_large_array_xd_with_np(...):
>     ...
> ```

CI on windows still stucks :(
2022-08-29T13:55:12Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1230343037
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5JVYt9
https://api.github.com/repos/huggingface/datasets/issues/comments/1230343037/reactions
2022-08-29T13:55:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1230343037
https://avatars.githubusercontent.com/u/48146603?v=4
https://api.github.com/users/Jiaxin-Wen/events{/privacy}
https://api.github.com/users/Jiaxin-Wen/followers
https://api.github.com/users/Jiaxin-Wen/following{/other_user}
https://api.github.com/users/Jiaxin-Wen/gists{/gist_id}
https://github.com/Jiaxin-Wen
Jiaxin-Wen
MDQ6VXNlcjQ4MTQ2NjAz
https://api.github.com/users/Jiaxin-Wen/orgs
https://api.github.com/users/Jiaxin-Wen/received_events
https://api.github.com/users/Jiaxin-Wen/repos
https://api.github.com/users/Jiaxin-Wen/starred{/owner}{/repo}
https://api.github.com/users/Jiaxin-Wen/subscriptions
User
https://api.github.com/users/Jiaxin-Wen
public
CONTRIBUTOR
@mariosasko Hi, could you please take a look at this issue
2022-09-03T04:56:35Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1236049076
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5JrJy0
https://api.github.com/repos/huggingface/datasets/issues/comments/1236049076/reactions
2022-09-03T04:56:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1236049076
https://avatars.githubusercontent.com/u/48146603?v=4
https://api.github.com/users/Jiaxin-Wen/events{/privacy}
https://api.github.com/users/Jiaxin-Wen/followers
https://api.github.com/users/Jiaxin-Wen/following{/other_user}
https://api.github.com/users/Jiaxin-Wen/gists{/gist_id}
https://github.com/Jiaxin-Wen
Jiaxin-Wen
MDQ6VXNlcjQ4MTQ2NjAz
https://api.github.com/users/Jiaxin-Wen/orgs
https://api.github.com/users/Jiaxin-Wen/received_events
https://api.github.com/users/Jiaxin-Wen/repos
https://api.github.com/users/Jiaxin-Wen/starred{/owner}{/repo}
https://api.github.com/users/Jiaxin-Wen/subscriptions
User
https://api.github.com/users/Jiaxin-Wen
public
CONTRIBUTOR
@mariosasko Hi, all checks have passed, and we are finally ready to merge this PR :)
2022-09-11T03:49:42Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1242879427
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5KFNXD
https://api.github.com/repos/huggingface/datasets/issues/comments/1242879427/reactions
2022-09-11T03:49:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1242879427
https://avatars.githubusercontent.com/u/48146603?v=4
https://api.github.com/users/Jiaxin-Wen/events{/privacy}
https://api.github.com/users/Jiaxin-Wen/followers
https://api.github.com/users/Jiaxin-Wen/following{/other_user}
https://api.github.com/users/Jiaxin-Wen/gists{/gist_id}
https://github.com/Jiaxin-Wen
Jiaxin-Wen
MDQ6VXNlcjQ4MTQ2NjAz
https://api.github.com/users/Jiaxin-Wen/orgs
https://api.github.com/users/Jiaxin-Wen/received_events
https://api.github.com/users/Jiaxin-Wen/repos
https://api.github.com/users/Jiaxin-Wen/starred{/owner}{/repo}
https://api.github.com/users/Jiaxin-Wen/subscriptions
User
https://api.github.com/users/Jiaxin-Wen
public
CONTRIBUTOR
@lhoestq @albertvillanova Perhaps other maintainers can take a look and merge this PR :)
2022-09-13T09:00:39Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1245109813
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5KNt41
https://api.github.com/repos/huggingface/datasets/issues/comments/1245109813/reactions
2022-09-13T09:00:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1245109813
https://avatars.githubusercontent.com/u/48146603?v=4
https://api.github.com/users/Jiaxin-Wen/events{/privacy}
https://api.github.com/users/Jiaxin-Wen/followers
https://api.github.com/users/Jiaxin-Wen/following{/other_user}
https://api.github.com/users/Jiaxin-Wen/gists{/gist_id}
https://github.com/Jiaxin-Wen
Jiaxin-Wen
MDQ6VXNlcjQ4MTQ2NjAz
https://api.github.com/users/Jiaxin-Wen/orgs
https://api.github.com/users/Jiaxin-Wen/received_events
https://api.github.com/users/Jiaxin-Wen/repos
https://api.github.com/users/Jiaxin-Wen/starred{/owner}{/repo}
https://api.github.com/users/Jiaxin-Wen/subscriptions
User
https://api.github.com/users/Jiaxin-Wen
public
NONE
same issus come from pyarrow.Is there a solution for this?
file parquet:50GB
datasets version: 2.14.4
pyarrow :12.0.1

Generating train split: 0 examples [01:22, ? examples/s]
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/datasets/builder.py", line 1925, in _prepare_split_single
    for _, table in generator:
  File "/opt/conda/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 79, in _generate_tables
    for batch_idx, record_batch in enumerate(
  File "pyarrow/_parquet.pyx", line 1315, in iter_batches
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: List index overflow.
2023-08-17T02:34:11Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-1681520841
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5kOfjJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1681520841/reactions
2023-08-17T02:34:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1681520841
https://avatars.githubusercontent.com/u/15654389?v=4
https://api.github.com/users/one-matrix/events{/privacy}
https://api.github.com/users/one-matrix/followers
https://api.github.com/users/one-matrix/following{/other_user}
https://api.github.com/users/one-matrix/gists{/gist_id}
https://github.com/one-matrix
one-matrix
MDQ6VXNlcjE1NjU0Mzg5
https://api.github.com/users/one-matrix/orgs
https://api.github.com/users/one-matrix/received_events
https://api.github.com/users/one-matrix/repos
https://api.github.com/users/one-matrix/starred{/owner}{/repo}
https://api.github.com/users/one-matrix/subscriptions
User
https://api.github.com/users/one-matrix
public
NONE
when this feature adds to the newest version?
2024-05-30T07:04:00Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-2138821754
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5_e9R6
https://api.github.com/repos/huggingface/datasets/issues/comments/2138821754/reactions
2024-05-30T07:04:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2138821754
https://avatars.githubusercontent.com/u/10741881?v=4
https://api.github.com/users/lkfo415579/events{/privacy}
https://api.github.com/users/lkfo415579/followers
https://api.github.com/users/lkfo415579/following{/other_user}
https://api.github.com/users/lkfo415579/gists{/gist_id}
https://github.com/lkfo415579
lkfo415579
MDQ6VXNlcjEwNzQxODgx
https://api.github.com/users/lkfo415579/orgs
https://api.github.com/users/lkfo415579/received_events
https://api.github.com/users/lkfo415579/repos
https://api.github.com/users/lkfo415579/starred{/owner}{/repo}
https://api.github.com/users/lkfo415579/subscriptions
User
https://api.github.com/users/lkfo415579
public
MEMBER
LargeListArray support is not ready yet, there is one remaining change:

> I think the key is to add the large parameter to Sequence and update the functions you modified in this PR to use pa.list_() if large is False, and pa.large_list otherwise
2024-05-30T10:07:20Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-2139217539
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps5_gd6D
https://api.github.com/repos/huggingface/datasets/issues/comments/2139217539/reactions
2024-05-30T10:07:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2139217539
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Gents, any move on this. Convert largse list of dicts to Datasets is a nightmare and took all RAM possible. Is there any other alternative?

Thanks,
Steve

2024-09-27T07:55:55Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-2378646200
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps6Nx0K4
https://api.github.com/repos/huggingface/datasets/issues/comments/2378646200/reactions
2024-09-27T07:55:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2378646200
https://avatars.githubusercontent.com/u/35662216?v=4
https://api.github.com/users/thusinh1969/events{/privacy}
https://api.github.com/users/thusinh1969/followers
https://api.github.com/users/thusinh1969/following{/other_user}
https://api.github.com/users/thusinh1969/gists{/gist_id}
https://github.com/thusinh1969
thusinh1969
MDQ6VXNlcjM1NjYyMjE2
https://api.github.com/users/thusinh1969/orgs
https://api.github.com/users/thusinh1969/received_events
https://api.github.com/users/thusinh1969/repos
https://api.github.com/users/thusinh1969/starred{/owner}{/repo}
https://api.github.com/users/thusinh1969/subscriptions
User
https://api.github.com/users/thusinh1969
public
MEMBER
Arrow large_list is supported since datasets 2.21.0. See: https://github.com/huggingface/datasets/releases/tag/2.21.0
- #7019
2024-09-27T09:54:16Z
https://github.com/huggingface/datasets/pull/4800#issuecomment-2378898891
https://api.github.com/repos/huggingface/datasets/issues/4800
IC_kwDODunzps6Nyx3L
https://api.github.com/repos/huggingface/datasets/issues/comments/2378898891/reactions
2024-09-27T09:54:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2378898891
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
```python
import numpy as np
import datasets
a = np.zeros((5000000, 768))
res = datasets.Dataset.from_dict({'embedding': a})

'''
  File '/home/wenjiaxin/anaconda3/envs/data/lib/python3.8/site-packages/datasets/arrow_writer.py', line 178, in __arrow_array__
    out = numpy_to_pyarrow_listarray(data)
  File "/home/wenjiaxin/anaconda3/envs/data/lib/python3.8/site-packages/datasets/features/features.py", line 1173, in numpy_to_pyarrow_listarray
    offsets = pa.array(np.arange(n_offsets + 1) * step_offsets, type=pa.int32())
  File "pyarrow/array.pxi", line 312, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 2147483904 not in range: -2147483648 to 2147483647
'''
```


Loading a large numpy array currently raises the error above as the type of offsets is `int32`. 
And pyarrow has supported [LargeListArray](https://arrow.apache.org/docs/python/generated/pyarrow.LargeListArray.html) for this case.
