Unblock NumPy 2.0
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6991). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2024-06-26T13:49:44Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2191758797
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6Co5XN
https://api.github.com/repos/huggingface/datasets/issues/comments/2191758797/reactions
2024-06-26T13:49:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2191758797
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
CONTRIBUTOR
@albertvillanova Any chance we could get this in before the next release?  Everything depending on HuggingFace has their NumPy upgrade blocked.
2024-07-10T17:57:06Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2221124149
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6EY6o1
https://api.github.com/repos/huggingface/datasets/issues/comments/2221124149/reactions
2024-07-10T17:57:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2221124149
https://avatars.githubusercontent.com/u/730137?v=4
https://api.github.com/users/NeilGirdhar/events{/privacy}
https://api.github.com/users/NeilGirdhar/followers
https://api.github.com/users/NeilGirdhar/following{/other_user}
https://api.github.com/users/NeilGirdhar/gists{/gist_id}
https://github.com/NeilGirdhar
NeilGirdhar
MDQ6VXNlcjczMDEzNw==
https://api.github.com/users/NeilGirdhar/orgs
https://api.github.com/users/NeilGirdhar/received_events
https://api.github.com/users/NeilGirdhar/repos
https://api.github.com/users/NeilGirdhar/starred{/owner}{/repo}
https://api.github.com/users/NeilGirdhar/subscriptions
User
https://api.github.com/users/NeilGirdhar
public
MEMBER
The incompatible libraries are:
- faiss-cpu 1.8.0.post1 requires numpy<2.0,>=1.0, but you have numpy 2.0.0 which is incompatible.
- tensorflow 2.16.2 requires numpy<2.0.0,>=1.23.5; python_version <= "3.11", but you have numpy 2.0.0 which is incompatible.
- transformers 4.42.3 requires numpy<2.0,>=1.17, but you have numpy 2.0.0 which is incompatible.
2024-07-11T10:08:26Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222541060
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6EeUkE
https://api.github.com/repos/huggingface/datasets/issues/comments/2222541060/reactions
2024-07-11T10:08:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222541060
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
Why is it installing numpy 2 if the dependencies don't support it?
2024-07-11T10:21:11Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222565227
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6Eeadr
https://api.github.com/repos/huggingface/datasets/issues/comments/2222565227/reactions
2024-07-11T10:21:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222565227
https://avatars.githubusercontent.com/u/730137?v=4
https://api.github.com/users/NeilGirdhar/events{/privacy}
https://api.github.com/users/NeilGirdhar/followers
https://api.github.com/users/NeilGirdhar/following{/other_user}
https://api.github.com/users/NeilGirdhar/gists{/gist_id}
https://github.com/NeilGirdhar
NeilGirdhar
MDQ6VXNlcjczMDEzNw==
https://api.github.com/users/NeilGirdhar/orgs
https://api.github.com/users/NeilGirdhar/received_events
https://api.github.com/users/NeilGirdhar/repos
https://api.github.com/users/NeilGirdhar/starred{/owner}{/repo}
https://api.github.com/users/NeilGirdhar/subscriptions
User
https://api.github.com/users/NeilGirdhar
public
CONTRIBUTOR
For me, I'm getting:
```
❯ uv pip install --system "datasets[tests] @ ."
Found existing alias for "uv pip install". You should use: "pipi"
Resolved 119 packages in 934ms
   Built datasets @ file:///Users/neil/src/datasets
Prepared 1 package in 1.28s
Uninstalled 1 package in 10ms
Installed 2 packages in 17ms
 - datasets==2.20.1.dev0 (from file:///Users/neil/src/datasets)
 + datasets==2.20.1.dev0 (from file:///Users/neil/src/datasets)
 + numpy==1.26.4
```
2024-07-11T10:42:34Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222603301
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6Eejwl
https://api.github.com/repos/huggingface/datasets/issues/comments/2222603301/reactions
2024-07-11T10:42:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222603301
https://avatars.githubusercontent.com/u/730137?v=4
https://api.github.com/users/NeilGirdhar/events{/privacy}
https://api.github.com/users/NeilGirdhar/followers
https://api.github.com/users/NeilGirdhar/following{/other_user}
https://api.github.com/users/NeilGirdhar/gists{/gist_id}
https://github.com/NeilGirdhar
NeilGirdhar
MDQ6VXNlcjczMDEzNw==
https://api.github.com/users/NeilGirdhar/orgs
https://api.github.com/users/NeilGirdhar/received_events
https://api.github.com/users/NeilGirdhar/repos
https://api.github.com/users/NeilGirdhar/starred{/owner}{/repo}
https://api.github.com/users/NeilGirdhar/subscriptions
User
https://api.github.com/users/NeilGirdhar
public
MEMBER
Which version on Python do you have?
2024-07-11T11:08:33Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222649715
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6EevFz
https://api.github.com/repos/huggingface/datasets/issues/comments/2222649715/reactions
2024-07-11T11:08:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222649715
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
3.12.4  I'll try on 3.10 now.
2024-07-11T11:11:27Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222654741
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6EewUV
https://api.github.com/repos/huggingface/datasets/issues/comments/2222654741/reactions
2024-07-11T11:11:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222654741
https://avatars.githubusercontent.com/u/730137?v=4
https://api.github.com/users/NeilGirdhar/events{/privacy}
https://api.github.com/users/NeilGirdhar/followers
https://api.github.com/users/NeilGirdhar/following{/other_user}
https://api.github.com/users/NeilGirdhar/gists{/gist_id}
https://github.com/NeilGirdhar
NeilGirdhar
MDQ6VXNlcjczMDEzNw==
https://api.github.com/users/NeilGirdhar/orgs
https://api.github.com/users/NeilGirdhar/received_events
https://api.github.com/users/NeilGirdhar/repos
https://api.github.com/users/NeilGirdhar/starred{/owner}{/repo}
https://api.github.com/users/NeilGirdhar/subscriptions
User
https://api.github.com/users/NeilGirdhar
public
MEMBER
Please, note that I obtained  the previous incompatible libraries in my local environment, by forcing the update of numpy.
2024-07-11T11:13:29Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222658262
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6EexLW
https://api.github.com/repos/huggingface/datasets/issues/comments/2222658262/reactions
2024-07-11T11:13:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222658262
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
In the Python 3.10 CI, the situation is different:
- for example, they install an older version of tensorflow (2.14.0), where probably the constraint on numpy was not yet implemented. See the details: https://github.com/huggingface/datasets/actions/runs/9879100332/job/27306903343?pr=6991
```
> uv pip install --system "datasets[tests] @ ."
...
 + faiss-cpu==1.8.0
...
 + numpy==2.0.0
...
 + tensorflow==2.14.0
```

See, CI installs:
- faiss-cpu 1.8.0 instead of 1.8.0.post1
- tensorflow 2.14.0 instead of 2.16.2
- transformers 4.41.2 instead of 4.42.3
2024-07-11T11:17:25Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222665108
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6Eey2U
https://api.github.com/repos/huggingface/datasets/issues/comments/2222665108/reactions
2024-07-11T11:24:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222665108
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
~~The main point is that we cannot support numpy 2.0 until tensorflow and faiss do.~~

Alternatively, we should ignore/select tests depending on the installed versions.
2024-07-11T11:19:52Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222669909
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6Ee0BV
https://api.github.com/repos/huggingface/datasets/issues/comments/2222669909/reactions
2024-07-11T11:30:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222669909
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
> Alternatively, we should ignore/select tests depending on the installed versions.

That works.

Alternatively, you could depend on tensorflow >= 2.16.2 (etc.) for the tests?
2024-07-11T11:33:40Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222694382
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6Ee5_u
https://api.github.com/repos/huggingface/datasets/issues/comments/2222694382/reactions
2024-07-11T11:34:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222694382
https://avatars.githubusercontent.com/u/730137?v=4
https://api.github.com/users/NeilGirdhar/events{/privacy}
https://api.github.com/users/NeilGirdhar/followers
https://api.github.com/users/NeilGirdhar/following{/other_user}
https://api.github.com/users/NeilGirdhar/gists{/gist_id}
https://github.com/NeilGirdhar
NeilGirdhar
MDQ6VXNlcjczMDEzNw==
https://api.github.com/users/NeilGirdhar/orgs
https://api.github.com/users/NeilGirdhar/received_events
https://api.github.com/users/NeilGirdhar/repos
https://api.github.com/users/NeilGirdhar/starred{/owner}{/repo}
https://api.github.com/users/NeilGirdhar/subscriptions
User
https://api.github.com/users/NeilGirdhar
public
MEMBER
Yes, I was thinking of a workaround solution.

The issue I see is that our CI will not test numpy 2.0 indeed.
2024-07-11T11:58:07Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222739863
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6EfFGX
https://api.github.com/repos/huggingface/datasets/issues/comments/2222739863/reactions
2024-07-11T11:58:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222739863
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
> The issue I see is that our CI will not test numpy 2.0 indeed.

Right, that's the advantage of the test skipping you wanted, I see your point.

Thing is, it won't be long before tensorflow supports numpy 2.0, and then the situation is resolved and your tests test numpy 2.0.  Do you really want to invest a lot of effort into testing numpy 2.0 for a few months benefit?
2024-07-11T11:59:43Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222743617
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6EfGBB
https://api.github.com/repos/huggingface/datasets/issues/comments/2222743617/reactions
2024-07-11T12:01:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222743617
https://avatars.githubusercontent.com/u/730137?v=4
https://api.github.com/users/NeilGirdhar/events{/privacy}
https://api.github.com/users/NeilGirdhar/followers
https://api.github.com/users/NeilGirdhar/following{/other_user}
https://api.github.com/users/NeilGirdhar/gists{/gist_id}
https://github.com/NeilGirdhar
NeilGirdhar
MDQ6VXNlcjczMDEzNw==
https://api.github.com/users/NeilGirdhar/orgs
https://api.github.com/users/NeilGirdhar/received_events
https://api.github.com/users/NeilGirdhar/repos
https://api.github.com/users/NeilGirdhar/starred{/owner}{/repo}
https://api.github.com/users/NeilGirdhar/subscriptions
User
https://api.github.com/users/NeilGirdhar
public
MEMBER
Without testing Numpy 2.0, we do not know if there are some other parts in the code broken.
2024-07-11T12:24:10Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222797938
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6EfTRy
https://api.github.com/repos/huggingface/datasets/issues/comments/2222797938/reactions
2024-07-11T12:24:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222797938
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
> Without testing Numpy 2.0, we do not know if there are some other parts in the code broken.

Yes, you're right.  I understand you're point, but you could say this for anything that your test dependencies don't support.

I guess the solution is to write tests that don't depend on tensorflow, etc., but still use numpy.  You could write some Jax tests for example.

That said, blocking numpy 2 isn't a good solution in my opinion. These dependencies are extremely late in supporting Numpy 2.  They were supposed to be testing against preview releases over three months ago.  I don't think the world should have to wait for them.
2024-07-11T12:37:55Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2222827471
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6EfafP
https://api.github.com/repos/huggingface/datasets/issues/comments/2222827471/reactions
2024-07-11T12:38:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2222827471
https://avatars.githubusercontent.com/u/730137?v=4
https://api.github.com/users/NeilGirdhar/events{/privacy}
https://api.github.com/users/NeilGirdhar/followers
https://api.github.com/users/NeilGirdhar/following{/other_user}
https://api.github.com/users/NeilGirdhar/gists{/gist_id}
https://github.com/NeilGirdhar
NeilGirdhar
MDQ6VXNlcjczMDEzNw==
https://api.github.com/users/NeilGirdhar/orgs
https://api.github.com/users/NeilGirdhar/received_events
https://api.github.com/users/NeilGirdhar/repos
https://api.github.com/users/NeilGirdhar/starred{/owner}{/repo}
https://api.github.com/users/NeilGirdhar/subscriptions
User
https://api.github.com/users/NeilGirdhar
public
MEMBER
> I guess the solution is to write tests that don't depend on tensorflow, etc., but still use numpy.
That is my point. What we cannot do is just blindly support Numpy 2.0 without knowing its consequences. We need to test it:
- to know if our core code works with it
- to know what optional libraries are incompatible

For example, while testing locally, I have discovered that librosa is also incompatible with numpy-2.0, due to its dependency on soxr:
- https://github.com/dofuuz/python-soxr/issues/28
2024-07-12T05:10:55Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2224664074
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6Ema4K
https://api.github.com/repos/huggingface/datasets/issues/comments/2224664074/reactions
2024-07-12T06:34:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2224664074
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
While testing locally, I have also discovered that pytorch does not support Numpy 2.0 on Windows platforms:
- https://github.com/pytorch/pytorch/issues/128860
2024-07-12T06:32:44Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2224887884
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6EnRhM
https://api.github.com/repos/huggingface/datasets/issues/comments/2224887884/reactions
2024-07-12T06:33:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2224887884
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I am adding Numpy 2.0 tests to your PR if you don't mind, before merging this PR.
2024-07-12T10:06:17Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2225249536
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6Eop0A
https://api.github.com/repos/huggingface/datasets/issues/comments/2225249536/reactions
2024-07-12T10:06:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2225249536
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
Awesome, thank you!  Please let me know if I need to do anything.
2024-07-12T10:34:05Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2225294955
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6Eo05r
https://api.github.com/repos/huggingface/datasets/issues/comments/2225294955/reactions
2024-07-12T10:34:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2225294955
https://avatars.githubusercontent.com/u/730137?v=4
https://api.github.com/users/NeilGirdhar/events{/privacy}
https://api.github.com/users/NeilGirdhar/followers
https://api.github.com/users/NeilGirdhar/following{/other_user}
https://api.github.com/users/NeilGirdhar/gists{/gist_id}
https://github.com/NeilGirdhar
NeilGirdhar
MDQ6VXNlcjczMDEzNw==
https://api.github.com/users/NeilGirdhar/orgs
https://api.github.com/users/NeilGirdhar/received_events
https://api.github.com/users/NeilGirdhar/repos
https://api.github.com/users/NeilGirdhar/starred{/owner}{/repo}
https://api.github.com/users/NeilGirdhar/subscriptions
User
https://api.github.com/users/NeilGirdhar
public
MEMBER
Now we test numpy 2.0 in the `test_py310_numpy2` CI tests: https://github.com/huggingface/datasets/actions/runs/9907254874/job/27370545495?pr=6991
```
 + numpy==2.0.0
```
2024-07-12T11:56:01Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2225427572
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6EpVR0
https://api.github.com/repos/huggingface/datasets/issues/comments/2225427572/reactions
2024-07-12T11:56:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2225427572
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005709 / 0.011353 (-0.005643) | 0.003947 / 0.011008 (-0.007061) | 0.064407 / 0.038508 (0.025899) | 0.029903 / 0.023109 (0.006794) | 0.244838 / 0.275898 (-0.031060) | 0.268894 / 0.323480 (-0.054586) | 0.003200 / 0.007986 (-0.004786) | 0.002867 / 0.004328 (-0.001461) | 0.050016 / 0.004250 (0.045765) | 0.047682 / 0.037052 (0.010629) | 0.252186 / 0.258489 (-0.006303) | 0.292050 / 0.293841 (-0.001791) | 0.030277 / 0.128546 (-0.098270) | 0.012283 / 0.075646 (-0.063364) | 0.205875 / 0.419271 (-0.213397) | 0.037202 / 0.043533 (-0.006331) | 0.246045 / 0.255139 (-0.009094) | 0.272422 / 0.283200 (-0.010777) | 0.020572 / 0.141683 (-0.121111) | 1.114343 / 1.452155 (-0.337812) | 1.169909 / 1.492716 (-0.322808) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096612 / 0.018006 (0.078605) | 0.303025 / 0.000490 (0.302535) | 0.000210 / 0.000200 (0.000010) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019292 / 0.037411 (-0.018119) | 0.062548 / 0.014526 (0.048023) | 0.076027 / 0.176557 (-0.100530) | 0.121752 / 0.737135 (-0.615383) | 0.076608 / 0.296338 (-0.219730) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283900 / 0.215209 (0.068691) | 2.829829 / 2.077655 (0.752174) | 1.428934 / 1.504120 (-0.075186) | 1.316796 / 1.541195 (-0.224399) | 1.330012 / 1.468490 (-0.138478) | 0.702245 / 4.584777 (-3.882532) | 2.380454 / 3.745712 (-1.365259) | 2.882881 / 5.269862 (-2.386980) | 1.920345 / 4.565676 (-2.645332) | 0.077860 / 0.424275 (-0.346415) | 0.005295 / 0.007607 (-0.002312) | 0.336968 / 0.226044 (0.110924) | 3.327808 / 2.268929 (1.058879) | 1.781958 / 55.444624 (-53.662666) | 1.489412 / 6.876477 (-5.387065) | 1.634829 / 2.142072 (-0.507243) | 0.787985 / 4.805227 (-4.017243) | 0.134397 / 6.500664 (-6.366267) | 0.042906 / 0.075469 (-0.032563) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.967647 / 1.841788 (-0.874141) | 11.714541 / 8.074308 (3.640233) | 9.350228 / 10.191392 (-0.841164) | 0.142675 / 0.680424 (-0.537749) | 0.014609 / 0.534201 (-0.519592) | 0.301970 / 0.579283 (-0.277314) | 0.262350 / 0.434364 (-0.172014) | 0.342933 / 0.540337 (-0.197404) | 0.437321 / 1.386936 (-0.949615) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005622 / 0.011353 (-0.005731) | 0.003958 / 0.011008 (-0.007050) | 0.050667 / 0.038508 (0.012159) | 0.032842 / 0.023109 (0.009733) | 0.252292 / 0.275898 (-0.023606) | 0.280602 / 0.323480 (-0.042878) | 0.004313 / 0.007986 (-0.003673) | 0.002870 / 0.004328 (-0.001458) | 0.049549 / 0.004250 (0.045299) | 0.040448 / 0.037052 (0.003396) | 0.270264 / 0.258489 (0.011775) | 0.302988 / 0.293841 (0.009147) | 0.030840 / 0.128546 (-0.097707) | 0.012131 / 0.075646 (-0.063515) | 0.060061 / 0.419271 (-0.359211) | 0.033025 / 0.043533 (-0.010507) | 0.251909 / 0.255139 (-0.003230) | 0.275511 / 0.283200 (-0.007689) | 0.018399 / 0.141683 (-0.123284) | 1.160744 / 1.452155 (-0.291411) | 1.188265 / 1.492716 (-0.304452) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.097719 / 0.018006 (0.079712) | 0.304389 / 0.000490 (0.303899) | 0.000217 / 0.000200 (0.000017) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022964 / 0.037411 (-0.014447) | 0.076897 / 0.014526 (0.062372) | 0.088930 / 0.176557 (-0.087626) | 0.128926 / 0.737135 (-0.608209) | 0.091049 / 0.296338 (-0.205290) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.285670 / 0.215209 (0.070461) | 2.806071 / 2.077655 (0.728416) | 1.527161 / 1.504120 (0.023041) | 1.410291 / 1.541195 (-0.130903) | 1.427071 / 1.468490 (-0.041419) | 0.705527 / 4.584777 (-3.879250) | 0.926915 / 3.745712 (-2.818797) | 2.893078 / 5.269862 (-2.376784) | 1.907113 / 4.565676 (-2.658564) | 0.077326 / 0.424275 (-0.346949) | 0.005182 / 0.007607 (-0.002425) | 0.332282 / 0.226044 (0.106237) | 3.312889 / 2.268929 (1.043960) | 1.853839 / 55.444624 (-53.590785) | 1.592013 / 6.876477 (-5.284464) | 1.620234 / 2.142072 (-0.521838) | 0.776894 / 4.805227 (-4.028333) | 0.132411 / 6.500664 (-6.368253) | 0.041430 / 0.075469 (-0.034039) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.003468 / 1.841788 (-0.838320) | 12.472251 / 8.074308 (4.397943) | 10.603243 / 10.191392 (0.411851) | 0.132561 / 0.680424 (-0.547863) | 0.015790 / 0.534201 (-0.518411) | 0.306724 / 0.579283 (-0.272559) | 0.125812 / 0.434364 (-0.308552) | 0.343782 / 0.540337 (-0.196555) | 0.445915 / 1.386936 (-0.941021) |

</details>
</details>

![](https://cml.dev/watermark.png#dfc2b1b14ab8f32730d2bc36c8016ecefbcbabd1 "CML watermark")

2024-07-12T12:11:17Z
https://github.com/huggingface/datasets/pull/6991#issuecomment-2225450583
https://api.github.com/repos/huggingface/datasets/issues/6991
IC_kwDODunzps6Epa5X
Iv1.05c79e9ad1f6bdfa
2018-07-30T09:30:17Z
Automate your workflow from idea to production
branch_protection_rule check_run check_suite create delete deployment deployment_status discussion discussion_comment fork gollum issues issue_comment label merge_group milestone page_build project project_card project_column public pull_request pull_request_review pull_request_review_comment push registry_package release repository repository_dispatch status watch workflow_dispatch workflow_run
https://help.github.com/en/actions
https://github.com/apps/github-actions
GitHub Actions
MDM6QXBwMTUzNjg=
https://avatars.githubusercontent.com/u/9919?v=4
https://api.github.com/users/github/events{/privacy}
https://api.github.com/users/github/followers
https://api.github.com/users/github/following{/other_user}
https://api.github.com/users/github/gists{/gist_id}
https://github.com/github
github
MDEyOk9yZ2FuaXphdGlvbjk5MTk=
https://api.github.com/users/github/orgs
https://api.github.com/users/github/received_events
https://api.github.com/users/github/repos
https://api.github.com/users/github/starred{/owner}{/repo}
https://api.github.com/users/github/subscriptions
Organization
https://api.github.com/users/github
public
write
read
write
write
write
write
write
write
write
read
read
write
write
write
write
write
write
write
read
github-actions
2025-03-07T16:35:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2225450583/reactions
2024-07-12T12:11:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2225450583
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
Fixes https://github.com/huggingface/datasets/issues/6980