Silence ruff deprecation messages
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6707). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2024-03-01T16:59:27Z
https://github.com/huggingface/datasets/pull/6707#issuecomment-1973546426
https://api.github.com/repos/huggingface/datasets/issues/6707
IC_kwDODunzps51oe26
https://api.github.com/repos/huggingface/datasets/issues/comments/1973546426/reactions
2024-03-01T16:59:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1973546426
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004728 / 0.011353 (-0.006624) | 0.002941 / 0.011008 (-0.008067) | 0.058270 / 0.038508 (0.019762) | 0.027418 / 0.023109 (0.004309) | 0.224993 / 0.275898 (-0.050905) | 0.243103 / 0.323480 (-0.080377) | 0.004668 / 0.007986 (-0.003318) | 0.002499 / 0.004328 (-0.001829) | 0.045020 / 0.004250 (0.040770) | 0.038006 / 0.037052 (0.000953) | 0.240807 / 0.258489 (-0.017682) | 0.264554 / 0.293841 (-0.029287) | 0.027018 / 0.128546 (-0.101529) | 0.009866 / 0.075646 (-0.065780) | 0.196578 / 0.419271 (-0.222694) | 0.034536 / 0.043533 (-0.008997) | 0.236535 / 0.255139 (-0.018604) | 0.248879 / 0.283200 (-0.034321) | 0.017140 / 0.141683 (-0.124543) | 1.046927 / 1.452155 (-0.405228) | 1.121209 / 1.492716 (-0.371507) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088267 / 0.018006 (0.070261) | 0.279774 / 0.000490 (0.279284) | 0.000214 / 0.000200 (0.000014) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.016521 / 0.037411 (-0.020890) | 0.056499 / 0.014526 (0.041974) | 0.067264 / 0.176557 (-0.109293) | 0.117270 / 0.737135 (-0.619865) | 0.069284 / 0.296338 (-0.227055) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.260679 / 0.215209 (0.045470) | 2.608971 / 2.077655 (0.531316) | 1.363139 / 1.504120 (-0.140981) | 1.262128 / 1.541195 (-0.279067) | 1.273619 / 1.468490 (-0.194871) | 0.523417 / 4.584777 (-4.061360) | 2.291145 / 3.745712 (-1.454567) | 2.540603 / 5.269862 (-2.729258) | 1.599090 / 4.565676 (-2.966586) | 0.058170 / 0.424275 (-0.366105) | 0.004556 / 0.007607 (-0.003051) | 0.308361 / 0.226044 (0.082316) | 3.069269 / 2.268929 (0.800340) | 1.698064 / 55.444624 (-53.746560) | 1.426631 / 6.876477 (-5.449846) | 1.463913 / 2.142072 (-0.678160) | 0.595234 / 4.805227 (-4.209993) | 0.107202 / 6.500664 (-6.393462) | 0.038183 / 0.075469 (-0.037286) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.905999 / 1.841788 (-0.935789) | 10.828492 / 8.074308 (2.754184) | 8.705635 / 10.191392 (-1.485757) | 0.121203 / 0.680424 (-0.559221) | 0.013789 / 0.534201 (-0.520412) | 0.268172 / 0.579283 (-0.311111) | 0.254277 / 0.434364 (-0.180086) | 0.310280 / 0.540337 (-0.230057) | 0.410490 / 1.386936 (-0.976446) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004696 / 0.011353 (-0.006657) | 0.002982 / 0.011008 (-0.008026) | 0.045333 / 0.038508 (0.006825) | 0.027483 / 0.023109 (0.004374) | 0.253438 / 0.275898 (-0.022460) | 0.272657 / 0.323480 (-0.050823) | 0.004060 / 0.007986 (-0.003926) | 0.002574 / 0.004328 (-0.001754) | 0.045462 / 0.004250 (0.041212) | 0.041260 / 0.037052 (0.004208) | 0.267919 / 0.258489 (0.009430) | 0.290935 / 0.293841 (-0.002906) | 0.026674 / 0.128546 (-0.101873) | 0.009370 / 0.075646 (-0.066276) | 0.053543 / 0.419271 (-0.365729) | 0.047390 / 0.043533 (0.003857) | 0.255774 / 0.255139 (0.000635) | 0.273909 / 0.283200 (-0.009291) | 0.017252 / 0.141683 (-0.124431) | 1.064298 / 1.452155 (-0.387857) | 1.125374 / 1.492716 (-0.367342) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091506 / 0.018006 (0.073499) | 0.298570 / 0.000490 (0.298080) | 0.000741 / 0.000200 (0.000541) | 0.000063 / 0.000054 (0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.020232 / 0.037411 (-0.017179) | 0.069640 / 0.014526 (0.055114) | 0.081360 / 0.176557 (-0.095197) | 0.116955 / 0.737135 (-0.620180) | 0.080920 / 0.296338 (-0.215418) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283000 / 0.215209 (0.067791) | 2.802526 / 2.077655 (0.724871) | 1.534631 / 1.504120 (0.030511) | 1.407260 / 1.541195 (-0.133935) | 1.409111 / 1.468490 (-0.059379) | 0.534892 / 4.584777 (-4.049885) | 2.350516 / 3.745712 (-1.395196) | 2.550444 / 5.269862 (-2.719418) | 1.661747 / 4.565676 (-2.903930) | 0.060978 / 0.424275 (-0.363297) | 0.005300 / 0.007607 (-0.002308) | 0.367418 / 0.226044 (0.141373) | 3.338046 / 2.268929 (1.069117) | 1.883914 / 55.444624 (-53.560710) | 1.638561 / 6.876477 (-5.237916) | 1.751547 / 2.142072 (-0.390526) | 0.633318 / 4.805227 (-4.171910) | 0.114971 / 6.500664 (-6.385693) | 0.040202 / 0.075469 (-0.035267) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.962053 / 1.841788 (-0.879735) | 11.600643 / 8.074308 (3.526334) | 9.526461 / 10.191392 (-0.664931) | 0.123909 / 0.680424 (-0.556515) | 0.015944 / 0.534201 (-0.518257) | 0.271542 / 0.579283 (-0.307741) | 0.254366 / 0.434364 (-0.179998) | 0.300499 / 0.540337 (-0.239838) | 0.409122 / 1.386936 (-0.977814) |

</details>
</details>

![](https://cml.dev/watermark.png#0acb27347a3c03efde612023235201a777e08e72 "CML watermark")

2024-03-01T17:32:13Z
https://github.com/huggingface/datasets/pull/6707#issuecomment-1973600862
https://api.github.com/repos/huggingface/datasets/issues/6707
IC_kwDODunzps51osJe
Iv1.05c79e9ad1f6bdfa
2018-07-30T09:30:17Z
Automate your workflow from idea to production
branch_protection_rule check_run check_suite create delete deployment deployment_status discussion discussion_comment fork gollum issues issue_comment label merge_group milestone page_build project project_card project_column public pull_request pull_request_review pull_request_review_comment push registry_package release repository repository_dispatch status watch workflow_dispatch workflow_run
https://help.github.com/en/actions
https://github.com/apps/github-actions
GitHub Actions
MDM6QXBwMTUzNjg=
https://avatars.githubusercontent.com/u/9919?v=4
https://api.github.com/users/github/events{/privacy}
https://api.github.com/users/github/followers
https://api.github.com/users/github/following{/other_user}
https://api.github.com/users/github/gists{/gist_id}
https://github.com/github
github
MDEyOk9yZ2FuaXphdGlvbjk5MTk=
https://api.github.com/users/github/orgs
https://api.github.com/users/github/received_events
https://api.github.com/users/github/repos
https://api.github.com/users/github/starred{/owner}{/repo}
https://api.github.com/users/github/subscriptions
Organization
https://api.github.com/users/github
public
write
read
write
write
write
write
write
write
write
read
read
write
write
write
write
write
write
write
read
github-actions
2025-03-07T16:35:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1973600862/reactions
2024-03-01T17:32:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1973600862
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public