Use a new low-memory approach for tf dataset index shuffling
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5863). All of your documentation changes will be reflected on that endpoint.
2023-05-15T15:32:54Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1548089581
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cRfjt
https://api.github.com/repos/huggingface/datasets/issues/comments/1548089581/reactions
2023-05-15T15:32:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548089581
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007764 / 0.011353 (-0.003588) | 0.005397 / 0.011008 (-0.005611) | 0.097995 / 0.038508 (0.059487) | 0.036360 / 0.023109 (0.013251) | 0.312148 / 0.275898 (0.036250) | 0.349427 / 0.323480 (0.025947) | 0.006635 / 0.007986 (-0.001350) | 0.004373 / 0.004328 (0.000044) | 0.074350 / 0.004250 (0.070099) | 0.054667 / 0.037052 (0.017614) | 0.301621 / 0.258489 (0.043132) | 0.364233 / 0.293841 (0.070392) | 0.035356 / 0.128546 (-0.093191) | 0.012512 / 0.075646 (-0.063134) | 0.333399 / 0.419271 (-0.085873) | 0.051363 / 0.043533 (0.007830) | 0.302372 / 0.255139 (0.047233) | 0.326542 / 0.283200 (0.043343) | 0.118610 / 0.141683 (-0.023073) | 1.438485 / 1.452155 (-0.013669) | 1.539131 / 1.492716 (0.046415) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.010920 / 0.018006 (-0.007086) | 0.561263 / 0.000490 (0.560773) | 0.003972 / 0.000200 (0.003772) | 0.000096 / 0.000054 (0.000042) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030333 / 0.037411 (-0.007078) | 0.113608 / 0.014526 (0.099083) | 0.125802 / 0.176557 (-0.050755) | 0.183885 / 0.737135 (-0.553250) | 0.130242 / 0.296338 (-0.166097) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.404147 / 0.215209 (0.188938) | 4.021990 / 2.077655 (1.944335) | 1.821450 / 1.504120 (0.317330) | 1.619032 / 1.541195 (0.077837) | 1.791267 / 1.468490 (0.322777) | 0.706683 / 4.584777 (-3.878094) | 3.819056 / 3.745712 (0.073344) | 3.485714 / 5.269862 (-1.784147) | 1.938968 / 4.565676 (-2.626709) | 0.086501 / 0.424275 (-0.337774) | 0.012300 / 0.007607 (0.004693) | 0.503600 / 0.226044 (0.277555) | 5.042123 / 2.268929 (2.773195) | 2.269712 / 55.444624 (-53.174912) | 1.944912 / 6.876477 (-4.931565) | 2.155196 / 2.142072 (0.013123) | 0.853434 / 4.805227 (-3.951793) | 0.175554 / 6.500664 (-6.325110) | 0.072005 / 0.075469 (-0.003464) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.203765 / 1.841788 (-0.638022) | 15.836634 / 8.074308 (7.762326) | 15.707348 / 10.191392 (5.515956) | 0.164828 / 0.680424 (-0.515596) | 0.018115 / 0.534201 (-0.516086) | 0.434591 / 0.579283 (-0.144692) | 0.437858 / 0.434364 (0.003495) | 0.524672 / 0.540337 (-0.015665) | 0.610535 / 1.386936 (-0.776401) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007558 / 0.011353 (-0.003795) | 0.005258 / 0.011008 (-0.005750) | 0.075263 / 0.038508 (0.036755) | 0.033915 / 0.023109 (0.010805) | 0.371368 / 0.275898 (0.095470) | 0.399239 / 0.323480 (0.075760) | 0.006547 / 0.007986 (-0.001439) | 0.004675 / 0.004328 (0.000347) | 0.074230 / 0.004250 (0.069980) | 0.054653 / 0.037052 (0.017601) | 0.376655 / 0.258489 (0.118166) | 0.438437 / 0.293841 (0.144596) | 0.035838 / 0.128546 (-0.092709) | 0.012641 / 0.075646 (-0.063005) | 0.087279 / 0.419271 (-0.331993) | 0.046311 / 0.043533 (0.002778) | 0.356649 / 0.255139 (0.101510) | 0.377876 / 0.283200 (0.094677) | 0.108097 / 0.141683 (-0.033586) | 1.478461 / 1.452155 (0.026306) | 1.560375 / 1.492716 (0.067658) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.316384 / 0.018006 (0.298378) | 0.539382 / 0.000490 (0.538892) | 0.002029 / 0.000200 (0.001829) | 0.000090 / 0.000054 (0.000036) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029950 / 0.037411 (-0.007462) | 0.111371 / 0.014526 (0.096846) | 0.125254 / 0.176557 (-0.051303) | 0.173064 / 0.737135 (-0.564071) | 0.130446 / 0.296338 (-0.165893) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.424882 / 0.215209 (0.209673) | 4.241575 / 2.077655 (2.163920) | 2.096216 / 1.504120 (0.592096) | 1.916017 / 1.541195 (0.374823) | 2.016318 / 1.468490 (0.547828) | 0.701197 / 4.584777 (-3.883580) | 3.762365 / 3.745712 (0.016652) | 3.307805 / 5.269862 (-1.962057) | 1.841752 / 4.565676 (-2.723925) | 0.086003 / 0.424275 (-0.338272) | 0.012247 / 0.007607 (0.004640) | 0.532926 / 0.226044 (0.306882) | 5.370509 / 2.268929 (3.101580) | 2.587853 / 55.444624 (-52.856772) | 2.264541 / 6.876477 (-4.611936) | 2.374833 / 2.142072 (0.232760) | 0.827751 / 4.805227 (-3.977476) | 0.169454 / 6.500664 (-6.331210) | 0.066340 / 0.075469 (-0.009129) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.319128 / 1.841788 (-0.522660) | 16.702085 / 8.074308 (8.627777) | 13.559957 / 10.191392 (3.368565) | 0.146659 / 0.680424 (-0.533765) | 0.017384 / 0.534201 (-0.516817) | 0.421126 / 0.579283 (-0.158157) | 0.422067 / 0.434364 (-0.012297) | 0.490615 / 0.540337 (-0.049723) | 0.587151 / 1.386936 (-0.799785) |

</details>
</details>

![](https://cml.dev/watermark.png#79f4b6de25128999f5fc0a7bde9aa71c461f518f "CML watermark")

2023-05-15T15:34:16Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1548091580
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cRgC8
https://api.github.com/repos/huggingface/datasets/issues/comments/1548091580/reactions
2023-05-15T15:34:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548091580
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006604 / 0.011353 (-0.004749) | 0.004508 / 0.011008 (-0.006500) | 0.098652 / 0.038508 (0.060144) | 0.028172 / 0.023109 (0.005063) | 0.366997 / 0.275898 (0.091099) | 0.403691 / 0.323480 (0.080211) | 0.005127 / 0.007986 (-0.002859) | 0.003340 / 0.004328 (-0.000989) | 0.075408 / 0.004250 (0.071157) | 0.038049 / 0.037052 (0.000996) | 0.367914 / 0.258489 (0.109425) | 0.410958 / 0.293841 (0.117118) | 0.030454 / 0.128546 (-0.098093) | 0.011422 / 0.075646 (-0.064224) | 0.325048 / 0.419271 (-0.094223) | 0.042959 / 0.043533 (-0.000574) | 0.374536 / 0.255139 (0.119397) | 0.394738 / 0.283200 (0.111538) | 0.090481 / 0.141683 (-0.051201) | 1.504858 / 1.452155 (0.052703) | 1.569072 / 1.492716 (0.076356) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.010062 / 0.018006 (-0.007945) | 0.408619 / 0.000490 (0.408130) | 0.002307 / 0.000200 (0.002107) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022898 / 0.037411 (-0.014514) | 0.096975 / 0.014526 (0.082449) | 0.103032 / 0.176557 (-0.073524) | 0.164877 / 0.737135 (-0.572259) | 0.107324 / 0.296338 (-0.189014) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.446652 / 0.215209 (0.231442) | 4.466939 / 2.077655 (2.389285) | 2.204590 / 1.504120 (0.700471) | 2.004048 / 1.541195 (0.462853) | 2.053035 / 1.468490 (0.584545) | 0.696617 / 4.584777 (-3.888160) | 3.391173 / 3.745712 (-0.354539) | 1.863306 / 5.269862 (-3.406556) | 1.160637 / 4.565676 (-3.405039) | 0.083115 / 0.424275 (-0.341160) | 0.012470 / 0.007607 (0.004862) | 0.547207 / 0.226044 (0.321163) | 5.500667 / 2.268929 (3.231739) | 2.656615 / 55.444624 (-52.788009) | 2.313281 / 6.876477 (-4.563195) | 2.395632 / 2.142072 (0.253559) | 0.815361 / 4.805227 (-3.989867) | 0.152112 / 6.500664 (-6.348552) | 0.067485 / 0.075469 (-0.007984) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.206975 / 1.841788 (-0.634813) | 13.684136 / 8.074308 (5.609828) | 13.919129 / 10.191392 (3.727737) | 0.140767 / 0.680424 (-0.539657) | 0.016445 / 0.534201 (-0.517756) | 0.379136 / 0.579283 (-0.200147) | 0.385395 / 0.434364 (-0.048969) | 0.445781 / 0.540337 (-0.094556) | 0.522056 / 1.386936 (-0.864880) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006370 / 0.011353 (-0.004983) | 0.004514 / 0.011008 (-0.006495) | 0.075671 / 0.038508 (0.037163) | 0.026723 / 0.023109 (0.003614) | 0.359819 / 0.275898 (0.083921) | 0.387935 / 0.323480 (0.064456) | 0.004888 / 0.007986 (-0.003098) | 0.004619 / 0.004328 (0.000290) | 0.075546 / 0.004250 (0.071295) | 0.039024 / 0.037052 (0.001971) | 0.361173 / 0.258489 (0.102684) | 0.411425 / 0.293841 (0.117584) | 0.030842 / 0.128546 (-0.097705) | 0.011555 / 0.075646 (-0.064091) | 0.084697 / 0.419271 (-0.334574) | 0.039281 / 0.043533 (-0.004252) | 0.370082 / 0.255139 (0.114943) | 0.382113 / 0.283200 (0.098913) | 0.091237 / 0.141683 (-0.050445) | 1.534185 / 1.452155 (0.082030) | 1.576488 / 1.492716 (0.083772) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.226568 / 0.018006 (0.208562) | 0.401566 / 0.000490 (0.401076) | 0.002915 / 0.000200 (0.002715) | 0.000076 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025357 / 0.037411 (-0.012054) | 0.099747 / 0.014526 (0.085221) | 0.106443 / 0.176557 (-0.070113) | 0.157147 / 0.737135 (-0.579989) | 0.110759 / 0.296338 (-0.185580) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.444648 / 0.215209 (0.229439) | 4.437930 / 2.077655 (2.360275) | 2.154033 / 1.504120 (0.649913) | 1.958351 / 1.541195 (0.417157) | 1.991031 / 1.468490 (0.522541) | 0.691440 / 4.584777 (-3.893337) | 3.369087 / 3.745712 (-0.376625) | 1.847103 / 5.269862 (-3.422758) | 1.152509 / 4.565676 (-3.413168) | 0.082519 / 0.424275 (-0.341756) | 0.012609 / 0.007607 (0.005001) | 0.547267 / 0.226044 (0.321222) | 5.501335 / 2.268929 (3.232407) | 2.621079 / 55.444624 (-52.823545) | 2.281332 / 6.876477 (-4.595145) | 2.300427 / 2.142072 (0.158354) | 0.803611 / 4.805227 (-4.001616) | 0.151784 / 6.500664 (-6.348880) | 0.067801 / 0.075469 (-0.007669) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.343201 / 1.841788 (-0.498587) | 13.901033 / 8.074308 (5.826725) | 13.114738 / 10.191392 (2.923346) | 0.149358 / 0.680424 (-0.531066) | 0.016596 / 0.534201 (-0.517605) | 0.377310 / 0.579283 (-0.201973) | 0.387045 / 0.434364 (-0.047319) | 0.441272 / 0.540337 (-0.099065) | 0.525783 / 1.386936 (-0.861153) |

</details>
</details>

![](https://cml.dev/watermark.png#c127e5575ab4e22648976ad268d76264ef5d04f8 "CML watermark")

2023-05-15T15:40:10Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1548101840
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cRijQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1548101840/reactions
2023-05-15T15:40:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548101840
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008147 / 0.011353 (-0.003205) | 0.005531 / 0.011008 (-0.005477) | 0.099796 / 0.038508 (0.061288) | 0.041574 / 0.023109 (0.018465) | 0.315752 / 0.275898 (0.039854) | 0.369846 / 0.323480 (0.046366) | 0.006489 / 0.007986 (-0.001497) | 0.004339 / 0.004328 (0.000010) | 0.074769 / 0.004250 (0.070519) | 0.051313 / 0.037052 (0.014261) | 0.313463 / 0.258489 (0.054974) | 0.369918 / 0.293841 (0.076077) | 0.035893 / 0.128546 (-0.092653) | 0.012487 / 0.075646 (-0.063159) | 0.336464 / 0.419271 (-0.082807) | 0.052870 / 0.043533 (0.009337) | 0.310795 / 0.255139 (0.055656) | 0.333146 / 0.283200 (0.049946) | 0.112813 / 0.141683 (-0.028870) | 1.488192 / 1.452155 (0.036038) | 1.563438 / 1.492716 (0.070721) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.015015 / 0.018006 (-0.002991) | 0.531783 / 0.000490 (0.531294) | 0.005039 / 0.000200 (0.004839) | 0.000103 / 0.000054 (0.000049) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030205 / 0.037411 (-0.007207) | 0.115997 / 0.014526 (0.101471) | 0.122958 / 0.176557 (-0.053599) | 0.186956 / 0.737135 (-0.550180) | 0.130268 / 0.296338 (-0.166071) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.402648 / 0.215209 (0.187439) | 3.996121 / 2.077655 (1.918466) | 1.811715 / 1.504120 (0.307595) | 1.640805 / 1.541195 (0.099610) | 1.810478 / 1.468490 (0.341988) | 0.699996 / 4.584777 (-3.884781) | 3.834020 / 3.745712 (0.088308) | 3.688364 / 5.269862 (-1.581498) | 1.973828 / 4.565676 (-2.591849) | 0.087085 / 0.424275 (-0.337190) | 0.012501 / 0.007607 (0.004894) | 0.498934 / 0.226044 (0.272889) | 4.977608 / 2.268929 (2.708680) | 2.258678 / 55.444624 (-53.185947) | 1.934251 / 6.876477 (-4.942226) | 2.177409 / 2.142072 (0.035337) | 0.873470 / 4.805227 (-3.931757) | 0.173132 / 6.500664 (-6.327532) | 0.069144 / 0.075469 (-0.006325) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.181554 / 1.841788 (-0.660234) | 15.694468 / 8.074308 (7.620160) | 15.026954 / 10.191392 (4.835562) | 0.167092 / 0.680424 (-0.513332) | 0.017921 / 0.534201 (-0.516280) | 0.425649 / 0.579283 (-0.153634) | 0.423225 / 0.434364 (-0.011139) | 0.522132 / 0.540337 (-0.018205) | 0.612806 / 1.386936 (-0.774130) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007896 / 0.011353 (-0.003457) | 0.005581 / 0.011008 (-0.005427) | 0.076338 / 0.038508 (0.037830) | 0.037064 / 0.023109 (0.013954) | 0.399706 / 0.275898 (0.123808) | 0.431698 / 0.323480 (0.108218) | 0.006846 / 0.007986 (-0.001140) | 0.006010 / 0.004328 (0.001682) | 0.075771 / 0.004250 (0.071520) | 0.058214 / 0.037052 (0.021161) | 0.395753 / 0.258489 (0.137264) | 0.459925 / 0.293841 (0.166084) | 0.036349 / 0.128546 (-0.092197) | 0.012720 / 0.075646 (-0.062926) | 0.087248 / 0.419271 (-0.332024) | 0.049405 / 0.043533 (0.005872) | 0.387576 / 0.255139 (0.132437) | 0.409861 / 0.283200 (0.126661) | 0.111639 / 0.141683 (-0.030043) | 1.482840 / 1.452155 (0.030685) | 1.574465 / 1.492716 (0.081749) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.320628 / 0.018006 (0.302622) | 0.556338 / 0.000490 (0.555848) | 0.000445 / 0.000200 (0.000245) | 0.000060 / 0.000054 (0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032905 / 0.037411 (-0.004507) | 0.121253 / 0.014526 (0.106727) | 0.127241 / 0.176557 (-0.049316) | 0.178090 / 0.737135 (-0.559045) | 0.143285 / 0.296338 (-0.153054) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.437852 / 0.215209 (0.222643) | 4.369770 / 2.077655 (2.292115) | 2.219932 / 1.504120 (0.715812) | 2.032520 / 1.541195 (0.491325) | 2.154300 / 1.468490 (0.685810) | 0.678942 / 4.584777 (-3.905835) | 3.768148 / 3.745712 (0.022436) | 2.152738 / 5.269862 (-3.117124) | 1.341480 / 4.565676 (-3.224197) | 0.084326 / 0.424275 (-0.339949) | 0.012288 / 0.007607 (0.004681) | 0.547677 / 0.226044 (0.321633) | 5.496777 / 2.268929 (3.227848) | 2.702267 / 55.444624 (-52.742357) | 2.388580 / 6.876477 (-4.487897) | 2.471673 / 2.142072 (0.329601) | 0.833645 / 4.805227 (-3.971582) | 0.167113 / 6.500664 (-6.333551) | 0.067658 / 0.075469 (-0.007811) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.282050 / 1.841788 (-0.559737) | 16.413677 / 8.074308 (8.339369) | 14.080910 / 10.191392 (3.889518) | 0.171782 / 0.680424 (-0.508642) | 0.018186 / 0.534201 (-0.516015) | 0.425244 / 0.579283 (-0.154039) | 0.430260 / 0.434364 (-0.004104) | 0.500838 / 0.540337 (-0.039499) | 0.591900 / 1.386936 (-0.795036) |

</details>
</details>

![](https://cml.dev/watermark.png#5fc5c538de84da400118e3712077acc580ce85c4 "CML watermark")

2023-05-15T15:41:35Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1548103971
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cRjEj
https://api.github.com/repos/huggingface/datasets/issues/comments/1548103971/reactions
2023-05-15T15:41:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548103971
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
MEMBER
The approach we take here is to no longer materialize the entire index array or shuffle buffer. Instead, we do the following:

1) Generate a dataset with `tf.data.Dataset.range`. This dataset is not materialized - it's basically a range iterator.
2) When we begin iterating over a dataset, generate a random seed. This value is constant for each pass over the dataset, and is regenerated if we start a new iteration or epoch over the dataset.
3) Map the range dataset and the random seed with `tf.random.index_shuffle`. This converts indices into the equivalent values in a permuted array. In other words `tf.random.index_shuffle(indices, maxval=50_000_000)` is equivalent to `np.random.permutation(50_000_000)[indices]`, but without ever materializing the `np.random.permutation(50_000_000)` array.

Using this approach gives us a complete iteration over the dataset that does not skip any samples, compiles in TF and also never materializes the complete index array, which should avoid the memory usage issues. I'm testing that now!
2023-05-15T15:42:45Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1548105647
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cRjev
https://api.github.com/repos/huggingface/datasets/issues/comments/1548105647/reactions
2023-05-15T15:44:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548105647
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008395 / 0.011353 (-0.002958) | 0.005893 / 0.011008 (-0.005115) | 0.117081 / 0.038508 (0.078573) | 0.040987 / 0.023109 (0.017878) | 0.394234 / 0.275898 (0.118336) | 0.447036 / 0.323480 (0.123556) | 0.006703 / 0.007986 (-0.001283) | 0.006085 / 0.004328 (0.001757) | 0.086479 / 0.004250 (0.082228) | 0.050192 / 0.037052 (0.013140) | 0.400958 / 0.258489 (0.142469) | 0.455551 / 0.293841 (0.161710) | 0.041481 / 0.128546 (-0.087065) | 0.014135 / 0.075646 (-0.061511) | 0.399929 / 0.419271 (-0.019343) | 0.060824 / 0.043533 (0.017291) | 0.395946 / 0.255139 (0.140807) | 0.428811 / 0.283200 (0.145611) | 0.120057 / 0.141683 (-0.021626) | 1.703244 / 1.452155 (0.251090) | 1.841153 / 1.492716 (0.348436) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.021826 / 0.018006 (0.003820) | 0.494279 / 0.000490 (0.493789) | 0.011258 / 0.000200 (0.011058) | 0.000382 / 0.000054 (0.000328) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031651 / 0.037411 (-0.005760) | 0.132871 / 0.014526 (0.118345) | 0.137388 / 0.176557 (-0.039169) | 0.205808 / 0.737135 (-0.531327) | 0.147585 / 0.296338 (-0.148753) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.474483 / 0.215209 (0.259274) | 4.726568 / 2.077655 (2.648914) | 2.136172 / 1.504120 (0.632052) | 1.918364 / 1.541195 (0.377169) | 2.068794 / 1.468490 (0.600304) | 0.836481 / 4.584777 (-3.748296) | 4.550583 / 3.745712 (0.804871) | 2.456287 / 5.269862 (-2.813574) | 1.563127 / 4.565676 (-3.002550) | 0.102541 / 0.424275 (-0.321734) | 0.014492 / 0.007607 (0.006885) | 0.598572 / 0.226044 (0.372528) | 5.953321 / 2.268929 (3.684392) | 2.695210 / 55.444624 (-52.749414) | 2.294317 / 6.876477 (-4.582160) | 2.456585 / 2.142072 (0.314513) | 1.019907 / 4.805227 (-3.785320) | 0.201225 / 6.500664 (-6.299439) | 0.077113 / 0.075469 (0.001644) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.497662 / 1.841788 (-0.344126) | 18.216941 / 8.074308 (10.142633) | 17.016638 / 10.191392 (6.825246) | 0.193271 / 0.680424 (-0.487153) | 0.020440 / 0.534201 (-0.513761) | 0.509361 / 0.579283 (-0.069922) | 0.513389 / 0.434364 (0.079025) | 0.622266 / 0.540337 (0.081928) | 0.741733 / 1.386936 (-0.645203) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008641 / 0.011353 (-0.002712) | 0.005792 / 0.011008 (-0.005216) | 0.086020 / 0.038508 (0.047512) | 0.040005 / 0.023109 (0.016896) | 0.435120 / 0.275898 (0.159222) | 0.480269 / 0.323480 (0.156789) | 0.006669 / 0.007986 (-0.001317) | 0.006039 / 0.004328 (0.001711) | 0.083468 / 0.004250 (0.079218) | 0.057700 / 0.037052 (0.020648) | 0.416418 / 0.258489 (0.157929) | 0.508286 / 0.293841 (0.214445) | 0.041198 / 0.128546 (-0.087349) | 0.014346 / 0.075646 (-0.061301) | 0.100553 / 0.419271 (-0.318718) | 0.054201 / 0.043533 (0.010668) | 0.438232 / 0.255139 (0.183093) | 0.454707 / 0.283200 (0.171508) | 0.118332 / 0.141683 (-0.023351) | 1.657607 / 1.452155 (0.205452) | 1.825510 / 1.492716 (0.332794) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.236156 / 0.018006 (0.218150) | 0.487612 / 0.000490 (0.487123) | 0.005747 / 0.000200 (0.005547) | 0.000111 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035127 / 0.037411 (-0.002284) | 0.132013 / 0.014526 (0.117487) | 0.142316 / 0.176557 (-0.034241) | 0.198627 / 0.737135 (-0.538508) | 0.145454 / 0.296338 (-0.150885) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.513041 / 0.215209 (0.297832) | 5.066197 / 2.077655 (2.988542) | 2.508779 / 1.504120 (1.004659) | 2.273901 / 1.541195 (0.732706) | 2.364958 / 1.468490 (0.896468) | 0.811367 / 4.584777 (-3.773410) | 4.504744 / 3.745712 (0.759032) | 2.499811 / 5.269862 (-2.770050) | 1.583349 / 4.565676 (-2.982328) | 0.101701 / 0.424275 (-0.322574) | 0.014379 / 0.007607 (0.006772) | 0.669506 / 0.226044 (0.443462) | 6.556702 / 2.268929 (4.287774) | 3.123457 / 55.444624 (-52.321167) | 2.731997 / 6.876477 (-4.144480) | 2.862866 / 2.142072 (0.720794) | 0.992956 / 4.805227 (-3.812271) | 0.200473 / 6.500664 (-6.300191) | 0.078780 / 0.075469 (0.003311) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.540718 / 1.841788 (-0.301070) | 18.749344 / 8.074308 (10.675036) | 15.648983 / 10.191392 (5.457591) | 0.174089 / 0.680424 (-0.506335) | 0.020441 / 0.534201 (-0.513760) | 0.503742 / 0.579283 (-0.075541) | 0.500648 / 0.434364 (0.066284) | 0.598558 / 0.540337 (0.058221) | 0.712093 / 1.386936 (-0.674843) |

</details>
</details>

![](https://cml.dev/watermark.png#621554280f964b5fe87ece1a46b794406d943b1e "CML watermark")

2023-05-15T16:10:23Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1548152347
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cRu4b
https://api.github.com/repos/huggingface/datasets/issues/comments/1548152347/reactions
2023-05-15T16:10:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548152347
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009940 / 0.011353 (-0.001412) | 0.006193 / 0.011008 (-0.004815) | 0.125874 / 0.038508 (0.087366) | 0.038664 / 0.023109 (0.015555) | 0.380013 / 0.275898 (0.104115) | 0.430152 / 0.323480 (0.106672) | 0.006961 / 0.007986 (-0.001025) | 0.004749 / 0.004328 (0.000420) | 0.099743 / 0.004250 (0.095492) | 0.052349 / 0.037052 (0.015297) | 0.433354 / 0.258489 (0.174865) | 0.436273 / 0.293841 (0.142433) | 0.053929 / 0.128546 (-0.074617) | 0.019369 / 0.075646 (-0.056278) | 0.421783 / 0.419271 (0.002511) | 0.062746 / 0.043533 (0.019213) | 0.377225 / 0.255139 (0.122086) | 0.413708 / 0.283200 (0.130508) | 0.111371 / 0.141683 (-0.030312) | 1.819166 / 1.452155 (0.367011) | 1.974527 / 1.492716 (0.481810) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090664 / 0.018006 (0.072658) | 0.566166 / 0.000490 (0.565676) | 0.079305 / 0.000200 (0.079105) | 0.000755 / 0.000054 (0.000700) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029720 / 0.037411 (-0.007691) | 0.126030 / 0.014526 (0.111504) | 0.146020 / 0.176557 (-0.030537) | 0.210354 / 0.737135 (-0.526781) | 0.149428 / 0.296338 (-0.146910) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.624371 / 0.215209 (0.409162) | 6.332839 / 2.077655 (4.255184) | 2.547784 / 1.504120 (1.043664) | 2.150508 / 1.541195 (0.609313) | 2.240816 / 1.468490 (0.772326) | 1.271131 / 4.584777 (-3.313646) | 5.642726 / 3.745712 (1.897014) | 3.212988 / 5.269862 (-2.056874) | 2.258123 / 4.565676 (-2.307553) | 0.149477 / 0.424275 (-0.274798) | 0.014603 / 0.007607 (0.006996) | 0.782155 / 0.226044 (0.556111) | 7.855191 / 2.268929 (5.586262) | 3.308638 / 55.444624 (-52.135986) | 2.548142 / 6.876477 (-4.328335) | 2.627374 / 2.142072 (0.485301) | 1.515170 / 4.805227 (-3.290058) | 0.262479 / 6.500664 (-6.238185) | 0.082181 / 0.075469 (0.006712) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.573169 / 1.841788 (-0.268618) | 18.105719 / 8.074308 (10.031411) | 22.015179 / 10.191392 (11.823787) | 0.254678 / 0.680424 (-0.425746) | 0.027098 / 0.534201 (-0.507103) | 0.578045 / 0.579283 (-0.001238) | 0.647130 / 0.434364 (0.212766) | 0.650522 / 0.540337 (0.110185) | 0.797713 / 1.386936 (-0.589223) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010376 / 0.011353 (-0.000977) | 0.005990 / 0.011008 (-0.005018) | 0.097144 / 0.038508 (0.058635) | 0.038205 / 0.023109 (0.015096) | 0.468347 / 0.275898 (0.192449) | 0.497646 / 0.323480 (0.174166) | 0.006916 / 0.007986 (-0.001069) | 0.004760 / 0.004328 (0.000431) | 0.109838 / 0.004250 (0.105587) | 0.048321 / 0.037052 (0.011269) | 0.437458 / 0.258489 (0.178969) | 0.534864 / 0.293841 (0.241023) | 0.053655 / 0.128546 (-0.074892) | 0.021915 / 0.075646 (-0.053732) | 0.121047 / 0.419271 (-0.298224) | 0.059694 / 0.043533 (0.016162) | 0.466937 / 0.255139 (0.211798) | 0.482030 / 0.283200 (0.198831) | 0.117458 / 0.141683 (-0.024225) | 1.835551 / 1.452155 (0.383396) | 1.965748 / 1.492716 (0.473031) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.234885 / 0.018006 (0.216879) | 0.529925 / 0.000490 (0.529436) | 0.000484 / 0.000200 (0.000284) | 0.000085 / 0.000054 (0.000031) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030959 / 0.037411 (-0.006453) | 0.128905 / 0.014526 (0.114379) | 0.136913 / 0.176557 (-0.039643) | 0.195133 / 0.737135 (-0.542002) | 0.147929 / 0.296338 (-0.148410) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.715661 / 0.215209 (0.500451) | 6.994125 / 2.077655 (4.916470) | 3.033178 / 1.504120 (1.529058) | 2.663709 / 1.541195 (1.122515) | 2.707558 / 1.468490 (1.239068) | 1.316195 / 4.584777 (-3.268582) | 5.688264 / 3.745712 (1.942552) | 3.260897 / 5.269862 (-2.008964) | 2.134985 / 4.565676 (-2.430691) | 0.153945 / 0.424275 (-0.270330) | 0.014727 / 0.007607 (0.007119) | 0.911339 / 0.226044 (0.685294) | 8.902640 / 2.268929 (6.633711) | 3.806606 / 55.444624 (-51.638018) | 3.052238 / 6.876477 (-3.824238) | 3.046945 / 2.142072 (0.904873) | 1.559837 / 4.805227 (-3.245390) | 0.272276 / 6.500664 (-6.228388) | 0.087728 / 0.075469 (0.012259) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.712691 / 1.841788 (-0.129097) | 18.127575 / 8.074308 (10.053267) | 19.734063 / 10.191392 (9.542671) | 0.235006 / 0.680424 (-0.445418) | 0.027581 / 0.534201 (-0.506620) | 0.551080 / 0.579283 (-0.028203) | 0.608564 / 0.434364 (0.174200) | 0.636578 / 0.540337 (0.096241) | 0.732374 / 1.386936 (-0.654562) |

</details>
</details>

![](https://cml.dev/watermark.png#36911ca06d9c4e37ce36da6228cb3af8b40c2add "CML watermark")

2023-05-15T16:15:25Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1548160258
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cRw0C
https://api.github.com/repos/huggingface/datasets/issues/comments/1548160258/reactions
2023-05-15T16:15:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548160258
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
MEMBER
Looks good in testing - this should be ready for review! cc @lhoestq @massquantity
2023-05-15T16:54:53Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1548212340
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cR9h0
https://api.github.com/repos/huggingface/datasets/issues/comments/1548212340/reactions
2023-05-15T16:54:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548212340
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
NONE
Looks good to me, though i doubt that very few people will upgrade to TF >= 2.9 unless their memory is full:)
2023-05-16T01:53:37Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1548850995
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cUZcz
https://api.github.com/repos/huggingface/datasets/issues/comments/1548850995/reactions
2023-05-16T02:14:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1548850995
https://avatars.githubusercontent.com/u/28751760?v=4
https://api.github.com/users/massquantity/events{/privacy}
https://api.github.com/users/massquantity/followers
https://api.github.com/users/massquantity/following{/other_user}
https://api.github.com/users/massquantity/gists{/gist_id}
https://github.com/massquantity
massquantity
MDQ6VXNlcjI4NzUxNzYw
https://api.github.com/users/massquantity/orgs
https://api.github.com/users/massquantity/received_events
https://api.github.com/users/massquantity/repos
https://api.github.com/users/massquantity/starred{/owner}{/repo}
https://api.github.com/users/massquantity/subscriptions
User
https://api.github.com/users/massquantity
public
MEMBER
Is it more efficient than using numpy to shuffle as in multiprocessing ? Why not use the same strategy ?
2023-05-16T08:29:38Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1549228360
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cV1lI
https://api.github.com/repos/huggingface/datasets/issues/comments/1549228360/reactions
2023-05-16T08:29:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549228360
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Good question, honestly! The NumPy strategy works fine, but requires us to handle multiple processes instead of doing everything in `tf.data`. We could just scrap this entire code path and always use the multiprocessing NumPy approach, but I think single-threaded throughput would be lower if we did that. If you prefer it for code simplicity, though, I can do that.

In the longer term, I'm hoping that `tf.data` gets native support for our data structures and we can transition the whole pipeline to pure `tf.data`, but that still hasn't happened 🫠
2023-05-16T12:34:09Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1549578030
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cXK8u
https://api.github.com/repos/huggingface/datasets/issues/comments/1549578030/reactions
2023-05-16T12:34:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549578030
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
And @massquantity TF 2.13 is going to release in a couple of days, so I hope most users are at least on TF 2.9 by now!
2023-05-16T12:35:14Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1549579570
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cXLUy
https://api.github.com/repos/huggingface/datasets/issues/comments/1549579570/reactions
2023-05-16T12:35:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549579570
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
Unless there is a big gap in performance I think code simplicity would be appreciated ^^
2023-05-16T13:29:26Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1549679601
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cXjvx
https://api.github.com/repos/huggingface/datasets/issues/comments/1549679601/reactions
2023-05-16T13:29:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549679601
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008638 / 0.011353 (-0.002715) | 0.006013 / 0.011008 (-0.004995) | 0.116456 / 0.038508 (0.077948) | 0.040419 / 0.023109 (0.017310) | 0.418374 / 0.275898 (0.142476) | 0.447693 / 0.323480 (0.124213) | 0.007002 / 0.007986 (-0.000984) | 0.006175 / 0.004328 (0.001847) | 0.087801 / 0.004250 (0.083550) | 0.051980 / 0.037052 (0.014928) | 0.393275 / 0.258489 (0.134786) | 0.449601 / 0.293841 (0.155760) | 0.041670 / 0.128546 (-0.086876) | 0.014396 / 0.075646 (-0.061251) | 0.399175 / 0.419271 (-0.020096) | 0.060635 / 0.043533 (0.017102) | 0.391449 / 0.255139 (0.136310) | 0.420713 / 0.283200 (0.137513) | 0.121369 / 0.141683 (-0.020314) | 1.692630 / 1.452155 (0.240475) | 1.815526 / 1.492716 (0.322810) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.244321 / 0.018006 (0.226315) | 0.487947 / 0.000490 (0.487458) | 0.004563 / 0.000200 (0.004363) | 0.000116 / 0.000054 (0.000061) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033425 / 0.037411 (-0.003987) | 0.134458 / 0.014526 (0.119932) | 0.138810 / 0.176557 (-0.037746) | 0.208871 / 0.737135 (-0.528264) | 0.147964 / 0.296338 (-0.148374) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.483347 / 0.215209 (0.268138) | 4.799550 / 2.077655 (2.721895) | 2.174149 / 1.504120 (0.670029) | 1.943276 / 1.541195 (0.402081) | 2.010884 / 1.468490 (0.542394) | 0.832030 / 4.584777 (-3.752747) | 4.716713 / 3.745712 (0.971001) | 4.615810 / 5.269862 (-0.654052) | 2.379600 / 4.565676 (-2.186077) | 0.103560 / 0.424275 (-0.320715) | 0.014683 / 0.007607 (0.007076) | 0.598558 / 0.226044 (0.372514) | 5.999126 / 2.268929 (3.730197) | 2.677819 / 55.444624 (-52.766805) | 2.320838 / 6.876477 (-4.555639) | 2.503684 / 2.142072 (0.361611) | 1.016459 / 4.805227 (-3.788769) | 0.201672 / 6.500664 (-6.298992) | 0.079310 / 0.075469 (0.003841) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.446374 / 1.841788 (-0.395413) | 19.219310 / 8.074308 (11.145002) | 17.294665 / 10.191392 (7.103273) | 0.246115 / 0.680424 (-0.434309) | 0.021406 / 0.534201 (-0.512795) | 0.524084 / 0.579283 (-0.055200) | 0.511254 / 0.434364 (0.076890) | 0.621304 / 0.540337 (0.080966) | 0.727088 / 1.386936 (-0.659848) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008907 / 0.011353 (-0.002446) | 0.006165 / 0.011008 (-0.004843) | 0.090786 / 0.038508 (0.052278) | 0.040893 / 0.023109 (0.017784) | 0.451252 / 0.275898 (0.175354) | 0.477811 / 0.323480 (0.154331) | 0.007418 / 0.007986 (-0.000568) | 0.005789 / 0.004328 (0.001461) | 0.087422 / 0.004250 (0.083171) | 0.061800 / 0.037052 (0.024748) | 0.459085 / 0.258489 (0.200596) | 0.488897 / 0.293841 (0.195056) | 0.048157 / 0.128546 (-0.080389) | 0.014676 / 0.075646 (-0.060970) | 0.104372 / 0.419271 (-0.314900) | 0.058066 / 0.043533 (0.014534) | 0.446131 / 0.255139 (0.190992) | 0.460428 / 0.283200 (0.177228) | 0.128492 / 0.141683 (-0.013191) | 1.811419 / 1.452155 (0.359265) | 1.894781 / 1.492716 (0.402064) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220527 / 0.018006 (0.202520) | 0.487663 / 0.000490 (0.487173) | 0.003864 / 0.000200 (0.003664) | 0.000162 / 0.000054 (0.000107) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036354 / 0.037411 (-0.001057) | 0.140469 / 0.014526 (0.125944) | 0.149990 / 0.176557 (-0.026566) | 0.212369 / 0.737135 (-0.524766) | 0.154000 / 0.296338 (-0.142338) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.514172 / 0.215209 (0.298963) | 5.129247 / 2.077655 (3.051593) | 2.536773 / 1.504120 (1.032653) | 2.317253 / 1.541195 (0.776058) | 2.424066 / 1.468490 (0.955576) | 0.836160 / 4.584777 (-3.748617) | 4.906235 / 3.745712 (1.160523) | 4.431395 / 5.269862 (-0.838467) | 2.332845 / 4.565676 (-2.232831) | 0.102867 / 0.424275 (-0.321409) | 0.014851 / 0.007607 (0.007244) | 0.644104 / 0.226044 (0.418060) | 6.415847 / 2.268929 (4.146918) | 3.186984 / 55.444624 (-52.257641) | 2.774125 / 6.876477 (-4.102352) | 2.848045 / 2.142072 (0.705972) | 1.018757 / 4.805227 (-3.786470) | 0.212333 / 6.500664 (-6.288331) | 0.079405 / 0.075469 (0.003936) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.748375 / 1.841788 (-0.093412) | 19.733829 / 8.074308 (11.659521) | 15.766665 / 10.191392 (5.575273) | 0.192087 / 0.680424 (-0.488337) | 0.027641 / 0.534201 (-0.506560) | 0.504101 / 0.579283 (-0.075182) | 0.493815 / 0.434364 (0.059451) | 0.583247 / 0.540337 (0.042910) | 0.697432 / 1.386936 (-0.689504) |

</details>
</details>

![](https://cml.dev/watermark.png#95c177e02ca20bf7bb3ed8f185d2d6f05a5e5f30 "CML watermark")

2023-05-16T14:55:02Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1549836406
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cYKB2
https://api.github.com/repos/huggingface/datasets/issues/comments/1549836406/reactions
2023-05-16T14:55:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549836406
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
MEMBER
Hi @lhoestq, I tried moving everything to the NumPy path but ran into issues - the `SharedMemory` constructs it depends on were only added in Python 3.8. As a result, if we move everything to that path then `to_tf_dataset` does not work on older Python versions.

For now, how do you feel about reverting and using my original solution, which has fallbacks for all versions of Python and TensorFlow? Once our minimum versions pass Python 3.8 or TF 2.9 we can remove the older code paths.
2023-05-16T15:09:41Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1549864031
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cYQxf
https://api.github.com/repos/huggingface/datasets/issues/comments/1549864031/reactions
2023-05-16T15:10:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1549864031
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
Gentle ping on this question @lhoestq!
2023-05-19T15:26:10Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1554752815
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5cq6Uv
https://api.github.com/repos/huggingface/datasets/issues/comments/1554752815/reactions
2023-05-19T15:26:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1554752815
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
Ah yes indeed. Feel free to revert and add comments to explain why you needed to have a different approach for single process
2023-05-19T15:55:05Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1554784413
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5crCCd
https://api.github.com/repos/huggingface/datasets/issues/comments/1554784413/reactions
2023-05-19T15:55:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1554784413
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008395 / 0.011353 (-0.002958) | 0.005773 / 0.011008 (-0.005235) | 0.115702 / 0.038508 (0.077194) | 0.039897 / 0.023109 (0.016788) | 0.483140 / 0.275898 (0.207242) | 0.531288 / 0.323480 (0.207808) | 0.006739 / 0.007986 (-0.001246) | 0.004419 / 0.004328 (0.000090) | 0.086374 / 0.004250 (0.082124) | 0.056498 / 0.037052 (0.019446) | 0.491589 / 0.258489 (0.233100) | 0.556366 / 0.293841 (0.262525) | 0.041366 / 0.128546 (-0.087181) | 0.014373 / 0.075646 (-0.061274) | 0.395504 / 0.419271 (-0.023767) | 0.094382 / 0.043533 (0.050849) | 0.483000 / 0.255139 (0.227861) | 0.522693 / 0.283200 (0.239494) | 0.138804 / 0.141683 (-0.002879) | 1.719563 / 1.452155 (0.267409) | 1.853470 / 1.492716 (0.360753) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235616 / 0.018006 (0.217610) | 0.483267 / 0.000490 (0.482777) | 0.008663 / 0.000200 (0.008463) | 0.000401 / 0.000054 (0.000347) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033124 / 0.037411 (-0.004287) | 0.128821 / 0.014526 (0.114295) | 0.138910 / 0.176557 (-0.037647) | 0.213570 / 0.737135 (-0.523566) | 0.146646 / 0.296338 (-0.149693) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.479998 / 0.215209 (0.264789) | 4.772325 / 2.077655 (2.694670) | 2.228424 / 1.504120 (0.724304) | 2.000915 / 1.541195 (0.459721) | 2.105799 / 1.468490 (0.637309) | 0.824235 / 4.584777 (-3.760542) | 4.511902 / 3.745712 (0.766189) | 4.723073 / 5.269862 (-0.546789) | 2.333442 / 4.565676 (-2.232235) | 0.101161 / 0.424275 (-0.323114) | 0.014403 / 0.007607 (0.006796) | 0.596395 / 0.226044 (0.370351) | 5.961046 / 2.268929 (3.692117) | 2.746679 / 55.444624 (-52.697946) | 2.352085 / 6.876477 (-4.524392) | 2.609812 / 2.142072 (0.467740) | 0.996950 / 4.805227 (-3.808277) | 0.197923 / 6.500664 (-6.302741) | 0.075546 / 0.075469 (0.000077) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.529896 / 1.841788 (-0.311892) | 18.183887 / 8.074308 (10.109578) | 16.352332 / 10.191392 (6.160940) | 0.213504 / 0.680424 (-0.466920) | 0.020388 / 0.534201 (-0.513813) | 0.497832 / 0.579283 (-0.081451) | 0.495477 / 0.434364 (0.061113) | 0.585984 / 0.540337 (0.045647) | 0.688726 / 1.386936 (-0.698210) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008422 / 0.011353 (-0.002931) | 0.005876 / 0.011008 (-0.005132) | 0.089310 / 0.038508 (0.050802) | 0.039769 / 0.023109 (0.016660) | 0.425279 / 0.275898 (0.149381) | 0.470818 / 0.323480 (0.147338) | 0.006519 / 0.007986 (-0.001467) | 0.006276 / 0.004328 (0.001948) | 0.085753 / 0.004250 (0.081503) | 0.053867 / 0.037052 (0.016815) | 0.429193 / 0.258489 (0.170704) | 0.480278 / 0.293841 (0.186437) | 0.040657 / 0.128546 (-0.087889) | 0.014055 / 0.075646 (-0.061591) | 0.101422 / 0.419271 (-0.317849) | 0.053803 / 0.043533 (0.010271) | 0.428348 / 0.255139 (0.173209) | 0.452193 / 0.283200 (0.168994) | 0.124914 / 0.141683 (-0.016769) | 1.750122 / 1.452155 (0.297968) | 1.850875 / 1.492716 (0.358159) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.249958 / 0.018006 (0.231952) | 0.485183 / 0.000490 (0.484694) | 0.000472 / 0.000200 (0.000272) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034563 / 0.037411 (-0.002848) | 0.135565 / 0.014526 (0.121039) | 0.143271 / 0.176557 (-0.033285) | 0.199080 / 0.737135 (-0.538056) | 0.149336 / 0.296338 (-0.147003) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.526170 / 0.215209 (0.310961) | 5.270960 / 2.077655 (3.193305) | 2.664585 / 1.504120 (1.160465) | 2.440027 / 1.541195 (0.898832) | 2.612764 / 1.468490 (1.144274) | 0.828965 / 4.584777 (-3.755812) | 4.769983 / 3.745712 (1.024271) | 2.441962 / 5.269862 (-2.827900) | 1.549032 / 4.565676 (-3.016644) | 0.100851 / 0.424275 (-0.323424) | 0.014425 / 0.007607 (0.006818) | 0.640908 / 0.226044 (0.414864) | 6.399041 / 2.268929 (4.130113) | 3.242424 / 55.444624 (-52.202200) | 2.836317 / 6.876477 (-4.040160) | 2.933010 / 2.142072 (0.790938) | 1.002277 / 4.805227 (-3.802950) | 0.201247 / 6.500664 (-6.299417) | 0.078777 / 0.075469 (0.003308) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.620415 / 1.841788 (-0.221373) | 19.153631 / 8.074308 (11.079323) | 16.744068 / 10.191392 (6.552676) | 0.167327 / 0.680424 (-0.513097) | 0.020186 / 0.534201 (-0.514015) | 0.503683 / 0.579283 (-0.075600) | 0.500051 / 0.434364 (0.065687) | 0.587188 / 0.540337 (0.046850) | 0.699975 / 1.386936 (-0.686961) |

</details>
</details>

![](https://cml.dev/watermark.png#291d7ffa695edb4b4e818c783b16d3466246cd56 "CML watermark")

2023-05-23T15:46:26Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1559704093
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5c9zId
https://api.github.com/repos/huggingface/datasets/issues/comments/1559704093/reactions
2023-05-23T15:46:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1559704093
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
MEMBER
This is probably ready, but likely conflicts with #5883. I'll wait for that PR to be merged and then rebase and merge this one.
2023-05-23T15:53:18Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1559719663
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5c927v
https://api.github.com/repos/huggingface/datasets/issues/comments/1559719663/reactions
2023-05-23T15:53:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1559719663
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008387 / 0.011353 (-0.002965) | 0.005824 / 0.011008 (-0.005184) | 0.117721 / 0.038508 (0.079213) | 0.040420 / 0.023109 (0.017311) | 0.404961 / 0.275898 (0.129063) | 0.426695 / 0.323480 (0.103215) | 0.006634 / 0.007986 (-0.001352) | 0.006033 / 0.004328 (0.001705) | 0.088652 / 0.004250 (0.084402) | 0.048075 / 0.037052 (0.011022) | 0.400683 / 0.258489 (0.142194) | 0.432489 / 0.293841 (0.138648) | 0.042065 / 0.128546 (-0.086482) | 0.014071 / 0.075646 (-0.061575) | 0.399398 / 0.419271 (-0.019873) | 0.066034 / 0.043533 (0.022501) | 0.400056 / 0.255139 (0.144918) | 0.421130 / 0.283200 (0.137930) | 0.119721 / 0.141683 (-0.021962) | 1.752166 / 1.452155 (0.300011) | 1.820161 / 1.492716 (0.327444) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.244264 / 0.018006 (0.226258) | 0.480882 / 0.000490 (0.480392) | 0.005604 / 0.000200 (0.005404) | 0.000175 / 0.000054 (0.000121) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032397 / 0.037411 (-0.005015) | 0.131632 / 0.014526 (0.117106) | 0.139765 / 0.176557 (-0.036792) | 0.213135 / 0.737135 (-0.524000) | 0.147891 / 0.296338 (-0.148447) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.474534 / 0.215209 (0.259325) | 4.730424 / 2.077655 (2.652770) | 2.163706 / 1.504120 (0.659586) | 1.936051 / 1.541195 (0.394857) | 2.012185 / 1.468490 (0.543695) | 0.826583 / 4.584777 (-3.758194) | 4.921494 / 3.745712 (1.175782) | 2.431401 / 5.269862 (-2.838460) | 1.566020 / 4.565676 (-2.999656) | 0.101255 / 0.424275 (-0.323020) | 0.014553 / 0.007607 (0.006946) | 0.608301 / 0.226044 (0.382256) | 6.089801 / 2.268929 (3.820873) | 2.691986 / 55.444624 (-52.752638) | 2.296498 / 6.876477 (-4.579979) | 2.455388 / 2.142072 (0.313315) | 0.984342 / 4.805227 (-3.820885) | 0.200447 / 6.500664 (-6.300217) | 0.077602 / 0.075469 (0.002133) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.445067 / 1.841788 (-0.396721) | 18.588670 / 8.074308 (10.514362) | 16.950216 / 10.191392 (6.758824) | 0.169688 / 0.680424 (-0.510736) | 0.020544 / 0.534201 (-0.513657) | 0.508506 / 0.579283 (-0.070777) | 0.516218 / 0.434364 (0.081854) | 0.646072 / 0.540337 (0.105734) | 0.763227 / 1.386936 (-0.623709) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008816 / 0.011353 (-0.002537) | 0.006016 / 0.011008 (-0.004992) | 0.090946 / 0.038508 (0.052438) | 0.040189 / 0.023109 (0.017080) | 0.446723 / 0.275898 (0.170825) | 0.494633 / 0.323480 (0.171153) | 0.007206 / 0.007986 (-0.000779) | 0.004508 / 0.004328 (0.000180) | 0.088477 / 0.004250 (0.084226) | 0.055587 / 0.037052 (0.018535) | 0.445349 / 0.258489 (0.186860) | 0.504940 / 0.293841 (0.211099) | 0.041976 / 0.128546 (-0.086570) | 0.014296 / 0.075646 (-0.061351) | 0.102835 / 0.419271 (-0.316436) | 0.054786 / 0.043533 (0.011253) | 0.444789 / 0.255139 (0.189651) | 0.472306 / 0.283200 (0.189106) | 0.123365 / 0.141683 (-0.018318) | 1.725803 / 1.452155 (0.273648) | 1.832216 / 1.492716 (0.339500) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.252680 / 0.018006 (0.234674) | 0.476719 / 0.000490 (0.476229) | 0.000461 / 0.000200 (0.000261) | 0.000067 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035961 / 0.037411 (-0.001450) | 0.135399 / 0.014526 (0.120873) | 0.147549 / 0.176557 (-0.029007) | 0.207468 / 0.737135 (-0.529667) | 0.151591 / 0.296338 (-0.144747) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.528143 / 0.215209 (0.312934) | 5.270766 / 2.077655 (3.193111) | 2.675644 / 1.504120 (1.171524) | 2.472855 / 1.541195 (0.931660) | 2.636020 / 1.468490 (1.167530) | 0.841325 / 4.584777 (-3.743452) | 4.702290 / 3.745712 (0.956578) | 2.523537 / 5.269862 (-2.746325) | 1.595617 / 4.565676 (-2.970059) | 0.102095 / 0.424275 (-0.322180) | 0.014568 / 0.007607 (0.006961) | 0.652090 / 0.226044 (0.426046) | 6.503086 / 2.268929 (4.234158) | 3.277025 / 55.444624 (-52.167599) | 2.931264 / 6.876477 (-3.945213) | 3.021667 / 2.142072 (0.879594) | 1.002560 / 4.805227 (-3.802668) | 0.202621 / 6.500664 (-6.298043) | 0.080583 / 0.075469 (0.005114) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.639281 / 1.841788 (-0.202507) | 18.911529 / 8.074308 (10.837220) | 17.082795 / 10.191392 (6.891403) | 0.179456 / 0.680424 (-0.500968) | 0.021740 / 0.534201 (-0.512460) | 0.526426 / 0.579283 (-0.052857) | 0.535083 / 0.434364 (0.100719) | 0.583304 / 0.540337 (0.042967) | 0.696733 / 1.386936 (-0.690203) |

</details>
</details>

![](https://cml.dev/watermark.png#757f19283f22eeb3e9aedefd82abc0aa2235f797 "CML watermark")

2023-05-23T16:09:26Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1559748285
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5c9969
https://api.github.com/repos/huggingface/datasets/issues/comments/1559748285/reactions
2023-05-23T16:09:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1559748285
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006823 / 0.011353 (-0.004530) | 0.004847 / 0.011008 (-0.006161) | 0.096038 / 0.038508 (0.057530) | 0.033037 / 0.023109 (0.009928) | 0.298379 / 0.275898 (0.022481) | 0.333319 / 0.323480 (0.009839) | 0.005343 / 0.007986 (-0.002643) | 0.003863 / 0.004328 (-0.000465) | 0.072928 / 0.004250 (0.068678) | 0.040898 / 0.037052 (0.003846) | 0.303116 / 0.258489 (0.044627) | 0.334021 / 0.293841 (0.040181) | 0.034780 / 0.128546 (-0.093767) | 0.011978 / 0.075646 (-0.063668) | 0.331642 / 0.419271 (-0.087629) | 0.052729 / 0.043533 (0.009196) | 0.298586 / 0.255139 (0.043447) | 0.319296 / 0.283200 (0.036097) | 0.097711 / 0.141683 (-0.043972) | 1.416899 / 1.452155 (-0.035256) | 1.546008 / 1.492716 (0.053292) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.234303 / 0.018006 (0.216296) | 0.492767 / 0.000490 (0.492278) | 0.004935 / 0.000200 (0.004736) | 0.000106 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030617 / 0.037411 (-0.006795) | 0.121203 / 0.014526 (0.106677) | 0.126677 / 0.176557 (-0.049879) | 0.186379 / 0.737135 (-0.550756) | 0.129849 / 0.296338 (-0.166490) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416324 / 0.215209 (0.201115) | 4.135563 / 2.077655 (2.057908) | 1.976182 / 1.504120 (0.472062) | 1.807611 / 1.541195 (0.266416) | 1.886282 / 1.468490 (0.417792) | 0.713006 / 4.584777 (-3.871771) | 3.899205 / 3.745712 (0.153493) | 2.283427 / 5.269862 (-2.986435) | 1.543088 / 4.565676 (-3.022589) | 0.086189 / 0.424275 (-0.338087) | 0.012908 / 0.007607 (0.005301) | 0.516156 / 0.226044 (0.290112) | 5.144199 / 2.268929 (2.875271) | 2.460142 / 55.444624 (-52.984482) | 2.209054 / 6.876477 (-4.667423) | 2.325277 / 2.142072 (0.183204) | 0.849890 / 4.805227 (-3.955337) | 0.173687 / 6.500664 (-6.326977) | 0.070178 / 0.075469 (-0.005291) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.241790 / 1.841788 (-0.599997) | 16.047257 / 8.074308 (7.972949) | 15.774146 / 10.191392 (5.582754) | 0.145871 / 0.680424 (-0.534553) | 0.018106 / 0.534201 (-0.516095) | 0.433642 / 0.579283 (-0.145641) | 0.425311 / 0.434364 (-0.009053) | 0.533963 / 0.540337 (-0.006375) | 0.638786 / 1.386936 (-0.748151) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007242 / 0.011353 (-0.004111) | 0.005599 / 0.011008 (-0.005410) | 0.073443 / 0.038508 (0.034935) | 0.033764 / 0.023109 (0.010655) | 0.365990 / 0.275898 (0.090092) | 0.392943 / 0.323480 (0.069463) | 0.005987 / 0.007986 (-0.001999) | 0.004312 / 0.004328 (-0.000016) | 0.072831 / 0.004250 (0.068580) | 0.048854 / 0.037052 (0.011802) | 0.362477 / 0.258489 (0.103988) | 0.399993 / 0.293841 (0.106152) | 0.035602 / 0.128546 (-0.092944) | 0.012445 / 0.075646 (-0.063202) | 0.085768 / 0.419271 (-0.333504) | 0.048544 / 0.043533 (0.005011) | 0.362246 / 0.255139 (0.107107) | 0.388753 / 0.283200 (0.105554) | 0.109829 / 0.141683 (-0.031854) | 1.546881 / 1.452155 (0.094726) | 1.619454 / 1.492716 (0.126737) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.189926 / 0.018006 (0.171920) | 0.447936 / 0.000490 (0.447446) | 0.002354 / 0.000200 (0.002155) | 0.000090 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031740 / 0.037411 (-0.005671) | 0.122595 / 0.014526 (0.108069) | 0.128389 / 0.176557 (-0.048168) | 0.180570 / 0.737135 (-0.556566) | 0.132939 / 0.296338 (-0.163399) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.425073 / 0.215209 (0.209863) | 4.238964 / 2.077655 (2.161309) | 2.095116 / 1.504120 (0.590996) | 1.913925 / 1.541195 (0.372730) | 2.024669 / 1.468490 (0.556179) | 0.699172 / 4.584777 (-3.885605) | 3.845807 / 3.745712 (0.100094) | 2.167502 / 5.269862 (-3.102360) | 1.375267 / 4.565676 (-3.190410) | 0.086739 / 0.424275 (-0.337536) | 0.012198 / 0.007607 (0.004591) | 0.525975 / 0.226044 (0.299931) | 5.249449 / 2.268929 (2.980521) | 2.550565 / 55.444624 (-52.894060) | 2.257557 / 6.876477 (-4.618920) | 2.298936 / 2.142072 (0.156863) | 0.850295 / 4.805227 (-3.954932) | 0.170506 / 6.500664 (-6.330158) | 0.065659 / 0.075469 (-0.009810) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.330556 / 1.841788 (-0.511231) | 16.920203 / 8.074308 (8.845894) | 15.966739 / 10.191392 (5.775347) | 0.164000 / 0.680424 (-0.516424) | 0.018211 / 0.534201 (-0.515990) | 0.436253 / 0.579283 (-0.143030) | 0.449666 / 0.434364 (0.015302) | 0.522287 / 0.540337 (-0.018050) | 0.615944 / 1.386936 (-0.770992) |

</details>
</details>

![](https://cml.dev/watermark.png#824f96c11a02b3817d6b1bf4dfed0abab27777f0 "CML watermark")

2023-05-24T15:50:03Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1561431575
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5dEY4X
https://api.github.com/repos/huggingface/datasets/issues/comments/1561431575/reactions
2023-05-24T15:50:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1561431575
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007273 / 0.011353 (-0.004080) | 0.005198 / 0.011008 (-0.005810) | 0.114362 / 0.038508 (0.075854) | 0.031113 / 0.023109 (0.008003) | 0.378568 / 0.275898 (0.102670) | 0.441695 / 0.323480 (0.118215) | 0.006037 / 0.007986 (-0.001949) | 0.005102 / 0.004328 (0.000774) | 0.098682 / 0.004250 (0.094432) | 0.042797 / 0.037052 (0.005745) | 0.360028 / 0.258489 (0.101539) | 0.435757 / 0.293841 (0.141916) | 0.041438 / 0.128546 (-0.087109) | 0.013728 / 0.075646 (-0.061918) | 0.376154 / 0.419271 (-0.043117) | 0.075324 / 0.043533 (0.031791) | 0.357221 / 0.255139 (0.102082) | 0.416378 / 0.283200 (0.133178) | 0.110707 / 0.141683 (-0.030975) | 1.603215 / 1.452155 (0.151061) | 1.736843 / 1.492716 (0.244127) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.249479 / 0.018006 (0.231473) | 0.513205 / 0.000490 (0.512715) | 0.003856 / 0.000200 (0.003656) | 0.000100 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027750 / 0.037411 (-0.009661) | 0.105437 / 0.014526 (0.090911) | 0.115903 / 0.176557 (-0.060653) | 0.179662 / 0.737135 (-0.557474) | 0.116305 / 0.296338 (-0.180033) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.551681 / 0.215209 (0.336472) | 5.544590 / 2.077655 (3.466935) | 2.193933 / 1.504120 (0.689813) | 1.898395 / 1.541195 (0.357201) | 1.877288 / 1.468490 (0.408798) | 0.858097 / 4.584777 (-3.726680) | 4.920982 / 3.745712 (1.175270) | 2.478220 / 5.269862 (-2.791641) | 1.779608 / 4.565676 (-2.786069) | 0.101321 / 0.424275 (-0.322954) | 0.012627 / 0.007607 (0.005020) | 0.674865 / 0.226044 (0.448820) | 6.808224 / 2.268929 (4.539295) | 2.822466 / 55.444624 (-52.622159) | 2.170379 / 6.876477 (-4.706098) | 2.224278 / 2.142072 (0.082205) | 1.032763 / 4.805227 (-3.772464) | 0.198851 / 6.500664 (-6.301813) | 0.069249 / 0.075469 (-0.006220) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.425987 / 1.841788 (-0.415801) | 16.212942 / 8.074308 (8.138634) | 18.945770 / 10.191392 (8.754378) | 0.192901 / 0.680424 (-0.487522) | 0.025343 / 0.534201 (-0.508858) | 0.465441 / 0.579283 (-0.113842) | 0.540966 / 0.434364 (0.106602) | 0.576736 / 0.540337 (0.036399) | 0.675717 / 1.386936 (-0.711219) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007426 / 0.011353 (-0.003927) | 0.005023 / 0.011008 (-0.005985) | 0.085083 / 0.038508 (0.046575) | 0.030559 / 0.023109 (0.007449) | 0.398461 / 0.275898 (0.122563) | 0.418998 / 0.323480 (0.095518) | 0.006697 / 0.007986 (-0.001288) | 0.004665 / 0.004328 (0.000337) | 0.087724 / 0.004250 (0.083473) | 0.045799 / 0.037052 (0.008747) | 0.395165 / 0.258489 (0.136676) | 0.430172 / 0.293841 (0.136331) | 0.040486 / 0.128546 (-0.088060) | 0.014237 / 0.075646 (-0.061409) | 0.099429 / 0.419271 (-0.319843) | 0.056006 / 0.043533 (0.012473) | 0.389046 / 0.255139 (0.133907) | 0.419559 / 0.283200 (0.136359) | 0.108550 / 0.141683 (-0.033132) | 1.614052 / 1.452155 (0.161897) | 1.677785 / 1.492716 (0.185069) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.202178 / 0.018006 (0.184172) | 0.486365 / 0.000490 (0.485875) | 0.003844 / 0.000200 (0.003644) | 0.000112 / 0.000054 (0.000058) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027963 / 0.037411 (-0.009449) | 0.110399 / 0.014526 (0.095873) | 0.122266 / 0.176557 (-0.054291) | 0.178551 / 0.737135 (-0.558585) | 0.129259 / 0.296338 (-0.167080) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.604178 / 0.215209 (0.388969) | 6.135943 / 2.077655 (4.058288) | 2.547576 / 1.504120 (1.043456) | 2.262470 / 1.541195 (0.721276) | 2.275402 / 1.468490 (0.806912) | 0.878804 / 4.584777 (-3.705972) | 5.152200 / 3.745712 (1.406488) | 2.553715 / 5.269862 (-2.716147) | 1.580959 / 4.565676 (-2.984717) | 0.107895 / 0.424275 (-0.316380) | 0.012751 / 0.007607 (0.005143) | 0.770678 / 0.226044 (0.544633) | 7.744303 / 2.268929 (5.475374) | 3.342037 / 55.444624 (-52.102588) | 2.756848 / 6.876477 (-4.119629) | 2.739357 / 2.142072 (0.597285) | 1.086330 / 4.805227 (-3.718897) | 0.230983 / 6.500664 (-6.269681) | 0.073771 / 0.075469 (-0.001698) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.493441 / 1.841788 (-0.348347) | 16.621611 / 8.074308 (8.547303) | 19.081000 / 10.191392 (8.889608) | 0.215623 / 0.680424 (-0.464801) | 0.025660 / 0.534201 (-0.508541) | 0.446490 / 0.579283 (-0.132793) | 0.560078 / 0.434364 (0.125714) | 0.527231 / 0.540337 (-0.013106) | 0.636551 / 1.386936 (-0.750385) |

</details>
</details>

![](https://cml.dev/watermark.png#b899ea45c0a7e724ceb5f43c3a8b9fdb081fa67a "CML watermark")

2023-05-24T16:04:43Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1561473287
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5dEjEH
https://api.github.com/repos/huggingface/datasets/issues/comments/1561473287/reactions
2023-05-24T16:04:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1561473287
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008266 / 0.011353 (-0.003087) | 0.005082 / 0.011008 (-0.005927) | 0.119858 / 0.038508 (0.081350) | 0.032907 / 0.023109 (0.009798) | 0.362816 / 0.275898 (0.086918) | 0.403684 / 0.323480 (0.080204) | 0.006296 / 0.007986 (-0.001690) | 0.006220 / 0.004328 (0.001891) | 0.095609 / 0.004250 (0.091359) | 0.048734 / 0.037052 (0.011682) | 0.385724 / 0.258489 (0.127235) | 0.424315 / 0.293841 (0.130475) | 0.042344 / 0.128546 (-0.086202) | 0.016147 / 0.075646 (-0.059500) | 0.409661 / 0.419271 (-0.009610) | 0.057900 / 0.043533 (0.014367) | 0.387013 / 0.255139 (0.131874) | 0.388901 / 0.283200 (0.105702) | 0.103920 / 0.141683 (-0.037762) | 1.732730 / 1.452155 (0.280575) | 1.863912 / 1.492716 (0.371196) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237406 / 0.018006 (0.219400) | 0.514398 / 0.000490 (0.513909) | 0.005941 / 0.000200 (0.005741) | 0.000109 / 0.000054 (0.000054) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027524 / 0.037411 (-0.009888) | 0.116498 / 0.014526 (0.101972) | 0.129034 / 0.176557 (-0.047522) | 0.218272 / 0.737135 (-0.518864) | 0.148389 / 0.296338 (-0.147950) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.604555 / 0.215209 (0.389346) | 5.921576 / 2.077655 (3.843921) | 2.410483 / 1.504120 (0.906363) | 2.220286 / 1.541195 (0.679092) | 2.138880 / 1.468490 (0.670390) | 0.934962 / 4.584777 (-3.649815) | 5.808855 / 3.745712 (2.063143) | 4.881554 / 5.269862 (-0.388308) | 2.536408 / 4.565676 (-2.029268) | 0.124260 / 0.424275 (-0.300015) | 0.017798 / 0.007607 (0.010190) | 0.778991 / 0.226044 (0.552947) | 7.899262 / 2.268929 (5.630333) | 3.208667 / 55.444624 (-52.235957) | 2.631182 / 6.876477 (-4.245295) | 2.676199 / 2.142072 (0.534127) | 1.165516 / 4.805227 (-3.639711) | 0.228751 / 6.500664 (-6.271913) | 0.081378 / 0.075469 (0.005909) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.522156 / 1.841788 (-0.319632) | 17.975381 / 8.074308 (9.901073) | 18.918882 / 10.191392 (8.727490) | 0.223984 / 0.680424 (-0.456440) | 0.025171 / 0.534201 (-0.509030) | 0.467894 / 0.579283 (-0.111389) | 0.559501 / 0.434364 (0.125137) | 0.550392 / 0.540337 (0.010055) | 0.696923 / 1.386936 (-0.690013) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008577 / 0.011353 (-0.002775) | 0.006735 / 0.011008 (-0.004273) | 0.095108 / 0.038508 (0.056600) | 0.035059 / 0.023109 (0.011950) | 0.448576 / 0.275898 (0.172677) | 0.492049 / 0.323480 (0.168569) | 0.006600 / 0.007986 (-0.001385) | 0.004760 / 0.004328 (0.000431) | 0.094670 / 0.004250 (0.090419) | 0.052543 / 0.037052 (0.015491) | 0.458927 / 0.258489 (0.200438) | 0.511522 / 0.293841 (0.217681) | 0.046046 / 0.128546 (-0.082500) | 0.015227 / 0.075646 (-0.060419) | 0.114585 / 0.419271 (-0.304686) | 0.057569 / 0.043533 (0.014036) | 0.441989 / 0.255139 (0.186850) | 0.487001 / 0.283200 (0.203801) | 0.115688 / 0.141683 (-0.025995) | 1.777366 / 1.452155 (0.325211) | 1.906216 / 1.492716 (0.413499) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224880 / 0.018006 (0.206874) | 0.504153 / 0.000490 (0.503664) | 0.001143 / 0.000200 (0.000943) | 0.000111 / 0.000054 (0.000056) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033618 / 0.037411 (-0.003793) | 0.127396 / 0.014526 (0.112870) | 0.135648 / 0.176557 (-0.040909) | 0.193140 / 0.737135 (-0.543995) | 0.142129 / 0.296338 (-0.154209) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.692845 / 0.215209 (0.477636) | 6.804897 / 2.077655 (4.727242) | 2.851041 / 1.504120 (1.346921) | 2.480698 / 1.541195 (0.939504) | 2.488619 / 1.468490 (1.020129) | 0.970439 / 4.584777 (-3.614338) | 5.466059 / 3.745712 (1.720347) | 2.790261 / 5.269862 (-2.479601) | 1.727638 / 4.565676 (-2.838039) | 0.116345 / 0.424275 (-0.307930) | 0.014348 / 0.007607 (0.006740) | 0.845510 / 0.226044 (0.619465) | 8.397198 / 2.268929 (6.128270) | 3.591998 / 55.444624 (-51.852626) | 2.858339 / 6.876477 (-4.018137) | 2.905075 / 2.142072 (0.763003) | 1.193569 / 4.805227 (-3.611658) | 0.243091 / 6.500664 (-6.257573) | 0.082198 / 0.075469 (0.006729) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.610327 / 1.841788 (-0.231461) | 17.191414 / 8.074308 (9.117106) | 20.176518 / 10.191392 (9.985126) | 0.246574 / 0.680424 (-0.433850) | 0.024343 / 0.534201 (-0.509858) | 0.482091 / 0.579283 (-0.097192) | 0.585241 / 0.434364 (0.150877) | 0.558833 / 0.540337 (0.018496) | 0.654811 / 1.386936 (-0.732125) |

</details>
</details>

![](https://cml.dev/watermark.png#81761dbfa738354a9c50309313dfe90bea26d872 "CML watermark")

2023-06-07T14:52:53Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1580990634
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5ePACq
https://api.github.com/repos/huggingface/datasets/issues/comments/1580990634/reactions
2023-06-07T14:52:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1580990634
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006353 / 0.011353 (-0.004999) | 0.004393 / 0.011008 (-0.006616) | 0.098751 / 0.038508 (0.060242) | 0.029090 / 0.023109 (0.005981) | 0.304169 / 0.275898 (0.028271) | 0.339879 / 0.323480 (0.016399) | 0.005577 / 0.007986 (-0.002408) | 0.003516 / 0.004328 (-0.000813) | 0.077347 / 0.004250 (0.073097) | 0.041935 / 0.037052 (0.004882) | 0.305865 / 0.258489 (0.047376) | 0.357063 / 0.293841 (0.063222) | 0.025245 / 0.128546 (-0.103301) | 0.008753 / 0.075646 (-0.066893) | 0.316734 / 0.419271 (-0.102538) | 0.043464 / 0.043533 (-0.000069) | 0.300944 / 0.255139 (0.045805) | 0.330091 / 0.283200 (0.046891) | 0.088593 / 0.141683 (-0.053090) | 1.588958 / 1.452155 (0.136803) | 1.641376 / 1.492716 (0.148660) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220290 / 0.018006 (0.202284) | 0.445430 / 0.000490 (0.444940) | 0.004800 / 0.000200 (0.004600) | 0.000075 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023828 / 0.037411 (-0.013583) | 0.103446 / 0.014526 (0.088920) | 0.110668 / 0.176557 (-0.065889) | 0.169604 / 0.737135 (-0.567531) | 0.114818 / 0.296338 (-0.181520) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416951 / 0.215209 (0.201742) | 4.138917 / 2.077655 (2.061263) | 1.891265 / 1.504120 (0.387145) | 1.687068 / 1.541195 (0.145873) | 1.726618 / 1.468490 (0.258128) | 0.546977 / 4.584777 (-4.037800) | 3.536153 / 3.745712 (-0.209560) | 1.795206 / 5.269862 (-3.474656) | 1.019845 / 4.565676 (-3.545831) | 0.067040 / 0.424275 (-0.357235) | 0.012038 / 0.007607 (0.004431) | 0.520583 / 0.226044 (0.294539) | 5.211520 / 2.268929 (2.942591) | 2.336136 / 55.444624 (-53.108488) | 2.011262 / 6.876477 (-4.865215) | 2.137311 / 2.142072 (-0.004762) | 0.654779 / 4.805227 (-4.150448) | 0.134555 / 6.500664 (-6.366109) | 0.066427 / 0.075469 (-0.009042) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.240187 / 1.841788 (-0.601600) | 14.104063 / 8.074308 (6.029755) | 13.369572 / 10.191392 (3.178180) | 0.147891 / 0.680424 (-0.532533) | 0.016993 / 0.534201 (-0.517208) | 0.364863 / 0.579283 (-0.214420) | 0.398684 / 0.434364 (-0.035680) | 0.430524 / 0.540337 (-0.109813) | 0.520920 / 1.386936 (-0.866016) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006845 / 0.011353 (-0.004508) | 0.004420 / 0.011008 (-0.006588) | 0.078334 / 0.038508 (0.039825) | 0.030566 / 0.023109 (0.007457) | 0.409568 / 0.275898 (0.133670) | 0.458389 / 0.323480 (0.134910) | 0.005739 / 0.007986 (-0.002247) | 0.005222 / 0.004328 (0.000893) | 0.076066 / 0.004250 (0.071816) | 0.049239 / 0.037052 (0.012187) | 0.409841 / 0.258489 (0.151352) | 0.472250 / 0.293841 (0.178409) | 0.025463 / 0.128546 (-0.103084) | 0.008738 / 0.075646 (-0.066909) | 0.083114 / 0.419271 (-0.336157) | 0.041233 / 0.043533 (-0.002300) | 0.407158 / 0.255139 (0.152019) | 0.438724 / 0.283200 (0.155524) | 0.097974 / 0.141683 (-0.043709) | 1.536514 / 1.452155 (0.084360) | 1.636704 / 1.492716 (0.143987) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.240589 / 0.018006 (0.222583) | 0.440328 / 0.000490 (0.439838) | 0.000937 / 0.000200 (0.000737) | 0.000076 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027559 / 0.037411 (-0.009853) | 0.109930 / 0.014526 (0.095405) | 0.113366 / 0.176557 (-0.063190) | 0.166849 / 0.737135 (-0.570286) | 0.118872 / 0.296338 (-0.177467) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.474120 / 0.215209 (0.258911) | 4.739222 / 2.077655 (2.661567) | 2.484386 / 1.504120 (0.980266) | 2.281937 / 1.541195 (0.740742) | 2.362974 / 1.468490 (0.894484) | 0.549897 / 4.584777 (-4.034879) | 3.425540 / 3.745712 (-0.320172) | 1.765810 / 5.269862 (-3.504051) | 1.008277 / 4.565676 (-3.557400) | 0.067288 / 0.424275 (-0.356987) | 0.011954 / 0.007607 (0.004347) | 0.577216 / 0.226044 (0.351172) | 5.790659 / 2.268929 (3.521731) | 2.946732 / 55.444624 (-52.497892) | 2.608835 / 6.876477 (-4.267641) | 2.642987 / 2.142072 (0.500915) | 0.652798 / 4.805227 (-4.152429) | 0.135909 / 6.500664 (-6.364755) | 0.068480 / 0.075469 (-0.006989) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.353550 / 1.841788 (-0.488237) | 14.732084 / 8.074308 (6.657775) | 14.439174 / 10.191392 (4.247782) | 0.131445 / 0.680424 (-0.548979) | 0.016608 / 0.534201 (-0.517593) | 0.368103 / 0.579283 (-0.211180) | 0.393918 / 0.434364 (-0.040446) | 0.423562 / 0.540337 (-0.116776) | 0.515041 / 1.386936 (-0.871895) |

</details>
</details>

![](https://cml.dev/watermark.png#8907bdb23f78545303eb3bb0561e33ec6787f96c "CML watermark")

2023-06-07T15:05:33Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1581016437
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5ePGV1
https://api.github.com/repos/huggingface/datasets/issues/comments/1581016437/reactions
2023-06-07T15:05:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1581016437
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006414 / 0.011353 (-0.004938) | 0.004704 / 0.011008 (-0.006305) | 0.096012 / 0.038508 (0.057504) | 0.032910 / 0.023109 (0.009800) | 0.290676 / 0.275898 (0.014778) | 0.319646 / 0.323480 (-0.003834) | 0.005806 / 0.007986 (-0.002180) | 0.004008 / 0.004328 (-0.000320) | 0.073982 / 0.004250 (0.069731) | 0.048985 / 0.037052 (0.011933) | 0.299498 / 0.258489 (0.041009) | 0.338118 / 0.293841 (0.044277) | 0.027680 / 0.128546 (-0.100866) | 0.009051 / 0.075646 (-0.066595) | 0.325051 / 0.419271 (-0.094221) | 0.051011 / 0.043533 (0.007478) | 0.292249 / 0.255139 (0.037110) | 0.315733 / 0.283200 (0.032533) | 0.100327 / 0.141683 (-0.041356) | 1.481862 / 1.452155 (0.029707) | 1.544884 / 1.492716 (0.052168) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.289610 / 0.018006 (0.271603) | 0.510164 / 0.000490 (0.509675) | 0.004726 / 0.000200 (0.004526) | 0.000090 / 0.000054 (0.000036) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027617 / 0.037411 (-0.009794) | 0.107593 / 0.014526 (0.093068) | 0.122783 / 0.176557 (-0.053774) | 0.181086 / 0.737135 (-0.556049) | 0.128030 / 0.296338 (-0.168308) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.403571 / 0.215209 (0.188362) | 4.002881 / 2.077655 (1.925227) | 1.805550 / 1.504120 (0.301430) | 1.619165 / 1.541195 (0.077971) | 1.606536 / 1.468490 (0.138046) | 0.518917 / 4.584777 (-4.065860) | 3.731498 / 3.745712 (-0.014214) | 3.206645 / 5.269862 (-2.063217) | 1.641615 / 4.565676 (-2.924062) | 0.065100 / 0.424275 (-0.359175) | 0.011396 / 0.007607 (0.003789) | 0.500597 / 0.226044 (0.274553) | 4.992293 / 2.268929 (2.723364) | 2.278726 / 55.444624 (-53.165898) | 1.960823 / 6.876477 (-4.915654) | 2.038684 / 2.142072 (-0.103388) | 0.640910 / 4.805227 (-4.164318) | 0.140597 / 6.500664 (-6.360067) | 0.062114 / 0.075469 (-0.013355) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.167366 / 1.841788 (-0.674422) | 14.748193 / 8.074308 (6.673884) | 13.592381 / 10.191392 (3.400989) | 0.165341 / 0.680424 (-0.515083) | 0.017360 / 0.534201 (-0.516841) | 0.393448 / 0.579283 (-0.185836) | 0.422951 / 0.434364 (-0.011413) | 0.460491 / 0.540337 (-0.079847) | 0.558238 / 1.386936 (-0.828698) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006373 / 0.011353 (-0.004980) | 0.004587 / 0.011008 (-0.006421) | 0.076421 / 0.038508 (0.037913) | 0.032162 / 0.023109 (0.009052) | 0.385531 / 0.275898 (0.109633) | 0.410424 / 0.323480 (0.086944) | 0.006154 / 0.007986 (-0.001832) | 0.005533 / 0.004328 (0.001205) | 0.077035 / 0.004250 (0.072784) | 0.051571 / 0.037052 (0.014519) | 0.393283 / 0.258489 (0.134794) | 0.433756 / 0.293841 (0.139915) | 0.028381 / 0.128546 (-0.100165) | 0.009034 / 0.075646 (-0.066613) | 0.083836 / 0.419271 (-0.335435) | 0.048246 / 0.043533 (0.004713) | 0.385437 / 0.255139 (0.130298) | 0.394187 / 0.283200 (0.110987) | 0.105453 / 0.141683 (-0.036230) | 1.459173 / 1.452155 (0.007018) | 1.575083 / 1.492716 (0.082367) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.320324 / 0.018006 (0.302318) | 0.502945 / 0.000490 (0.502455) | 0.004470 / 0.000200 (0.004270) | 0.000107 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028118 / 0.037411 (-0.009293) | 0.111430 / 0.014526 (0.096904) | 0.123141 / 0.176557 (-0.053415) | 0.175215 / 0.737135 (-0.561920) | 0.126429 / 0.296338 (-0.169909) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.433407 / 0.215209 (0.218198) | 4.329945 / 2.077655 (2.252291) | 2.096822 / 1.504120 (0.592702) | 1.908173 / 1.541195 (0.366978) | 1.967167 / 1.468490 (0.498676) | 0.529207 / 4.584777 (-4.055570) | 3.798424 / 3.745712 (0.052712) | 3.050716 / 5.269862 (-2.219146) | 1.445009 / 4.565676 (-3.120668) | 0.066467 / 0.424275 (-0.357809) | 0.011698 / 0.007607 (0.004090) | 0.528660 / 0.226044 (0.302615) | 5.282069 / 2.268929 (3.013141) | 2.535501 / 55.444624 (-52.909124) | 2.202856 / 6.876477 (-4.673621) | 2.293225 / 2.142072 (0.151153) | 0.640216 / 4.805227 (-4.165011) | 0.140884 / 6.500664 (-6.359780) | 0.064231 / 0.075469 (-0.011238) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.292129 / 1.841788 (-0.549659) | 15.371370 / 8.074308 (7.297062) | 15.114854 / 10.191392 (4.923462) | 0.176870 / 0.680424 (-0.503554) | 0.017380 / 0.534201 (-0.516821) | 0.398156 / 0.579283 (-0.181127) | 0.442277 / 0.434364 (0.007913) | 0.467093 / 0.540337 (-0.073244) | 0.561599 / 1.386936 (-0.825337) |

</details>
</details>

![](https://cml.dev/watermark.png#323747a5ff7d9b204ea3c4989d658af7102f7bbd "CML watermark")

2023-06-07T15:18:26Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1581043682
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5ePM_i
https://api.github.com/repos/huggingface/datasets/issues/comments/1581043682/reactions
2023-06-07T15:18:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1581043682
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009360 / 0.011353 (-0.001993) | 0.006297 / 0.011008 (-0.004712) | 0.133131 / 0.038508 (0.094623) | 0.040261 / 0.023109 (0.017152) | 0.419101 / 0.275898 (0.143203) | 0.453087 / 0.323480 (0.129607) | 0.007718 / 0.007986 (-0.000268) | 0.005698 / 0.004328 (0.001369) | 0.102261 / 0.004250 (0.098010) | 0.055147 / 0.037052 (0.018095) | 0.428355 / 0.258489 (0.169866) | 0.505241 / 0.293841 (0.211400) | 0.046745 / 0.128546 (-0.081802) | 0.015559 / 0.075646 (-0.060088) | 0.441775 / 0.419271 (0.022503) | 0.070165 / 0.043533 (0.026632) | 0.421957 / 0.255139 (0.166818) | 0.445156 / 0.283200 (0.161957) | 0.126321 / 0.141683 (-0.015362) | 1.900486 / 1.452155 (0.448331) | 2.088630 / 1.492716 (0.595913) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.260244 / 0.018006 (0.242237) | 0.606317 / 0.000490 (0.605828) | 0.006827 / 0.000200 (0.006627) | 0.000117 / 0.000054 (0.000063) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031958 / 0.037411 (-0.005453) | 0.139362 / 0.014526 (0.124836) | 0.148748 / 0.176557 (-0.027809) | 0.226269 / 0.737135 (-0.510866) | 0.161145 / 0.296338 (-0.135194) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.666287 / 0.215209 (0.451078) | 6.588707 / 2.077655 (4.511053) | 2.736155 / 1.504120 (1.232035) | 2.329601 / 1.541195 (0.788406) | 2.324991 / 1.468490 (0.856501) | 0.943608 / 4.584777 (-3.641169) | 6.051653 / 3.745712 (2.305941) | 2.929150 / 5.269862 (-2.340711) | 1.804461 / 4.565676 (-2.761216) | 0.113302 / 0.424275 (-0.310973) | 0.015245 / 0.007607 (0.007638) | 0.827029 / 0.226044 (0.600984) | 8.211536 / 2.268929 (5.942608) | 3.445231 / 55.444624 (-51.999393) | 2.756728 / 6.876477 (-4.119748) | 2.904039 / 2.142072 (0.761966) | 1.162339 / 4.805227 (-3.642888) | 0.231168 / 6.500664 (-6.269496) | 0.089038 / 0.075469 (0.013569) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.640619 / 1.841788 (-0.201169) | 20.034157 / 8.074308 (11.959849) | 22.346006 / 10.191392 (12.154614) | 0.255300 / 0.680424 (-0.425124) | 0.031452 / 0.534201 (-0.502749) | 0.563290 / 0.579283 (-0.015993) | 0.653556 / 0.434364 (0.219192) | 0.687663 / 0.540337 (0.147326) | 0.816432 / 1.386936 (-0.570504) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010340 / 0.011353 (-0.001013) | 0.006245 / 0.011008 (-0.004764) | 0.128012 / 0.038508 (0.089504) | 0.041799 / 0.023109 (0.018690) | 0.533340 / 0.275898 (0.257442) | 0.592243 / 0.323480 (0.268763) | 0.009256 / 0.007986 (0.001271) | 0.005310 / 0.004328 (0.000982) | 0.110973 / 0.004250 (0.106722) | 0.065465 / 0.037052 (0.028412) | 0.533845 / 0.258489 (0.275356) | 0.602190 / 0.293841 (0.308349) | 0.060245 / 0.128546 (-0.068301) | 0.016954 / 0.075646 (-0.058693) | 0.119727 / 0.419271 (-0.299545) | 0.064628 / 0.043533 (0.021095) | 0.558229 / 0.255139 (0.303090) | 0.563696 / 0.283200 (0.280496) | 0.137225 / 0.141683 (-0.004458) | 2.038605 / 1.452155 (0.586451) | 2.158655 / 1.492716 (0.665939) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.327067 / 0.018006 (0.309061) | 0.628812 / 0.000490 (0.628323) | 0.010259 / 0.000200 (0.010059) | 0.000123 / 0.000054 (0.000069) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037023 / 0.037411 (-0.000388) | 0.142462 / 0.014526 (0.127936) | 0.158165 / 0.176557 (-0.018392) | 0.220808 / 0.737135 (-0.516328) | 0.163608 / 0.296338 (-0.132731) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.776119 / 0.215209 (0.560910) | 7.813044 / 2.077655 (5.735389) | 3.610901 / 1.504120 (2.106781) | 3.195144 / 1.541195 (1.653950) | 3.218245 / 1.468490 (1.749755) | 1.092732 / 4.584777 (-3.492045) | 5.965526 / 3.745712 (2.219813) | 2.914683 / 5.269862 (-2.355179) | 1.848397 / 4.565676 (-2.717280) | 0.114436 / 0.424275 (-0.309839) | 0.014794 / 0.007607 (0.007187) | 0.887141 / 0.226044 (0.661096) | 9.009743 / 2.268929 (6.740815) | 4.180143 / 55.444624 (-51.264481) | 3.452194 / 6.876477 (-3.424283) | 3.493520 / 2.142072 (1.351448) | 1.233327 / 4.805227 (-3.571900) | 0.235390 / 6.500664 (-6.265274) | 0.099544 / 0.075469 (0.024075) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.853482 / 1.841788 (0.011694) | 20.071177 / 8.074308 (11.996869) | 24.507618 / 10.191392 (14.316226) | 0.260164 / 0.680424 (-0.420260) | 0.028433 / 0.534201 (-0.505768) | 0.549181 / 0.579283 (-0.030102) | 0.650069 / 0.434364 (0.215705) | 0.629541 / 0.540337 (0.089203) | 0.808932 / 1.386936 (-0.578004) |

</details>
</details>

![](https://cml.dev/watermark.png#f39ba76af62c8037de3f464e87cbb095f8729062 "CML watermark")

2023-06-07T15:18:38Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1581044016
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5ePNEw
https://api.github.com/repos/huggingface/datasets/issues/comments/1581044016/reactions
2023-06-07T15:18:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1581044016
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009537 / 0.011353 (-0.001816) | 0.006036 / 0.011008 (-0.004972) | 0.141210 / 0.038508 (0.102701) | 0.037493 / 0.023109 (0.014384) | 0.404285 / 0.275898 (0.128386) | 0.458906 / 0.323480 (0.135427) | 0.007224 / 0.007986 (-0.000761) | 0.005148 / 0.004328 (0.000819) | 0.103889 / 0.004250 (0.099639) | 0.048877 / 0.037052 (0.011824) | 0.413220 / 0.258489 (0.154731) | 0.458153 / 0.293841 (0.164312) | 0.046008 / 0.128546 (-0.082538) | 0.015116 / 0.075646 (-0.060531) | 0.439836 / 0.419271 (0.020565) | 0.067527 / 0.043533 (0.023994) | 0.435794 / 0.255139 (0.180656) | 0.451687 / 0.283200 (0.168487) | 0.121274 / 0.141683 (-0.020409) | 1.950199 / 1.452155 (0.498044) | 2.035589 / 1.492716 (0.542873) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.247056 / 0.018006 (0.229050) | 0.550348 / 0.000490 (0.549858) | 0.005504 / 0.000200 (0.005305) | 0.000116 / 0.000054 (0.000061) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032171 / 0.037411 (-0.005240) | 0.135983 / 0.014526 (0.121457) | 0.149587 / 0.176557 (-0.026970) | 0.233414 / 0.737135 (-0.503722) | 0.152598 / 0.296338 (-0.143740) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.634813 / 0.215209 (0.419604) | 6.453619 / 2.077655 (4.375964) | 2.582070 / 1.504120 (1.077951) | 2.214292 / 1.541195 (0.673097) | 2.220012 / 1.468490 (0.751522) | 0.987374 / 4.584777 (-3.597403) | 5.543760 / 3.745712 (1.798047) | 2.808865 / 5.269862 (-2.460996) | 1.714713 / 4.565676 (-2.850963) | 0.111016 / 0.424275 (-0.313259) | 0.014688 / 0.007607 (0.007081) | 0.842542 / 0.226044 (0.616498) | 8.414336 / 2.268929 (6.145407) | 3.501021 / 55.444624 (-51.943604) | 2.665335 / 6.876477 (-4.211142) | 2.843706 / 2.142072 (0.701633) | 1.196398 / 4.805227 (-3.608829) | 0.245508 / 6.500664 (-6.255156) | 0.086970 / 0.075469 (0.011501) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.590244 / 1.841788 (-0.251544) | 18.694141 / 8.074308 (10.619833) | 21.752463 / 10.191392 (11.561071) | 0.264511 / 0.680424 (-0.415913) | 0.028713 / 0.534201 (-0.505488) | 0.531102 / 0.579283 (-0.048181) | 0.626302 / 0.434364 (0.191938) | 0.624541 / 0.540337 (0.084203) | 0.745745 / 1.386936 (-0.641191) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010097 / 0.011353 (-0.001256) | 0.005558 / 0.011008 (-0.005451) | 0.111326 / 0.038508 (0.072818) | 0.036465 / 0.023109 (0.013356) | 0.472116 / 0.275898 (0.196218) | 0.524479 / 0.323480 (0.200999) | 0.007466 / 0.007986 (-0.000520) | 0.005440 / 0.004328 (0.001112) | 0.103482 / 0.004250 (0.099231) | 0.053217 / 0.037052 (0.016165) | 0.476685 / 0.258489 (0.218196) | 0.554011 / 0.293841 (0.260170) | 0.047157 / 0.128546 (-0.081390) | 0.015895 / 0.075646 (-0.059751) | 0.115997 / 0.419271 (-0.303274) | 0.062290 / 0.043533 (0.018758) | 0.474166 / 0.255139 (0.219027) | 0.498854 / 0.283200 (0.215655) | 0.121798 / 0.141683 (-0.019885) | 1.956583 / 1.452155 (0.504428) | 2.069620 / 1.492716 (0.576904) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.278637 / 0.018006 (0.260631) | 0.555295 / 0.000490 (0.554805) | 0.007401 / 0.000200 (0.007201) | 0.000121 / 0.000054 (0.000066) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033576 / 0.037411 (-0.003835) | 0.136479 / 0.014526 (0.121954) | 0.153960 / 0.176557 (-0.022597) | 0.203422 / 0.737135 (-0.533713) | 0.154159 / 0.296338 (-0.142180) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.672561 / 0.215209 (0.457352) | 6.956675 / 2.077655 (4.879020) | 3.063636 / 1.504120 (1.559516) | 2.668256 / 1.541195 (1.127061) | 2.794793 / 1.468490 (1.326303) | 0.964242 / 4.584777 (-3.620535) | 5.785992 / 3.745712 (2.040279) | 2.850079 / 5.269862 (-2.419782) | 1.782491 / 4.565676 (-2.783186) | 0.114859 / 0.424275 (-0.309416) | 0.015229 / 0.007607 (0.007622) | 0.858406 / 0.226044 (0.632362) | 8.646296 / 2.268929 (6.377367) | 3.842133 / 55.444624 (-51.602492) | 3.180017 / 6.876477 (-3.696460) | 3.241315 / 2.142072 (1.099243) | 1.248988 / 4.805227 (-3.556239) | 0.235075 / 6.500664 (-6.265589) | 0.087192 / 0.075469 (0.011723) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.783877 / 1.841788 (-0.057910) | 19.477223 / 8.074308 (11.402914) | 22.926734 / 10.191392 (12.735342) | 0.246970 / 0.680424 (-0.433454) | 0.026386 / 0.534201 (-0.507815) | 0.517599 / 0.579283 (-0.061684) | 0.626504 / 0.434364 (0.192140) | 0.606943 / 0.540337 (0.066606) | 0.739115 / 1.386936 (-0.647821) |

</details>
</details>

![](https://cml.dev/watermark.png#e8f051a41454f8625091338e6b53119a5eb9b2a0 "CML watermark")

2023-06-07T15:21:31Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1581049397
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5ePOY1
https://api.github.com/repos/huggingface/datasets/issues/comments/1581049397/reactions
2023-06-07T15:21:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1581049397
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008085 / 0.011353 (-0.003268) | 0.005568 / 0.011008 (-0.005440) | 0.119674 / 0.038508 (0.081166) | 0.040452 / 0.023109 (0.017343) | 0.360288 / 0.275898 (0.084390) | 0.409448 / 0.323480 (0.085968) | 0.007281 / 0.007986 (-0.000705) | 0.004931 / 0.004328 (0.000602) | 0.089956 / 0.004250 (0.085706) | 0.056088 / 0.037052 (0.019036) | 0.384708 / 0.258489 (0.126219) | 0.423506 / 0.293841 (0.129665) | 0.033280 / 0.128546 (-0.095266) | 0.010696 / 0.075646 (-0.064951) | 0.394851 / 0.419271 (-0.024421) | 0.058412 / 0.043533 (0.014879) | 0.361514 / 0.255139 (0.106375) | 0.399121 / 0.283200 (0.115921) | 0.117927 / 0.141683 (-0.023756) | 1.791499 / 1.452155 (0.339344) | 1.889000 / 1.492716 (0.396284) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.253324 / 0.018006 (0.235318) | 0.536151 / 0.000490 (0.535661) | 0.010450 / 0.000200 (0.010250) | 0.000171 / 0.000054 (0.000117) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034646 / 0.037411 (-0.002765) | 0.145999 / 0.014526 (0.131473) | 0.153793 / 0.176557 (-0.022763) | 0.232871 / 0.737135 (-0.504265) | 0.161151 / 0.296338 (-0.135188) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.471407 / 0.215209 (0.256197) | 4.715702 / 2.077655 (2.638047) | 2.228939 / 1.504120 (0.724819) | 2.008511 / 1.541195 (0.467317) | 2.135182 / 1.468490 (0.666692) | 0.620720 / 4.584777 (-3.964057) | 4.960731 / 3.745712 (1.215019) | 2.222469 / 5.269862 (-3.047393) | 1.284467 / 4.565676 (-3.281209) | 0.077931 / 0.424275 (-0.346344) | 0.013935 / 0.007607 (0.006328) | 0.593164 / 0.226044 (0.367120) | 5.940829 / 2.268929 (3.671900) | 2.664277 / 55.444624 (-52.780347) | 2.290655 / 6.876477 (-4.585822) | 2.496664 / 2.142072 (0.354592) | 0.759166 / 4.805227 (-4.046061) | 0.168011 / 6.500664 (-6.332653) | 0.077993 / 0.075469 (0.002524) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.440663 / 1.841788 (-0.401125) | 19.105377 / 8.074308 (11.031069) | 16.068118 / 10.191392 (5.876726) | 0.193024 / 0.680424 (-0.487400) | 0.022348 / 0.534201 (-0.511853) | 0.517454 / 0.579283 (-0.061829) | 0.528072 / 0.434364 (0.093708) | 0.565293 / 0.540337 (0.024955) | 0.676578 / 1.386936 (-0.710358) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008089 / 0.011353 (-0.003264) | 0.005287 / 0.011008 (-0.005721) | 0.087964 / 0.038508 (0.049456) | 0.041548 / 0.023109 (0.018439) | 0.437733 / 0.275898 (0.161835) | 0.487878 / 0.323480 (0.164398) | 0.006898 / 0.007986 (-0.001087) | 0.004649 / 0.004328 (0.000320) | 0.086982 / 0.004250 (0.082732) | 0.056874 / 0.037052 (0.019822) | 0.437397 / 0.258489 (0.178908) | 0.490636 / 0.293841 (0.196795) | 0.033550 / 0.128546 (-0.094997) | 0.010430 / 0.075646 (-0.065216) | 0.096076 / 0.419271 (-0.323196) | 0.054028 / 0.043533 (0.010495) | 0.450262 / 0.255139 (0.195123) | 0.465566 / 0.283200 (0.182366) | 0.119987 / 0.141683 (-0.021696) | 1.764428 / 1.452155 (0.312273) | 1.841547 / 1.492716 (0.348831) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.271427 / 0.018006 (0.253420) | 0.506386 / 0.000490 (0.505896) | 0.001213 / 0.000200 (0.001013) | 0.000125 / 0.000054 (0.000070) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036159 / 0.037411 (-0.001253) | 0.140578 / 0.014526 (0.126053) | 0.147517 / 0.176557 (-0.029040) | 0.206215 / 0.737135 (-0.530921) | 0.152560 / 0.296338 (-0.143779) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.522833 / 0.215209 (0.307624) | 5.215732 / 2.077655 (3.138077) | 2.553406 / 1.504120 (1.049286) | 2.344815 / 1.541195 (0.803620) | 2.422377 / 1.468490 (0.953886) | 0.631197 / 4.584777 (-3.953580) | 4.906216 / 3.745712 (1.160504) | 2.212923 / 5.269862 (-3.056938) | 1.352937 / 4.565676 (-3.212740) | 0.079141 / 0.424275 (-0.345135) | 0.013691 / 0.007607 (0.006084) | 0.634939 / 0.226044 (0.408895) | 6.578770 / 2.268929 (4.309842) | 3.080339 / 55.444624 (-52.364286) | 2.710243 / 6.876477 (-4.166234) | 2.740476 / 2.142072 (0.598404) | 0.783610 / 4.805227 (-4.021617) | 0.171589 / 6.500664 (-6.329075) | 0.077311 / 0.075469 (0.001842) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.584847 / 1.841788 (-0.256941) | 19.510132 / 8.074308 (11.435824) | 18.074572 / 10.191392 (7.883180) | 0.173494 / 0.680424 (-0.506930) | 0.021149 / 0.534201 (-0.513052) | 0.469026 / 0.579283 (-0.110258) | 0.518463 / 0.434364 (0.084099) | 0.550363 / 0.540337 (0.010026) | 0.667087 / 1.386936 (-0.719849) |

</details>
</details>

![](https://cml.dev/watermark.png#5dfcd876c25cc0ffbd6b5b518b017419390a8ada "CML watermark")

2023-06-07T15:22:14Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1581051185
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5ePO0x
https://api.github.com/repos/huggingface/datasets/issues/comments/1581051185/reactions
2023-06-07T15:22:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1581051185
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007144 / 0.011353 (-0.004209) | 0.004783 / 0.011008 (-0.006225) | 0.103991 / 0.038508 (0.065483) | 0.039098 / 0.023109 (0.015989) | 0.319851 / 0.275898 (0.043952) | 0.356104 / 0.323480 (0.032625) | 0.007077 / 0.007986 (-0.000909) | 0.004188 / 0.004328 (-0.000141) | 0.078360 / 0.004250 (0.074109) | 0.050951 / 0.037052 (0.013899) | 0.321791 / 0.258489 (0.063302) | 0.356123 / 0.293841 (0.062283) | 0.028967 / 0.128546 (-0.099579) | 0.009091 / 0.075646 (-0.066555) | 0.355265 / 0.419271 (-0.064007) | 0.052521 / 0.043533 (0.008988) | 0.317333 / 0.255139 (0.062194) | 0.340747 / 0.283200 (0.057547) | 0.104354 / 0.141683 (-0.037329) | 1.522791 / 1.452155 (0.070636) | 1.579835 / 1.492716 (0.087118) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.260539 / 0.018006 (0.242532) | 0.454230 / 0.000490 (0.453740) | 0.036588 / 0.000200 (0.036388) | 0.000289 / 0.000054 (0.000235) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028375 / 0.037411 (-0.009036) | 0.118939 / 0.014526 (0.104413) | 0.126553 / 0.176557 (-0.050004) | 0.184596 / 0.737135 (-0.552539) | 0.130583 / 0.296338 (-0.165755) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417353 / 0.215209 (0.202144) | 4.171595 / 2.077655 (2.093940) | 1.855096 / 1.504120 (0.350976) | 1.673941 / 1.541195 (0.132747) | 1.761370 / 1.468490 (0.292880) | 0.544081 / 4.584777 (-4.040696) | 3.851877 / 3.745712 (0.106165) | 1.896661 / 5.269862 (-3.373200) | 1.093303 / 4.565676 (-3.472373) | 0.067967 / 0.424275 (-0.356308) | 0.012313 / 0.007607 (0.004706) | 0.532316 / 0.226044 (0.306272) | 5.336016 / 2.268929 (3.067087) | 2.344780 / 55.444624 (-53.099845) | 1.993909 / 6.876477 (-4.882568) | 2.167324 / 2.142072 (0.025251) | 0.670334 / 4.805227 (-4.134893) | 0.147705 / 6.500664 (-6.352959) | 0.067634 / 0.075469 (-0.007835) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.251005 / 1.841788 (-0.590783) | 15.405531 / 8.074308 (7.331223) | 14.197019 / 10.191392 (4.005627) | 0.144230 / 0.680424 (-0.536193) | 0.018352 / 0.534201 (-0.515849) | 0.427536 / 0.579283 (-0.151748) | 0.433135 / 0.434364 (-0.001229) | 0.502624 / 0.540337 (-0.037713) | 0.612312 / 1.386936 (-0.774624) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007011 / 0.011353 (-0.004342) | 0.004857 / 0.011008 (-0.006151) | 0.077797 / 0.038508 (0.039289) | 0.035411 / 0.023109 (0.012302) | 0.368234 / 0.275898 (0.092336) | 0.408359 / 0.323480 (0.084879) | 0.005883 / 0.007986 (-0.002102) | 0.004311 / 0.004328 (-0.000017) | 0.077216 / 0.004250 (0.072966) | 0.052062 / 0.037052 (0.015010) | 0.368502 / 0.258489 (0.110013) | 0.428681 / 0.293841 (0.134840) | 0.028889 / 0.128546 (-0.099657) | 0.009146 / 0.075646 (-0.066501) | 0.085515 / 0.419271 (-0.333756) | 0.050216 / 0.043533 (0.006683) | 0.359562 / 0.255139 (0.104423) | 0.378335 / 0.283200 (0.095135) | 0.106351 / 0.141683 (-0.035332) | 1.538943 / 1.452155 (0.086788) | 1.663572 / 1.492716 (0.170855) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216917 / 0.018006 (0.198911) | 0.444130 / 0.000490 (0.443641) | 0.002640 / 0.000200 (0.002440) | 0.000093 / 0.000054 (0.000038) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032509 / 0.037411 (-0.004902) | 0.123955 / 0.014526 (0.109430) | 0.133236 / 0.176557 (-0.043321) | 0.187408 / 0.737135 (-0.549727) | 0.136696 / 0.296338 (-0.159643) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.443714 / 0.215209 (0.228505) | 4.416973 / 2.077655 (2.339318) | 2.145279 / 1.504120 (0.641159) | 1.946669 / 1.541195 (0.405474) | 2.044105 / 1.468490 (0.575614) | 0.534463 / 4.584777 (-4.050314) | 3.824926 / 3.745712 (0.079214) | 3.151796 / 5.269862 (-2.118066) | 1.497513 / 4.565676 (-3.068164) | 0.066799 / 0.424275 (-0.357476) | 0.012408 / 0.007607 (0.004801) | 0.544182 / 0.226044 (0.318138) | 5.419403 / 2.268929 (3.150474) | 2.605191 / 55.444624 (-52.839433) | 2.285354 / 6.876477 (-4.591123) | 2.359520 / 2.142072 (0.217448) | 0.655489 / 4.805227 (-4.149738) | 0.143496 / 6.500664 (-6.357168) | 0.066782 / 0.075469 (-0.008687) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.329370 / 1.841788 (-0.512418) | 16.058019 / 8.074308 (7.983711) | 15.119769 / 10.191392 (4.928377) | 0.147967 / 0.680424 (-0.532457) | 0.018360 / 0.534201 (-0.515841) | 0.436847 / 0.579283 (-0.142436) | 0.435136 / 0.434364 (0.000773) | 0.507176 / 0.540337 (-0.033161) | 0.610627 / 1.386936 (-0.776309) |

</details>
</details>

![](https://cml.dev/watermark.png#b4cc3ee6d8945052283076854eb77575d52b7432 "CML watermark")

2023-06-07T15:25:54Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1581058200
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5ePQiY
https://api.github.com/repos/huggingface/datasets/issues/comments/1581058200/reactions
2023-06-07T15:25:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1581058200
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006425 / 0.011353 (-0.004927) | 0.003710 / 0.011008 (-0.007298) | 0.102072 / 0.038508 (0.063564) | 0.033974 / 0.023109 (0.010865) | 0.273146 / 0.275898 (-0.002752) | 0.313254 / 0.323480 (-0.010226) | 0.004889 / 0.007986 (-0.003096) | 0.004803 / 0.004328 (0.000475) | 0.067359 / 0.004250 (0.063109) | 0.040281 / 0.037052 (0.003228) | 0.302106 / 0.258489 (0.043617) | 0.318039 / 0.293841 (0.024198) | 0.028839 / 0.128546 (-0.099707) | 0.008726 / 0.075646 (-0.066921) | 0.322532 / 0.419271 (-0.096739) | 0.048845 / 0.043533 (0.005312) | 0.299836 / 0.255139 (0.044697) | 0.300983 / 0.283200 (0.017784) | 0.103384 / 0.141683 (-0.038299) | 1.417245 / 1.452155 (-0.034910) | 1.538819 / 1.492716 (0.046102) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219798 / 0.018006 (0.201792) | 0.442297 / 0.000490 (0.441807) | 0.013792 / 0.000200 (0.013592) | 0.000101 / 0.000054 (0.000046) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024996 / 0.037411 (-0.012416) | 0.098558 / 0.014526 (0.084032) | 0.116423 / 0.176557 (-0.060133) | 0.163481 / 0.737135 (-0.573654) | 0.115031 / 0.296338 (-0.181308) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.392411 / 0.215209 (0.177202) | 4.025992 / 2.077655 (1.948337) | 1.850809 / 1.504120 (0.346690) | 1.668330 / 1.541195 (0.127136) | 1.627041 / 1.468490 (0.158551) | 0.510721 / 4.584777 (-4.074055) | 3.841318 / 3.745712 (0.095606) | 3.416979 / 5.269862 (-1.852883) | 1.640796 / 4.565676 (-2.924880) | 0.061968 / 0.424275 (-0.362307) | 0.010281 / 0.007607 (0.002674) | 0.485592 / 0.226044 (0.259548) | 4.872205 / 2.268929 (2.603277) | 2.146753 / 55.444624 (-53.297871) | 1.832087 / 6.876477 (-5.044390) | 1.920928 / 2.142072 (-0.221144) | 0.606363 / 4.805227 (-4.198864) | 0.134351 / 6.500664 (-6.366313) | 0.057583 / 0.075469 (-0.017886) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.153048 / 1.841788 (-0.688739) | 14.165743 / 8.074308 (6.091435) | 12.237798 / 10.191392 (2.046406) | 0.159815 / 0.680424 (-0.520608) | 0.018226 / 0.534201 (-0.515975) | 0.372390 / 0.579283 (-0.206893) | 0.396552 / 0.434364 (-0.037811) | 0.439445 / 0.540337 (-0.100892) | 0.521924 / 1.386936 (-0.865012) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006162 / 0.011353 (-0.005191) | 0.004006 / 0.011008 (-0.007002) | 0.067226 / 0.038508 (0.028718) | 0.030285 / 0.023109 (0.007176) | 0.361220 / 0.275898 (0.085322) | 0.386783 / 0.323480 (0.063303) | 0.005202 / 0.007986 (-0.002784) | 0.003453 / 0.004328 (-0.000876) | 0.068299 / 0.004250 (0.064048) | 0.041433 / 0.037052 (0.004381) | 0.360222 / 0.258489 (0.101733) | 0.399327 / 0.293841 (0.105486) | 0.026066 / 0.128546 (-0.102480) | 0.008025 / 0.075646 (-0.067621) | 0.079588 / 0.419271 (-0.339683) | 0.042616 / 0.043533 (-0.000917) | 0.347639 / 0.255139 (0.092500) | 0.386092 / 0.283200 (0.102893) | 0.100869 / 0.141683 (-0.040814) | 1.386901 / 1.452155 (-0.065254) | 1.471523 / 1.492716 (-0.021193) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.217020 / 0.018006 (0.199014) | 0.431033 / 0.000490 (0.430543) | 0.002902 / 0.000200 (0.002702) | 0.000092 / 0.000054 (0.000037) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027396 / 0.037411 (-0.010015) | 0.114154 / 0.014526 (0.099629) | 0.117918 / 0.176557 (-0.058638) | 0.173342 / 0.737135 (-0.563794) | 0.125812 / 0.296338 (-0.170526) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.424843 / 0.215209 (0.209634) | 4.324828 / 2.077655 (2.247174) | 2.188263 / 1.504120 (0.684143) | 1.912288 / 1.541195 (0.371094) | 2.011621 / 1.468490 (0.543131) | 0.560944 / 4.584777 (-4.023833) | 3.975047 / 3.745712 (0.229335) | 3.130242 / 5.269862 (-2.139619) | 1.667902 / 4.565676 (-2.897775) | 0.062245 / 0.424275 (-0.362030) | 0.011300 / 0.007607 (0.003692) | 0.498571 / 0.226044 (0.272527) | 5.024887 / 2.268929 (2.755958) | 2.482967 / 55.444624 (-52.961657) | 2.216125 / 6.876477 (-4.660352) | 2.175856 / 2.142072 (0.033783) | 0.615207 / 4.805227 (-4.190021) | 0.133808 / 6.500664 (-6.366856) | 0.058681 / 0.075469 (-0.016788) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.370150 / 1.841788 (-0.471637) | 14.580907 / 8.074308 (6.506599) | 14.209955 / 10.191392 (4.018563) | 0.139738 / 0.680424 (-0.540686) | 0.018722 / 0.534201 (-0.515479) | 0.375755 / 0.579283 (-0.203528) | 0.428335 / 0.434364 (-0.006029) | 0.438957 / 0.540337 (-0.101380) | 0.541130 / 1.386936 (-0.845806) |

</details>
</details>

![](https://cml.dev/watermark.png#c14806a42a20f44a60f3663642bae1de199ab1ec "CML watermark")

2023-06-07T16:01:16Z
https://github.com/huggingface/datasets/pull/5863#issuecomment-1581119983
https://api.github.com/repos/huggingface/datasets/issues/5863
IC_kwDODunzps5ePfnv
https://api.github.com/repos/huggingface/datasets/issues/comments/1581119983/reactions
2023-06-07T16:01:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1581119983
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
This PR tries out a new approach to generating the index tensor in `to_tf_dataset`, which should reduce memory usage for very large datasets. I'll need to do some testing before merging it!

Fixes #5855