Bump pyarrow to 8.0.0
NONE
_The documentation is not available anymore as the PR was closed or merged._
2023-03-07T13:35:44Z
https://github.com/huggingface/datasets/pull/5620#issuecomment-1458185284
https://api.github.com/repos/huggingface/datasets/issues/5620
IC_kwDODunzps5W6iRE
https://api.github.com/repos/huggingface/datasets/issues/comments/1458185284/reactions
2023-03-08T13:57:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1458185284
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009873 / 0.011353 (-0.001480) | 0.005180 / 0.011008 (-0.005828) | 0.099587 / 0.038508 (0.061079) | 0.035674 / 0.023109 (0.012565) | 0.299156 / 0.275898 (0.023258) | 0.361253 / 0.323480 (0.037773) | 0.008159 / 0.007986 (0.000173) | 0.004245 / 0.004328 (-0.000084) | 0.076809 / 0.004250 (0.072559) | 0.045251 / 0.037052 (0.008199) | 0.306002 / 0.258489 (0.047513) | 0.345758 / 0.293841 (0.051917) | 0.037826 / 0.128546 (-0.090721) | 0.011887 / 0.075646 (-0.063759) | 0.333804 / 0.419271 (-0.085467) | 0.047859 / 0.043533 (0.004326) | 0.291866 / 0.255139 (0.036727) | 0.319356 / 0.283200 (0.036157) | 0.104241 / 0.141683 (-0.037442) | 1.443816 / 1.452155 (-0.008338) | 1.514654 / 1.492716 (0.021938) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.009846 / 0.018006 (-0.008160) | 0.439488 / 0.000490 (0.438999) | 0.003227 / 0.000200 (0.003028) | 0.000092 / 0.000054 (0.000037) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027553 / 0.037411 (-0.009858) | 0.105337 / 0.014526 (0.090811) | 0.116203 / 0.176557 (-0.060354) | 0.161140 / 0.737135 (-0.575995) | 0.123002 / 0.296338 (-0.173336) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400102 / 0.215209 (0.184893) | 3.976748 / 2.077655 (1.899094) | 1.794763 / 1.504120 (0.290643) | 1.602477 / 1.541195 (0.061282) | 1.703689 / 1.468490 (0.235199) | 0.696751 / 4.584777 (-3.888026) | 3.713832 / 3.745712 (-0.031880) | 2.124536 / 5.269862 (-3.145326) | 1.313005 / 4.565676 (-3.252671) | 0.086130 / 0.424275 (-0.338146) | 0.012085 / 0.007607 (0.004477) | 0.512976 / 0.226044 (0.286932) | 5.135313 / 2.268929 (2.866384) | 2.318173 / 55.444624 (-53.126451) | 1.996360 / 6.876477 (-4.880117) | 2.060150 / 2.142072 (-0.081922) | 0.853534 / 4.805227 (-3.951693) | 0.165586 / 6.500664 (-6.335078) | 0.062365 / 0.075469 (-0.013104) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.178843 / 1.841788 (-0.662945) | 14.541639 / 8.074308 (6.467331) | 14.090782 / 10.191392 (3.899390) | 0.158717 / 0.680424 (-0.521707) | 0.028825 / 0.534201 (-0.505376) | 0.441427 / 0.579283 (-0.137856) | 0.439856 / 0.434364 (0.005492) | 0.530610 / 0.540337 (-0.009727) | 0.634044 / 1.386936 (-0.752892) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007502 / 0.011353 (-0.003851) | 0.005208 / 0.011008 (-0.005801) | 0.075020 / 0.038508 (0.036512) | 0.033297 / 0.023109 (0.010188) | 0.342218 / 0.275898 (0.066320) | 0.376716 / 0.323480 (0.053236) | 0.005906 / 0.007986 (-0.002080) | 0.005320 / 0.004328 (0.000992) | 0.073531 / 0.004250 (0.069281) | 0.049091 / 0.037052 (0.012039) | 0.344202 / 0.258489 (0.085713) | 0.380556 / 0.293841 (0.086715) | 0.037500 / 0.128546 (-0.091047) | 0.012404 / 0.075646 (-0.063242) | 0.087254 / 0.419271 (-0.332017) | 0.055145 / 0.043533 (0.011612) | 0.344112 / 0.255139 (0.088973) | 0.359052 / 0.283200 (0.075852) | 0.108337 / 0.141683 (-0.033345) | 1.450332 / 1.452155 (-0.001822) | 1.553607 / 1.492716 (0.060891) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216335 / 0.018006 (0.198329) | 0.436813 / 0.000490 (0.436323) | 0.005055 / 0.000200 (0.004855) | 0.000088 / 0.000054 (0.000033) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030037 / 0.037411 (-0.007374) | 0.110854 / 0.014526 (0.096329) | 0.121967 / 0.176557 (-0.054589) | 0.174029 / 0.737135 (-0.563107) | 0.128340 / 0.296338 (-0.167998) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.424463 / 0.215209 (0.209254) | 4.201822 / 2.077655 (2.124167) | 2.043075 / 1.504120 (0.538956) | 1.851841 / 1.541195 (0.310647) | 1.947790 / 1.468490 (0.479300) | 0.684110 / 4.584777 (-3.900667) | 3.763536 / 3.745712 (0.017824) | 3.106988 / 5.269862 (-2.162873) | 1.498305 / 4.565676 (-3.067372) | 0.085079 / 0.424275 (-0.339196) | 0.012241 / 0.007607 (0.004634) | 0.520877 / 0.226044 (0.294832) | 5.181455 / 2.268929 (2.912527) | 2.443038 / 55.444624 (-53.001586) | 2.130823 / 6.876477 (-4.745654) | 2.217901 / 2.142072 (0.075829) | 0.837116 / 4.805227 (-3.968111) | 0.166581 / 6.500664 (-6.334083) | 0.065510 / 0.075469 (-0.009959) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.289317 / 1.841788 (-0.552471) | 15.122019 / 8.074308 (7.047710) | 13.919670 / 10.191392 (3.728278) | 0.150047 / 0.680424 (-0.530377) | 0.017612 / 0.534201 (-0.516589) | 0.426239 / 0.579283 (-0.153044) | 0.425686 / 0.434364 (-0.008678) | 0.521436 / 0.540337 (-0.018901) | 0.618217 / 1.386936 (-0.768719) |

</details>
</details>

![](https://cml.dev/watermark.png#879fc6d5186ce593fe819f1e9e67897a1873766b "CML watermark")

2023-03-07T13:37:10Z
https://github.com/huggingface/datasets/pull/5620#issuecomment-1458187412
https://api.github.com/repos/huggingface/datasets/issues/5620
IC_kwDODunzps5W6iyU
https://api.github.com/repos/huggingface/datasets/issues/comments/1458187412/reactions
2023-03-07T13:37:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1458187412
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
COLLABORATOR
We haven't updated the minimal version requirement for PyArrow in a while, so it's ok to make a bigger leap IMO, e.g., PyArrow 8.0 (Colab installs 9.0). With this change, we should also remove the PyArrow version check in `folder_based_builder.py`, and the ones in `table.py`/`arrow_dataset.py` regarding the `to_reader` API if we decide to bump PyArrow to version 8.0.
2023-03-07T13:53:19Z
https://github.com/huggingface/datasets/pull/5620#issuecomment-1458212369
https://api.github.com/repos/huggingface/datasets/issues/5620
IC_kwDODunzps5W6o4R
https://api.github.com/repos/huggingface/datasets/issues/comments/1458212369/reactions
2023-03-07T13:53:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1458212369
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
I think it's a good opportunity to bump the version to 8.0 which offers higher performance anyway, I wouldn't bother trying to support 6.0.1 anymore. Only 1% of users based on 6.0.1 use the latest `datasets` version 2.10.1

Bumping to 8.0 if it sounds good to you
2023-03-07T14:41:04Z
https://github.com/huggingface/datasets/pull/5620#issuecomment-1458292088
https://api.github.com/repos/huggingface/datasets/issues/5620
IC_kwDODunzps5W68V4
https://api.github.com/repos/huggingface/datasets/issues/comments/1458292088/reactions
2023-03-07T14:41:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1458292088
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Sure, it is OK for those other reasons. I would just not stress that the increase of the minimum version is to support pandas 2.0 though...
2023-03-07T14:42:46Z
https://github.com/huggingface/datasets/pull/5620#issuecomment-1458294828
https://api.github.com/repos/huggingface/datasets/issues/5620
IC_kwDODunzps5W69As
https://api.github.com/repos/huggingface/datasets/issues/comments/1458294828/reactions
2023-03-08T08:33:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1458294828
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
If requiring min 8.0, do you know the percentage of people using 7.0 and latest datasets version?
2023-03-07T14:47:50Z
https://github.com/huggingface/datasets/pull/5620#issuecomment-1458302768
https://api.github.com/repos/huggingface/datasets/issues/5620
IC_kwDODunzps5W6-8w
https://api.github.com/repos/huggingface/datasets/issues/comments/1458302768/reactions
2023-03-07T14:47:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1458302768
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Around 10% of users have 7.0.0, and 25% among them use the latest datasets version
2023-03-07T15:57:22Z
https://github.com/huggingface/datasets/pull/5620#issuecomment-1458413445
https://api.github.com/repos/huggingface/datasets/issues/5620
IC_kwDODunzps5W7Z-F
https://api.github.com/repos/huggingface/datasets/issues/comments/1458413445/reactions
2023-03-07T15:57:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1458413445
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006744 / 0.011353 (-0.004609) | 0.004585 / 0.011008 (-0.006423) | 0.097828 / 0.038508 (0.059320) | 0.028230 / 0.023109 (0.005121) | 0.302190 / 0.275898 (0.026292) | 0.335022 / 0.323480 (0.011542) | 0.005107 / 0.007986 (-0.002878) | 0.004648 / 0.004328 (0.000320) | 0.076842 / 0.004250 (0.072592) | 0.038291 / 0.037052 (0.001239) | 0.313286 / 0.258489 (0.054797) | 0.342534 / 0.293841 (0.048693) | 0.031325 / 0.128546 (-0.097221) | 0.011632 / 0.075646 (-0.064014) | 0.321879 / 0.419271 (-0.097392) | 0.042204 / 0.043533 (-0.001329) | 0.304442 / 0.255139 (0.049303) | 0.330912 / 0.283200 (0.047712) | 0.085446 / 0.141683 (-0.056237) | 1.469990 / 1.452155 (0.017835) | 1.551147 / 1.492716 (0.058431) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.185961 / 0.018006 (0.167955) | 0.404675 / 0.000490 (0.404186) | 0.003212 / 0.000200 (0.003012) | 0.000074 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023876 / 0.037411 (-0.013535) | 0.097820 / 0.014526 (0.083295) | 0.107382 / 0.176557 (-0.069174) | 0.167598 / 0.737135 (-0.569537) | 0.108789 / 0.296338 (-0.187550) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.455004 / 0.215209 (0.239795) | 4.529104 / 2.077655 (2.451449) | 2.180068 / 1.504120 (0.675948) | 1.982109 / 1.541195 (0.440914) | 2.041856 / 1.468490 (0.573366) | 0.702029 / 4.584777 (-3.882747) | 3.368613 / 3.745712 (-0.377099) | 1.932303 / 5.269862 (-3.337559) | 1.278340 / 4.565676 (-3.287336) | 0.082836 / 0.424275 (-0.341439) | 0.012349 / 0.007607 (0.004742) | 0.548197 / 0.226044 (0.322153) | 5.509982 / 2.268929 (3.241053) | 2.612889 / 55.444624 (-52.831736) | 2.278157 / 6.876477 (-4.598320) | 2.386923 / 2.142072 (0.244851) | 0.803332 / 4.805227 (-4.001896) | 0.151222 / 6.500664 (-6.349442) | 0.066673 / 0.075469 (-0.008796) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.209453 / 1.841788 (-0.632335) | 13.649733 / 8.074308 (5.575424) | 14.065917 / 10.191392 (3.874525) | 0.128872 / 0.680424 (-0.551551) | 0.016773 / 0.534201 (-0.517428) | 0.385475 / 0.579283 (-0.193809) | 0.386208 / 0.434364 (-0.048156) | 0.475144 / 0.540337 (-0.065194) | 0.564183 / 1.386936 (-0.822753) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006629 / 0.011353 (-0.004724) | 0.004433 / 0.011008 (-0.006575) | 0.076008 / 0.038508 (0.037500) | 0.027471 / 0.023109 (0.004362) | 0.339837 / 0.275898 (0.063939) | 0.376857 / 0.323480 (0.053377) | 0.004930 / 0.007986 (-0.003055) | 0.003312 / 0.004328 (-0.001016) | 0.075070 / 0.004250 (0.070820) | 0.035897 / 0.037052 (-0.001156) | 0.342398 / 0.258489 (0.083909) | 0.380202 / 0.293841 (0.086361) | 0.031781 / 0.128546 (-0.096766) | 0.011697 / 0.075646 (-0.063950) | 0.085926 / 0.419271 (-0.333345) | 0.041599 / 0.043533 (-0.001934) | 0.343098 / 0.255139 (0.087959) | 0.371275 / 0.283200 (0.088076) | 0.090489 / 0.141683 (-0.051194) | 1.483738 / 1.452155 (0.031584) | 1.554973 / 1.492716 (0.062256) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.183703 / 0.018006 (0.165697) | 0.395105 / 0.000490 (0.394616) | 0.002162 / 0.000200 (0.001963) | 0.000074 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025432 / 0.037411 (-0.011979) | 0.101322 / 0.014526 (0.086796) | 0.107839 / 0.176557 (-0.068718) | 0.160328 / 0.737135 (-0.576807) | 0.109899 / 0.296338 (-0.186440) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.448001 / 0.215209 (0.232792) | 4.485321 / 2.077655 (2.407666) | 2.157064 / 1.504120 (0.652944) | 1.966141 / 1.541195 (0.424947) | 2.032808 / 1.468490 (0.564318) | 0.705684 / 4.584777 (-3.879093) | 3.359802 / 3.745712 (-0.385910) | 2.694952 / 5.269862 (-2.574910) | 1.471309 / 4.565676 (-3.094368) | 0.084185 / 0.424275 (-0.340090) | 0.012330 / 0.007607 (0.004723) | 0.554083 / 0.226044 (0.328038) | 5.569137 / 2.268929 (3.300208) | 2.586009 / 55.444624 (-52.858615) | 2.234920 / 6.876477 (-4.641557) | 2.285128 / 2.142072 (0.143056) | 0.818825 / 4.805227 (-3.986402) | 0.152604 / 6.500664 (-6.348060) | 0.067722 / 0.075469 (-0.007747) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.305571 / 1.841788 (-0.536217) | 13.687471 / 8.074308 (5.613163) | 13.305401 / 10.191392 (3.114009) | 0.140477 / 0.680424 (-0.539947) | 0.018138 / 0.534201 (-0.516063) | 0.377255 / 0.579283 (-0.202028) | 0.379522 / 0.434364 (-0.054842) | 0.458489 / 0.540337 (-0.081849) | 0.543767 / 1.386936 (-0.843169) |

</details>
</details>

![](https://cml.dev/watermark.png#02570894db6ecc46bf25b7fa1cb1bcdc1dede853 "CML watermark")

2023-03-07T17:03:17Z
https://github.com/huggingface/datasets/pull/5620#issuecomment-1458521424
https://api.github.com/repos/huggingface/datasets/issues/5620
IC_kwDODunzps5W70VQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1458521424/reactions
2023-03-07T17:03:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1458521424
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009606 / 0.011353 (-0.001747) | 0.006795 / 0.011008 (-0.004213) | 0.133738 / 0.038508 (0.095230) | 0.043379 / 0.023109 (0.020270) | 0.412917 / 0.275898 (0.137019) | 0.418790 / 0.323480 (0.095310) | 0.007290 / 0.007986 (-0.000696) | 0.004960 / 0.004328 (0.000632) | 0.095496 / 0.004250 (0.091246) | 0.057607 / 0.037052 (0.020555) | 0.402638 / 0.258489 (0.144149) | 0.436206 / 0.293841 (0.142365) | 0.056023 / 0.128546 (-0.072523) | 0.019909 / 0.075646 (-0.055737) | 0.463958 / 0.419271 (0.044687) | 0.064073 / 0.043533 (0.020541) | 0.398337 / 0.255139 (0.143198) | 0.421786 / 0.283200 (0.138586) | 0.131563 / 0.141683 (-0.010120) | 1.840217 / 1.452155 (0.388063) | 1.912013 / 1.492716 (0.419296) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230519 / 0.018006 (0.212513) | 0.550506 / 0.000490 (0.550017) | 0.003649 / 0.000200 (0.003449) | 0.000107 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029713 / 0.037411 (-0.007698) | 0.129913 / 0.014526 (0.115387) | 0.131543 / 0.176557 (-0.045013) | 0.203571 / 0.737135 (-0.533565) | 0.141483 / 0.296338 (-0.154856) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.626383 / 0.215209 (0.411174) | 6.193043 / 2.077655 (4.115388) | 2.442728 / 1.504120 (0.938608) | 2.079049 / 1.541195 (0.537855) | 2.117761 / 1.468490 (0.649271) | 1.315296 / 4.584777 (-3.269481) | 5.643709 / 3.745712 (1.897997) | 5.245789 / 5.269862 (-0.024073) | 2.757442 / 4.565676 (-1.808235) | 0.151655 / 0.424275 (-0.272620) | 0.014686 / 0.007607 (0.007079) | 0.779937 / 0.226044 (0.553893) | 7.796685 / 2.268929 (5.527756) | 3.349580 / 55.444624 (-52.095045) | 2.493750 / 6.876477 (-4.382727) | 2.506200 / 2.142072 (0.364128) | 1.534964 / 4.805227 (-3.270263) | 0.260001 / 6.500664 (-6.240663) | 0.080543 / 0.075469 (0.005074) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.541940 / 1.841788 (-0.299848) | 17.851935 / 8.074308 (9.777627) | 22.418859 / 10.191392 (12.227467) | 0.258602 / 0.680424 (-0.421822) | 0.027679 / 0.534201 (-0.506522) | 0.548379 / 0.579283 (-0.030904) | 0.625505 / 0.434364 (0.191141) | 0.664074 / 0.540337 (0.123737) | 0.797418 / 1.386936 (-0.589518) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009800 / 0.011353 (-0.001553) | 0.006178 / 0.011008 (-0.004830) | 0.105667 / 0.038508 (0.067159) | 0.039380 / 0.023109 (0.016271) | 0.419528 / 0.275898 (0.143630) | 0.469857 / 0.323480 (0.146377) | 0.006672 / 0.007986 (-0.001314) | 0.004745 / 0.004328 (0.000417) | 0.101647 / 0.004250 (0.097397) | 0.048531 / 0.037052 (0.011478) | 0.433364 / 0.258489 (0.174875) | 0.459719 / 0.293841 (0.165878) | 0.054291 / 0.128546 (-0.074256) | 0.020406 / 0.075646 (-0.055240) | 0.122321 / 0.419271 (-0.296951) | 0.059719 / 0.043533 (0.016186) | 0.416083 / 0.255139 (0.160944) | 0.455277 / 0.283200 (0.172077) | 0.119342 / 0.141683 (-0.022341) | 1.862544 / 1.452155 (0.410390) | 2.001428 / 1.492716 (0.508712) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.240951 / 0.018006 (0.222945) | 0.516958 / 0.000490 (0.516468) | 0.000449 / 0.000200 (0.000249) | 0.000092 / 0.000054 (0.000037) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032725 / 0.037411 (-0.004686) | 0.130291 / 0.014526 (0.115765) | 0.139834 / 0.176557 (-0.036723) | 0.214995 / 0.737135 (-0.522140) | 0.150925 / 0.296338 (-0.145414) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.652062 / 0.215209 (0.436853) | 6.584447 / 2.077655 (4.506793) | 2.654838 / 1.504120 (1.150718) | 2.297209 / 1.541195 (0.756015) | 2.420394 / 1.468490 (0.951904) | 1.299285 / 4.584777 (-3.285492) | 5.605849 / 3.745712 (1.860137) | 3.166103 / 5.269862 (-2.103759) | 2.138123 / 4.565676 (-2.427554) | 0.152562 / 0.424275 (-0.271713) | 0.015499 / 0.007607 (0.007892) | 0.816300 / 0.226044 (0.590256) | 8.308746 / 2.268929 (6.039817) | 3.482982 / 55.444624 (-51.961642) | 2.689247 / 6.876477 (-4.187229) | 2.792728 / 2.142072 (0.650656) | 1.566320 / 4.805227 (-3.238907) | 0.264110 / 6.500664 (-6.236554) | 0.083652 / 0.075469 (0.008183) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.643027 / 1.841788 (-0.198760) | 18.612349 / 8.074308 (10.538041) | 19.460644 / 10.191392 (9.269252) | 0.260795 / 0.680424 (-0.419629) | 0.026050 / 0.534201 (-0.508151) | 0.539750 / 0.579283 (-0.039533) | 0.620791 / 0.434364 (0.186428) | 0.645023 / 0.540337 (0.104686) | 0.765604 / 1.386936 (-0.621332) |

</details>
</details>

![](https://cml.dev/watermark.png#e6dcf4c50e14ee6dbc6d763ed1b7ce3501460863 "CML watermark")

2023-03-07T17:07:06Z
https://github.com/huggingface/datasets/pull/5620#issuecomment-1458527451
https://api.github.com/repos/huggingface/datasets/issues/5620
IC_kwDODunzps5W71zb
https://api.github.com/repos/huggingface/datasets/issues/comments/1458527451/reactions
2023-03-07T17:07:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1458527451
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
MEMBER
ready for re-review :)
2023-03-07T17:29:37Z
https://github.com/huggingface/datasets/pull/5620#issuecomment-1458559496
https://api.github.com/repos/huggingface/datasets/issues/5620
IC_kwDODunzps5W79oI
https://api.github.com/repos/huggingface/datasets/issues/comments/1458559496/reactions
2023-03-07T17:29:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1458559496
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006388 / 0.011353 (-0.004965) | 0.004469 / 0.011008 (-0.006540) | 0.097082 / 0.038508 (0.058573) | 0.028005 / 0.023109 (0.004895) | 0.364797 / 0.275898 (0.088899) | 0.399671 / 0.323480 (0.076191) | 0.005062 / 0.007986 (-0.002923) | 0.004580 / 0.004328 (0.000252) | 0.075670 / 0.004250 (0.071420) | 0.038328 / 0.037052 (0.001276) | 0.365948 / 0.258489 (0.107459) | 0.402631 / 0.293841 (0.108790) | 0.031378 / 0.128546 (-0.097168) | 0.011443 / 0.075646 (-0.064203) | 0.321590 / 0.419271 (-0.097682) | 0.042263 / 0.043533 (-0.001270) | 0.368238 / 0.255139 (0.113099) | 0.389928 / 0.283200 (0.106728) | 0.085203 / 0.141683 (-0.056480) | 1.462820 / 1.452155 (0.010665) | 1.529207 / 1.492716 (0.036490) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.197194 / 0.018006 (0.179188) | 0.410897 / 0.000490 (0.410407) | 0.003394 / 0.000200 (0.003194) | 0.000075 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022911 / 0.037411 (-0.014500) | 0.097012 / 0.014526 (0.082486) | 0.102247 / 0.176557 (-0.074309) | 0.163363 / 0.737135 (-0.573772) | 0.106897 / 0.296338 (-0.189441) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416303 / 0.215209 (0.201094) | 4.159325 / 2.077655 (2.081671) | 1.844893 / 1.504120 (0.340773) | 1.646131 / 1.541195 (0.104936) | 1.706763 / 1.468490 (0.238273) | 0.699607 / 4.584777 (-3.885170) | 3.462048 / 3.745712 (-0.283664) | 1.939076 / 5.269862 (-3.330786) | 1.324744 / 4.565676 (-3.240932) | 0.082949 / 0.424275 (-0.341326) | 0.012327 / 0.007607 (0.004720) | 0.513812 / 0.226044 (0.287768) | 5.171021 / 2.268929 (2.902093) | 2.288039 / 55.444624 (-53.156585) | 1.957403 / 6.876477 (-4.919074) | 1.990060 / 2.142072 (-0.152013) | 0.805571 / 4.805227 (-3.999656) | 0.152641 / 6.500664 (-6.348023) | 0.068169 / 0.075469 (-0.007300) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.200624 / 1.841788 (-0.641164) | 13.836334 / 8.074308 (5.762026) | 14.065340 / 10.191392 (3.873948) | 0.143406 / 0.680424 (-0.537018) | 0.016709 / 0.534201 (-0.517492) | 0.380080 / 0.579283 (-0.199204) | 0.398414 / 0.434364 (-0.035950) | 0.479192 / 0.540337 (-0.061145) | 0.572508 / 1.386936 (-0.814428) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006622 / 0.011353 (-0.004731) | 0.004511 / 0.011008 (-0.006497) | 0.076454 / 0.038508 (0.037946) | 0.027431 / 0.023109 (0.004322) | 0.339041 / 0.275898 (0.063143) | 0.375691 / 0.323480 (0.052211) | 0.004854 / 0.007986 (-0.003131) | 0.004654 / 0.004328 (0.000325) | 0.075300 / 0.004250 (0.071049) | 0.036469 / 0.037052 (-0.000583) | 0.341357 / 0.258489 (0.082868) | 0.381561 / 0.293841 (0.087720) | 0.031754 / 0.128546 (-0.096792) | 0.011544 / 0.075646 (-0.064102) | 0.085956 / 0.419271 (-0.333315) | 0.041704 / 0.043533 (-0.001828) | 0.340088 / 0.255139 (0.084950) | 0.364037 / 0.283200 (0.080838) | 0.091016 / 0.141683 (-0.050667) | 1.483515 / 1.452155 (0.031360) | 1.562878 / 1.492716 (0.070162) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228019 / 0.018006 (0.210013) | 0.404809 / 0.000490 (0.404320) | 0.000384 / 0.000200 (0.000184) | 0.000060 / 0.000054 (0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025230 / 0.037411 (-0.012181) | 0.099790 / 0.014526 (0.085264) | 0.107923 / 0.176557 (-0.068634) | 0.157651 / 0.737135 (-0.579484) | 0.112525 / 0.296338 (-0.183813) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440360 / 0.215209 (0.225151) | 4.387749 / 2.077655 (2.310094) | 2.077592 / 1.504120 (0.573472) | 1.872532 / 1.541195 (0.331337) | 1.941607 / 1.468490 (0.473117) | 0.699394 / 4.584777 (-3.885383) | 3.411210 / 3.745712 (-0.334502) | 1.901816 / 5.269862 (-3.368046) | 1.177042 / 4.565676 (-3.388634) | 0.083536 / 0.424275 (-0.340739) | 0.012418 / 0.007607 (0.004811) | 0.548463 / 0.226044 (0.322419) | 5.487107 / 2.268929 (3.218178) | 2.548076 / 55.444624 (-52.896548) | 2.215012 / 6.876477 (-4.661465) | 2.253472 / 2.142072 (0.111400) | 0.812925 / 4.805227 (-3.992302) | 0.152935 / 6.500664 (-6.347729) | 0.068144 / 0.075469 (-0.007325) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.267914 / 1.841788 (-0.573873) | 14.015185 / 8.074308 (5.940877) | 13.153967 / 10.191392 (2.962575) | 0.140666 / 0.680424 (-0.539758) | 0.016718 / 0.534201 (-0.517483) | 0.383411 / 0.579283 (-0.195872) | 0.395424 / 0.434364 (-0.038940) | 0.466069 / 0.540337 (-0.074269) | 0.553825 / 1.386936 (-0.833111) |

</details>
</details>

![](https://cml.dev/watermark.png#14568bf072b38e3b295f29774c874c8e78b9fe37 "CML watermark")

2023-03-08T12:03:47Z
https://github.com/huggingface/datasets/pull/5620#issuecomment-1460055219
https://api.github.com/repos/huggingface/datasets/issues/5620
IC_kwDODunzps5XBqyz
https://api.github.com/repos/huggingface/datasets/issues/comments/1460055219/reactions
2023-03-08T12:03:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1460055219
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007463 / 0.011353 (-0.003890) | 0.005017 / 0.011008 (-0.005991) | 0.098777 / 0.038508 (0.060269) | 0.033859 / 0.023109 (0.010750) | 0.298569 / 0.275898 (0.022670) | 0.343717 / 0.323480 (0.020237) | 0.005806 / 0.007986 (-0.002180) | 0.005403 / 0.004328 (0.001074) | 0.075840 / 0.004250 (0.071590) | 0.046539 / 0.037052 (0.009487) | 0.300058 / 0.258489 (0.041569) | 0.345036 / 0.293841 (0.051195) | 0.036258 / 0.128546 (-0.092288) | 0.011992 / 0.075646 (-0.063654) | 0.334986 / 0.419271 (-0.084286) | 0.050427 / 0.043533 (0.006894) | 0.295319 / 0.255139 (0.040180) | 0.318980 / 0.283200 (0.035780) | 0.098407 / 0.141683 (-0.043276) | 1.437626 / 1.452155 (-0.014529) | 1.562548 / 1.492716 (0.069832) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231502 / 0.018006 (0.213496) | 0.441550 / 0.000490 (0.441060) | 0.005863 / 0.000200 (0.005663) | 0.000724 / 0.000054 (0.000670) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027501 / 0.037411 (-0.009911) | 0.111490 / 0.014526 (0.096964) | 0.117503 / 0.176557 (-0.059054) | 0.173849 / 0.737135 (-0.563286) | 0.124521 / 0.296338 (-0.171818) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419266 / 0.215209 (0.204057) | 4.170337 / 2.077655 (2.092683) | 2.015883 / 1.504120 (0.511763) | 1.832683 / 1.541195 (0.291488) | 1.950195 / 1.468490 (0.481705) | 0.698150 / 4.584777 (-3.886627) | 3.775601 / 3.745712 (0.029889) | 2.094581 / 5.269862 (-3.175281) | 1.325437 / 4.565676 (-3.240240) | 0.085382 / 0.424275 (-0.338894) | 0.012151 / 0.007607 (0.004544) | 0.526441 / 0.226044 (0.300397) | 5.256124 / 2.268929 (2.987196) | 2.488408 / 55.444624 (-52.956216) | 2.157228 / 6.876477 (-4.719249) | 2.228991 / 2.142072 (0.086919) | 0.837002 / 4.805227 (-3.968225) | 0.167520 / 6.500664 (-6.333144) | 0.066435 / 0.075469 (-0.009035) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.174544 / 1.841788 (-0.667243) | 14.684207 / 8.074308 (6.609899) | 14.494676 / 10.191392 (4.303284) | 0.143423 / 0.680424 (-0.537001) | 0.017289 / 0.534201 (-0.516912) | 0.424727 / 0.579283 (-0.154556) | 0.417077 / 0.434364 (-0.017287) | 0.498955 / 0.540337 (-0.041383) | 0.584838 / 1.386936 (-0.802098) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007666 / 0.011353 (-0.003687) | 0.005269 / 0.011008 (-0.005739) | 0.073548 / 0.038508 (0.035040) | 0.033683 / 0.023109 (0.010573) | 0.342646 / 0.275898 (0.066747) | 0.380948 / 0.323480 (0.057468) | 0.005737 / 0.007986 (-0.002248) | 0.005366 / 0.004328 (0.001038) | 0.073228 / 0.004250 (0.068978) | 0.050065 / 0.037052 (0.013013) | 0.348593 / 0.258489 (0.090104) | 0.393930 / 0.293841 (0.100089) | 0.037411 / 0.128546 (-0.091135) | 0.012476 / 0.075646 (-0.063170) | 0.084884 / 0.419271 (-0.334387) | 0.049368 / 0.043533 (0.005835) | 0.343142 / 0.255139 (0.088003) | 0.362828 / 0.283200 (0.079628) | 0.102962 / 0.141683 (-0.038721) | 1.505703 / 1.452155 (0.053549) | 1.580695 / 1.492716 (0.087979) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.207621 / 0.018006 (0.189615) | 0.437678 / 0.000490 (0.437188) | 0.003931 / 0.000200 (0.003731) | 0.000093 / 0.000054 (0.000038) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029079 / 0.037411 (-0.008332) | 0.108600 / 0.014526 (0.094074) | 0.124787 / 0.176557 (-0.051770) | 0.173354 / 0.737135 (-0.563781) | 0.126124 / 0.296338 (-0.170214) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.427911 / 0.215209 (0.212702) | 4.254227 / 2.077655 (2.176572) | 2.052142 / 1.504120 (0.548022) | 1.857042 / 1.541195 (0.315848) | 1.965244 / 1.468490 (0.496754) | 0.707994 / 4.584777 (-3.876783) | 3.807593 / 3.745712 (0.061880) | 3.387588 / 5.269862 (-1.882274) | 1.844853 / 4.565676 (-2.720824) | 0.088548 / 0.424275 (-0.335727) | 0.012398 / 0.007607 (0.004791) | 0.565896 / 0.226044 (0.339851) | 5.228024 / 2.268929 (2.959095) | 2.467220 / 55.444624 (-52.977405) | 2.144413 / 6.876477 (-4.732064) | 2.214049 / 2.142072 (0.071977) | 0.869381 / 4.805227 (-3.935846) | 0.170991 / 6.500664 (-6.329673) | 0.064932 / 0.075469 (-0.010537) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.246661 / 1.841788 (-0.595127) | 14.902743 / 8.074308 (6.828435) | 13.264294 / 10.191392 (3.072902) | 0.165328 / 0.680424 (-0.515095) | 0.017567 / 0.534201 (-0.516634) | 0.425491 / 0.579283 (-0.153792) | 0.427327 / 0.434364 (-0.007037) | 0.526475 / 0.540337 (-0.013862) | 0.627309 / 1.386936 (-0.759627) |

</details>
</details>

![](https://cml.dev/watermark.png#dd31bce76b554447bccb2b1447440e1f8ddba035 "CML watermark")

2023-03-08T14:01:26Z
https://github.com/huggingface/datasets/pull/5620#issuecomment-1460199760
https://api.github.com/repos/huggingface/datasets/issues/5620
IC_kwDODunzps5XCOFQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1460199760/reactions
2023-03-08T14:01:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1460199760
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
Fix those for Pandas 2.0 (tested [here](https://github.com/huggingface/datasets/actions/runs/4346221280/jobs/7592010397) with pandas==2.0.0.rc0):

```python
=========================== short test summary info ============================
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_to_parquet_in_memory - ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
A suitable version of pyarrow or fastparquet is required for parquet support.
Trying to import the above resulted in these errors:
 - Pandas requires version '7.0.0' or newer of 'pyarrow' (version '6.0.1' currently installed).
 - Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet.
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_to_parquet_on_disk - ImportError: Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'.
A suitable version of pyarrow or fastparquet is required for parquet support.
Trying to import the above resulted in these errors:
 - Pandas requires version '7.0.0' or newer of 'pyarrow' (version '6.0.1' currently installed).
 - Missing optional dependency 'fastparquet'. fastparquet is required for parquet support. Use pip or conda to install fastparquet.
===== 2 failed, 2137 passed, 18 skipped, 32 warnings in 212.76s (0:03:32) ======
```

EDIT: also for performance - with 8.0 we can use `.to_reader()`