Apply ruff flake8-comprehension checks
NONE
_The documentation is not available anymore as the PR was closed or merged._
2023-02-21T13:13:31Z
https://github.com/huggingface/datasets/pull/5549#issuecomment-1438469858
https://api.github.com/repos/huggingface/datasets/issues/5549
IC_kwDODunzps5VvU7i
https://api.github.com/repos/huggingface/datasets/issues/comments/1438469858/reactions
2023-02-23T14:02:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1438469858
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009598 / 0.011353 (-0.001755) | 0.005115 / 0.011008 (-0.005893) | 0.100100 / 0.038508 (0.061592) | 0.036193 / 0.023109 (0.013083) | 0.296478 / 0.275898 (0.020580) | 0.355997 / 0.323480 (0.032517) | 0.007846 / 0.007986 (-0.000140) | 0.004082 / 0.004328 (-0.000247) | 0.076949 / 0.004250 (0.072699) | 0.044304 / 0.037052 (0.007252) | 0.310775 / 0.258489 (0.052286) | 0.333914 / 0.293841 (0.040073) | 0.037783 / 0.128546 (-0.090763) | 0.012023 / 0.075646 (-0.063623) | 0.333311 / 0.419271 (-0.085961) | 0.047568 / 0.043533 (0.004035) | 0.295567 / 0.255139 (0.040428) | 0.315707 / 0.283200 (0.032507) | 0.102675 / 0.141683 (-0.039008) | 1.471546 / 1.452155 (0.019391) | 1.507991 / 1.492716 (0.015274) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208658 / 0.018006 (0.190651) | 0.445026 / 0.000490 (0.444536) | 0.002593 / 0.000200 (0.002393) | 0.000084 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026968 / 0.037411 (-0.010444) | 0.108188 / 0.014526 (0.093662) | 0.117965 / 0.176557 (-0.058592) | 0.182769 / 0.737135 (-0.554366) | 0.121671 / 0.296338 (-0.174667) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.400677 / 0.215209 (0.185468) | 4.012577 / 2.077655 (1.934922) | 1.821324 / 1.504120 (0.317204) | 1.624438 / 1.541195 (0.083244) | 1.731886 / 1.468490 (0.263396) | 0.698089 / 4.584777 (-3.886688) | 3.786165 / 3.745712 (0.040453) | 2.079742 / 5.269862 (-3.190119) | 1.325032 / 4.565676 (-3.240644) | 0.085229 / 0.424275 (-0.339046) | 0.012017 / 0.007607 (0.004410) | 0.511779 / 0.226044 (0.285734) | 5.114358 / 2.268929 (2.845430) | 2.324763 / 55.444624 (-53.119861) | 2.011864 / 6.876477 (-4.864612) | 2.075875 / 2.142072 (-0.066198) | 0.853475 / 4.805227 (-3.951752) | 0.166949 / 6.500664 (-6.333715) | 0.064669 / 0.075469 (-0.010800) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.230212 / 1.841788 (-0.611576) | 14.942371 / 8.074308 (6.868063) | 14.075795 / 10.191392 (3.884403) | 0.156920 / 0.680424 (-0.523504) | 0.029002 / 0.534201 (-0.505199) | 0.442213 / 0.579283 (-0.137070) | 0.436888 / 0.434364 (0.002524) | 0.519725 / 0.540337 (-0.020613) | 0.604634 / 1.386936 (-0.782303) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007649 / 0.011353 (-0.003704) | 0.005298 / 0.011008 (-0.005710) | 0.076559 / 0.038508 (0.038050) | 0.033723 / 0.023109 (0.010614) | 0.334946 / 0.275898 (0.059048) | 0.372785 / 0.323480 (0.049305) | 0.006032 / 0.007986 (-0.001953) | 0.004125 / 0.004328 (-0.000204) | 0.075366 / 0.004250 (0.071116) | 0.049061 / 0.037052 (0.012009) | 0.338188 / 0.258489 (0.079699) | 0.389693 / 0.293841 (0.095852) | 0.037246 / 0.128546 (-0.091301) | 0.012530 / 0.075646 (-0.063116) | 0.088053 / 0.419271 (-0.331219) | 0.049844 / 0.043533 (0.006311) | 0.338476 / 0.255139 (0.083337) | 0.361672 / 0.283200 (0.078473) | 0.101982 / 0.141683 (-0.039701) | 1.479550 / 1.452155 (0.027396) | 1.541031 / 1.492716 (0.048315) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.226162 / 0.018006 (0.208156) | 0.439108 / 0.000490 (0.438618) | 0.001102 / 0.000200 (0.000902) | 0.000076 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030240 / 0.037411 (-0.007171) | 0.113754 / 0.014526 (0.099229) | 0.122839 / 0.176557 (-0.053717) | 0.192531 / 0.737135 (-0.544604) | 0.129455 / 0.296338 (-0.166884) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.424701 / 0.215209 (0.209492) | 4.208161 / 2.077655 (2.130507) | 2.045733 / 1.504120 (0.541613) | 1.892369 / 1.541195 (0.351174) | 1.997024 / 1.468490 (0.528534) | 0.739883 / 4.584777 (-3.844894) | 3.760939 / 3.745712 (0.015227) | 3.195748 / 5.269862 (-2.074113) | 1.731480 / 4.565676 (-2.834197) | 0.087013 / 0.424275 (-0.337262) | 0.012550 / 0.007607 (0.004943) | 0.540829 / 0.226044 (0.314785) | 5.329933 / 2.268929 (3.061005) | 2.507572 / 55.444624 (-52.937052) | 2.167761 / 6.876477 (-4.708716) | 2.250298 / 2.142072 (0.108226) | 0.868718 / 4.805227 (-3.936510) | 0.181643 / 6.500664 (-6.319021) | 0.064817 / 0.075469 (-0.010653) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.295001 / 1.841788 (-0.546787) | 15.236413 / 8.074308 (7.162105) | 13.692212 / 10.191392 (3.500820) | 0.186330 / 0.680424 (-0.494094) | 0.017492 / 0.534201 (-0.516709) | 0.427365 / 0.579283 (-0.151919) | 0.427781 / 0.434364 (-0.006583) | 0.533763 / 0.540337 (-0.006575) | 0.636011 / 1.386936 (-0.750925) |

</details>
</details>

![](https://cml.dev/watermark.png#94b16b674111ca5e1a03ddcb71dc0b53acc2f934 "CML watermark")

2023-02-23T14:06:39Z
https://github.com/huggingface/datasets/pull/5549#issuecomment-1441861177
https://api.github.com/repos/huggingface/datasets/issues/5549
IC_kwDODunzps5V8Q45
https://api.github.com/repos/huggingface/datasets/issues/comments/1441861177/reactions
2023-02-23T14:06:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1441861177
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
Fix #5548 

Apply ruff's flake8-comprehension checks for better performance, and more readable code.