improved message error row formatting
NONE
_The documentation is not available anymore as the PR was closed or merged._
2023-02-21T12:57:36Z
https://github.com/huggingface/datasets/pull/5553#issuecomment-1438438287
https://api.github.com/repos/huggingface/datasets/issues/5553
IC_kwDODunzps5VvNOP
https://api.github.com/repos/huggingface/datasets/issues/comments/1438438287/reactions
2023-02-21T13:00:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1438438287
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.014953 / 0.011353 (0.003600) | 0.006936 / 0.011008 (-0.004072) | 0.144039 / 0.038508 (0.105531) | 0.046719 / 0.023109 (0.023610) | 0.408832 / 0.275898 (0.132934) | 0.501419 / 0.323480 (0.177939) | 0.010190 / 0.007986 (0.002204) | 0.007618 / 0.004328 (0.003290) | 0.108553 / 0.004250 (0.104303) | 0.048484 / 0.037052 (0.011432) | 0.451586 / 0.258489 (0.193097) | 0.469864 / 0.293841 (0.176023) | 0.062159 / 0.128546 (-0.066387) | 0.019937 / 0.075646 (-0.055710) | 0.473718 / 0.419271 (0.054446) | 0.064777 / 0.043533 (0.021244) | 0.428675 / 0.255139 (0.173536) | 0.467665 / 0.283200 (0.184465) | 0.133528 / 0.141683 (-0.008155) | 1.978084 / 1.452155 (0.525930) | 1.965878 / 1.492716 (0.473162) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.290112 / 0.018006 (0.272106) | 0.629481 / 0.000490 (0.628992) | 0.003600 / 0.000200 (0.003400) | 0.000144 / 0.000054 (0.000089) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030806 / 0.037411 (-0.006605) | 0.142376 / 0.014526 (0.127850) | 0.150020 / 0.176557 (-0.026537) | 0.193679 / 0.737135 (-0.543457) | 0.151329 / 0.296338 (-0.145009) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.629725 / 0.215209 (0.414516) | 6.656313 / 2.077655 (4.578659) | 2.712160 / 1.504120 (1.208041) | 2.328461 / 1.541195 (0.787266) | 2.452502 / 1.468490 (0.984012) | 1.353183 / 4.584777 (-3.231594) | 5.981521 / 3.745712 (2.235809) | 3.707186 / 5.269862 (-1.562676) | 2.460583 / 4.565676 (-2.105094) | 0.178300 / 0.424275 (-0.245975) | 0.020357 / 0.007607 (0.012750) | 0.813564 / 0.226044 (0.587520) | 8.465600 / 2.268929 (6.196671) | 3.491507 / 55.444624 (-51.953117) | 2.810781 / 6.876477 (-4.065695) | 3.100182 / 2.142072 (0.958110) | 1.539321 / 4.805227 (-3.265906) | 0.257735 / 6.500664 (-6.242929) | 0.082785 / 0.075469 (0.007316) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.766586 / 1.841788 (-0.075201) | 20.534638 / 8.074308 (12.460330) | 24.066176 / 10.191392 (13.874784) | 0.272419 / 0.680424 (-0.408005) | 0.048940 / 0.534201 (-0.485261) | 0.606004 / 0.579283 (0.026721) | 0.669684 / 0.434364 (0.235320) | 0.716858 / 0.540337 (0.176521) | 0.949394 / 1.386936 (-0.437542) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010865 / 0.011353 (-0.000488) | 0.009855 / 0.011008 (-0.001153) | 0.105973 / 0.038508 (0.067465) | 0.039818 / 0.023109 (0.016709) | 0.544505 / 0.275898 (0.268607) | 0.511253 / 0.323480 (0.187773) | 0.007350 / 0.007986 (-0.000635) | 0.006950 / 0.004328 (0.002622) | 0.106548 / 0.004250 (0.102298) | 0.062740 / 0.037052 (0.025688) | 0.465881 / 0.258489 (0.207392) | 0.524426 / 0.293841 (0.230585) | 0.056052 / 0.128546 (-0.072495) | 0.020906 / 0.075646 (-0.054741) | 0.125337 / 0.419271 (-0.293935) | 0.064689 / 0.043533 (0.021156) | 0.483055 / 0.255139 (0.227916) | 0.518878 / 0.283200 (0.235678) | 0.127288 / 0.141683 (-0.014394) | 1.936246 / 1.452155 (0.484092) | 2.162532 / 1.492716 (0.669816) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.253691 / 0.018006 (0.235685) | 0.606244 / 0.000490 (0.605754) | 0.004251 / 0.000200 (0.004051) | 0.000126 / 0.000054 (0.000071) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038356 / 0.037411 (0.000944) | 0.146690 / 0.014526 (0.132164) | 0.146545 / 0.176557 (-0.030012) | 0.218452 / 0.737135 (-0.518684) | 0.165314 / 0.296338 (-0.131025) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.645768 / 0.215209 (0.430559) | 7.229186 / 2.077655 (5.151531) | 3.484778 / 1.504120 (1.980658) | 2.585310 / 1.541195 (1.044116) | 2.727670 / 1.468490 (1.259180) | 1.393416 / 4.584777 (-3.191361) | 6.448707 / 3.745712 (2.702995) | 3.433652 / 5.269862 (-1.836209) | 2.106450 / 4.565676 (-2.459226) | 0.143899 / 0.424275 (-0.280376) | 0.015097 / 0.007607 (0.007490) | 0.860960 / 0.226044 (0.634916) | 9.509725 / 2.268929 (7.240797) | 3.881601 / 55.444624 (-51.563024) | 3.156018 / 6.876477 (-3.720459) | 3.556330 / 2.142072 (1.414257) | 1.525940 / 4.805227 (-3.279287) | 0.264588 / 6.500664 (-6.236076) | 0.090327 / 0.075469 (0.014858) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.829761 / 1.841788 (-0.012027) | 21.037774 / 8.074308 (12.963466) | 24.464737 / 10.191392 (14.273345) | 0.394165 / 0.680424 (-0.286259) | 0.039286 / 0.534201 (-0.494915) | 0.546412 / 0.579283 (-0.032871) | 0.741760 / 0.434364 (0.307396) | 0.683969 / 0.540337 (0.143632) | 0.831392 / 1.386936 (-0.555544) |

</details>
</details>

![](https://cml.dev/watermark.png#e453eeac5239d0ff3e98adcba59a6724ee68b46b "CML watermark")

2023-02-21T13:08:24Z
https://github.com/huggingface/datasets/pull/5553#issuecomment-1438458466
https://api.github.com/repos/huggingface/datasets/issues/5553
IC_kwDODunzps5VvSJi
https://api.github.com/repos/huggingface/datasets/issues/comments/1438458466/reactions
2023-02-21T13:08:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1438458466
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
Solves  #5539