More hub centric docs
NONE
_The documentation is not available anymore as the PR was closed or merged._
2023-10-26T17:01:41Z
https://github.com/huggingface/datasets/pull/6355#issuecomment-1781505031
https://api.github.com/repos/huggingface/datasets/issues/6355
IC_kwDODunzps5qL5wH
https://api.github.com/repos/huggingface/datasets/issues/comments/1781505031/reactions
2023-10-30T17:33:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1781505031
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006941 / 0.011353 (-0.004412) | 0.004255 / 0.011008 (-0.006753) | 0.085237 / 0.038508 (0.046729) | 0.080962 / 0.023109 (0.057853) | 0.312016 / 0.275898 (0.036118) | 0.353161 / 0.323480 (0.029681) | 0.005756 / 0.007986 (-0.002230) | 0.003591 / 0.004328 (-0.000738) | 0.065416 / 0.004250 (0.061166) | 0.057837 / 0.037052 (0.020785) | 0.316169 / 0.258489 (0.057680) | 0.372345 / 0.293841 (0.078504) | 0.031958 / 0.128546 (-0.096588) | 0.008798 / 0.075646 (-0.066848) | 0.294764 / 0.419271 (-0.124507) | 0.053954 / 0.043533 (0.010421) | 0.310961 / 0.255139 (0.055822) | 0.330063 / 0.283200 (0.046864) | 0.025298 / 0.141683 (-0.116385) | 1.454715 / 1.452155 (0.002560) | 1.557915 / 1.492716 (0.065198) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.274830 / 0.018006 (0.256824) | 0.565890 / 0.000490 (0.565400) | 0.009242 / 0.000200 (0.009042) | 0.000321 / 0.000054 (0.000266) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031092 / 0.037411 (-0.006320) | 0.087558 / 0.014526 (0.073033) | 0.103395 / 0.176557 (-0.073162) | 0.160078 / 0.737135 (-0.577057) | 0.102356 / 0.296338 (-0.193983) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.402912 / 0.215209 (0.187703) | 4.029374 / 2.077655 (1.951719) | 2.048237 / 1.504120 (0.544117) | 1.887470 / 1.541195 (0.346276) | 1.994807 / 1.468490 (0.526316) | 0.491109 / 4.584777 (-4.093668) | 3.645059 / 3.745712 (-0.100653) | 3.516376 / 5.269862 (-1.753486) | 2.103267 / 4.565676 (-2.462409) | 0.058072 / 0.424275 (-0.366203) | 0.007796 / 0.007607 (0.000189) | 0.480544 / 0.226044 (0.254499) | 4.795422 / 2.268929 (2.526494) | 2.507770 / 55.444624 (-52.936854) | 2.187106 / 6.876477 (-4.689371) | 2.271005 / 2.142072 (0.128933) | 0.585376 / 4.805227 (-4.219851) | 0.134741 / 6.500664 (-6.365923) | 0.060684 / 0.075469 (-0.014785) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.264349 / 1.841788 (-0.577439) | 19.448735 / 8.074308 (11.374427) | 14.521197 / 10.191392 (4.329805) | 0.167295 / 0.680424 (-0.513129) | 0.018352 / 0.534201 (-0.515849) | 0.396345 / 0.579283 (-0.182938) | 0.418690 / 0.434364 (-0.015674) | 0.469703 / 0.540337 (-0.070635) | 0.637852 / 1.386936 (-0.749084) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006939 / 0.011353 (-0.004414) | 0.004196 / 0.011008 (-0.006812) | 0.064719 / 0.038508 (0.026211) | 0.077517 / 0.023109 (0.054407) | 0.401977 / 0.275898 (0.126079) | 0.431089 / 0.323480 (0.107609) | 0.005624 / 0.007986 (-0.002362) | 0.003680 / 0.004328 (-0.000649) | 0.065817 / 0.004250 (0.061567) | 0.058297 / 0.037052 (0.021245) | 0.399614 / 0.258489 (0.141125) | 0.440089 / 0.293841 (0.146248) | 0.032492 / 0.128546 (-0.096054) | 0.008974 / 0.075646 (-0.066672) | 0.071311 / 0.419271 (-0.347961) | 0.048001 / 0.043533 (0.004468) | 0.394763 / 0.255139 (0.139624) | 0.416754 / 0.283200 (0.133554) | 0.023730 / 0.141683 (-0.117953) | 1.509677 / 1.452155 (0.057522) | 1.605711 / 1.492716 (0.112994) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.265490 / 0.018006 (0.247483) | 0.561745 / 0.000490 (0.561255) | 0.004616 / 0.000200 (0.004417) | 0.000105 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033371 / 0.037411 (-0.004040) | 0.092763 / 0.014526 (0.078238) | 0.108905 / 0.176557 (-0.067652) | 0.160380 / 0.737135 (-0.576756) | 0.106968 / 0.296338 (-0.189370) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.430268 / 0.215209 (0.215059) | 4.299313 / 2.077655 (2.221658) | 2.308971 / 1.504120 (0.804851) | 2.155855 / 1.541195 (0.614661) | 2.392698 / 1.468490 (0.924208) | 0.498464 / 4.584777 (-4.086313) | 3.694473 / 3.745712 (-0.051239) | 3.409625 / 5.269862 (-1.860236) | 2.106144 / 4.565676 (-2.459532) | 0.058992 / 0.424275 (-0.365283) | 0.007395 / 0.007607 (-0.000212) | 0.511291 / 0.226044 (0.285247) | 5.101806 / 2.268929 (2.832877) | 2.853100 / 55.444624 (-52.591524) | 2.527216 / 6.876477 (-4.349260) | 2.819380 / 2.142072 (0.677308) | 0.635155 / 4.805227 (-4.170072) | 0.135816 / 6.500664 (-6.364848) | 0.062056 / 0.075469 (-0.013413) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.353479 / 1.841788 (-0.488308) | 20.318513 / 8.074308 (12.244205) | 15.105336 / 10.191392 (4.913944) | 0.166186 / 0.680424 (-0.514238) | 0.020742 / 0.534201 (-0.513459) | 0.399286 / 0.579283 (-0.179997) | 0.431785 / 0.434364 (-0.002579) | 0.478667 / 0.540337 (-0.061671) | 0.654683 / 1.386936 (-0.732253) |

</details>
</details>

![](https://cml.dev/watermark.png#b39d1ce0b8f231649752f28cb724971f4df1c7ae "CML watermark")

2023-10-26T17:02:18Z
https://github.com/huggingface/datasets/pull/6355#issuecomment-1781505892
https://api.github.com/repos/huggingface/datasets/issues/6355
IC_kwDODunzps5qL59k
https://api.github.com/repos/huggingface/datasets/issues/comments/1781505892/reactions
2023-10-26T17:02:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1781505892
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
MEMBER
Yea I think some of it should be in the Hub docs indeed, let me open a new PR there.

Then I'll update the `datasets` docs anyway to avoid redundant stuff and add redirects instead
2023-10-30T17:32:57Z
https://github.com/huggingface/datasets/pull/6355#issuecomment-1785725325
https://api.github.com/repos/huggingface/datasets/issues/6355
IC_kwDODunzps5qcAGN
https://api.github.com/repos/huggingface/datasets/issues/comments/1785725325/reactions
2023-10-30T17:32:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1785725325
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Let's have more hub-centric documentation in the datasets docs

Tutorials
- Add “Configure the dataset viewer” page
- Change order:
  - Overview
    - and more focused on the Hub rather than the library
  - Then all the hub related things
    - and mention how to read/write with other tools like pandas
  - Then all the datasets lib related things in a subsection

Also:
- Rename “know your dataset” page to “Explore your dataset”
- Remove “Evaluate Predictions” page since it's 'evaluate' stuff (or move to legacy section ?)

TODO:
- [ ] write the “Configure the dataset viewer” page