Add IterableDataset `__repr__`
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6480). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2023-12-07T16:41:34Z
https://github.com/huggingface/datasets/pull/6480#issuecomment-1845676565
https://api.github.com/repos/huggingface/datasets/issues/6480
IC_kwDODunzps5uAsoV
https://api.github.com/repos/huggingface/datasets/issues/comments/1845676565/reactions
2023-12-07T16:41:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1845676565
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005392 / 0.011353 (-0.005960) | 0.003120 / 0.011008 (-0.007888) | 0.062017 / 0.038508 (0.023509) | 0.048824 / 0.023109 (0.025715) | 0.232300 / 0.275898 (-0.043598) | 0.262045 / 0.323480 (-0.061435) | 0.002909 / 0.007986 (-0.005077) | 0.003916 / 0.004328 (-0.000413) | 0.049469 / 0.004250 (0.045218) | 0.038965 / 0.037052 (0.001913) | 0.247841 / 0.258489 (-0.010648) | 0.268259 / 0.293841 (-0.025582) | 0.027588 / 0.128546 (-0.100958) | 0.010334 / 0.075646 (-0.065312) | 0.205811 / 0.419271 (-0.213460) | 0.035456 / 0.043533 (-0.008077) | 0.242774 / 0.255139 (-0.012365) | 0.260377 / 0.283200 (-0.022823) | 0.017469 / 0.141683 (-0.124214) | 1.199665 / 1.452155 (-0.252489) | 1.259316 / 1.492716 (-0.233400) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092357 / 0.018006 (0.074350) | 0.303745 / 0.000490 (0.303255) | 0.000212 / 0.000200 (0.000012) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018820 / 0.037411 (-0.018592) | 0.061548 / 0.014526 (0.047022) | 0.072527 / 0.176557 (-0.104030) | 0.119696 / 0.737135 (-0.617440) | 0.074153 / 0.296338 (-0.222185) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283952 / 0.215209 (0.068743) | 2.769844 / 2.077655 (0.692189) | 1.526100 / 1.504120 (0.021980) | 1.417584 / 1.541195 (-0.123611) | 1.440523 / 1.468490 (-0.027967) | 0.556994 / 4.584777 (-4.027783) | 2.400392 / 3.745712 (-1.345320) | 2.727794 / 5.269862 (-2.542068) | 1.724671 / 4.565676 (-2.841006) | 0.062111 / 0.424275 (-0.362164) | 0.004925 / 0.007607 (-0.002682) | 0.342748 / 0.226044 (0.116704) | 3.376790 / 2.268929 (1.107862) | 1.856498 / 55.444624 (-53.588127) | 1.574143 / 6.876477 (-5.302334) | 1.591828 / 2.142072 (-0.550245) | 0.644416 / 4.805227 (-4.160811) | 0.116862 / 6.500664 (-6.383802) | 0.041484 / 0.075469 (-0.033985) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975704 / 1.841788 (-0.866084) | 11.196447 / 8.074308 (3.122139) | 10.567518 / 10.191392 (0.376126) | 0.126786 / 0.680424 (-0.553638) | 0.013768 / 0.534201 (-0.520433) | 0.284531 / 0.579283 (-0.294752) | 0.260855 / 0.434364 (-0.173509) | 0.328888 / 0.540337 (-0.211450) | 0.439911 / 1.386936 (-0.947025) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005108 / 0.011353 (-0.006245) | 0.003006 / 0.011008 (-0.008003) | 0.048673 / 0.038508 (0.010165) | 0.051066 / 0.023109 (0.027957) | 0.279578 / 0.275898 (0.003680) | 0.298356 / 0.323480 (-0.025123) | 0.003965 / 0.007986 (-0.004020) | 0.002662 / 0.004328 (-0.001667) | 0.049037 / 0.004250 (0.044786) | 0.039385 / 0.037052 (0.002333) | 0.284545 / 0.258489 (0.026055) | 0.314240 / 0.293841 (0.020399) | 0.028493 / 0.128546 (-0.100053) | 0.010400 / 0.075646 (-0.065247) | 0.057375 / 0.419271 (-0.361896) | 0.032382 / 0.043533 (-0.011151) | 0.283163 / 0.255139 (0.028024) | 0.298967 / 0.283200 (0.015768) | 0.017564 / 0.141683 (-0.124119) | 1.172425 / 1.452155 (-0.279730) | 1.219975 / 1.492716 (-0.272742) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090664 / 0.018006 (0.072658) | 0.298419 / 0.000490 (0.297929) | 0.000211 / 0.000200 (0.000011) | 0.000045 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021739 / 0.037411 (-0.015672) | 0.068274 / 0.014526 (0.053748) | 0.080820 / 0.176557 (-0.095736) | 0.119809 / 0.737135 (-0.617326) | 0.081612 / 0.296338 (-0.214727) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.303346 / 0.215209 (0.088137) | 2.971648 / 2.077655 (0.893993) | 1.634828 / 1.504120 (0.130708) | 1.510851 / 1.541195 (-0.030344) | 1.515236 / 1.468490 (0.046745) | 0.558487 / 4.584777 (-4.026289) | 2.436263 / 3.745712 (-1.309449) | 2.718525 / 5.269862 (-2.551336) | 1.727421 / 4.565676 (-2.838255) | 0.061396 / 0.424275 (-0.362879) | 0.004951 / 0.007607 (-0.002656) | 0.352950 / 0.226044 (0.126906) | 3.473766 / 2.268929 (1.204838) | 1.971299 / 55.444624 (-53.473325) | 1.712173 / 6.876477 (-5.164304) | 1.711334 / 2.142072 (-0.430738) | 0.627291 / 4.805227 (-4.177936) | 0.113779 / 6.500664 (-6.386885) | 0.046561 / 0.075469 (-0.028908) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.989507 / 1.841788 (-0.852280) | 11.777883 / 8.074308 (3.703575) | 10.525453 / 10.191392 (0.334061) | 0.129118 / 0.680424 (-0.551306) | 0.014989 / 0.534201 (-0.519212) | 0.282324 / 0.579283 (-0.296959) | 0.280688 / 0.434364 (-0.153676) | 0.322579 / 0.540337 (-0.217758) | 0.554327 / 1.386936 (-0.832609) |

</details>
</details>

![](https://cml.dev/watermark.png#79e94fcdf3d4378ddcdf7e130bb1ae23d99c6fce "CML watermark")

2023-12-08T13:33:06Z
https://github.com/huggingface/datasets/pull/6480#issuecomment-1847172665
https://api.github.com/repos/huggingface/datasets/issues/6480
IC_kwDODunzps5uGZ45
Iv1.05c79e9ad1f6bdfa
2018-07-30T09:30:17Z
Automate your workflow from idea to production
branch_protection_rule check_run check_suite create delete deployment deployment_status discussion discussion_comment fork gollum issues issue_comment label merge_group milestone page_build project project_card project_column public pull_request pull_request_review pull_request_review_comment push registry_package release repository repository_dispatch status watch workflow_dispatch workflow_run
https://help.github.com/en/actions
https://github.com/apps/github-actions
GitHub Actions
MDM6QXBwMTUzNjg=
https://avatars.githubusercontent.com/u/9919?v=4
https://api.github.com/users/github/events{/privacy}
https://api.github.com/users/github/followers
https://api.github.com/users/github/following{/other_user}
https://api.github.com/users/github/gists{/gist_id}
https://github.com/github
github
MDEyOk9yZ2FuaXphdGlvbjk5MTk=
https://api.github.com/users/github/orgs
https://api.github.com/users/github/received_events
https://api.github.com/users/github/repos
https://api.github.com/users/github/starred{/owner}{/repo}
https://api.github.com/users/github/subscriptions
Organization
https://api.github.com/users/github
public
write
read
write
write
write
write
write
write
write
read
read
write
write
write
write
write
write
write
read
github-actions
2025-03-07T16:35:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1847172665/reactions
2023-12-08T13:33:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1847172665
https://avatars.githubusercontent.com/in/15368?v=4
https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}
https://api.github.com/users/github-actions%5Bbot%5D/followers
https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}
https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}
https://github.com/apps/github-actions
github-actions[bot]
MDM6Qm90NDE4OTgyODI=
https://api.github.com/users/github-actions%5Bbot%5D/orgs
https://api.github.com/users/github-actions%5Bbot%5D/received_events
https://api.github.com/users/github-actions%5Bbot%5D/repos
https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}
https://api.github.com/users/github-actions%5Bbot%5D/subscriptions
Bot
https://api.github.com/users/github-actions%5Bbot%5D
public
Example for glue sst2:

Dataset

```
DatasetDict({
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1821
    })
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 872
    })
})
```

IterableDataset (new)

```
IterableDatasetDict({
    test: IterableDataset({
        features: ['sentence', 'label', 'idx'],
        n_shards: 1
    })
    train: IterableDataset({
        features: ['sentence', 'label', 'idx'],
        n_shards: 1
    })
    validation: IterableDataset({
        features: ['sentence', 'label', 'idx'],
        n_shards: 1
    })
})
```

IterableDataset (before)

```
{'test': <datasets.iterable_dataset.IterableDataset object at 0x130d421f0>, 'train': <datasets.iterable_dataset.IterableDataset object at 0x136f3aaf0>, 'validation': <datasets.iterable_dataset.IterableDataset object at 0x136f4b100>}
{'sentence': 'hide new secretions from the parental units ', 'label': 0, 'idx': 0}
```