Improve dataset `.skip()` speed in streaming mode
COLLABORATOR
Hi! I agree `skip` can be inefficient to use in the current state.

To make it fast, we could use "statistics" stored in Parquet metadata and read only the chunks needed to form a dataset. 

And thanks to the "datasets-server" project, which aims to store the Parquet versions of the Hub datasets (only the smaller datasets are covered currently), this solution can also be applied to datasets stored in formats other than Parquet. (cc @severo)
2022-12-20T18:19:36Z
https://github.com/huggingface/datasets/issues/5380#issuecomment-1359951367
https://api.github.com/repos/huggingface/datasets/issues/5380
IC_kwDODunzps5RDzYH
https://api.github.com/repos/huggingface/datasets/issues/comments/1359951367/reactions
2022-12-20T18:19:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1359951367
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
COLLABORATOR
@mariosasko do the current parquet files created by the datasets-server already have the required "statistics"? If not, please open an issue on https://github.com/huggingface/datasets-server with some details to make sure we implement it.
2022-12-20T20:22:35Z
https://github.com/huggingface/datasets/issues/5380#issuecomment-1360132667
https://api.github.com/repos/huggingface/datasets/issues/5380
IC_kwDODunzps5REfo7
https://api.github.com/repos/huggingface/datasets/issues/comments/1360132667/reactions
2022-12-20T20:22:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1360132667
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
COLLABORATOR
Yes, nothing has to be changed on the datasets-server side. What I mean by "statistics" is that we can use the "row_group" metadata embedded in a Parquet file (by default) to fetch the requested rows more efficiently.
2022-12-21T17:43:58Z
https://github.com/huggingface/datasets/issues/5380#issuecomment-1361746042
https://api.github.com/repos/huggingface/datasets/issues/5380
IC_kwDODunzps5RKph6
https://api.github.com/repos/huggingface/datasets/issues/comments/1361746042/reactions
2022-12-21T17:43:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1361746042
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Glad to see the feature could be of interest. 

I'm sure there are many possible ways to implement this feature. I don't know enough about the datasets-server, but I guess that it is not instantaneous, in the sense that user-owned private datasets might need hours or days until they are ported to the datasets-server (if at all), which could be cumbersome. Having optionally that information in the `dataset_infos.json` file would make it easier for users to control the skip process a bit.
2022-12-27T12:23:50Z
https://github.com/huggingface/datasets/issues/5380#issuecomment-1365865553
https://api.github.com/repos/huggingface/datasets/issues/5380
IC_kwDODunzps5RaXRR
https://api.github.com/repos/huggingface/datasets/issues/comments/1365865553/reactions
2022-12-27T12:23:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1365865553
https://avatars.githubusercontent.com/u/173537?v=4
https://api.github.com/users/versae/events{/privacy}
https://api.github.com/users/versae/followers
https://api.github.com/users/versae/following{/other_user}
https://api.github.com/users/versae/gists{/gist_id}
https://github.com/versae
versae
MDQ6VXNlcjE3MzUzNw==
https://api.github.com/users/versae/orgs
https://api.github.com/users/versae/received_events
https://api.github.com/users/versae/repos
https://api.github.com/users/versae/starred{/owner}{/repo}
https://api.github.com/users/versae/subscriptions
User
https://api.github.com/users/versae
public
COLLABORATOR
re: statistics:

- https://arrow.apache.org/docs/python/generated/pyarrow.parquet.FileMetaData.html
- https://arrow.apache.org/docs/python/generated/pyarrow.parquet.RowGroupMetaData.html

```python
>>> import pyarrow.parquet as pq
>>> import hffs
>>> fs = hffs.HfFileSystem("glue", repo_type="dataset", revision="refs/convert/parquet")
>>> metadata = pq.read_metadata("ax/glue-test.parquet", filesystem=fs)
>>> metadata
<pyarrow._parquet.FileMetaData object at 0x7f4537cec400>
  created_by: parquet-cpp-arrow version 7.0.0
  num_columns: 4
  num_rows: 1104
  num_row_groups: 2
  format_version: 1.0
  serialized_size: 2902
>>> metadata.row_group(0)
<pyarrow._parquet.RowGroupMetaData object at 0x7f45564bcbd0>
  num_columns: 4
  num_rows: 1000
  total_byte_size: 164474
>>> metadata.row_group(1)
<pyarrow._parquet.RowGroupMetaData object at 0x7f455005c400>
  num_columns: 4
  num_rows: 104
  total_byte_size: 13064
```
2022-12-27T13:07:18Z
https://github.com/huggingface/datasets/issues/5380#issuecomment-1365888835
https://api.github.com/repos/huggingface/datasets/issues/5380
IC_kwDODunzps5Rac9D
https://api.github.com/repos/huggingface/datasets/issues/comments/1365888835/reactions
2022-12-27T13:07:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1365888835
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
COLLABORATOR
> user-owned private datasets might need hours or days until they are ported to the datasets-server (if at all)

private datasets are not supported yet (https://github.com/huggingface/datasets-server/issues/39)
2022-12-27T13:08:46Z
https://github.com/huggingface/datasets/issues/5380#issuecomment-1365889825
https://api.github.com/repos/huggingface/datasets/issues/5380
IC_kwDODunzps5RadMh
https://api.github.com/repos/huggingface/datasets/issues/comments/1365889825/reactions
2022-12-27T13:08:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1365889825
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
COLLABORATOR
@versae `Dataset.push_to_hub` writes shards in Parquet, so this solution would also work for such datasets (immediately after the push). 
2023-01-13T13:34:48Z
https://github.com/huggingface/datasets/issues/5380#issuecomment-1381857857
https://api.github.com/repos/huggingface/datasets/issues/5380
IC_kwDODunzps5SXXpB
https://api.github.com/repos/huggingface/datasets/issues/comments/1381857857/reactions
2023-01-13T13:34:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1381857857
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
@mariosasko that is right. However, there are still a good amount of datasets for which the shards are created manually. In our very specific case, we create medium-sized datasets (rarely over 100-200GB) of both text and audio, we prepare the shards by hand and then upload then. It would be great to have immediate access to this download skipping feature for them too.
2023-01-17T08:44:56Z
https://github.com/huggingface/datasets/issues/5380#issuecomment-1385033568
https://api.github.com/repos/huggingface/datasets/issues/5380
IC_kwDODunzps5Sje9g
https://api.github.com/repos/huggingface/datasets/issues/comments/1385033568/reactions
2023-01-17T08:44:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1385033568
https://avatars.githubusercontent.com/u/173537?v=4
https://api.github.com/users/versae/events{/privacy}
https://api.github.com/users/versae/followers
https://api.github.com/users/versae/following{/other_user}
https://api.github.com/users/versae/gists{/gist_id}
https://github.com/versae
versae
MDQ6VXNlcjE3MzUzNw==
https://api.github.com/users/versae/orgs
https://api.github.com/users/versae/received_events
https://api.github.com/users/versae/repos
https://api.github.com/users/versae/starred{/owner}{/repo}
https://api.github.com/users/versae/subscriptions
User
https://api.github.com/users/versae
public
COLLABORATOR
From looking at Arrow's source, it seems Parquet stores metadata at the end, which means one needs to iterate over a Parquet file's data before accessing its metadata. We could mimic Dask to address this "limitation" and write metadata in a `_metadata`/`_common_metadata` file in `to_parquet`/`push_to_hub`, which we could then use to optimize reads (if present). Plus, it's handy that PyArrow can also parse these metadata files.
2023-02-24T19:06:38Z
https://github.com/huggingface/datasets/issues/5380#issuecomment-1444281177
https://api.github.com/repos/huggingface/datasets/issues/5380
IC_kwDODunzps5WFftZ
https://api.github.com/repos/huggingface/datasets/issues/comments/1444281177/reactions
2023-02-26T15:03:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1444281177
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
So if Parquet metadata needs to be in its own file anyway, why not implement this skipping feature by storing the example counts per shard in `dataset_infos.json`? That would allow:
- Support both private and public datasets
- Immediate access to the feature upon uploading of shards
- Use any dataset, not only those uploaded using `.push_to_hub()`

A proper Parquet metadata file could still be created and "overwrite" the `dataset_infos.json` info in the datasets-server.
2023-03-03T11:00:27Z
https://github.com/huggingface/datasets/issues/5380#issuecomment-1453354304
https://api.github.com/repos/huggingface/datasets/issues/5380
IC_kwDODunzps5WoG1A
https://api.github.com/repos/huggingface/datasets/issues/comments/1453354304/reactions
2023-03-08T10:47:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1453354304
https://avatars.githubusercontent.com/u/173537?v=4
https://api.github.com/users/versae/events{/privacy}
https://api.github.com/users/versae/followers
https://api.github.com/users/versae/following{/other_user}
https://api.github.com/users/versae/gists{/gist_id}
https://github.com/versae
versae
MDQ6VXNlcjE3MzUzNw==
https://api.github.com/users/versae/orgs
https://api.github.com/users/versae/received_events
https://api.github.com/users/versae/repos
https://api.github.com/users/versae/starred{/owner}{/repo}
https://api.github.com/users/versae/subscriptions
User
https://api.github.com/users/versae
public
### Feature request

Add extra information to the `dataset_infos.json` file to include the number of samples/examples in each shard, for example in a new field `num_examples` alongside `num_bytes`. The `.skip()` function could use this information to ignore the download of a shard when in streaming mode, which AFAICT it should speed up the skipping process.

### Motivation

When resuming from a checkpoint after a crashed run, using `dataset.skip()` is very convenient to recover the exact state of the data and to not train again over the same examples (assuming same seed, no shuffling). However, I have noticed that for audio datasets in streaming mode this is very costly in terms of time, as shards need to be downloaded every time before skipping the right number of examples.

### Your contribution

I took a look already at the code, but it seems a change like this is way deeper than I am able to manage, as it touches the library in several parts. I could give it a try but might need some guidance on the internals.