IterableDataset columns and feature types
MEMBER
#self-assign
2022-11-23T15:31:02Z
https://github.com/huggingface/datasets/issues/3888#issuecomment-1325258716
https://api.github.com/repos/huggingface/datasets/issues/3888
IC_kwDODunzps5O_dfc
https://api.github.com/repos/huggingface/datasets/issues/comments/1325258716/reactions
2022-11-23T15:31:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1325258716
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
COLLABORATOR
@alvarobartt I've assigned you the issue since I'm not actively working on it.
2022-11-23T15:41:58Z
https://github.com/huggingface/datasets/issues/3888#issuecomment-1325277124
https://api.github.com/repos/huggingface/datasets/issues/3888
IC_kwDODunzps5O_h_E
https://api.github.com/repos/huggingface/datasets/issues/comments/1325277124/reactions
2022-11-23T15:41:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1325277124
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Cool thanks @mariosasko I'll try to fix it in the upcoming days, thanks!
2022-11-23T15:43:17Z
https://github.com/huggingface/datasets/issues/3888#issuecomment-1325278979
https://api.github.com/repos/huggingface/datasets/issues/3888
IC_kwDODunzps5O_icD
https://api.github.com/repos/huggingface/datasets/issues/comments/1325278979/reactions
2022-11-23T15:43:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1325278979
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
@lhoestq so in order to address what’s not completed in this issue, do you think it makes sense to add a param `features` to `IterableDataset.map` so that the output features right after the `map` are defined there? 
2022-11-28T15:47:37Z
https://github.com/huggingface/datasets/issues/3888#issuecomment-1329327570
https://api.github.com/repos/huggingface/datasets/issues/3888
IC_kwDODunzps5PO-3S
https://api.github.com/repos/huggingface/datasets/issues/comments/1329327570/reactions
2022-11-28T15:47:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1329327570
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
Yes that would be ideal IMO, thanks again for the help :)
2022-11-28T15:58:54Z
https://github.com/huggingface/datasets/issues/3888#issuecomment-1329341884
https://api.github.com/repos/huggingface/datasets/issues/3888
IC_kwDODunzps5PPCW8
https://api.github.com/repos/huggingface/datasets/issues/comments/1329341884/reactions
2022-11-28T15:58:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1329341884
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq cool then if you agree I can work on that! I’ll also update the docs accordingly once done, thanks!
2022-11-28T17:18:52Z
https://github.com/huggingface/datasets/issues/3888#issuecomment-1329463131
https://api.github.com/repos/huggingface/datasets/issues/3888
IC_kwDODunzps5PPf9b
https://api.github.com/repos/huggingface/datasets/issues/comments/1329463131/reactions
2022-11-28T17:18:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1329463131
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
I've already started with a PR as a draft @lhoestq, should we also try to look for a way to explicitly request pre-fetching right after a map operation is applied, so that the features are inferred if the user says explicitly so? Thanks!
2022-11-29T11:16:57Z
https://github.com/huggingface/datasets/issues/3888#issuecomment-1330470387
https://api.github.com/repos/huggingface/datasets/issues/3888
IC_kwDODunzps5PTV3z
https://api.github.com/repos/huggingface/datasets/issues/comments/1330470387/reactions
2022-11-29T11:16:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1330470387
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
> should we also try to look for a way to explicitly request pre-fetching right after a map operation is applied, so that the features are inferred if the user says explicitly so?

Right now one can use `ds = ds._resolve_features()` do to so. It can be used after `map` or `load_dataset` if the features are not known. Maybe we can make this method public ?
2022-11-29T11:39:24Z
https://github.com/huggingface/datasets/issues/3888#issuecomment-1330495533
https://api.github.com/repos/huggingface/datasets/issues/3888
IC_kwDODunzps5PTcAt
https://api.github.com/repos/huggingface/datasets/issues/comments/1330495533/reactions
2022-11-29T11:39:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1330495533
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Right now, an IterableDataset (e.g. when streaming a dataset) doesn't require to know the list of columns it contains, nor their types: `my_iterable_dataset.features` may be `None`

However it's often interesting to know the column types and types. This helps knowing what's inside your dataset without having to manually check a few examples, and this is useful to prepare a processing pipeline or to train models.

Here are a few cases that lead to `features` being `None`:

1. when loading a dataset with `load_dataset` on CSV, JSON Lines, etc. files: type inference is only done when iterating over the dataset
2. when calling `map`, because we don't know in advance what's the output of the user's function passed to `map`
3. when calling `rename_columns`, `remove_columns`, etc. because they rely on `map`

Things we can consider, for each point above:

1.a infer the type automatically from the first samples on the dataset using prefetching, when the dataset builder doesn't provide the `features`
2.a allow the user to specify the `features` as an argument to `map` (this would be consistent with the non-streaming API)
2.b prefetch the first output value to infer the type
3.a don't rely on `map` directly and reuse the previous `features` and rename/remove the corresponding ones

The thing is that prefetching can take a few seconds, while the operations above are instantaneous since no data are downloaded. Therefore I'm not sure whether this solution may be worth it. Maybe prefetching could also be done when explicitly asked by the user

cc @mariosasko @albertvillanova 