Features of IterableDataset set to None by remove column
MEMBER
Related to https://github.com/huggingface/datasets/issues/5245
2022-11-23T13:48:49Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1325093384
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5O-1II
https://api.github.com/repos/huggingface/datasets/issues/comments/1325093384/reactions
2022-11-23T13:48:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1325093384
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
#self-assign
2022-11-23T15:30:35Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1325258082
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5O_dVi
https://api.github.com/repos/huggingface/datasets/issues/comments/1325258082/reactions
2022-11-23T15:30:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1325258082
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
CONTRIBUTOR
Thanks @lhoestq and @alvarobartt!

This would be extremely helpful to have working for the Whisper fine-tuning event - we're **only** training using streaming mode, so it'll be quite important to have this feature working to make training as easy as possible!

_c.f._ https://twitter.com/sanchitgandhi99/status/1592188332171493377
2022-11-25T16:35:03Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1327683379
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5PItcz
https://api.github.com/repos/huggingface/datasets/issues/comments/1327683379/reactions
2022-11-25T16:37:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1327683379
https://avatars.githubusercontent.com/u/93869735?v=4
https://api.github.com/users/sanchit-gandhi/events{/privacy}
https://api.github.com/users/sanchit-gandhi/followers
https://api.github.com/users/sanchit-gandhi/following{/other_user}
https://api.github.com/users/sanchit-gandhi/gists{/gist_id}
https://github.com/sanchit-gandhi
sanchit-gandhi
U_kgDOBZhWpw
https://api.github.com/users/sanchit-gandhi/orgs
https://api.github.com/users/sanchit-gandhi/received_events
https://api.github.com/users/sanchit-gandhi/repos
https://api.github.com/users/sanchit-gandhi/starred{/owner}{/repo}
https://api.github.com/users/sanchit-gandhi/subscriptions
User
https://api.github.com/users/sanchit-gandhi
public
MEMBER
> Thanks @lhoestq and @alvarobartt!
> 
> 
> 
> This would be extremely helpful to have working for the Whisper fine-tuning event - we're **only** training using streaming mode, so it'll be quite important to have this feature working to make training as easy as possible!
> 
> 
> 
> _c.f._ https://twitter.com/sanchitgandhi99/status/1592188332171493377

I'm almost done with at least a temporary fix to `rename_column`, `rename_columns`, and `remove_columns`, just trying to figure out how to extend it to the `map` function itself!

I'll probably open the PR for review either tomorrow or Sunday hopefully! Glad I can help you and HuggingFace 🤗 
2022-11-25T16:40:14Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1327687586
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5PIuei
https://api.github.com/repos/huggingface/datasets/issues/comments/1327687586/reactions
2022-11-25T16:40:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1327687586
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
CONTRIBUTOR
Awesome - thank you so much for this PR @alvarobartt! Is much appreciated!
2022-11-25T16:43:57Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1327690007
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5PIvEX
https://api.github.com/repos/huggingface/datasets/issues/comments/1327690007/reactions
2022-11-25T16:43:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1327690007
https://avatars.githubusercontent.com/u/93869735?v=4
https://api.github.com/users/sanchit-gandhi/events{/privacy}
https://api.github.com/users/sanchit-gandhi/followers
https://api.github.com/users/sanchit-gandhi/following{/other_user}
https://api.github.com/users/sanchit-gandhi/gists{/gist_id}
https://github.com/sanchit-gandhi
sanchit-gandhi
U_kgDOBZhWpw
https://api.github.com/users/sanchit-gandhi/orgs
https://api.github.com/users/sanchit-gandhi/received_events
https://api.github.com/users/sanchit-gandhi/repos
https://api.github.com/users/sanchit-gandhi/starred{/owner}{/repo}
https://api.github.com/users/sanchit-gandhi/subscriptions
User
https://api.github.com/users/sanchit-gandhi
public
MEMBER
@sanchit-gandhi PR is ready and open for review at #5287, but there's still one issue I may need @lhoestq's input :hugs:
2022-11-26T10:07:58Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1328018516
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5PJ_RU
https://api.github.com/repos/huggingface/datasets/issues/comments/1328018516/reactions
2022-11-26T10:07:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1328018516
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
Let us know @sanchit-gandhi if you need a new release of `datasets` soon with this fix included :)
2022-11-28T12:55:32Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1329044964
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5PN53k
https://api.github.com/repos/huggingface/datasets/issues/comments/1329044964/reactions
2022-11-28T12:55:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1329044964
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks for the fix guys! We can direct people to install `datasets` from main if that's easier!
2022-11-28T15:18:08Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1329281585
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5POzox
https://api.github.com/repos/huggingface/datasets/issues/comments/1329281585/reactions
2022-11-28T15:18:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1329281585
https://avatars.githubusercontent.com/u/93869735?v=4
https://api.github.com/users/sanchit-gandhi/events{/privacy}
https://api.github.com/users/sanchit-gandhi/followers
https://api.github.com/users/sanchit-gandhi/following{/other_user}
https://api.github.com/users/sanchit-gandhi/gists{/gist_id}
https://github.com/sanchit-gandhi
sanchit-gandhi
U_kgDOBZhWpw
https://api.github.com/users/sanchit-gandhi/orgs
https://api.github.com/users/sanchit-gandhi/received_events
https://api.github.com/users/sanchit-gandhi/repos
https://api.github.com/users/sanchit-gandhi/starred{/owner}{/repo}
https://api.github.com/users/sanchit-gandhi/subscriptions
User
https://api.github.com/users/sanchit-gandhi
public
NONE
Hey guys, any update around this? I'm facing the same issue with a streamable dataset. 
2023-02-02T04:03:47Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1413121546
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5UOoYK
https://api.github.com/repos/huggingface/datasets/issues/comments/1413121546/reactions
2023-02-02T04:03:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1413121546
https://avatars.githubusercontent.com/u/5939429?v=4
https://api.github.com/users/asennoussi/events{/privacy}
https://api.github.com/users/asennoussi/followers
https://api.github.com/users/asennoussi/following{/other_user}
https://api.github.com/users/asennoussi/gists{/gist_id}
https://github.com/asennoussi
asennoussi
MDQ6VXNlcjU5Mzk0Mjk=
https://api.github.com/users/asennoussi/orgs
https://api.github.com/users/asennoussi/received_events
https://api.github.com/users/asennoussi/repos
https://api.github.com/users/asennoussi/starred{/owner}{/repo}
https://api.github.com/users/asennoussi/subscriptions
User
https://api.github.com/users/asennoussi
public
MEMBER
Hi @asennoussi so this was already fixed and released as part of https://github.com/huggingface/datasets/releases/tag/2.8.0, so you should be able to install it as `pip install datasets==2.8.0` or just to use `pip install datasets --upgrade` to get the latest version, as of now, the https://github.com/huggingface/datasets/releases/tag/2.9.0 released last week! 🤗
2023-02-02T07:41:33Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1413280383
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5UPPJ_
https://api.github.com/repos/huggingface/datasets/issues/comments/1413280383/reactions
2023-02-02T07:41:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1413280383
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
NONE
Still facing the same issue though: 
```
from datasets import IterableDatasetDict, load_dataset

raw_datasets = vectorized_datasets = IterableDatasetDict()


raw_datasets["train"] = load_dataset("asennoussi/private", split="train", use_auth_token=True, streaming=True)
raw_datasets["test"] = load_dataset("asennoussi/private", split="test", use_auth_token=True, streaming=True)

print("Original features: ", raw_datasets['train'].features.keys())

...

def prepare_dataset(batch):

    # load and (possibly) resample audio datato 16kHz
    audio = batch["audio"]

    # compute log-Mel input features from input audio array 
    batch["input_features"] = processor.feature_extractor(audio["array"], sampling_rate=audio["sampling_rate"]).input_features[0]
    # compute input length of audio sample in seconds
    batch["input_length"] = len(audio["array"]) / audio["sampling_rate"]
    
    # optional pre-processing steps
    transcription = batch["sentence"]
    
    # encode target text to label ids
    batch["labels"] = processor.tokenizer(transcription).input_ids
    batch["labels_length"] = len(batch["labels"])
    return batch
...
vectorized_datasets = vectorized_datasets.remove_columns(['input_length', 'labels_length']+list(next(iter(raw_datasets.values())).features))
print("Processed features: ", vectorized_datasets['train'].features)
print("First sample:", next(iter(vectorized_datasets['train'])))

```

Output: 
```
Original features:  dict_keys(['path', 'audio', 'sentence'])
Processed features:  None
```
2023-02-02T07:54:40Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1413292374
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5UPSFW
https://api.github.com/repos/huggingface/datasets/issues/comments/1413292374/reactions
2023-02-02T07:54:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1413292374
https://avatars.githubusercontent.com/u/5939429?v=4
https://api.github.com/users/asennoussi/events{/privacy}
https://api.github.com/users/asennoussi/followers
https://api.github.com/users/asennoussi/following{/other_user}
https://api.github.com/users/asennoussi/gists{/gist_id}
https://github.com/asennoussi
asennoussi
MDQ6VXNlcjU5Mzk0Mjk=
https://api.github.com/users/asennoussi/orgs
https://api.github.com/users/asennoussi/received_events
https://api.github.com/users/asennoussi/repos
https://api.github.com/users/asennoussi/starred{/owner}{/repo}
https://api.github.com/users/asennoussi/subscriptions
User
https://api.github.com/users/asennoussi
public
MEMBER
Hmm weird, could you try to print

```python
print("Processed features: ", vectorized_datasets['train'].features)
```

again after iterating over the `vectorized_datasets`? In the code above, should be last line :)
2023-02-02T07:58:12Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1413295625
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5UPS4J
https://api.github.com/repos/huggingface/datasets/issues/comments/1413295625/reactions
2023-02-02T07:58:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1413295625
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
NONE
Didn't seem to fix it: 
```
Original features:  dict_keys(['path', 'audio', 'sentence'])
Processed features:  None
Processed features:  None
```
2023-02-02T08:05:08Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1413303319
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5UPUwX
https://api.github.com/repos/huggingface/datasets/issues/comments/1413303319/reactions
2023-02-02T08:05:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1413303319
https://avatars.githubusercontent.com/u/5939429?v=4
https://api.github.com/users/asennoussi/events{/privacy}
https://api.github.com/users/asennoussi/followers
https://api.github.com/users/asennoussi/following{/other_user}
https://api.github.com/users/asennoussi/gists{/gist_id}
https://github.com/asennoussi
asennoussi
MDQ6VXNlcjU5Mzk0Mjk=
https://api.github.com/users/asennoussi/orgs
https://api.github.com/users/asennoussi/received_events
https://api.github.com/users/asennoussi/repos
https://api.github.com/users/asennoussi/starred{/owner}{/repo}
https://api.github.com/users/asennoussi/subscriptions
User
https://api.github.com/users/asennoussi
public
NONE
Actually the culprit looks to be this one: 
`vectorized_datasets = raw_datasets.map(prepare_dataset).with_format("torch")`
When I remove this line: `vectorized_datasets = vectorized_datasets.remove_columns(['input_length', 'labels_length']+list(next(iter(raw_datasets.values())).features))`

I still get 
```
Processed features:  None
```
2023-02-02T08:07:13Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1413305772
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5UPVWs
https://api.github.com/repos/huggingface/datasets/issues/comments/1413305772/reactions
2023-02-02T08:07:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1413305772
https://avatars.githubusercontent.com/u/5939429?v=4
https://api.github.com/users/asennoussi/events{/privacy}
https://api.github.com/users/asennoussi/followers
https://api.github.com/users/asennoussi/following{/other_user}
https://api.github.com/users/asennoussi/gists{/gist_id}
https://github.com/asennoussi
asennoussi
MDQ6VXNlcjU5Mzk0Mjk=
https://api.github.com/users/asennoussi/orgs
https://api.github.com/users/asennoussi/received_events
https://api.github.com/users/asennoussi/repos
https://api.github.com/users/asennoussi/starred{/owner}{/repo}
https://api.github.com/users/asennoussi/subscriptions
User
https://api.github.com/users/asennoussi
public
NONE
The culprit is definitely `.map` 
Just validated it. 
Any idea please? 
2023-02-02T08:39:51Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1413343503
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5UPekP
https://api.github.com/repos/huggingface/datasets/issues/comments/1413343503/reactions
2023-02-02T08:39:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1413343503
https://avatars.githubusercontent.com/u/5939429?v=4
https://api.github.com/users/asennoussi/events{/privacy}
https://api.github.com/users/asennoussi/followers
https://api.github.com/users/asennoussi/following{/other_user}
https://api.github.com/users/asennoussi/gists{/gist_id}
https://github.com/asennoussi
asennoussi
MDQ6VXNlcjU5Mzk0Mjk=
https://api.github.com/users/asennoussi/orgs
https://api.github.com/users/asennoussi/received_events
https://api.github.com/users/asennoussi/repos
https://api.github.com/users/asennoussi/starred{/owner}{/repo}
https://api.github.com/users/asennoussi/subscriptions
User
https://api.github.com/users/asennoussi
public
MEMBER
> The culprit is definitely `.map` Just validated it. Any idea please?

Yes, indeed `.map` losses the features, because AFAIK pre-fetching the data to infer the features is expensive and not ideal, that's part of this issue https://github.com/huggingface/datasets/issues/3888

Anyway, now you can pass the `features` as a param to `.map` as follows:

```python
from datasets import Features
vectorized_datasets = raw_datasets.map(
    prepare_dataset,
    features=Features(
        {"path": raw_datasets["train"].info.features["path"], "audio": raw_datasets["train"].info.features["audio"], "sentence": raw_datasets["train"].info.features["sentence"]}
    ),
).with_format("torch")
```

Also, to let you know, when calling `.remove_columns` over an `IterableDataset`, the `features` are not lost, as well as `.rename_column` and `rename_columns` :)

More information about the latter at https://github.com/huggingface/datasets/pull/5287
2023-02-02T09:01:32Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1413369746
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5UPk-S
https://api.github.com/repos/huggingface/datasets/issues/comments/1413369746/reactions
2023-02-02T09:01:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1413369746
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
@asennoussi alternatively you can just call `._resolve_features()` from your `IterableDataset` and it will pre-fetch the data to resolve the features, but note that feature-inference is not as accurate as if you manually specify which features and feature-types the `IterableDataset` has, as mentioned in the comment above, the alternative is to provide `features` param to `.map` :hugs:
2023-02-02T09:04:42Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1413374606
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5UPmKO
https://api.github.com/repos/huggingface/datasets/issues/comments/1413374606/reactions
2023-02-02T09:04:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1413374606
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
NONE
Got it thanks a lot! 
2023-02-02T09:05:50Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-1413376420
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps5UPmmk
https://api.github.com/repos/huggingface/datasets/issues/comments/1413376420/reactions
2023-02-02T09:05:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1413376420
https://avatars.githubusercontent.com/u/5939429?v=4
https://api.github.com/users/asennoussi/events{/privacy}
https://api.github.com/users/asennoussi/followers
https://api.github.com/users/asennoussi/following{/other_user}
https://api.github.com/users/asennoussi/gists{/gist_id}
https://github.com/asennoussi
asennoussi
MDQ6VXNlcjU5Mzk0Mjk=
https://api.github.com/users/asennoussi/orgs
https://api.github.com/users/asennoussi/received_events
https://api.github.com/users/asennoussi/repos
https://api.github.com/users/asennoussi/starred{/owner}{/repo}
https://api.github.com/users/asennoussi/subscriptions
User
https://api.github.com/users/asennoussi
public
NONE
> > The culprit is definitely `.map` Just validated it. Any idea please?
> 
> Yes, indeed `.map` losses the features, because AFAIK pre-fetching the data to infer the features is expensive and not ideal, that's part of this issue [#3888](https://github.com/huggingface/datasets/issues/3888)
> 
> Anyway, now you can pass the `features` as a param to `.map` as follows:
> 
> from datasets import Features
> vectorized_datasets = raw_datasets.map(
>     prepare_dataset,
>     features=Features(
>         {"path": raw_datasets["train"].info.features["path"], "audio": raw_datasets["train"].info.features["audio"], "sentence": raw_datasets["train"].info.features["sentence"]}
>     ),
> ).with_format("torch")
> Also, to let you know, when calling `.remove_columns` over an `IterableDataset`, the `features` are not lost, as well as `.rename_column` and `rename_columns` :)
> 
> More information about the latter at [#5287](https://github.com/huggingface/datasets/pull/5287)

I am very late to the game, but facing the same issue still after almost 2 years. Our dataset type is IterableDatasetDict and after mapping we still get the none features. We needed the feature names to be able to in later stage use interleave_datasets() to combine two datasets together, otherwise we got errors. I just want to mention here, if somebody is using a function like prepare_dataset(batch), for whatever feature you produce here, you need to only cast these features manually to your features. Just as an example:

```python
ds = IterableDatasetDict()

ds["train"] = load_dataset(str(DATA.DIR), "default", split="train", trust_remote_code=True, streaming=True)

ds = ds.cast_column("audio_filepath", Audio(sampling_rate=None))

def prepare_dataset(batch):
            audio = batch["audio_filepath"]
            # compute log-Mel input features from input audio array
            batch["input_features"] = feature_extractor(
                audio["array"], sampling_rate=audio["sampling_rate"]
            ).input_features[0]
            # encode target text to label ids
            batch["labels"] = tokenizer(batch["text"]).input_ids
            return batch

add_new_features = Features({'input_features': Sequence(feature=Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), length=-1, id=None),
   'labels': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)}) # features you produce in your prepare_dataset()

ds = ds.map(prepare_dataset)  
ds = ds.cast(add_new_features)
```
2025-02-07T11:28:30Z
https://github.com/huggingface/datasets/issues/5284#issuecomment-2642660570
https://api.github.com/repos/huggingface/datasets/issues/5284
IC_kwDODunzps6dg8za
https://api.github.com/repos/huggingface/datasets/issues/comments/2642660570/reactions
2025-02-07T11:36:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2642660570
https://avatars.githubusercontent.com/u/34571650?v=4
https://api.github.com/users/SahebehDadboud/events{/privacy}
https://api.github.com/users/SahebehDadboud/followers
https://api.github.com/users/SahebehDadboud/following{/other_user}
https://api.github.com/users/SahebehDadboud/gists{/gist_id}
https://github.com/SahebehDadboud
SahebehDadboud
MDQ6VXNlcjM0NTcxNjUw
https://api.github.com/users/SahebehDadboud/orgs
https://api.github.com/users/SahebehDadboud/received_events
https://api.github.com/users/SahebehDadboud/repos
https://api.github.com/users/SahebehDadboud/starred{/owner}{/repo}
https://api.github.com/users/SahebehDadboud/subscriptions
User
https://api.github.com/users/SahebehDadboud
public
### Describe the bug

The `remove_column` method of the IterableDataset sets the dataset features to None.

### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset

# load LS in streaming mode
dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)

# check original features
print("Original features: ", dataset.features.keys())

# define features to remove: we KEEP audio and text
COLUMNS_TO_REMOVE = ['chapter_id', 'speaker_id', 'file', 'id']

dataset = dataset.remove_columns(COLUMNS_TO_REMOVE)

# check processed features, uh-oh!
print("Processed features: ", dataset.features)

# streaming the first audio sample still works
print("First sample:", next(iter(ds)))
```

**Print Output:**

```
Original features:  dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'])
Processed features:  None
First sample: {'audio': {'path': '2277-149896-0000.flac', 'array': array([ 0.00186157,  0.0005188 ,  0.00024414, ..., -0.00097656,
       -0.00109863, -0.00146484]), 'sampling_rate': 16000}, 'text': "HE WAS IN A FEVERED STATE OF MIND OWING TO THE BLIGHT HIS WIFE'S ACTION THREATENED TO CAST UPON HIS ENTIRE FUTURE"}
```

### Expected behavior

The features should be those **not** removed by the `remove_column` method, i.e. audio and text.

### Environment info

- `datasets` version: 2.7.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- PyArrow version: 9.0.0
- Pandas version: 1.3.5

(Running on Google Colab for a blog post: https://colab.research.google.com/drive/1ySCQREPZEl4msLfxb79pYYOWjUZhkr9y#scrollTo=8pRDGiVmH2ml)

cc @polinaeterna @lhoestq 