Unable to rename columns in streaming dataset
MEMBER
Hi @peregilk this bug is directly related to https://github.com/huggingface/datasets/issues/3888, and still not fixed... But I'll try to have a look!
2022-11-16T09:10:26Z
https://github.com/huggingface/datasets/issues/5245#issuecomment-1316647542
https://api.github.com/repos/huggingface/datasets/issues/5245
IC_kwDODunzps5OenJ2
https://api.github.com/repos/huggingface/datasets/issues/comments/1316647542/reactions
2022-11-16T09:10:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1316647542
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
NONE
Thanks @alvarobartt. It is great if you are able to fix it, but when reading the explanation it seems like it is possible to work around it.

We also tried keeping the 'info.features' and then adding a modified version back after the remove/rename. Unforutunately that leads to a dataset that is not possible to iterate over.
2022-11-16T10:09:02Z
https://github.com/huggingface/datasets/issues/5245#issuecomment-1316735501
https://api.github.com/repos/huggingface/datasets/issues/5245
IC_kwDODunzps5Oe8oN
https://api.github.com/repos/huggingface/datasets/issues/comments/1316735501/reactions
2022-11-16T10:09:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1316735501
https://avatars.githubusercontent.com/u/9079808?v=4
https://api.github.com/users/peregilk/events{/privacy}
https://api.github.com/users/peregilk/followers
https://api.github.com/users/peregilk/following{/other_user}
https://api.github.com/users/peregilk/gists{/gist_id}
https://github.com/peregilk
peregilk
MDQ6VXNlcjkwNzk4MDg=
https://api.github.com/users/peregilk/orgs
https://api.github.com/users/peregilk/received_events
https://api.github.com/users/peregilk/repos
https://api.github.com/users/peregilk/starred{/owner}{/repo}
https://api.github.com/users/peregilk/subscriptions
User
https://api.github.com/users/peregilk
public
MEMBER
So if you iterate over the `IterableDataset` as `next(iter(ds))` and then run `rename_columns` when checking that data it will work, but in the end, it's just renaming the column one example/batch at a time, not renaming the column name for all the entries in the dataset, which is the ideal.
2022-11-16T10:24:58Z
https://github.com/huggingface/datasets/issues/5245#issuecomment-1316757383
https://api.github.com/repos/huggingface/datasets/issues/5245
IC_kwDODunzps5OfB-H
https://api.github.com/repos/huggingface/datasets/issues/comments/1316757383/reactions
2022-11-16T10:25:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1316757383
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
NONE
@alvarobartt Thanks. My use case was that I wanted to do multiple things, ie removing all unnecessary columns, renaming some valid columns, and then using cast (in my case checking if the audio is not 16K and casting it). It is just convenient to look into the info.features between each of these operations. Alternatively, I will just plan ahead...;) To me it seems like all the operations are working.

Thanks for the advice. It was very useful.
2022-11-16T11:24:27Z
https://github.com/huggingface/datasets/issues/5245#issuecomment-1316840197
https://api.github.com/repos/huggingface/datasets/issues/5245
IC_kwDODunzps5OfWMF
https://api.github.com/repos/huggingface/datasets/issues/comments/1316840197/reactions
2022-11-16T11:24:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1316840197
https://avatars.githubusercontent.com/u/9079808?v=4
https://api.github.com/users/peregilk/events{/privacy}
https://api.github.com/users/peregilk/followers
https://api.github.com/users/peregilk/following{/other_user}
https://api.github.com/users/peregilk/gists{/gist_id}
https://github.com/peregilk
peregilk
MDQ6VXNlcjkwNzk4MDg=
https://api.github.com/users/peregilk/orgs
https://api.github.com/users/peregilk/received_events
https://api.github.com/users/peregilk/repos
https://api.github.com/users/peregilk/starred{/owner}{/repo}
https://api.github.com/users/peregilk/subscriptions
User
https://api.github.com/users/peregilk
public
MEMBER
If we know the features before renaming, then we know the features after renaming, so we can pass the new features to the returned dataset in `rename_column` indeed ! If anyone is interested in contributing, feel free to open a PR and I'd be happy to help / give some pointers :)
2022-11-16T15:03:43Z
https://github.com/huggingface/datasets/issues/5245#issuecomment-1317163366
https://api.github.com/repos/huggingface/datasets/issues/5245
IC_kwDODunzps5OglFm
https://api.github.com/repos/huggingface/datasets/issues/comments/1317163366/reactions
2022-11-16T15:03:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1317163366
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Sure @lhoestq thanks! I’ll try to work on that
2022-11-16T17:20:20Z
https://github.com/huggingface/datasets/issues/5245#issuecomment-1317379674
https://api.github.com/repos/huggingface/datasets/issues/5245
IC_kwDODunzps5OhZ5a
https://api.github.com/repos/huggingface/datasets/issues/comments/1317379674/reactions
2022-11-16T17:20:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1317379674
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
#self-assign
2022-11-23T15:30:45Z
https://github.com/huggingface/datasets/issues/5245#issuecomment-1325258322
https://api.github.com/repos/huggingface/datasets/issues/5245
IC_kwDODunzps5O_dZS
https://api.github.com/repos/huggingface/datasets/issues/comments/1325258322/reactions
2022-11-23T15:30:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1325258322
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
### Describe the bug

Trying to rename column in a streaming datasets, destroys the features object.

### Steps to reproduce the bug

The following code illustrates the error:

```
from datasets import load_dataset
dataset = load_dataset('mc4', 'en', streaming=True, split='train')
dataset.info.features
# {'text': Value(dtype='string', id=None), 'timestamp': Value(dtype='string', id=None), 'url': Value(dtype='string', id=None)}
dataset = dataset.rename_column("text", "content")
dataset.info.features
# This returned object is now None!
```



### Expected behavior

This should just alter the renamed column.

### Environment info

datasets 2.6.1