Add some iteration method on a dataset column (specific for inference)
MEMBER
Thanks for the suggestion ! I agree it would be nice to have something directly in `datasets` to do something as simple as that

cc @albertvillanova @mariosasko @polinaeterna What do you think if we have something similar to pandas `Series` that wouldn't bring everything in memory when doing `dataset["audio"]` ? Currently it returns a list with all the decoded audio data in memory.

It would be a breaking change though, since `isinstance(dataset["audio"], list)` wouldn't work anymore, but we could implement a `Sequence` so that `dataset["audio"][0]` still works and only loads one item in memory.

Your alternative suggestion with `iterate` is also sensible, though maybe less satisfactory in terms of experience IMO
2022-04-19T15:39:43Z
https://github.com/huggingface/datasets/issues/4180#issuecomment-1102803891
https://api.github.com/repos/huggingface/datasets/issues/4180
IC_kwDODunzps5Bu3Oz
https://api.github.com/repos/huggingface/datasets/issues/comments/1102803891/reactions
2022-04-19T15:39:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102803891
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I agree that current behavior (decoding all audio file sin the dataset when accessing `dataset["audio"]`) is not useful, IMHO. Indeed in our docs, we are constantly warning our collaborators not to do that.

Therefore I upvote for a "useful" behavior of `dataset["audio"]`. I don't think the breaking change is important in this case, as I guess no many people use it with its current behavior. Therefore, for me it seems reasonable to return a generator (instead of an in-memeory list) for "special" features, like Audio/Image.

@lhoestq on the other hand I don't understand your proposal about Pandas-like... 
2022-04-19T15:54:44Z
https://github.com/huggingface/datasets/issues/4180#issuecomment-1102819728
https://api.github.com/repos/huggingface/datasets/issues/4180
IC_kwDODunzps5Bu7GQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1102819728/reactions
2022-04-19T15:55:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1102819728
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
COLLABORATOR
I recall I had the same idea while working on the `Image` feature, so I agree implementing something similar to `pd.Series` that lazily brings elements in memory would be beneficial.
2022-04-20T12:01:35Z
https://github.com/huggingface/datasets/issues/4180#issuecomment-1103850907
https://api.github.com/repos/huggingface/datasets/issues/4180
IC_kwDODunzps5By22b
https://api.github.com/repos/huggingface/datasets/issues/comments/1103850907/reactions
2022-04-20T12:01:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1103850907
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
@lhoestq @mariosasko Could you please give a link to that new feature of `pandas.Series`? As far as I remember since I worked with pandas for more than 6 years, there was no lazy in-memory feature; it was everything in-memory; that was the reason why other frameworks were created, like Vaex or Dask, e.g. 
2022-04-20T12:15:44Z
https://github.com/huggingface/datasets/issues/4180#issuecomment-1103863213
https://api.github.com/repos/huggingface/datasets/issues/4180
IC_kwDODunzps5By52t
https://api.github.com/repos/huggingface/datasets/issues/comments/1103863213/reactions
2022-04-20T12:16:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1103863213
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Yea pandas doesn't do lazy loading. I was referring to pandas.Series to say that they have a dedicated class to represent a column ;)
2022-04-21T10:30:57Z
https://github.com/huggingface/datasets/issues/4180#issuecomment-1105031263
https://api.github.com/repos/huggingface/datasets/issues/4180
IC_kwDODunzps5B3XBf
https://api.github.com/repos/huggingface/datasets/issues/comments/1105031263/reactions
2022-04-21T10:30:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1105031263
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

Currently, `dataset["audio"]` will load EVERY element in the dataset in RAM, which can be quite big for an audio dataset.
Having an iterator (or sequence) type of object, would make inference with `transformers` 's `pipeline` easier to use and not so memory hungry.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

For a non breaking change:

```python
for audio in dataset.iterate("audio"):
    # {"array": np.array(...), "sampling_rate":...}
```

For a  breaking change solution (not necessary), changing the type of `dataset["audio"]` to a sequence type so that

```python
pipe = pipeline(model="...")
for out in pipe(dataset["audio"]):
    # {"text":....}
```
could work

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

```python
def iterate(dataset, key):
    for item in dataset:
        yield dataset[key]

for out in pipeline(iterate(dataset, "audio")):
    # {"array": ...}
```

This works but requires the helper function which feels slightly clunky.

**Additional context**
Add any other context about the feature request here.

The context is actually to showcase better integration between  `pipeline` and `datasets` in the Quicktour demo: https://github.com/huggingface/transformers/pull/16723/files

@lhoestq 
