Add video feature
COLLABORATOR
@NielsRogge @rwightman may have additional requirements regarding this feature.

When adding a new (decodable) type, the hardest part is choosing the right decoding library. What I mean by "right" here is that it has all the features we need and is easy to install (with GPU support?).

Some candidates/options:
* [`decord`](https://github.com/dmlc/decord): no longer [maintained](https://github.com/dmlc/decord/issues/214), not trivial to install with GPU support
* [`pyAV`](https://github.com/PyAV-Org/PyAV):  used for CPU decoding in `torchvision`, GPU decoding not supported if I'm not mistaken, otherwise the best candidate probably
* [`video_reader`](https://github.com/pytorch/vision/blob/de350bc01ad2193ea2888f0ce8a6a346d3cba5a9/torchvision/csrc/io/video_reader/video_reader.cpp): used for GPU decoding in `torchvision`, depends on `torch'
* OpenCV: uses `ffmpeg` for video decoding under the hood
* ...

And the last resort is building our own library, which is the most flexible solution but also requires the most work.

PS: I'm adding a link to an article that compares various video decoding libraries: https://towardsdatascience.com/lightning-fast-video-reading-in-python-c1438771c4e6
2022-11-29T20:11:20Z
https://github.com/huggingface/datasets/issues/5225#issuecomment-1331238855
https://api.github.com/repos/huggingface/datasets/issues/5225
IC_kwDODunzps5PWRfH
https://api.github.com/repos/huggingface/datasets/issues/comments/1331238855/reactions
2022-11-29T20:11:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1331238855
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
@mariosasko is GPU decoding a hard requirement here? Do we really need it? (I don't know)

Something to consider with `decord` is that it doesn't (AFAIK) support writing videos, so you'd still need something else for that. also I've noticed [issues](https://github.com/dmlc/decord/issues/242) with decord's ability to decode stereo audio streams along side the video (which you don't run into with PyAV).

---

I think PyAV should be able to do the job just fine to start. If we write the video io utilities as their own functions, we can hot swap them later if we find/write a different solution that's faster/better.
2022-11-29T20:50:43Z
https://github.com/huggingface/datasets/issues/5225#issuecomment-1331285995
https://api.github.com/repos/huggingface/datasets/issues/5225
IC_kwDODunzps5PWc_r
https://api.github.com/repos/huggingface/datasets/issues/comments/1331285995/reactions
2022-11-29T20:50:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1331285995
https://avatars.githubusercontent.com/u/32437151?v=4
https://api.github.com/users/nateraw/events{/privacy}
https://api.github.com/users/nateraw/followers
https://api.github.com/users/nateraw/following{/other_user}
https://api.github.com/users/nateraw/gists{/gist_id}
https://github.com/nateraw
nateraw
MDQ6VXNlcjMyNDM3MTUx
https://api.github.com/users/nateraw/orgs
https://api.github.com/users/nateraw/received_events
https://api.github.com/users/nateraw/repos
https://api.github.com/users/nateraw/starred{/owner}{/repo}
https://api.github.com/users/nateraw/subscriptions
User
https://api.github.com/users/nateraw
public
NONE
Video is still a bit of a mess, but I'd say pyAV is likely the best approach (or supporting all three via pytorchvideo, but that adds a middle man dependency).

Being able to decode on the GPU, into memory that could be passed off to a Tensor in whatever framework is being used would be the dream, I don't think there is any interop of that nature working right now. Number of decoder instances per GPU is limited so it's not clear if balancing load btw GPU decoders and CPUs would be needed in say large scale video training.

Any of these solutions is less than ideal due to the nature of video, having a simple Python interface video / start -> end results in lots of extra memory (you need to decode whole range of the clips into a buffer before using anything). Any scalable video system would be streaming on the fly (issuing frames via callbacks as soon as the stream is far enough along to have re-ordered the frames and synced audio+video+other metadata (sensors, CC, etc).


2022-11-29T21:18:19Z
https://github.com/huggingface/datasets/issues/5225#issuecomment-1331322523
https://api.github.com/repos/huggingface/datasets/issues/5225
IC_kwDODunzps5PWl6b
https://api.github.com/repos/huggingface/datasets/issues/comments/1331322523/reactions
2022-11-29T21:18:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1331322523
https://avatars.githubusercontent.com/u/5702664?v=4
https://api.github.com/users/rwightman/events{/privacy}
https://api.github.com/users/rwightman/followers
https://api.github.com/users/rwightman/following{/other_user}
https://api.github.com/users/rwightman/gists{/gist_id}
https://github.com/rwightman
rwightman
MDQ6VXNlcjU3MDI2NjQ=
https://api.github.com/users/rwightman/orgs
https://api.github.com/users/rwightman/received_events
https://api.github.com/users/rwightman/repos
https://api.github.com/users/rwightman/starred{/owner}{/repo}
https://api.github.com/users/rwightman/subscriptions
User
https://api.github.com/users/rwightman
public
MEMBER
For standalone usage, decoding on GPU could be ideal but isn't async processing of inputs on CPUs while letting the accelerator busy for training the de-facto? Of course, I am aware of other advanced mechanisms such as CPU offloading, but I think my point is conveyed. 
2022-11-30T03:39:29Z
https://github.com/huggingface/datasets/issues/5225#issuecomment-1331604683
https://api.github.com/repos/huggingface/datasets/issues/5225
IC_kwDODunzps5PXqzL
https://api.github.com/repos/huggingface/datasets/issues/comments/1331604683/reactions
2022-11-30T03:39:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1331604683
https://avatars.githubusercontent.com/u/22957388?v=4
https://api.github.com/users/sayakpaul/events{/privacy}
https://api.github.com/users/sayakpaul/followers
https://api.github.com/users/sayakpaul/following{/other_user}
https://api.github.com/users/sayakpaul/gists{/gist_id}
https://github.com/sayakpaul
sayakpaul
MDQ6VXNlcjIyOTU3Mzg4
https://api.github.com/users/sayakpaul/orgs
https://api.github.com/users/sayakpaul/received_events
https://api.github.com/users/sayakpaul/repos
https://api.github.com/users/sayakpaul/starred{/owner}{/repo}
https://api.github.com/users/sayakpaul/subscriptions
User
https://api.github.com/users/sayakpaul
public
CONTRIBUTOR
Here's a minimal implementation of the helper functions we'd need from PyAV, a lot of which I borrowed from `pytorchvideo`, stripping out the `torch` specific stuff:

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/nateraw/c327cb6ff6b074e6ddc8068d19c0367d/pyav-io.ipynb)
    
It's not too much code...@mariosasko we could probably just maintain these helper fns within the `datasets` library, right? 
2022-12-02T06:39:21Z
https://github.com/huggingface/datasets/issues/5225#issuecomment-1334823028
https://api.github.com/repos/huggingface/datasets/issues/5225
IC_kwDODunzps5Pj8h0
https://api.github.com/repos/huggingface/datasets/issues/comments/1334823028/reactions
2022-12-02T06:39:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1334823028
https://avatars.githubusercontent.com/u/32437151?v=4
https://api.github.com/users/nateraw/events{/privacy}
https://api.github.com/users/nateraw/followers
https://api.github.com/users/nateraw/following{/other_user}
https://api.github.com/users/nateraw/gists{/gist_id}
https://github.com/nateraw
nateraw
MDQ6VXNlcjMyNDM3MTUx
https://api.github.com/users/nateraw/orgs
https://api.github.com/users/nateraw/received_events
https://api.github.com/users/nateraw/repos
https://api.github.com/users/nateraw/starred{/owner}{/repo}
https://api.github.com/users/nateraw/subscriptions
User
https://api.github.com/users/nateraw
public
CONTRIBUTOR
Also wanted to note I added a PR for video classification in `transformers` here, which uses `decord`. It's still open...should we make a decision now to align the libraries we are using between `datasets` and `transformers`? (CC @Narsil )

https://github.com/huggingface/transformers/pull/20151
2022-12-02T06:41:22Z
https://github.com/huggingface/datasets/issues/5225#issuecomment-1334824218
https://api.github.com/repos/huggingface/datasets/issues/5225
IC_kwDODunzps5Pj80a
https://api.github.com/repos/huggingface/datasets/issues/comments/1334824218/reactions
2022-12-02T06:52:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1334824218
https://avatars.githubusercontent.com/u/32437151?v=4
https://api.github.com/users/nateraw/events{/privacy}
https://api.github.com/users/nateraw/followers
https://api.github.com/users/nateraw/following{/other_user}
https://api.github.com/users/nateraw/gists{/gist_id}
https://github.com/nateraw
nateraw
MDQ6VXNlcjMyNDM3MTUx
https://api.github.com/users/nateraw/orgs
https://api.github.com/users/nateraw/received_events
https://api.github.com/users/nateraw/repos
https://api.github.com/users/nateraw/starred{/owner}{/repo}
https://api.github.com/users/nateraw/subscriptions
User
https://api.github.com/users/nateraw
public
CONTRIBUTOR
Fully agree on at least trying to unite things.

Making clear function boundaries to help us change dependency if needed seems like a good idea since there doesn't seem to be a clear winner.

I also happen to like directly calling ffmpeg. For some reason it was a lot faster than pyav. 
2022-12-02T15:13:15Z
https://github.com/huggingface/datasets/issues/5225#issuecomment-1335392859
https://api.github.com/repos/huggingface/datasets/issues/5225
IC_kwDODunzps5PmHpb
https://api.github.com/repos/huggingface/datasets/issues/comments/1335392859/reactions
2022-12-02T15:13:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1335392859
https://avatars.githubusercontent.com/u/204321?v=4
https://api.github.com/users/Narsil/events{/privacy}
https://api.github.com/users/Narsil/followers
https://api.github.com/users/Narsil/following{/other_user}
https://api.github.com/users/Narsil/gists{/gist_id}
https://github.com/Narsil
Narsil
MDQ6VXNlcjIwNDMyMQ==
https://api.github.com/users/Narsil/orgs
https://api.github.com/users/Narsil/received_events
https://api.github.com/users/Narsil/repos
https://api.github.com/users/Narsil/starred{/owner}{/repo}
https://api.github.com/users/Narsil/subscriptions
User
https://api.github.com/users/Narsil
public
### Feature request

Add a `Video` feature to the library so folks can include videos in their datasets.

### Motivation

Being able to load Video data would be quite helpful. However, there are some challenges when it comes to videos:

1. Videos, unlike images, can end up being extremely large files
2. Often times when training video models, you need to do some very specific sampling. Videos might end up needing to be broken down into X number of clips used for training/inference
3. Videos have an additional audio stream, which must be accounted for
4. The feature needs to be able to encode/decode videos (with right video settings) from bytes.

### Your contribution

I did work on this a while back in [this (now closed) PR](https://github.com/huggingface/datasets/pull/4532). It used a library I made called [encoded_video](https://github.com/nateraw/encoded-video), which is basically the utils from [pytorchvideo](https://github.com/facebookresearch/pytorchvideo), but without the `torch` dep. It included the ability to read/write from bytes, as we need to do here. We don't want to be using a sketchy library that I made as a dependency in this repo, though.

Would love to use this issue as a place to:
- brainstorm ideas on how to do this right
- list ways/examples to work around it for now

CC @sayakpaul @mariosasko @fcakyon