Torchcodec decoding
CONTRIBUTOR
@lhoestq any updates on when this will be merged? Let me know if theres anything you need from my end.
2025-06-17T13:52:48Z
https://github.com/huggingface/datasets/pull/7616#issuecomment-2980477096
https://api.github.com/repos/huggingface/datasets/issues/7616
IC_kwDODunzps6xpnio
https://api.github.com/repos/huggingface/datasets/issues/comments/2980477096/reactions
2025-06-17T13:52:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2980477096
https://avatars.githubusercontent.com/u/49127578?v=4
https://api.github.com/users/TyTodd/events{/privacy}
https://api.github.com/users/TyTodd/followers
https://api.github.com/users/TyTodd/following{/other_user}
https://api.github.com/users/TyTodd/gists{/gist_id}
https://github.com/TyTodd
TyTodd
MDQ6VXNlcjQ5MTI3NTc4
https://api.github.com/users/TyTodd/orgs
https://api.github.com/users/TyTodd/received_events
https://api.github.com/users/TyTodd/repos
https://api.github.com/users/TyTodd/starred{/owner}{/repo}
https://api.github.com/users/TyTodd/subscriptions
User
https://api.github.com/users/TyTodd
public
MEMBER
Btw I plan to release `datasets` 4.0 after your PR, this will be a major milestone :)
2025-06-17T14:36:49Z
https://github.com/huggingface/datasets/pull/7616#issuecomment-2980641019
https://api.github.com/repos/huggingface/datasets/issues/7616
IC_kwDODunzps6xqPj7
https://api.github.com/repos/huggingface/datasets/issues/comments/2980641019/reactions
2025-06-17T14:36:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2980641019
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq just pushed the new changes.
2025-06-17T19:05:55Z
https://github.com/huggingface/datasets/pull/7616#issuecomment-2981535016
https://api.github.com/repos/huggingface/datasets/issues/7616
IC_kwDODunzps6xtp0o
https://api.github.com/repos/huggingface/datasets/issues/comments/2981535016/reactions
2025-06-17T19:05:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2981535016
https://avatars.githubusercontent.com/u/49127578?v=4
https://api.github.com/users/TyTodd/events{/privacy}
https://api.github.com/users/TyTodd/followers
https://api.github.com/users/TyTodd/following{/other_user}
https://api.github.com/users/TyTodd/gists{/gist_id}
https://github.com/TyTodd
TyTodd
MDQ6VXNlcjQ5MTI3NTc4
https://api.github.com/users/TyTodd/orgs
https://api.github.com/users/TyTodd/received_events
https://api.github.com/users/TyTodd/repos
https://api.github.com/users/TyTodd/starred{/owner}{/repo}
https://api.github.com/users/TyTodd/subscriptions
User
https://api.github.com/users/TyTodd
public
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7616). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
2025-06-18T14:32:11Z
https://github.com/huggingface/datasets/pull/7616#issuecomment-2984464684
https://api.github.com/repos/huggingface/datasets/issues/7616
IC_kwDODunzps6x41Es
https://api.github.com/repos/huggingface/datasets/issues/comments/2984464684/reactions
2025-06-18T14:32:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2984464684
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Great ! I took the liberty to move the AudioDecoder to its own file and make small edits in the docs and docstrings

If it looks good to you I think we can merge :)
2025-06-18T16:28:32Z
https://github.com/huggingface/datasets/pull/7616#issuecomment-2984926736
https://api.github.com/repos/huggingface/datasets/issues/7616
IC_kwDODunzps6x6l4Q
https://api.github.com/repos/huggingface/datasets/issues/comments/2984926736/reactions
2025-06-18T16:28:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2984926736
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Closes #7607
## New signatures
### Audio
```python
Audio(sampling_rate: Optional[int] = None, mono: bool = True, decode: bool = True, stream_index: Optional[int] = None)

Audio.encode_example(self, value: Union[str, bytes, bytearray, dict, "AudioDecoder"]) -> dict

Audio.decode_example(self, value: dict, token_per_repo_id: Optional[dict[str, Union[str, bool, None]]] = None) -> "AudioDecoder":
```

### Video
```python
Video(decode: bool = True, stream_index: Optional[int] = None, dimension_order: Literal['NCHW', 'NHWC'] = 'NCHW', num_ffmpeg_threads: int = 1, device: Optional[Union[str, "torch.device"]] = 'cpu', seek_mode: Literal['exact', 'approximate'] = 'exact')

Video.encode_example(self, value: Union[str, bytes, bytearray, Example, np.ndarray, "VideoDecoder"]) -> Example:

Video.decode_example(self, value: Union[str, Example], token_per_repo_id: Optional[dict[str, Union[bool, str]]] = None, ) -> "VideoDecoder":
```



## Notes
Audio features constructor takes in 1 new optional param stream_index which is passed to the AudioDecoder constructor to select the stream index of a file.
Audio feature can now take in torchcodec.decoders.AudioDecoder as input to encode_example()
Audio feature decode_example() returns torchcodec.decoders.AudioDecoder

Video feature constructor takes in 5 new optional params stream_index, dimension_order, num_ffmpeg_threads, device, seek_mode all of which are passed to VideoDecoder constructor 
Video feature decode_example() returns torchcodec.decoders.VideoDecoder
Video feature can now take in torchcodec.decoders.VideoDecoder as input to encode_example() 

All test cases have been updated to reflect these changes
All documentation has also been updated to reflect these changes.

Both VideoDecoder and AudioDecoder when formatted with (np_formatter, tf_formatter, etc) will ignore the type and return themselves. Formatting test cases were updated accordingly to reflect this. (Pretty simple to make this not the case if we want though)

## Errors
This test case from `tests/packaged_modules/test_audiofolder.py`
```python
@require_librosa
@require_sndfile
@pytest.mark.parametrize("streaming", [False, True])
def test_data_files_with_metadata_and_archives(streaming, cache_dir, data_files_with_zip_archives):
    audiofolder = AudioFolder(data_files=data_files_with_zip_archives, cache_dir=cache_dir)
    audiofolder.download_and_prepare()
    datasets = audiofolder.as_streaming_dataset() if streaming else audiofolder.as_dataset()
    for split, data_files in data_files_with_zip_archives.items():
        num_of_archives = len(data_files)  # the metadata file is inside the archive
        expected_num_of_audios = 2 * num_of_archives
        assert split in datasets
        dataset = list(datasets[split])
        assert len(dataset) == expected_num_of_audios
        # make sure each sample has its own audio (all arrays are different) and metadata
        assert (
            sum(np.array_equal(dataset[0]["audio"].get_all_samples().data.numpy(), example["audio"].get_all_samples().data.numpy()) for example in dataset[1:])
            == 0
        )
        assert len({example["text"] for example in dataset}) == expected_num_of_audios
        assert all(example["text"] is not None for example in dataset)
```
Fails now because AudioDecoder needs to access the files after the lines below are run, but there seems to be some context issues. The file the decoder is trying to read is closed before the decoder gets the chance to decode it.
```python
audiofolder.download_and_prepare()
datasets = audiofolder.as_streaming_dataset() if streaming else audiofolder.as_dataset()
```