Add AudioFolder packaged loader
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-06-22T13:03:51Z
https://github.com/huggingface/datasets/pull/4530#issuecomment-1163071443
https://api.github.com/repos/huggingface/datasets/issues/4530
IC_kwDODunzps5FUw_T
https://api.github.com/repos/huggingface/datasets/issues/comments/1163071443/reactions
2022-08-22T14:36:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1163071443
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
CONTRIBUTOR
@lhoestq @mariosasko I don't know what to do with the test, do you have any ideas? :)
2022-06-22T18:05:05Z
https://github.com/huggingface/datasets/pull/4530#issuecomment-1163447577
https://api.github.com/repos/huggingface/datasets/issues/4530
IC_kwDODunzps5FWM0Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1163447577/reactions
2022-06-22T18:11:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1163447577
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
CONTRIBUTOR
also it's passed in `pyarrow_latest_WIN`
2022-06-23T11:05:19Z
https://github.com/huggingface/datasets/pull/4530#issuecomment-1164273737
https://api.github.com/repos/huggingface/datasets/issues/4530
IC_kwDODunzps5FZWhJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1164273737/reactions
2022-06-23T11:05:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1164273737
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
MEMBER
If the error only happens on 3.6, maybe #4460 can help ^^' It seems to work in 3.7 on the windows CI

> inferring labels is not the default behavior (drop_labels is set to True in config)

I think it a missed opportunity to have a consistent API between imagefolder and audiofolder, since they do everything the same way. Can you give more details why you think we should drop the labels by default ?
2022-06-23T12:35:19Z
https://github.com/huggingface/datasets/pull/4530#issuecomment-1164352433
https://api.github.com/repos/huggingface/datasets/issues/4530
IC_kwDODunzps5FZpux
https://api.github.com/repos/huggingface/datasets/issues/comments/1164352433/reactions
2022-06-23T12:36:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1164352433
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Considering audio classification in audio is not as common as image classification in image, I'm ok with having different config defaults as long as they are properly documented (check [Papers With Code](https://paperswithcode.com/datasets) for stats and compare the classification numbers to the other tasks, do this for both modalities)

Also, WDYT about creating a generic folder loader that ImageFolder and AudioFolder then subclass to avoid having to update both of them when there is something to update/fix?
2022-06-23T13:30:01Z
https://github.com/huggingface/datasets/pull/4530#issuecomment-1164411395
https://api.github.com/repos/huggingface/datasets/issues/4530
IC_kwDODunzps5FZ4ID
https://api.github.com/repos/huggingface/datasets/issues/comments/1164411395/reactions
2022-06-23T13:30:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1164411395
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
@lhoestq I think it doesn't change the API itself, it just doesn't infer labels by default, but you can **still** set `drop_labels=False` to `load_dataset` and the labels will be inferred. 
Suppose that one has data structured as follows:
```
data/
   train/
      audio/
          file1.wav
          file2.wav
          file3.wav
      metadata.jsonl
   test/
      audio/
          file1.wav
          file2.wav
          file3.wav
      metadata.jsonl
```
If users load this dataset with `load_dataset("audiofolder", data_dir="data")` (the most native way), they will get a `label` feature that will always be equal to 0 (= "audio"). To mitigate this, they will have to always specify `load_dataset("audiofolder", data_dir="data", drop_labels=True)` explicitly and I believe it's not convenient. 

At the same time, `label` column can be added just as easy as adding one argument:` load_dataset("audiofolder", data_dir="data", drop_labels=False)`. As classification task is not as common, I think it should require more symbols to be added to the code :D  

But this is definitely should be explained in the docs, which I've forgotten to update... I'll add this section soon.

Also +to the generic loader, will work on it. 


2022-06-23T14:10:04Z
https://github.com/huggingface/datasets/pull/4530#issuecomment-1164456361
https://api.github.com/repos/huggingface/datasets/issues/4530
IC_kwDODunzps5FaDGp
https://api.github.com/repos/huggingface/datasets/issues/comments/1164456361/reactions
2022-06-23T14:14:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1164456361
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
MEMBER
If a metadata.jsonl file is present, then it doesn't have to infer the labels I agree. Note that this is already the case for imagefolder ;) in your case `load_dataset("audiofolder", data_dir="data")` won't return labels !

Labels are only inferred if there are no metadata.jsonl
2022-06-23T14:17:15Z
https://github.com/huggingface/datasets/pull/4530#issuecomment-1164466528
https://api.github.com/repos/huggingface/datasets/issues/4530
IC_kwDODunzps5FaFlg
https://api.github.com/repos/huggingface/datasets/issues/comments/1164466528/reactions
2022-06-23T14:17:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1164466528
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Feel free to merge the `main` branch into yours after updating your fork of `datasets`: https://github.com/huggingface/datasets/issues/4629

This should fix some errors in the CI
2022-07-07T13:01:42Z
https://github.com/huggingface/datasets/pull/4530#issuecomment-1177577654
https://api.github.com/repos/huggingface/datasets/issues/4530
IC_kwDODunzps5GMGi2
https://api.github.com/repos/huggingface/datasets/issues/comments/1177577654/reactions
2022-07-07T13:01:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1177577654
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@mariosasko  could you please review this PR again? :)

most of the tests for AutoFolder (base class for AudioFolder and ImageFolder) are now basically copied from Image/AudioFolder (their tests are also almost identical too) and adapted to test other methods. it should be refactored but i think this is not that important for now and might be done in the future PR, wdyt?
2022-08-09T12:28:41Z
https://github.com/huggingface/datasets/pull/4530#issuecomment-1209318353
https://api.github.com/repos/huggingface/datasets/issues/4530
IC_kwDODunzps5IFLvR
https://api.github.com/repos/huggingface/datasets/issues/comments/1209318353/reactions
2022-08-09T12:28:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1209318353
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
CONTRIBUTOR
@mariosasko thank you for the review! I'm sorry I accidentally asked for the review again, ignore it.
2022-08-11T11:47:01Z
https://github.com/huggingface/datasets/pull/4530#issuecomment-1211880396
https://api.github.com/repos/huggingface/datasets/issues/4530
IC_kwDODunzps5IO9PM
https://api.github.com/repos/huggingface/datasets/issues/comments/1211880396/reactions
2022-08-11T11:47:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1211880396
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
will close  #3964

AudioFolder is almost identical to ImageFolder except for inferring labels is not the default behavior (`drop_labels` is set to True in config), the option of inferring them is preserved though.

The weird thing is happening with the `test_data_files_with_metadata_and_archives` when `streaming` is `True`. Here is the log from the CI:
```

../.pyenv/versions/3.6.15/lib/python3.6/site-packages/datasets/features/audio.py:237: in _decode_non_mp3_path_like
    array, sampling_rate = librosa.load(f, sr=self.sampling_rate, mono=self.mono)
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/librosa/util/decorators.py:88: in inner_f
    return f(*args, **kwargs)
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/librosa/core/audio.py:176: in load
    raise (exc)
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/librosa/core/audio.py:155: in load
    context = sf.SoundFile(path)
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/soundfile.py:629: in __init__
    self._file = self._open(file, mode_int, closefd)
../.pyenv/versions/3.6.15/lib/python3.6/site-packages/soundfile.py:1184: in _open
    "Error opening {0!r}: ".format(self.name))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

err = 72
prefix = "Error opening <zipfile.ZipExtFile name='audio_file.wav' mode='r' compress_type=deflate>: "

    def _error_check(err, prefix=""):
        """Pretty-print a numerical error code if there is an error."""
        if err != 0:
            err_str = _snd.sf_error_number(err)
>           raise RuntimeError(prefix + _ffi.string(err_str).decode('utf-8', 'replace'))
E           RuntimeError: Error opening <zipfile.ZipExtFile name='audio_file.wav' mode='r' compress_type=deflate>: Error in WAV file. No 'data' chunk marker.
```
I hadn't been able to reproduce this locally until I created the same test environment (I mean with `pip install .[tests]`) with python3.6. The same env but with python3.8 passes the test! I didn't manage to figure out what's wrong, I also tried simply to replace the test wav file and still got the same error. Versions of `soundfile`, `librosa` and `libsndfile` are identical. Might it be something with zip compression? Sounds weird but I don't have any other ideas... 
 
TODO:

- [x] align with #4622
- [x] documentation
- [x] tests for AutoFolder?