Add imagefolder dataset
CONTRIBUTOR
@lhoestq @albertvillanova it would be super cool if we could get the Image Classification task to work with this. I'm not sure how to have the dataset find the unique label names _after_ the dataset has been loaded. Is that even possible? 

My hacky community version [here](https://huggingface.co/datasets/nateraw/image-folder) does this, but it wouldn't pass the test suite here. Any thoughts?
2021-09-02T21:16:19Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-912063563
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps42XPxL
https://api.github.com/repos/huggingface/datasets/issues/comments/912063563/reactions
2021-09-02T21:16:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/912063563
https://avatars.githubusercontent.com/u/32437151?v=4
https://api.github.com/users/nateraw/events{/privacy}
https://api.github.com/users/nateraw/followers
https://api.github.com/users/nateraw/following{/other_user}
https://api.github.com/users/nateraw/gists{/gist_id}
https://github.com/nateraw
nateraw
MDQ6VXNlcjMyNDM3MTUx
https://api.github.com/users/nateraw/orgs
https://api.github.com/users/nateraw/received_events
https://api.github.com/users/nateraw/repos
https://api.github.com/users/nateraw/starred{/owner}{/repo}
https://api.github.com/users/nateraw/subscriptions
User
https://api.github.com/users/nateraw
public
MEMBER
Hi ! Dataset builders that require some `data_files` like `csv` or `json` are handled differently that actual dataset scripts.

In particular:
- they are placed directly in the `src` folder of the lib so that you can use it without internet connection (more exactly in `src/datasets/packaged_modules/<builder_name>.py`). So feel free to move the dataset python file there. You also need to register it in `src/datasets/packaked_modules.__init__.py`
- they are handled a bit differently in our test suite (see the `PackagedDatasetTest` class in `test_dataset_common.py`). To be able to test the builder with your dummy data, you just need to modify `get_packaged_dataset_dummy_data_files` in `test_dataset_common.py` to return the right `data_files` for your builder. The dummy data can stay in `datasets/image_folder/dummy`

Let me know if you have questions or if I can help !
2021-09-06T16:11:39Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-913751545
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps42dr35
https://api.github.com/repos/huggingface/datasets/issues/comments/913751545/reactions
2021-09-06T16:11:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/913751545
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hey @lhoestq , I actually already did both of those things. I'm trying to get the `image-classification` task to work now.  

For example...When you run `ds = load_dataset('imagefolder', data_files='my_files')`, with a directory called `./my_files` that looks like this:

```
my_files
----| Cat
--------| image1.jpg
--------| ...
----| Dog
--------| image1.jpg
--------| ...
```

...We should set the dataset's `labels` feature to `datasets.features.ClassLabel(names=['cat', 'dog'])`  dynamically with class names we find by getting a list of directories in `my_files` (via `data_files`). Otherwise the `datasets.tasks.ImageClassification` task will break, as the `labels` feature is not a `ClassLabel`.

I couldn't figure out how to access the `data_files` in the builder's `_info` function in a way that would pass in the test suite. 
2021-09-07T02:59:55Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-913953518
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps42edLu
https://api.github.com/repos/huggingface/datasets/issues/comments/913953518/reactions
2021-09-07T03:06:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/913953518
https://avatars.githubusercontent.com/u/32437151?v=4
https://api.github.com/users/nateraw/events{/privacy}
https://api.github.com/users/nateraw/followers
https://api.github.com/users/nateraw/following{/other_user}
https://api.github.com/users/nateraw/gists{/gist_id}
https://github.com/nateraw
nateraw
MDQ6VXNlcjMyNDM3MTUx
https://api.github.com/users/nateraw/orgs
https://api.github.com/users/nateraw/received_events
https://api.github.com/users/nateraw/repos
https://api.github.com/users/nateraw/starred{/owner}{/repo}
https://api.github.com/users/nateraw/subscriptions
User
https://api.github.com/users/nateraw
public
MEMBER
Nice ! Then maybe you can use `self.config.data_files` in `_info()` ?
What error are you getting in the test suite ?

Also note that `data_files` was first developed to accept paths to actual files, not directories. In particular, it fetches the metadata of all the data_files to get a unique hash for the caching mechanism. So we may need to do a few changes first.
2021-09-07T09:06:41Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-914129768
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps42fINo
https://api.github.com/repos/huggingface/datasets/issues/comments/914129768/reactions
2021-09-07T09:06:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/914129768
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I'm trying to make it work by getting the label names in the _info automatically.
I'll let you know tomorrow how it goes :)

Also cc @mariosasko since we're going to use #3163 

Right now I'm getting the label name per file by taking the first word (from regex `\w+`) after the common prefix of all the files per split
2021-11-15T16:47:47Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-969102325
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps45w1P1
https://api.github.com/repos/huggingface/datasets/issues/comments/969102325/reactions
2021-11-15T16:50:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/969102325
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Data files resolution takes too much time on my side for a dataset of a few 10,000s of examples. I'll speed it up with some multihreading tomorrow, and maybe by removing the unnecessary checksum verification
2021-11-15T17:54:28Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-969167512
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps45xFKY
https://api.github.com/repos/huggingface/datasets/issues/comments/969167512/reactions
2021-11-15T17:54:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/969167512
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
The code is a bit ugly for my taste. I'll try to simplify it tomorrow by avoiding the `os.path.commonprefix` computation and do something similar to @nateraw's ImageFolder instead, where only the second-to-last path component is considered a label (and see if I can update the class labels lazily in `_generate_examples`).

Also, as discussed offline with @lhoestq,  I reverted the automatic directory globbing change in `data_files.py` and will investigate if we can use `data_dir` for that (e.g. `load_dataset("imagefolder", data_dir="path/to/data")` would be equal to `load_dataset("imagefolder", data_files=["path/to/data/**/*", "path/to/data/*"])`. The only problem with `data_dir`  that it's equal to `dl_manager.manual_dir`, which would break scripts with `manul_download_instructions`, so maybe we can limit this behavior only to the packaged loaders? WDYT?
2022-02-22T19:47:01Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-1048152299
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps4-eYjr
https://api.github.com/repos/huggingface/datasets/issues/comments/1048152299/reactions
2022-02-22T19:47:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1048152299
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
COLLABORATOR
An updated example of usage: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1zJkAQm0Dk372EHcBq6hYgHuunsK4rR0r?usp=sharing)
2022-02-23T14:45:38Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-1048858706
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps4-hFBS
https://api.github.com/repos/huggingface/datasets/issues/comments/1048858706/reactions
2022-02-23T14:45:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1048858706
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
> The code is a bit ugly for my taste. I'll try to simplify it tomorrow by avoiding the os.path.commonprefix computation and do something similar to @nateraw's ImageFolder instead, where only the second-to-last path component is considered a label (and see if I can update the class labels lazily in _generate_examples).

Sounds good ! It's fine if we just support the same format as pytorch ImageFolder.

Regarding the `data_dir` parameter, what do you think is best ?

1. `dl_manager.data_dir = data_dir`
2. `dl_manager.data_files = resolve(os.path.join(data_dir, "**"))`

or something else ?

> The only problem with data_dir that it's equal to dl_manager.manual_dir, which would break scripts with manul_download_instructions, so maybe we can limit this behavior only to the packaged loaders? WDYT?

We can still have `dl_manager.manual_dir = data_dir` though
2022-02-23T14:58:25Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-1048871825
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps4-hIOR
https://api.github.com/repos/huggingface/datasets/issues/comments/1048871825/reactions
2022-02-23T14:58:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1048871825
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
The example colab is amazing !
2022-02-23T14:59:14Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-1048872719
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps4-hIcP
https://api.github.com/repos/huggingface/datasets/issues/comments/1048872719/reactions
2022-02-23T14:59:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1048872719
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
@lhoestq 
>Regarding the `data_dir` parameter, what do you think is best ?
>
>1. `dl_manager.data_dir = data_dir`
>2. `dl_manager.data_files = resolve(os.path.join(data_dir, "**"))`

The second option. Basically, I would like `data_files` to be equal to:
```python
def _split_generators(self, dl_manager):
    data_files = self.config.data_files
    if data_files is None:    
        data_files = glob.glob("{self.config.data_dir}/**", recursive=True)
    else:
        raise ValueError(f"At least one data file must be specified, but got data_files={data_files}")
```
in the scripts of packaged modules. It's probably better to do the resolution in `data_files.py` tho (to handle relative file paths on the Hub, for instance)
2022-02-23T16:50:50Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-1048989764
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps4-hlBE
https://api.github.com/repos/huggingface/datasets/issues/comments/1048989764/reactions
2022-02-23T16:51:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1048989764
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
> The second option. Basically, I would like data_files to be equal to:
> ```python
> def _split_generators(self, dl_manager):
>     data_files = self.config.data_files
>     if data_files is None:    
>         data_files = glob.glob("{self.config.data_dir}/**", recursive=True)
>     else:
>         raise ValueError(f"At least one data file must be specified, but got data_files={data_files}")
> ```
> in the scripts of packaged modules. It's probably better to do the resolution in data_files.py tho (to handle relative file paths on the Hub, for instance)

sounds good !
2022-02-23T17:30:50Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-1049030836
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps4-hvC0
https://api.github.com/repos/huggingface/datasets/issues/comments/1049030836/reactions
2022-02-23T17:30:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1049030836
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
🙌
2022-02-24T16:16:00Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-1050021903
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps4-lhAP
https://api.github.com/repos/huggingface/datasets/issues/comments/1050021903/reactions
2022-02-24T16:16:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1050021903
https://avatars.githubusercontent.com/u/8995957?v=4
https://api.github.com/users/davanstrien/events{/privacy}
https://api.github.com/users/davanstrien/followers
https://api.github.com/users/davanstrien/following{/other_user}
https://api.github.com/users/davanstrien/gists{/gist_id}
https://github.com/davanstrien
davanstrien
MDQ6VXNlcjg5OTU5NTc=
https://api.github.com/users/davanstrien/orgs
https://api.github.com/users/davanstrien/received_events
https://api.github.com/users/davanstrien/repos
https://api.github.com/users/davanstrien/starred{/owner}{/repo}
https://api.github.com/users/davanstrien/subscriptions
User
https://api.github.com/users/davanstrien
public
CONTRIBUTOR
Hey @mariosasko are we still actually able to load an image folder?

For example...

```
! wget https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip
! unzip kagglecatsanddogs_3367a.zip
```

followed by

```python
from datasets import load_dataset

# Does not work
ds = load_dataset('imagefolder', data_files='/PetImages')

# Also doesn't work
ds = load_dataset('imagefolder', data_dir='/PetImages')
```

Are we going forward with the assumption that the user always wants to download from URL and that they won't have a dataset locally already? This at least gets us part of the way, but is technically not an "imagefolder" as intended. 

Either way, was delighted to see the colab notebook work smoothly outside of the case I just described above. ❤️ thanks so much for the work here.
2022-02-24T23:22:42Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-1050357376
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps4-my6A
https://api.github.com/repos/huggingface/datasets/issues/comments/1050357376/reactions
2022-02-24T23:22:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1050357376
https://avatars.githubusercontent.com/u/32437151?v=4
https://api.github.com/users/nateraw/events{/privacy}
https://api.github.com/users/nateraw/followers
https://api.github.com/users/nateraw/following{/other_user}
https://api.github.com/users/nateraw/gists{/gist_id}
https://github.com/nateraw
nateraw
MDQ6VXNlcjMyNDM3MTUx
https://api.github.com/users/nateraw/orgs
https://api.github.com/users/nateraw/received_events
https://api.github.com/users/nateraw/repos
https://api.github.com/users/nateraw/starred{/owner}{/repo}
https://api.github.com/users/nateraw/subscriptions
User
https://api.github.com/users/nateraw
public
MEMBER
> Hey @mariosasko are we still actually able to load an image folder?
> 
> For example...
> 
> ```
> ! wget https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip
> ! unzip kagglecatsanddogs_3367a.zip
> ```
> 
> followed by
> 
> ```python
> from datasets import load_dataset
> 
> # Does not work
> ds = load_dataset('imagefolder', data_files='/PetImages')

I ran into this too when I was trying to out. At the moment you can still load from a local on disk directory using a glob pattern i.e. 

```python
from datasets import load_dataset
ds = load_dataset("imagefolder", data_files="PetImages/**/*")
```
[Colab example](https://colab.research.google.com/drive/1IvAyYSAADHphzbtJMt02OXmnGwGtku3k?usp=sharing). I'm not sure if that is the intended behaviour or not. If it is, I think it would be good to document this because I also assumed the approach @nateraw used would work.
2022-02-25T10:26:17Z
https://github.com/huggingface/datasets/pull/2830#issuecomment-1050728662
https://api.github.com/repos/huggingface/datasets/issues/2830
IC_kwDODunzps4-oNjW
https://api.github.com/repos/huggingface/datasets/issues/comments/1050728662/reactions
2022-02-25T10:33:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1050728662
https://avatars.githubusercontent.com/u/8995957?v=4
https://api.github.com/users/davanstrien/events{/privacy}
https://api.github.com/users/davanstrien/followers
https://api.github.com/users/davanstrien/following{/other_user}
https://api.github.com/users/davanstrien/gists{/gist_id}
https://github.com/davanstrien
davanstrien
MDQ6VXNlcjg5OTU5NTc=
https://api.github.com/users/davanstrien/orgs
https://api.github.com/users/davanstrien/received_events
https://api.github.com/users/davanstrien/repos
https://api.github.com/users/davanstrien/starred{/owner}{/repo}
https://api.github.com/users/davanstrien/subscriptions
User
https://api.github.com/users/davanstrien
public
A generic imagefolder dataset inspired by `torchvision.datasets.ImageFolder`. 

Resolves #2508 

---

Example Usage:

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/nateraw/954fa8cba4ff806f6147a782fa9efd1a/imagefolder-official-example.ipynb)