load_dataset can't work correct with my image data 
CONTRIBUTOR
Can you post a reproducible code snippet of what you tried to do?


2023-03-20T09:40:47Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1475905454
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5X-Ieu
https://api.github.com/repos/huggingface/datasets/issues/comments/1475905454/reactions
2023-03-20T09:40:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1475905454
https://avatars.githubusercontent.com/u/26489385?v=4
https://api.github.com/users/Plutone11011/events{/privacy}
https://api.github.com/users/Plutone11011/followers
https://api.github.com/users/Plutone11011/following{/other_user}
https://api.github.com/users/Plutone11011/gists{/gist_id}
https://github.com/Plutone11011
Plutone11011
MDQ6VXNlcjI2NDg5Mzg1
https://api.github.com/users/Plutone11011/orgs
https://api.github.com/users/Plutone11011/received_events
https://api.github.com/users/Plutone11011/repos
https://api.github.com/users/Plutone11011/starred{/owner}{/repo}
https://api.github.com/users/Plutone11011/subscriptions
User
https://api.github.com/users/Plutone11011
public
NONE
> Can you post a reproducible code snippet of what you tried to do?
> 
> 

```python
from datasets import load_dataset

dataset = load_dataset("my_folder_name", split="train")
```
2023-03-20T11:42:29Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1476074931
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5X-x2z
https://api.github.com/repos/huggingface/datasets/issues/comments/1476074931/reactions
2023-03-20T17:47:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1476074931
https://avatars.githubusercontent.com/u/41611046?v=4
https://api.github.com/users/WiNE-iNEFF/events{/privacy}
https://api.github.com/users/WiNE-iNEFF/followers
https://api.github.com/users/WiNE-iNEFF/following{/other_user}
https://api.github.com/users/WiNE-iNEFF/gists{/gist_id}
https://github.com/WiNE-iNEFF
WiNE-iNEFF
MDQ6VXNlcjQxNjExMDQ2
https://api.github.com/users/WiNE-iNEFF/orgs
https://api.github.com/users/WiNE-iNEFF/received_events
https://api.github.com/users/WiNE-iNEFF/repos
https://api.github.com/users/WiNE-iNEFF/starred{/owner}{/repo}
https://api.github.com/users/WiNE-iNEFF/subscriptions
User
https://api.github.com/users/WiNE-iNEFF
public
CONTRIBUTOR
hi @WiNE-iNEFF ! can you please also tell a bit more about how your data is structured (directory structure and filenames patterns)?
2023-03-20T13:17:31Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1476218742
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5X_U92
https://api.github.com/repos/huggingface/datasets/issues/comments/1476218742/reactions
2023-03-20T13:17:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1476218742
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
NONE
> hi @WiNE-iNEFF ! can you please also tell a bit more about how your data is structured (directory structure and filenames patterns)?

All file have format .png converted in RGBA. 
In main folder "MyData" contain 4 folder with images. In function load_dataset i use folder "MyData"
2023-03-20T13:54:24Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1476277916
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5X_jac
https://api.github.com/repos/huggingface/datasets/issues/comments/1476277916/reactions
2023-03-20T13:54:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1476277916
https://avatars.githubusercontent.com/u/41611046?v=4
https://api.github.com/users/WiNE-iNEFF/events{/privacy}
https://api.github.com/users/WiNE-iNEFF/followers
https://api.github.com/users/WiNE-iNEFF/following{/other_user}
https://api.github.com/users/WiNE-iNEFF/gists{/gist_id}
https://github.com/WiNE-iNEFF
WiNE-iNEFF
MDQ6VXNlcjQxNjExMDQ2
https://api.github.com/users/WiNE-iNEFF/orgs
https://api.github.com/users/WiNE-iNEFF/received_events
https://api.github.com/users/WiNE-iNEFF/repos
https://api.github.com/users/WiNE-iNEFF/starred{/owner}{/repo}
https://api.github.com/users/WiNE-iNEFF/subscriptions
User
https://api.github.com/users/WiNE-iNEFF
public
CONTRIBUTOR
@WiNE-iNEFF I'm sorry there is still not enough information to answer your question :( For now I can only assume that your [filenames contain split names](https://huggingface.co/docs/datasets/repository_structure#splits-and-file-names) which are somehow incorrectly parsed. 
What would be the output if you omit `split` while loading? Like just
```python
ds = load_dataset("MyData")
print(ds)
```


2023-03-20T17:35:24Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1476661263
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5YBBAP
https://api.github.com/repos/huggingface/datasets/issues/comments/1476661263/reactions
2023-03-20T17:35:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1476661263
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
NONE
> @WiNE-iNEFF I'm sorry there is still not enough information to answer your question :( For now I can only assume that your [filenames contain split names](https://huggingface.co/docs/datasets/repository_structure#splits-and-file-names) which are somehow incorrectly parsed. 
> What would be the output if you omit `split` while loading? Like just
> ```python
> ds = load_dataset("MyData")
> print(ds)
> ```
> 
> 

```python
Dataset({
    features: ['image', 'label'],
    num_rows: 4
})
```
2023-03-20T17:47:15Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1476678052
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5YBFGk
https://api.github.com/repos/huggingface/datasets/issues/comments/1476678052/reactions
2023-03-20T17:47:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1476678052
https://avatars.githubusercontent.com/u/41611046?v=4
https://api.github.com/users/WiNE-iNEFF/events{/privacy}
https://api.github.com/users/WiNE-iNEFF/followers
https://api.github.com/users/WiNE-iNEFF/following{/other_user}
https://api.github.com/users/WiNE-iNEFF/gists{/gist_id}
https://github.com/WiNE-iNEFF
WiNE-iNEFF
MDQ6VXNlcjQxNjExMDQ2
https://api.github.com/users/WiNE-iNEFF/orgs
https://api.github.com/users/WiNE-iNEFF/received_events
https://api.github.com/users/WiNE-iNEFF/repos
https://api.github.com/users/WiNE-iNEFF/starred{/owner}{/repo}
https://api.github.com/users/WiNE-iNEFF/subscriptions
User
https://api.github.com/users/WiNE-iNEFF
public
CONTRIBUTOR
@WiNE-iNEFF  My only guess is that 4 images in your data have `"train"` string in their names (something like `"train_image_0.png"`) and others do not and the loader ignores all the files that do not contain split name in filename. If it's true, please try to remove "train" from filenames. Or maybe they are inside a directory named "train", then the directory should be renamed (unless you want to put only these 4 specific images to the train but apparently you do not).

If there is a bug I cannot investigate it unfortunately because I cannot reproduce your case without some data samples. 
2023-03-21T13:49:22Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1477871998
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5YFol-
https://api.github.com/repos/huggingface/datasets/issues/comments/1477871998/reactions
2023-03-21T13:49:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1477871998
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
NONE
> @WiNE-iNEFF  My only guess is that 4 images in your data have `"train"` string in their names (something like `"train_image_0.png"`) and others do not and the loader ignores all the files that do not contain split name in filename. If it's true, please try to remove "train" from filenames. Or maybe they are inside a directory named "train", then the directory should be renamed (unless you want to put only these 4 specific images to the train but apparently you do not).
> 
> If there is a bug I cannot investigate it unfortunately because I cannot reproduce your case without some data samples. 

I checked my files and some of them do have the words train, valid and test in their names, but the number of such images is more than 500, not 4.
2023-03-21T13:53:45Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1477879050
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5YFqUK
https://api.github.com/repos/huggingface/datasets/issues/comments/1477879050/reactions
2023-03-21T13:53:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1477879050
https://avatars.githubusercontent.com/u/41611046?v=4
https://api.github.com/users/WiNE-iNEFF/events{/privacy}
https://api.github.com/users/WiNE-iNEFF/followers
https://api.github.com/users/WiNE-iNEFF/following{/other_user}
https://api.github.com/users/WiNE-iNEFF/gists{/gist_id}
https://github.com/WiNE-iNEFF
WiNE-iNEFF
MDQ6VXNlcjQxNjExMDQ2
https://api.github.com/users/WiNE-iNEFF/orgs
https://api.github.com/users/WiNE-iNEFF/received_events
https://api.github.com/users/WiNE-iNEFF/repos
https://api.github.com/users/WiNE-iNEFF/starred{/owner}{/repo}
https://api.github.com/users/WiNE-iNEFF/subscriptions
User
https://api.github.com/users/WiNE-iNEFF
public
CONTRIBUTOR
@WiNE-iNEFF Probably they are named inconsistently so that the correct pattern for which files should correspond to which split cannot be inferred. You can make it clearer to the loader by removing split names from filenames and putting files in separate folder for each split (you can take a look at the [documentation for imagefolder](https://huggingface.co/docs/datasets/image_dataset#imagefolder)):
```
 Fuaimeanna2/
├─ test
│   ├─ label_0
│   │   ├── filename_0.jpg
│   │   └── filename_1.jpg
│   │   └── ...
│   ├─ label_1
│   │   └── ...
│   ├─ label_2
│   │   └── ...
│   └─ label_3
│          └── ...
├─ train
│   ├─ label_0
│   │   └── ...
│   ├─ label_1
│   │   └── ...
│   ├─ label_2
│   │   └── ...
│   └─ label_3
│          └── ...
└── validation
      ├─ label_0
      │   └── ...
      ├─ label_1
      │   └── ...
      ├─ label_2
      │   └── ...
      └─ label_3
            └── ...
```
2023-03-22T12:32:57Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1479493268
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5YL0aU
https://api.github.com/repos/huggingface/datasets/issues/comments/1479493268/reactions
2023-03-22T12:33:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1479493268
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
NONE
> @WiNE-iNEFF Probably they are named inconsistently so that the correct pattern for which files should correspond to which split cannot be inferred. You can make it clearer to the loader by removing split names from filenames and putting files in separate folder for each split (you can take a look at the [documentation for imagefolder](https://huggingface.co/docs/datasets/image_dataset#imagefolder)):
> ```
>  Fuaimeanna2/
> ├─ test
> │   ├─ label_0
> │   │   ├── filename_0.jpg
> │   │   └── filename_1.jpg
> │   │   └── ...
> │   ├─ label_1
> │   │   └── ...
> │   ├─ label_2
> │   │   └── ...
> │   └─ label_3
> │          └── ...
> ├─ train
> │   ├─ label_0
> │   │   └── ...
> │   ├─ label_1
> │   │   └── ...
> │   ├─ label_2
> │   │   └── ...
> │   └─ label_3
> │          └── ...
> └── validation
>       ├─ label_0
>       │   └── ...
>       ├─ label_1
>       │   └── ...
>       ├─ label_2
>       │   └── ...
>       └─ label_3
>             └── ...
> ```

I have read this documentation more than once. It just wasn't a problem before.
2023-03-22T12:41:48Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1479504550
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5YL3Km
https://api.github.com/repos/huggingface/datasets/issues/comments/1479504550/reactions
2023-03-22T12:41:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1479504550
https://avatars.githubusercontent.com/u/41611046?v=4
https://api.github.com/users/WiNE-iNEFF/events{/privacy}
https://api.github.com/users/WiNE-iNEFF/followers
https://api.github.com/users/WiNE-iNEFF/following{/other_user}
https://api.github.com/users/WiNE-iNEFF/gists{/gist_id}
https://github.com/WiNE-iNEFF
WiNE-iNEFF
MDQ6VXNlcjQxNjExMDQ2
https://api.github.com/users/WiNE-iNEFF/orgs
https://api.github.com/users/WiNE-iNEFF/received_events
https://api.github.com/users/WiNE-iNEFF/repos
https://api.github.com/users/WiNE-iNEFF/starred{/owner}{/repo}
https://api.github.com/users/WiNE-iNEFF/subscriptions
User
https://api.github.com/users/WiNE-iNEFF
public
CONTRIBUTOR
Hi,

You need to use:
```
from datasets import load_dataset

dataset = load_dataset("imagefolder", split="train", data_dir="path_to_your_folder")
```
instead of 
```
from datasets import load_dataset

dataset = load_dataset("my_folder_name", split="train")
```
To create an image dataset from your local folders.
2023-03-23T09:43:42Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1480881027
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5YRHOD
https://api.github.com/repos/huggingface/datasets/issues/comments/1480881027/reactions
2023-03-23T09:43:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1480881027
https://avatars.githubusercontent.com/u/48327001?v=4
https://api.github.com/users/NielsRogge/events{/privacy}
https://api.github.com/users/NielsRogge/followers
https://api.github.com/users/NielsRogge/following{/other_user}
https://api.github.com/users/NielsRogge/gists{/gist_id}
https://github.com/NielsRogge
NielsRogge
MDQ6VXNlcjQ4MzI3MDAx
https://api.github.com/users/NielsRogge/orgs
https://api.github.com/users/NielsRogge/received_events
https://api.github.com/users/NielsRogge/repos
https://api.github.com/users/NielsRogge/starred{/owner}{/repo}
https://api.github.com/users/NielsRogge/subscriptions
User
https://api.github.com/users/NielsRogge
public
NONE
> Hi,
> 
> You need to use:
> 
> ```
> from datasets import load_dataset
> 
> dataset = load_dataset("imagefolder", split="train", data_dir="path_to_your_folder")
> ```
> 
> instead of
> 
> ```
> from datasets import load_dataset
> 
> dataset = load_dataset("my_folder_name", split="train")
> ```
> 
> To create an image dataset from your local folders.

Thank you, but even using the method that you wrote above absolutely nothing changes, especially without using data_dir on my other data everything works fine
2023-03-24T16:55:06Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1483118430
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5YZpde
https://api.github.com/repos/huggingface/datasets/issues/comments/1483118430/reactions
2023-03-24T16:55:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1483118430
https://avatars.githubusercontent.com/u/41611046?v=4
https://api.github.com/users/WiNE-iNEFF/events{/privacy}
https://api.github.com/users/WiNE-iNEFF/followers
https://api.github.com/users/WiNE-iNEFF/following{/other_user}
https://api.github.com/users/WiNE-iNEFF/gists{/gist_id}
https://github.com/WiNE-iNEFF
WiNE-iNEFF
MDQ6VXNlcjQxNjExMDQ2
https://api.github.com/users/WiNE-iNEFF/orgs
https://api.github.com/users/WiNE-iNEFF/received_events
https://api.github.com/users/WiNE-iNEFF/repos
https://api.github.com/users/WiNE-iNEFF/starred{/owner}{/repo}
https://api.github.com/users/WiNE-iNEFF/subscriptions
User
https://api.github.com/users/WiNE-iNEFF
public
CONTRIBUTOR
@WiNE-iNEFF have you tried the suggestion I posted above? with removing split names from filenames and structuring files in folders? 


> even using the method that you wrote above absolutely nothing changes

fyi - nothing changed because these two approaches are basically the same. it's just that when you pass your data directory as a dataset name (`load_dataset("my_folder_name"`), not as `data_dir` (`load_dataset("imagefolder", data_dir="my_folder_name"`), `datasets` infers what module to use (`imagefolder` in your case) automatically, by file extensions.
2023-03-27T11:39:20Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1484990020
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5YgyZE
https://api.github.com/repos/huggingface/datasets/issues/comments/1484990020/reactions
2023-03-27T11:39:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1484990020
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
CONTRIBUTOR
Oh I didn't know that! OK but in any case, not sure why the image builder isn't working for @WiNE-iNEFF. But it's hard for us to help if we can't reproduce. I'd just check the structure of the folders, see if the splits are correctly set up, etc.
2023-03-27T17:11:48Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1485528178
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5Yi1xy
https://api.github.com/repos/huggingface/datasets/issues/comments/1485528178/reactions
2023-03-27T17:11:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1485528178
https://avatars.githubusercontent.com/u/48327001?v=4
https://api.github.com/users/NielsRogge/events{/privacy}
https://api.github.com/users/NielsRogge/followers
https://api.github.com/users/NielsRogge/following{/other_user}
https://api.github.com/users/NielsRogge/gists{/gist_id}
https://github.com/NielsRogge
NielsRogge
MDQ6VXNlcjQ4MzI3MDAx
https://api.github.com/users/NielsRogge/orgs
https://api.github.com/users/NielsRogge/received_events
https://api.github.com/users/NielsRogge/repos
https://api.github.com/users/NielsRogge/starred{/owner}{/repo}
https://api.github.com/users/NielsRogge/subscriptions
User
https://api.github.com/users/NielsRogge
public
NONE
> @WiNE-iNEFF have you tried the suggestion I posted above? with removing split names from filenames and structuring files in folders? 
> 
> 
> > even using the method that you wrote above absolutely nothing changes
> 
> fyi - nothing changed because these two approaches are basically the same. it's just that when you pass your data directory as a dataset name (`load_dataset("my_folder_name"`), not as `data_dir` (`load_dataset("imagefolder", data_dir="my_folder_name"`), `datasets` infers what module to use (`imagefolder` in your case) automatically, by file extensions.

I'll try to try your method over the next few days, then I'll write it turned out 
2023-03-27T17:14:15Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1485531438
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5Yi2ku
https://api.github.com/repos/huggingface/datasets/issues/comments/1485531438/reactions
2023-03-27T17:14:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1485531438
https://avatars.githubusercontent.com/u/41611046?v=4
https://api.github.com/users/WiNE-iNEFF/events{/privacy}
https://api.github.com/users/WiNE-iNEFF/followers
https://api.github.com/users/WiNE-iNEFF/following{/other_user}
https://api.github.com/users/WiNE-iNEFF/gists{/gist_id}
https://github.com/WiNE-iNEFF
WiNE-iNEFF
MDQ6VXNlcjQxNjExMDQ2
https://api.github.com/users/WiNE-iNEFF/orgs
https://api.github.com/users/WiNE-iNEFF/received_events
https://api.github.com/users/WiNE-iNEFF/repos
https://api.github.com/users/WiNE-iNEFF/starred{/owner}{/repo}
https://api.github.com/users/WiNE-iNEFF/subscriptions
User
https://api.github.com/users/WiNE-iNEFF
public
NONE
> @WiNE-iNEFF have you tried the suggestion I posted above? with removing split names from filenames and structuring files in folders? 
> 
> 
> > even using the method that you wrote above absolutely nothing changes
> 
> fyi - nothing changed because these two approaches are basically the same. it's just that when you pass your data directory as a dataset name (`load_dataset("my_folder_name"`), not as `data_dir` (`load_dataset("imagefolder", data_dir="my_folder_name"`), `datasets` infers what module to use (`imagefolder` in your case) automatically, by file extensions.

I tried creating a `train` folder and put my image folders in it. As a result, all 18,000 images were loaded. 
2023-03-27T17:23:26Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1485542704
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5Yi5Uw
https://api.github.com/repos/huggingface/datasets/issues/comments/1485542704/reactions
2023-03-27T17:23:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1485542704
https://avatars.githubusercontent.com/u/41611046?v=4
https://api.github.com/users/WiNE-iNEFF/events{/privacy}
https://api.github.com/users/WiNE-iNEFF/followers
https://api.github.com/users/WiNE-iNEFF/following{/other_user}
https://api.github.com/users/WiNE-iNEFF/gists{/gist_id}
https://github.com/WiNE-iNEFF
WiNE-iNEFF
MDQ6VXNlcjQxNjExMDQ2
https://api.github.com/users/WiNE-iNEFF/orgs
https://api.github.com/users/WiNE-iNEFF/received_events
https://api.github.com/users/WiNE-iNEFF/repos
https://api.github.com/users/WiNE-iNEFF/starred{/owner}{/repo}
https://api.github.com/users/WiNE-iNEFF/subscriptions
User
https://api.github.com/users/WiNE-iNEFF
public
CONTRIBUTOR
@WiNE-iNEFF great! So to explain what happened according to my assumptions:

When you use a standard packaged loader (like `imagefolder`, `csv`, `jsonl`, and so on) and load your data like `load_dataset("my_folder_name")` or `load_dataset("imagefolder", data_dir="my_folder_name"`, the library searches for patterns to divide files into splits. This is described a bit in [this doc](https://huggingface.co/docs/datasets/v2.10.0/en/repository_structure#splits-and-file-names). And the order to search for patterns is the following:
1. first it checks for [pattern like `data/<split_name>-xxxxx-of-xxxxx`](https://huggingface.co/docs/datasets/v2.10.0/en/repository_structure#custom-split-names) (which allows to pass custom split names)
2. then for directories named as splits (if you have directories named `train`, `test` etc.)
3. then for [splits in filenames](https://huggingface.co/docs/datasets/v2.10.0/en/repository_structure#splits-and-file-names) (like if you have files named `train-image.jpg`, `test_0.jpg`, ...)
4. then if no pattern was found, it treats all files as belonging to a single `train` split

The code is [here](https://github.com/huggingface/datasets/blob/main/src/datasets/data_files.py#L215).
So I assume that in your case, since you didn't have directories for splits (pattern 2), some files that included split keywords (pattern 3) were included and others were ignored as not matching the pattern. And when you added `train` directory, the pattern for directories (pattern 2) was triggered first and everything worked as expected. Everything worked in your previous cases probably because you didn't have split names keywords in filenames, so all the files ended up being a part of a single train split (pattern 4).

Another way to mitigate this apart from structuring your data according to the patterns is to explicitly state with files belong to which splits by passing them with `data_files` parameter:
```python
load_dataset("my_folder_name", data_files={"train": "**"})   # to tell that all files should be included  
```

Now I see that this order should be explained in documentation and also referenced in sections for packaged modules like `imagefolder`, thank you for pointing this out.  

 

2023-03-29T11:20:18Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1488412527
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5Yt19v
https://api.github.com/repos/huggingface/datasets/issues/comments/1488412527/reactions
2023-03-29T11:20:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1488412527
https://avatars.githubusercontent.com/u/16348744?v=4
https://api.github.com/users/polinaeterna/events{/privacy}
https://api.github.com/users/polinaeterna/followers
https://api.github.com/users/polinaeterna/following{/other_user}
https://api.github.com/users/polinaeterna/gists{/gist_id}
https://github.com/polinaeterna
polinaeterna
MDQ6VXNlcjE2MzQ4NzQ0
https://api.github.com/users/polinaeterna/orgs
https://api.github.com/users/polinaeterna/received_events
https://api.github.com/users/polinaeterna/repos
https://api.github.com/users/polinaeterna/starred{/owner}{/repo}
https://api.github.com/users/polinaeterna/subscriptions
User
https://api.github.com/users/polinaeterna
public
NONE
@NielsRogge @polinaeterna I have a similar problem when reading my dataset. I want to use DETR for object detection, but my data is in YOLO format. With a dataset of 10k images, yolo format involves having 10k labels. As far as I read regarding [COCO format](https://auto.gluon.ai/stable/tutorials/multimodal/object_detection/data_preparation/convert_data_to_coco_format.html), there must be one JSON per split. However, as I post in the [Hugging Face forum](https://discuss.huggingface.co/t/prepare-dataset-from-yolo-format-to-coco-for-detr/34894), when it is read, the number of rows is 1, which does not make sense. 
The instruction to read the train-val-test splits are: 
```python
from datasets import load_dataset
data_files = {
	"train": './train_labels.json',
	"validation": './val_labels.json',
	"test": './test_labels.json'
}
dataset = load_dataset("json", data_files=data_files)
```
An example of the short version of the json file I read, to reproduce my error, is the following: 

``` json
{
    "info": {},
    "licenses": [],
    "images": [
        {
            "id": 1,
            "file_name": "aceca_100.mp4frame21.png",
            "width": 1280,
            "height": 720,
            "pixel_values": null,
            "pixel_mask": null
            },
        {
            "id": 2,
            "file_name": "aceca_100.mp4frame24.png",
            "width": 1280,
            "height": 720,
            "pixel_values": null,
            "pixel_mask": null
        },
        {
            "id": 3,
            "file_name": "aceca_100.mp4frame25.png",
            "width": 1280,
            "height": 720,
            "pixel_values": null,
            "pixel_mask": null}],
            "annotations": [
                {
                    "id": 1,
                    "image_id": 1,
                    "category_id": 0,
                    "bbox": [0.0, 278.21896388398557, 86.94096523844935, 156.0293445072134],
                    "area": 13565.341816979679,
                    "iscrowd": 0
                },
                {
                    "id": 2,
                    "image_id": 2,
                    "category_id": 0,
                    "bbox": [149.28851295721816, 297.6359759754418, 34.76802347007475, 98.03908698442889],
                    "area": 3408.625277259324,
                    "iscrowd": 0
                },
                {
                    "id": 3,
                    "image_id": 3,
                    "category_id": 0,
                    "bbox": [153.3817197549372, 300.168969412891, 31.787555842913775, 89.69583163436312],
                    "area": 2851.2112569539095,
                    "iscrowd": 0
                }
            ],
            "categories": [
                {
                    "id": 0, "name": "person"
                }
            ]
        }
```
If full files required, my email is aruigui98@gmail.com
2023-03-31T08:06:57Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1491490433
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5Y5laB
https://api.github.com/repos/huggingface/datasets/issues/comments/1491490433/reactions
2023-03-31T08:17:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1491490433
https://avatars.githubusercontent.com/u/91409881?v=4
https://api.github.com/users/Alberto1404/events{/privacy}
https://api.github.com/users/Alberto1404/followers
https://api.github.com/users/Alberto1404/following{/other_user}
https://api.github.com/users/Alberto1404/gists{/gist_id}
https://github.com/Alberto1404
Alberto1404
MDQ6VXNlcjkxNDA5ODgx
https://api.github.com/users/Alberto1404/orgs
https://api.github.com/users/Alberto1404/received_events
https://api.github.com/users/Alberto1404/repos
https://api.github.com/users/Alberto1404/starred{/owner}{/repo}
https://api.github.com/users/Alberto1404/subscriptions
User
https://api.github.com/users/Alberto1404
public
CONTRIBUTOR
Hi @Alberto1404, to load an object detection dataset it's recommended to make use of the metadata feature as explained [here](https://huggingface.co/docs/datasets/image_dataset#object-detection).  
2023-03-31T15:40:46Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1492158247
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5Y8Icn
https://api.github.com/repos/huggingface/datasets/issues/comments/1492158247/reactions
2023-03-31T15:40:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1492158247
https://avatars.githubusercontent.com/u/48327001?v=4
https://api.github.com/users/NielsRogge/events{/privacy}
https://api.github.com/users/NielsRogge/followers
https://api.github.com/users/NielsRogge/following{/other_user}
https://api.github.com/users/NielsRogge/gists{/gist_id}
https://github.com/NielsRogge
NielsRogge
MDQ6VXNlcjQ4MzI3MDAx
https://api.github.com/users/NielsRogge/orgs
https://api.github.com/users/NielsRogge/received_events
https://api.github.com/users/NielsRogge/repos
https://api.github.com/users/NielsRogge/starred{/owner}{/repo}
https://api.github.com/users/NielsRogge/subscriptions
User
https://api.github.com/users/NielsRogge
public
NONE
Thank you @NielsRogge! It works!!!
2023-04-03T14:03:03Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1494381883
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5ZEnU7
https://api.github.com/repos/huggingface/datasets/issues/comments/1494381883/reactions
2023-04-03T14:03:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1494381883
https://avatars.githubusercontent.com/u/91409881?v=4
https://api.github.com/users/Alberto1404/events{/privacy}
https://api.github.com/users/Alberto1404/followers
https://api.github.com/users/Alberto1404/following{/other_user}
https://api.github.com/users/Alberto1404/gists{/gist_id}
https://github.com/Alberto1404
Alberto1404
MDQ6VXNlcjkxNDA5ODgx
https://api.github.com/users/Alberto1404/orgs
https://api.github.com/users/Alberto1404/received_events
https://api.github.com/users/Alberto1404/repos
https://api.github.com/users/Alberto1404/starred{/owner}{/repo}
https://api.github.com/users/Alberto1404/subscriptions
User
https://api.github.com/users/Alberto1404
public
COLLABORATOR
You can now refer to https://huggingface.co/docs/datasets/repository_structure to learn about the `datasets`' data files inference, so I'm closing this issue.
2023-07-24T14:13:01Z
https://github.com/huggingface/datasets/issues/5650#issuecomment-1647999990
https://api.github.com/repos/huggingface/datasets/issues/5650
IC_kwDODunzps5iOnv2
https://api.github.com/repos/huggingface/datasets/issues/comments/1647999990/reactions
2023-07-24T14:13:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1647999990
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
I have about 20000 images in my folder which divided into 4 folders with class names.
When i use load_dataset("my_folder_name", split="train") this function create dataset in which there are only 4 images, the remaining 19000 images were not added there. What is the problem and did not understand. Tried converting images and the like but absolutely nothing worked