Fix key error in webdataset
MEMBER
hi ! What version of `datasets` are you using ? Is this issue also happening with `datasets==3.0.0` ?
Asking because we made sure to replicate the official webdataset logic, which is to use the latest dot as separator between the sample base name and the key
2024-09-12T14:48:58Z
https://github.com/huggingface/datasets/pull/7144#issuecomment-2346515957
https://api.github.com/repos/huggingface/datasets/issues/7144
IC_kwDODunzps6L3P31
https://api.github.com/repos/huggingface/datasets/issues/comments/2346515957/reactions
2024-09-12T14:48:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2346515957
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi, yes this is still a problem on `datasets==3.0.0`.

I was using `datasets=2.20.0` and in that version you get the key error.

I just upgraded to `datasets==3.0.0` and in this version, you do not get a key error because it sets all keys to none by default   in `_generate_examples` function:

```python
if field_name not in example:
    example[field_name] = None
```

However, the behaviour is still incorrect. This `if` condition is triggered because the filename is not split properly and it returns the data as `None` when it shouldn't.

> we made sure to replicate the official webdataset logic, which is to use the latest dot as separator
 
Ah, but that's not what `split(, 1)` does though. This is exactly why I'm suggesting to use `rsplit` instead. In general, using `rsplit` should not be a breaking change I believe.
2024-09-13T00:26:58Z
https://github.com/huggingface/datasets/pull/7144#issuecomment-2347576374
https://api.github.com/repos/huggingface/datasets/issues/7144
IC_kwDODunzps6L7Sw2
https://api.github.com/repos/huggingface/datasets/issues/comments/2347576374/reactions
2024-09-13T00:32:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2347576374
https://avatars.githubusercontent.com/u/26804893?v=4
https://api.github.com/users/ragavsachdeva/events{/privacy}
https://api.github.com/users/ragavsachdeva/followers
https://api.github.com/users/ragavsachdeva/following{/other_user}
https://api.github.com/users/ragavsachdeva/gists{/gist_id}
https://github.com/ragavsachdeva
ragavsachdeva
MDQ6VXNlcjI2ODA0ODkz
https://api.github.com/users/ragavsachdeva/orgs
https://api.github.com/users/ragavsachdeva/received_events
https://api.github.com/users/ragavsachdeva/repos
https://api.github.com/users/ragavsachdeva/starred{/owner}{/repo}
https://api.github.com/users/ragavsachdeva/subscriptions
User
https://api.github.com/users/ragavsachdeva
public
MEMBER
Hi  @ragavsachdeva,

We already had this discussion in the issue you have linked:
- #6880
- I even opened a PR with your proposed fix:
  - #6888

However, we decided not to implement this feature because it is NOT aligned with the behavior of the `webdataset` library:
> The prefix of a file is all directory components of the file plus the file name component up to the *first* “.” in the file name.
```python
In [1]: import webdataset as wds

In [2]: wds.tariterators.base_plus_ext("22.05.png")
Out[2]: ('22', '05.png')
```


2024-09-13T04:31:37Z
https://github.com/huggingface/datasets/pull/7144#issuecomment-2348012361
https://api.github.com/repos/huggingface/datasets/issues/7144
IC_kwDODunzps6L89NJ
https://api.github.com/repos/huggingface/datasets/issues/comments/2348012361/reactions
2024-09-13T04:38:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2348012361
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Ah, my apologies I missed https://github.com/huggingface/datasets/pull/6888 (clearly didn't do my due diligence). It's such a weird convention to have though. My keys are `/some/path/22.0/1.1.png` and it splits them at  `/some/path/22` and `.0/1.1.png`(!) I'm okay with this PR not being merged though. Thanks for your time.
2024-09-13T08:09:14Z
https://github.com/huggingface/datasets/pull/7144#issuecomment-2348307792
https://api.github.com/repos/huggingface/datasets/issues/7144
IC_kwDODunzps6L-FVQ
https://api.github.com/repos/huggingface/datasets/issues/comments/2348307792/reactions
2024-09-13T08:09:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2348307792
https://avatars.githubusercontent.com/u/26804893?v=4
https://api.github.com/users/ragavsachdeva/events{/privacy}
https://api.github.com/users/ragavsachdeva/followers
https://api.github.com/users/ragavsachdeva/following{/other_user}
https://api.github.com/users/ragavsachdeva/gists{/gist_id}
https://github.com/ragavsachdeva
ragavsachdeva
MDQ6VXNlcjI2ODA0ODkz
https://api.github.com/users/ragavsachdeva/orgs
https://api.github.com/users/ragavsachdeva/received_events
https://api.github.com/users/ragavsachdeva/repos
https://api.github.com/users/ragavsachdeva/starred{/owner}{/repo}
https://api.github.com/users/ragavsachdeva/subscriptions
User
https://api.github.com/users/ragavsachdeva
public
MEMBER
Actually `datasets` is not behaving correctly in this case and should not split as `.0/1.1.png` - even webdataset handles this correctly via their regex `^((?:.*/|)[^.]+)[.]([^/]*)$` in `wds.tariterators.base_plus_ext` here:

https://github.com/webdataset/webdataset/blob/87bd5aa41602d57f070f65a670893ee625702f2f/webdataset/tariterators.py#L36
2024-09-13T13:29:23Z
https://github.com/huggingface/datasets/pull/7144#issuecomment-2348967609
https://api.github.com/repos/huggingface/datasets/issues/7144
IC_kwDODunzps6MAma5
https://api.github.com/repos/huggingface/datasets/issues/comments/2348967609/reactions
2024-09-13T13:31:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2348967609
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Oh.. the intention with that regex is to capture "multi-part" extensions e.g. `.tar.gz`. Makes sense. So `rsplit` isn't the solution then and neither is `split`. This expression makes so much more sense. Nice find! I'm assuming you'll add a patch?
2024-09-13T14:01:33Z
https://github.com/huggingface/datasets/pull/7144#issuecomment-2349038443
https://api.github.com/repos/huggingface/datasets/issues/7144
IC_kwDODunzps6MA3tr
https://api.github.com/repos/huggingface/datasets/issues/comments/2349038443/reactions
2024-09-13T14:01:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2349038443
https://avatars.githubusercontent.com/u/26804893?v=4
https://api.github.com/users/ragavsachdeva/events{/privacy}
https://api.github.com/users/ragavsachdeva/followers
https://api.github.com/users/ragavsachdeva/following{/other_user}
https://api.github.com/users/ragavsachdeva/gists{/gist_id}
https://github.com/ragavsachdeva
ragavsachdeva
MDQ6VXNlcjI2ODA0ODkz
https://api.github.com/users/ragavsachdeva/orgs
https://api.github.com/users/ragavsachdeva/received_events
https://api.github.com/users/ragavsachdeva/repos
https://api.github.com/users/ragavsachdeva/starred{/owner}{/repo}
https://api.github.com/users/ragavsachdeva/subscriptions
User
https://api.github.com/users/ragavsachdeva
public
MEMBER
Issue addressed by:
- #7151
2024-09-16T06:23:36Z
https://github.com/huggingface/datasets/pull/7144#issuecomment-2352097904
https://api.github.com/repos/huggingface/datasets/issues/7144
IC_kwDODunzps6MMipw
https://api.github.com/repos/huggingface/datasets/issues/comments/2352097904/reactions
2024-09-16T06:23:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2352097904
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
 Maybe the libray should suggest/warn users about **file naming**, in particular, file naming **with dots**. Indeed, otherwise a naive naming approach like mentioned above would lead to

```python
ValueError: The TAR archives of the dataset should be in WebDataset format, but the files in the archive don't share the same prefix or the same types.
```

with the following code

```ipython
In [1]: from datasets import load_dataset

In [2]: !tar tvf easy_trouble/001.tar
drwxr-xr-x phunc20/phunc20   0 2025-01-15 17:57 001/
-rw-r--r-- phunc20/phunc20  20 2025-01-15 17:54 001/1.2.jpg
-rw-r--r-- phunc20/phunc20  20 2025-01-15 17:53 001/1.1.jpg
-rw-r--r-- phunc20/phunc20  20 2025-01-15 17:57 001/2.1.jpg
-rw-r--r-- phunc20/phunc20  20 2025-01-15 17:57 001/2.2.jpg
-rw-r--r-- phunc20/phunc20  20 2025-01-15 17:57 001/2.3.jpg

In [29]: dataset = load_dataset("webdataset", data_dir="easy_trouble")
```

BTW, @ragavsachdeva , why would you find including a `.tar.gz` inside a `.tar` makes sense? I personally find switching `base_plus_ext` into sth similar to the following makes more sense, though. (That is, no multi-part extension.)

```python
from pathlib import Path

def base_plus_ext(path):
    """Split off all file extensions.

    Returns base, allext.
    """
    p = Path(path)
    base = (p.parent / p.stem).as_posix()
    ext = p.suffix.split(".")[-1]
    return base, ext
```
2025-01-15T10:24:24Z
https://github.com/huggingface/datasets/pull/7144#issuecomment-2592247761
https://api.github.com/repos/huggingface/datasets/issues/7144
IC_kwDODunzps6ago_R
https://api.github.com/repos/huggingface/datasets/issues/comments/2592247761/reactions
2025-01-15T10:32:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2592247761
https://avatars.githubusercontent.com/u/44637303?v=4
https://api.github.com/users/phunc20/events{/privacy}
https://api.github.com/users/phunc20/followers
https://api.github.com/users/phunc20/following{/other_user}
https://api.github.com/users/phunc20/gists{/gist_id}
https://github.com/phunc20
phunc20
MDQ6VXNlcjQ0NjM3MzAz
https://api.github.com/users/phunc20/orgs
https://api.github.com/users/phunc20/received_events
https://api.github.com/users/phunc20/repos
https://api.github.com/users/phunc20/starred{/owner}{/repo}
https://api.github.com/users/phunc20/subscriptions
User
https://api.github.com/users/phunc20
public
I was running into

```
example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
KeyError: 'png'
```

The issue is that a filename may have multiple "." e.g. `22.05.png`. Changing `split` to `rsplit` fixes it.

Related https://github.com/huggingface/datasets/issues/6880