[WebDataset] KeyError with user-defined `Features` when a field is missing in an example
CONTRIBUTOR
@lhoestq How difficult of fix is this?
2024-06-24T21:36:13Z
https://github.com/huggingface/datasets/issues/6900#issuecomment-2187450240
https://api.github.com/repos/huggingface/datasets/issues/6900
IC_kwDODunzps6CYdeA
https://api.github.com/repos/huggingface/datasets/issues/comments/2187450240/reactions
2024-06-24T21:36:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2187450240
https://avatars.githubusercontent.com/u/10626398?v=4
https://api.github.com/users/ProGamerGov/events{/privacy}
https://api.github.com/users/ProGamerGov/followers
https://api.github.com/users/ProGamerGov/following{/other_user}
https://api.github.com/users/ProGamerGov/gists{/gist_id}
https://github.com/ProGamerGov
ProGamerGov
MDQ6VXNlcjEwNjI2Mzk4
https://api.github.com/users/ProGamerGov/orgs
https://api.github.com/users/ProGamerGov/received_events
https://api.github.com/users/ProGamerGov/repos
https://api.github.com/users/ProGamerGov/starred{/owner}{/repo}
https://api.github.com/users/ProGamerGov/subscriptions
User
https://api.github.com/users/ProGamerGov
public
MEMBER
It shouldn't be difficult, I think it's just a matter of adding the missing fields from `self.config.features` in `example` here: before it iterates on image_field_names and audio_field_names. A missing field should have a value set to None

https://github.com/huggingface/datasets/blob/768cb35ede5a6c35fa7545aa3671f3e321c96440/src/datasets/packaged_modules/webdataset/webdataset.py#L113-L116
2024-06-25T10:26:44Z
https://github.com/huggingface/datasets/issues/6900#issuecomment-2188547531
https://api.github.com/repos/huggingface/datasets/issues/6900
IC_kwDODunzps6CcpXL
https://api.github.com/repos/huggingface/datasets/issues/comments/2188547531/reactions
2024-06-25T10:28:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2188547531
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq So like this then?

```    
def _generate_examples(self, tar_paths, tar_iterators):
        image_field_names = [
            field_name for field_name, feature in self.info.features.items() if isinstance(feature, datasets.Image)
        ]
        audio_field_names = [
            field_name for field_name, feature in self.info.features.items() if isinstance(feature, datasets.Audio)
        ]
	
        all_field_names = list(self.config.features.keys())
    
        for tar_idx, (tar_path, tar_iterator) in enumerate(zip(tar_paths, tar_iterators)):
            for example_idx, example in enumerate(self._get_pipeline_from_tar(tar_path, tar_iterator)):
                for field_name in all_field_names:
                    if field_name not in example:
                        if field_name in self.config.features:
                            example[field_name] = self.config.features[field_name]
                        else:
                            example[field_name] = None
    
                # Process image and audio fields
                for field_name in image_field_names + audio_field_names:
                    if example[field_name] is not None:
                        example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
    
                yield f"{tar_idx}_{example_idx}", example
```

Or should we avoid trying add the missing values and just set them to None?

```
            for field_name in all_field_names:
                if field_name not in example:
                    example[field_name] = None
```
2024-06-25T18:00:06Z
https://github.com/huggingface/datasets/issues/6900#issuecomment-2189629215
https://api.github.com/repos/huggingface/datasets/issues/6900
IC_kwDODunzps6Cgxcf
https://api.github.com/repos/huggingface/datasets/issues/comments/2189629215/reactions
2024-06-25T18:00:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2189629215
https://avatars.githubusercontent.com/u/10626398?v=4
https://api.github.com/users/ProGamerGov/events{/privacy}
https://api.github.com/users/ProGamerGov/followers
https://api.github.com/users/ProGamerGov/following{/other_user}
https://api.github.com/users/ProGamerGov/gists{/gist_id}
https://github.com/ProGamerGov
ProGamerGov
MDQ6VXNlcjEwNjI2Mzk4
https://api.github.com/users/ProGamerGov/orgs
https://api.github.com/users/ProGamerGov/received_events
https://api.github.com/users/ProGamerGov/repos
https://api.github.com/users/ProGamerGov/starred{/owner}{/repo}
https://api.github.com/users/ProGamerGov/subscriptions
User
https://api.github.com/users/ProGamerGov
public
MEMBER
Yup this is the solution !

```python
            for field_name in all_field_names:
                if field_name not in example:
                    example[field_name] = None
```
2024-06-26T11:47:56Z
https://github.com/huggingface/datasets/issues/6900#issuecomment-2191493322
https://api.github.com/repos/huggingface/datasets/issues/6900
IC_kwDODunzps6Cn4jK
https://api.github.com/repos/huggingface/datasets/issues/comments/2191493322/reactions
2024-06-26T11:47:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2191493322
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq Awesome, thanks! I made a PR with the fixes
2024-06-26T19:05:10Z
https://github.com/huggingface/datasets/issues/6900#issuecomment-2192442789
https://api.github.com/repos/huggingface/datasets/issues/6900
IC_kwDODunzps6CrgWl
https://api.github.com/repos/huggingface/datasets/issues/comments/2192442789/reactions
2024-06-26T19:05:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2192442789
https://avatars.githubusercontent.com/u/10626398?v=4
https://api.github.com/users/ProGamerGov/events{/privacy}
https://api.github.com/users/ProGamerGov/followers
https://api.github.com/users/ProGamerGov/following{/other_user}
https://api.github.com/users/ProGamerGov/gists{/gist_id}
https://github.com/ProGamerGov
ProGamerGov
MDQ6VXNlcjEwNjI2Mzk4
https://api.github.com/users/ProGamerGov/orgs
https://api.github.com/users/ProGamerGov/received_events
https://api.github.com/users/ProGamerGov/repos
https://api.github.com/users/ProGamerGov/starred{/owner}{/repo}
https://api.github.com/users/ProGamerGov/subscriptions
User
https://api.github.com/users/ProGamerGov
public
reported at https://huggingface.co/datasets/ProGamerGov/synthetic-dataset-1m-dalle3-high-quality-captions/discussions/1

```
File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/packaged_modules/webdataset/webdataset.py", line 109, in _generate_examples
    example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
```