Add ignore_decode_errors option to Image feature for robust decoding #7612
CONTRIBUTOR
cc @lhoestq
2025-06-24T16:48:03Z
https://github.com/huggingface/datasets/pull/7638#issuecomment-3001189357
https://api.github.com/repos/huggingface/datasets/issues/7638
IC_kwDODunzps6y4oPt
https://api.github.com/repos/huggingface/datasets/issues/comments/3001189357/reactions
2025-06-24T16:48:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3001189357
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
NONE
I think splitting the error handling for the main image decoding process and the metadata decoding process is possibly a bit nicer, as some images do render correctly, but their metadata might be invalid and cause the pipeline to fail, which I've encountered recently as in #7668.

The [`decode_image`](https://docs.pytorch.org/vision/main/generated/torchvision.io.decode_image.html) function in `torchvision` handles similar cases by using the `apply_exif_orientation` flag to turn off the exif metadata processing entirely.
2025-07-03T12:35:39Z
https://github.com/huggingface/datasets/pull/7638#issuecomment-3032106052
https://api.github.com/repos/huggingface/datasets/issues/7638
IC_kwDODunzps60ukRE
https://api.github.com/repos/huggingface/datasets/issues/comments/3032106052/reactions
2025-07-03T12:39:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3032106052
https://avatars.githubusercontent.com/u/30485844?v=4
https://api.github.com/users/Seas0/events{/privacy}
https://api.github.com/users/Seas0/followers
https://api.github.com/users/Seas0/following{/other_user}
https://api.github.com/users/Seas0/gists{/gist_id}
https://github.com/Seas0
Seas0
MDQ6VXNlcjMwNDg1ODQ0
https://api.github.com/users/Seas0/orgs
https://api.github.com/users/Seas0/received_events
https://api.github.com/users/Seas0/repos
https://api.github.com/users/Seas0/starred{/owner}{/repo}
https://api.github.com/users/Seas0/subscriptions
User
https://api.github.com/users/Seas0
public
CONTRIBUTOR
> I think splitting the error handling for the main image decoding process and the metadata decoding process is possibly a bit nicer, as some images do render correctly, but their metadata might be invalid and cause the pipeline to fail, which I've encountered recently as in #7668.
> The [`decode_image`](https://docs.pytorch.org/vision/main/generated/torchvision.io.decode_image.html) function in `torchvision` handles similar cases by using the `apply_exif_orientation` flag to turn off the exif metadata processing entirely.
 
 @lhoestq & @Seas0 — that makes total sense.
 
Currently, if EXIF metadata like `.getexif()` fails (due to malformed tags), the whole image gets dropped even if it renders correctly — not ideal.
 
To address this, I'm planning to split the EXIF handling into a separate `try/except` block, like:
```python
try:
    exif = image.getexif()
    if exif.get(PIL.Image.ExifTags.Base.Orientation) is not None:
        image = PIL.ImageOps.exif_transpose(image)
except Exception as exif_err:
    if self.ignore_decode_errors:
        warnings.warn(f"[Image.decode_example] Skipped EXIF metadata: {exif_err}")
    else:
        raise
```

So that, Valid but EXIF-broken images will still be returned & EXIF failures will be skipped only if ignore_decode_errors=True. 

Sounds good??
2025-07-03T16:34:52Z
https://github.com/huggingface/datasets/pull/7638#issuecomment-3032888234
https://api.github.com/repos/huggingface/datasets/issues/7638
IC_kwDODunzps60xjOq
https://api.github.com/repos/huggingface/datasets/issues/comments/3032888234/reactions
2025-07-03T16:37:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3032888234
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
CONTRIBUTOR
With the recent EXIF decoding isolation logic added, this PR now fully addresses:

- ✅ #7612 – Robust iteration over corrupt samples (especially useful in `.streaming=True`)
- ✅ #7632 – Graceful handling of invalid image files when using `cast_column(..., Image(...))`
- ✅ #7668 – Broken EXIF metadata no longer crashes decoding; images are returned if usable

All decoding errors (including `.getexif()` and image file loading) are now skipped with a warning when `ignore_decode_errors=True`. This enables safe, scalable image preprocessing pipelines.
2025-07-04T07:07:29Z
https://github.com/huggingface/datasets/pull/7638#issuecomment-3034769764
https://api.github.com/repos/huggingface/datasets/issues/7638
IC_kwDODunzps604ulk
https://api.github.com/repos/huggingface/datasets/issues/comments/3034769764/reactions
2025-07-04T07:07:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/3034769764
https://avatars.githubusercontent.com/u/142811259?v=4
https://api.github.com/users/ArjunJagdale/events{/privacy}
https://api.github.com/users/ArjunJagdale/followers
https://api.github.com/users/ArjunJagdale/following{/other_user}
https://api.github.com/users/ArjunJagdale/gists{/gist_id}
https://github.com/ArjunJagdale
ArjunJagdale
U_kgDOCIMgew
https://api.github.com/users/ArjunJagdale/orgs
https://api.github.com/users/ArjunJagdale/received_events
https://api.github.com/users/ArjunJagdale/repos
https://api.github.com/users/ArjunJagdale/starred{/owner}{/repo}
https://api.github.com/users/ArjunJagdale/subscriptions
User
https://api.github.com/users/ArjunJagdale
public
This PR implements support for robust image decoding in the `Image` feature, as discussed in issue #7612.

## 🔧 What was added
- A new boolean field: `ignore_decode_errors` (default: `False`)
- If set to `True`, any exceptions during decoding will be caught, and `None` will be returned instead of raising an error

```python
features = Features({
    "image": Image(decode=True, ignore_decode_errors=True),
})
````

This enables robust iteration over potentially corrupted datasets — especially useful when streaming datasets like WebDataset or image-heavy public sets where sample corruption is common.

## 🧪 Behavior

* If `ignore_decode_errors=False` (default), decoding behaves exactly as before
* If `True`, decoding errors are caught, and a warning is emitted:

  ```
  [Image.decode_example] Skipped corrupted image: ...
  ```

## 🧵 Linked issue

Closes #7612

Let me know if you'd like a follow-up test PR. Happy to write one!