Add filename in error message when ReadError or similar occur
MEMBER
Hi Elisa, please share the error traceback here, and if you manage to find the location in the `datasets` code where the error occurs, feel free to open a PR to add the necessary logging / improve the error message.
2024-11-18T11:40:23Z
https://github.com/huggingface/datasets/issues/7280#issuecomment-2482797690
https://api.github.com/repos/huggingface/datasets/issues/7280
IC_kwDODunzps6T_Hx6
https://api.github.com/repos/huggingface/datasets/issues/comments/2482797690/reactions
2024-11-18T11:40:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2482797690
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> please share the error traceback

I don't have access to it but it should be during [this exception](https://github.com/huggingface/datasets/blob/2049c00921c59cdeb835137a1c49639cf175af07/src/datasets/builder.py#L1643) which happens during the loading of a dataset. If one of the downloaded files is corrupted, the for loop will not yield correctly, and the error will come from, say, in the case of tar files, [this iterable](https://github.com/huggingface/datasets/blob/2049c00921c59cdeb835137a1c49639cf175af07/src/datasets/utils/file_utils.py#L1293) which has no explicit error handling that leaves clues as to which file has failed.

I only know the case for tar files, but I consider this issue could be happening across different file types too.
2024-11-18T14:19:57Z
https://github.com/huggingface/datasets/issues/7280#issuecomment-2483179382
https://api.github.com/repos/huggingface/datasets/issues/7280
IC_kwDODunzps6UAk92
https://api.github.com/repos/huggingface/datasets/issues/comments/2483179382/reactions
2024-11-18T14:19:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2483179382
https://avatars.githubusercontent.com/u/37046039?v=4
https://api.github.com/users/elisa-aleman/events{/privacy}
https://api.github.com/users/elisa-aleman/followers
https://api.github.com/users/elisa-aleman/following{/other_user}
https://api.github.com/users/elisa-aleman/gists{/gist_id}
https://github.com/elisa-aleman
elisa-aleman
MDQ6VXNlcjM3MDQ2MDM5
https://api.github.com/users/elisa-aleman/orgs
https://api.github.com/users/elisa-aleman/received_events
https://api.github.com/users/elisa-aleman/repos
https://api.github.com/users/elisa-aleman/starred{/owner}{/repo}
https://api.github.com/users/elisa-aleman/subscriptions
User
https://api.github.com/users/elisa-aleman
public
MEMBER
I think having a better error handling for this tar iterable would be useful already, maybe a simple try/except in `_iter_from_urlpath` that checks for `tarfile.ReadError` and raises an error with the `urlpath` mentioned in the error ?
2024-11-18T15:07:32Z
https://github.com/huggingface/datasets/issues/7280#issuecomment-2483321337
https://api.github.com/repos/huggingface/datasets/issues/7280
IC_kwDODunzps6UBHn5
https://api.github.com/repos/huggingface/datasets/issues/comments/2483321337/reactions
2024-11-18T15:07:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2483321337
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I think not just from higher calls like the `_iter_from_urlpath` but directly wherever a file is attempted to be opened would be the best case, as the traceback would simply lead to that.
2024-11-19T05:03:01Z
https://github.com/huggingface/datasets/issues/7280#issuecomment-2484715143
https://api.github.com/repos/huggingface/datasets/issues/7280
IC_kwDODunzps6UGb6H
https://api.github.com/repos/huggingface/datasets/issues/comments/2484715143/reactions
2024-11-19T05:03:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2484715143
https://avatars.githubusercontent.com/u/37046039?v=4
https://api.github.com/users/elisa-aleman/events{/privacy}
https://api.github.com/users/elisa-aleman/followers
https://api.github.com/users/elisa-aleman/following{/other_user}
https://api.github.com/users/elisa-aleman/gists{/gist_id}
https://github.com/elisa-aleman
elisa-aleman
MDQ6VXNlcjM3MDQ2MDM5
https://api.github.com/users/elisa-aleman/orgs
https://api.github.com/users/elisa-aleman/received_events
https://api.github.com/users/elisa-aleman/repos
https://api.github.com/users/elisa-aleman/starred{/owner}{/repo}
https://api.github.com/users/elisa-aleman/subscriptions
User
https://api.github.com/users/elisa-aleman
public
MEMBER
so maybe there should be better error messages in each dataset builder definition ? e.g. in https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/webdataset/webdataset.py for webdataset TAR archives
2024-11-20T13:23:10Z
https://github.com/huggingface/datasets/issues/7280#issuecomment-2488572334
https://api.github.com/repos/huggingface/datasets/issues/7280
IC_kwDODunzps6UVJmu
https://api.github.com/repos/huggingface/datasets/issues/comments/2488572334/reactions
2024-11-20T13:23:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2488572334
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Please update error messages to include relevant information for debugging when loading datasets with `load_dataset()` that may have a few corrupted files. 

Whenever downloading a full dataset, some files might be corrupted (either at the source or from downloading corruption). 

However the errors often only let me know it was a tar file if `tarfile.ReadError` appears on the traceback, and I imagine similarly for other file types. 

This makes it really hard to debug which file is corrupted, and when dealing with very large datasets, it shouldn't be necessary to force download everything again.