How to skip a example when running dataset.map
MEMBER
Hi @xixiaoyao,
Depending on what you want to do you can:
- use a first step of `filter` to filter out the invalid examples: https://huggingface.co/docs/datasets/processing.html#filtering-rows-select-and-filter
- or directly detect the invalid examples inside the callable used with `map` and return them unchanged or even remove them at the same time if you are using `map` in batched mode. Here is an example where we use `map` in batched mode to add new rows on the fly but you can also use it to remove examples on the fly (that's what `filter` actually do under-the-hood): https://huggingface.co/docs/datasets/processing.html#augmenting-the-dataset
2020-09-25T12:00:01Z
https://github.com/huggingface/datasets/issues/669#issuecomment-698889036
https://api.github.com/repos/huggingface/datasets/issues/669
MDEyOklzc3VlQ29tbWVudDY5ODg4OTAzNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/698889036/reactions
2020-09-25T12:00:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/698889036
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Closing this one.
Feel free to re-open if you have other questions
2020-10-05T16:28:12Z
https://github.com/huggingface/datasets/issues/669#issuecomment-703742751
https://api.github.com/repos/huggingface/datasets/issues/669
MDEyOklzc3VlQ29tbWVudDcwMzc0Mjc1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/703742751/reactions
2020-10-05T16:28:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/703742751
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Letting finders-of-this-thread know that the new link is: https://huggingface.co/docs/datasets/process#data-augmentation

2022-06-17T21:45:02Z
https://github.com/huggingface/datasets/issues/669#issuecomment-1159247781
https://api.github.com/repos/huggingface/datasets/issues/669
IC_kwDODunzps5FGLel
https://api.github.com/repos/huggingface/datasets/issues/comments/1159247781/reactions
2022-06-17T21:45:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1159247781
https://avatars.githubusercontent.com/u/99206017?v=4
https://api.github.com/users/plamb-viso/events{/privacy}
https://api.github.com/users/plamb-viso/followers
https://api.github.com/users/plamb-viso/following{/other_user}
https://api.github.com/users/plamb-viso/gists{/gist_id}
https://github.com/plamb-viso
plamb-viso
U_kgDOBenDgQ
https://api.github.com/users/plamb-viso/orgs
https://api.github.com/users/plamb-viso/received_events
https://api.github.com/users/plamb-viso/repos
https://api.github.com/users/plamb-viso/starred{/owner}{/repo}
https://api.github.com/users/plamb-viso/subscriptions
User
https://api.github.com/users/plamb-viso
public
in processing func, I process examples and detect some invalid examples, which I did not want it to be added into train dataset. However I did not find how to skip this recognized invalid example when doing dataset.map. 