Stop dropping columns in to_tf_dataset() before we load batches
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-06-23T18:33:05Z
https://github.com/huggingface/datasets/pull/4553#issuecomment-1164740332
https://api.github.com/repos/huggingface/datasets/issues/4553
IC_kwDODunzps5FbIbs
https://api.github.com/repos/huggingface/datasets/issues/comments/1164740332/reactions
2022-07-04T19:00:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1164740332
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
@lhoestq Rebasing fixed the test failures, so this should be ready to review now! There's still a failure on Win but it seems unrelated.
2022-06-27T14:49:56Z
https://github.com/huggingface/datasets/pull/4553#issuecomment-1167450717
https://api.github.com/repos/huggingface/datasets/issues/4553
IC_kwDODunzps5FleJd
https://api.github.com/repos/huggingface/datasets/issues/comments/1167450717/reactions
2022-06-27T14:49:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1167450717
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
Gentle ping @lhoestq ! This is a simple fix (dropping columns after loading a batch from the dataset rather than with `.remove_columns()` to make sure we don't break transforms), and tests are green so we're ready for review!
2022-06-29T17:59:55Z
https://github.com/huggingface/datasets/pull/4553#issuecomment-1170313325
https://api.github.com/repos/huggingface/datasets/issues/4553
IC_kwDODunzps5FwZBt
https://api.github.com/repos/huggingface/datasets/issues/comments/1170313325/reactions
2022-06-29T17:59:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1170313325
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
@lhoestq Test is in!
2022-07-01T20:36:30Z
https://github.com/huggingface/datasets/pull/4553#issuecomment-1172696411
https://api.github.com/repos/huggingface/datasets/issues/4553
IC_kwDODunzps5F5e1b
https://api.github.com/repos/huggingface/datasets/issues/comments/1172696411/reactions
2022-07-01T20:36:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1172696411
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
`to_tf_dataset()` dropped unnecessary columns before loading batches from the dataset, but this is causing problems when using a transform, because the dropped columns might be needed to compute the transform. Since there's no real way to check which columns the transform might need, we skip dropping columns and instead drop keys from the batch after we load it.

cc @amyeroberts and https://github.com/huggingface/notebooks/pull/202