to_tf_dataset rewrite
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-04-14T11:45:06Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1099101904
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5BgvbQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1099101904/reactions
2022-06-06T14:31:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1099101904
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
[Magic is now banned](https://www.youtube.com/watch?v=WIn58XoY728#t=36s) by decree of @sgugger. This is honestly much cleaner, and the functionality will make much more sense in `transformers` anyway!
2022-04-14T14:20:54Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1099237764
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5BhQmE
https://api.github.com/repos/huggingface/datasets/issues/comments/1099237764/reactions
2022-04-14T14:20:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1099237764
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
@gante I renamed the default collator to `minimal_tf_collate_fn`!
2022-04-14T16:13:08Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1099355102
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5BhtPe
https://api.github.com/repos/huggingface/datasets/issues/comments/1099355102/reactions
2022-04-14T16:13:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1099355102
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
@lhoestq @sgugger @gante 

I think this should now be ready, it looks good in testing! I'll try a few more notebooks today and tomorrow to be sure before I merge. Key changes are:

- No column autodetection magic (will make a separate PR to add this as a `transformers` function)
- Drops non-numerical features automatically (this is more of a 'DataLoader' method, we'll have a separate method to expose 'raw' datasets to `tf.data`)
- Better autodetection of numerical features.
- Shouldn't randomly crash mid-function :skull: 

We definitely have some questions still to resolve about how to handle making a 'DataLoader' dataset versus a 'raw' dataset - see [the Notion doc](https://www.notion.so/huggingface2/Splitting-to_tf_dataset-c2e0773c4bec484384064b30ed634383) if you're interested. Still, since this PR is just fixes/improvements to an existing method which never supported non-numerical features anyway, we can merge it before we've resolved those issues, and then think about how to name and split things afterwards.
2022-04-25T18:19:08Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1108892167
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5CGFoH
https://api.github.com/repos/huggingface/datasets/issues/comments/1108892167/reactions
2022-04-25T18:19:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1108892167
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
P.S. I'll take out the region comments at the end before I merge, I promise! They're just helpful while I'm editing it
2022-04-26T14:00:36Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1109835582
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5CJr8-
https://api.github.com/repos/huggingface/datasets/issues/comments/1109835582/reactions
2022-04-26T14:00:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1109835582
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
+1 for the tests

> Drops non-numerical features automatically

Can you give more details on how this work and the rationale as well ? This is not explained in the docs

Also why are you adding `error_on_missing` and `auto_fix_label_names ` ? The rationale is not clear to me. In particular I think it is sensible enough to expect users to not ask columns that don't exist, and to rename a label column when required.
2022-05-02T16:56:06Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1115119944
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5Cd2FI
https://api.github.com/repos/huggingface/datasets/issues/comments/1115119944/reactions
2022-05-02T17:00:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1115119944
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq I rewrote those parts - they were causing some other issues too! `error_on_missing` and `auto_fix_label_names` have been removed. The new logic is to simply drop (before batch collation) all columns the user doesn't ask for, but not to raise errors if the user asked for columns not in the dataset, as they may be added by the collator. Hopefully this cleans it up and matches the documentation better!
2022-05-05T16:38:12Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1118792781
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5Cr2xN
https://api.github.com/repos/huggingface/datasets/issues/comments/1118792781/reactions
2022-05-05T16:38:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1118792781
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
@lhoestq New tests are now in!
2022-05-17T14:50:17Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1128968768
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5DSrJA
https://api.github.com/repos/huggingface/datasets/issues/comments/1128968768/reactions
2022-05-17T14:50:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1128968768
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
Seeing some other random tests failing that don't look to be associated with this PR.
2022-05-17T16:44:21Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1129091773
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5DTJK9
https://api.github.com/repos/huggingface/datasets/issues/comments/1129091773/reactions
2022-05-17T16:44:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1129091773
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
@lhoestq I can't figure out these test failures! They don't seem related to this PR at all, but I rebased to the latest version and they keep happening, even though they're not visible on master.
2022-05-19T16:03:14Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1131908485
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5Dd42F
https://api.github.com/repos/huggingface/datasets/issues/comments/1131908485/reactions
2022-05-19T16:03:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1131908485
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
Thanks for the ping, will take a look tomorrow :)

Maybe the rebase didn't go well for the code recently merged about label alignment from https://github.com/huggingface/datasets/pull/4277 ?
2022-05-19T16:44:06Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1131947907
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5DeCeD
https://api.github.com/repos/huggingface/datasets/issues/comments/1131947907/reactions
2022-05-19T16:44:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1131947907
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
It's very strange! The rebase looks fine to me. I might try to move my changes to a new branch from `master` and see if I can figure out which change causes this problem to appear.
2022-05-20T13:08:28Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1132881550
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5DhmaO
https://api.github.com/repos/huggingface/datasets/issues/comments/1132881550/reactions
2022-05-20T13:08:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1132881550
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
@lhoestq Got it! It was caused by a name collision - I was importing `typing.Sequence`, but the code also needed `features.Sequence`. The tests from that PR were expecting the latter but got the former, and then crashed.
2022-05-20T13:22:16Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1132895365
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5DhpyF
https://api.github.com/repos/huggingface/datasets/issues/comments/1132895365/reactions
2022-05-20T13:23:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1132895365
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
@lhoestq Thanks! Also, when you're ready, don't merge it immediately! I'd like to do a quick round of manual testing with the very final build once you're happy to make sure it still works in our notebooks and examples.
2022-05-25T16:17:03Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1137492150
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5DzMC2
https://api.github.com/repos/huggingface/datasets/issues/comments/1137492150/reactions
2022-05-25T16:17:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1137492150
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
@lhoestq Tests look good to me, merging now!
2022-06-06T14:21:59Z
https://github.com/huggingface/datasets/pull/4170#issuecomment-1147503987
https://api.github.com/repos/huggingface/datasets/issues/4170
IC_kwDODunzps5EZYVz
https://api.github.com/repos/huggingface/datasets/issues/comments/1147503987/reactions
2022-06-06T14:21:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1147503987
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
This PR rewrites almost all of `to_tf_dataset()`, which makes it kind of hard to list all the changes, but the most critical ones are:

- Much better stability and no more dropping unexpected column names (Sorry @NielsRogge)
- Doesn't clobber custom transforms on the data (Sorry @NielsRogge again)
- Much better handling of the situation when the `collate_fn` adds columns that aren't in the dataset.
- Better inference of shapes and data types
- Lots of hacky special-casing code removed
- Can return string columns (as `tf.String`)
- Most arguments have default values, calling the method should be much simpler
- ~~Can accept a `model` argument and only return columns that are valid inputs to that model~~
- Drops the `dummy_labels` argument - this was a workaround for Keras issues that have been resolved by changes in `transformers`. Also remove it from tests and the Overview notebook.

I still have a couple of TODOs remaining and some testing to do, so don't merge yet, but it should be mostly ready for review at this point!