to_tf_dataset() refactor
MEMBER
Also, please don't merge yet - I need to make sure all the code samples and notebooks have a collate_fn specified, since we're removing the ability for this method to work without one!
2021-12-01T14:56:57Z
https://github.com/huggingface/datasets/pull/3356#issuecomment-983722177
https://api.github.com/repos/huggingface/datasets/issues/3356
IC_kwDODunzps46omjB
https://api.github.com/repos/huggingface/datasets/issues/comments/983722177/reactions
2021-12-01T14:56:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/983722177
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
Hi @lhoestq @mariosasko, the other PRs this was depending on in Transformers and huggingface/notebooks are now merged, so this is ready to go. Do you want to take one more look at it, or are you happy at this point?
2021-12-08T14:16:59Z
https://github.com/huggingface/datasets/pull/3356#issuecomment-988852700
https://api.github.com/repos/huggingface/datasets/issues/3356
IC_kwDODunzps468LHc
https://api.github.com/repos/huggingface/datasets/issues/comments/988852700/reactions
2021-12-08T14:16:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/988852700
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
The documentation for the method is fine, it doesn't need to be changed, but the tutorial notebook definitely looks a little out of date. Let me see what I can do!
2021-12-08T15:56:04Z
https://github.com/huggingface/datasets/pull/3356#issuecomment-988939929
https://api.github.com/repos/huggingface/datasets/issues/3356
IC_kwDODunzps468gaZ
https://api.github.com/repos/huggingface/datasets/issues/comments/988939929/reactions
2021-12-08T15:56:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/988939929
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
@lhoestq I rewrote the last bit of the notebook - let me know what you think!
2021-12-08T18:06:00Z
https://github.com/huggingface/datasets/pull/3356#issuecomment-989049036
https://api.github.com/repos/huggingface/datasets/issues/3356
IC_kwDODunzps4687DM
https://api.github.com/repos/huggingface/datasets/issues/comments/989049036/reactions
2021-12-08T18:06:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/989049036
https://avatars.githubusercontent.com/u/12866554?v=4
https://api.github.com/users/Rocketknight1/events{/privacy}
https://api.github.com/users/Rocketknight1/followers
https://api.github.com/users/Rocketknight1/following{/other_user}
https://api.github.com/users/Rocketknight1/gists{/gist_id}
https://github.com/Rocketknight1
Rocketknight1
MDQ6VXNlcjEyODY2NTU0
https://api.github.com/users/Rocketknight1/orgs
https://api.github.com/users/Rocketknight1/received_events
https://api.github.com/users/Rocketknight1/repos
https://api.github.com/users/Rocketknight1/starred{/owner}{/repo}
https://api.github.com/users/Rocketknight1/subscriptions
User
https://api.github.com/users/Rocketknight1
public
MEMBER
Cool thank you ! It's much nicer that what we had :)

I also spotted other things I'd like to update in the notebook (especially the beginning) but it can be fixed later
2021-12-09T10:24:09Z
https://github.com/huggingface/datasets/pull/3356#issuecomment-989717088
https://api.github.com/repos/huggingface/datasets/issues/3356
IC_kwDODunzps46_eJg
https://api.github.com/repos/huggingface/datasets/issues/comments/989717088/reactions
2021-12-09T10:24:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/989717088
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
This is the promised cleanup to `to_tf_dataset()` now that the course is out of the way! The main changes are:

- A collator is always required (there was way too much hackiness making things like labels work without it)
- Lots of cleanup and a lot of code moved to `_get_output_signature`
- Should now handle it gracefully when the data collator adds unexpected columns