use ragged tensor by default
MEMBER
Yes I agree. Maybe something that lets specify different format depending on the column ? Especially to better control dtype and shape (and ragged for tf)

Oh and I forgot: this one should also fix the second issue found in #477 for the next release
2020-08-24T19:19:48Z
https://github.com/huggingface/datasets/pull/530#issuecomment-679316827
https://api.github.com/repos/huggingface/datasets/issues/530
MDEyOklzc3VlQ29tbWVudDY3OTMxNjgyNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/679316827/reactions
2020-08-24T19:19:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/679316827
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I am running into the same issue with the error message on my local windows machine -
AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'to_tensor'. Tensorflow version is 2.6. Anything that I can do to fix it?
train_features = {x: tf_train_dataset[x].to_tensor() for x in tokenizer.model_input_names}
train_tf_dataset = tf.data.Dataset.from_tensor_slices((train_features, tf_train_dataset["label"]))
train_tf_dataset = train_tf_dataset.shuffle(len(tf_train_dataset)).batch(8)

eval_features = {x: tf_eval_dataset[x].to_tensor() for x in tokenizer.model_input_names}
eval_tf_dataset = tf.data.Dataset.from_tensor_slices((eval_features, tf_eval_dataset["label"]))
eval_tf_dataset = eval_tf_dataset.batch(8)

ttributeError                            Traceback (most recent call last)
<ipython-input-59-f50e45c2c0dc> in <module>
----> 1 train_features = {x: tf_train_dataset[x].convert_to_tensor() for x in tokenizer.model_input_names}
      2 train_tf_dataset = tf.data.Dataset.from_tensor_slices((train_features, tf_train_dataset["label"]))
      3 train_tf_dataset = train_tf_dataset.shuffle(len(tf_train_dataset)).batch(8)
      4 
      5 eval_features = {x: tf_eval_dataset[x].to_tensor() for x in tokenizer.model_input_names}

<ipython-input-59-f50e45c2c0dc> in <dictcomp>(.0)
----> 1 train_features = {x: tf_train_dataset[x].convert_to_tensor() for x in tokenizer.model_input_names}
      2 train_tf_dataset = tf.data.Dataset.from_tensor_slices((train_features, tf_train_dataset["label"]))
      3 train_tf_dataset = train_tf_dataset.shuffle(len(tf_train_dataset)).batch(8)
      4 
      5 eval_features = {x: tf_eval_dataset[x].to_tensor() for x in tokenizer.model_input_names}

~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\framework\ops.py in __getattr__(self, name)
    399         from tensorflow.python.ops.numpy_ops import np_config
    400         np_config.enable_numpy_behavior()""".format(type(self).__name__, name))
--> 401     self.__getattribute__(name)
    402 
    403   @staticmethod

AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'convert_to_tensor'


2021-10-14T21:03:14Z
https://github.com/huggingface/datasets/pull/530#issuecomment-943730441
https://api.github.com/repos/huggingface/datasets/issues/530
IC_kwDODunzps44QC8J
https://api.github.com/repos/huggingface/datasets/issues/comments/943730441/reactions
2021-10-14T21:03:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/943730441
https://avatars.githubusercontent.com/u/25391924?v=4
https://api.github.com/users/mahalaxminanya/events{/privacy}
https://api.github.com/users/mahalaxminanya/followers
https://api.github.com/users/mahalaxminanya/following{/other_user}
https://api.github.com/users/mahalaxminanya/gists{/gist_id}
https://github.com/mahalaxminanya
mahalaxminanya
MDQ6VXNlcjI1MzkxOTI0
https://api.github.com/users/mahalaxminanya/orgs
https://api.github.com/users/mahalaxminanya/received_events
https://api.github.com/users/mahalaxminanya/repos
https://api.github.com/users/mahalaxminanya/starred{/owner}{/repo}
https://api.github.com/users/mahalaxminanya/subscriptions
User
https://api.github.com/users/mahalaxminanya
public
MEMBER
Hi ! Before calling `to_tensor`, make sure that your object is a RaggedTensor, because it may already be a regular Tensor if the shapes of your examples are all the same
2021-10-22T09:38:59Z
https://github.com/huggingface/datasets/pull/530#issuecomment-949457845
https://api.github.com/repos/huggingface/datasets/issues/530
IC_kwDODunzps44l5O1
https://api.github.com/repos/huggingface/datasets/issues/comments/949457845/reactions
2021-10-22T09:38:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/949457845
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Okay. i am not familiar with how to check the difference between the two. I will research on this.
2021-10-22T19:38:40Z
https://github.com/huggingface/datasets/pull/530#issuecomment-949912877
https://api.github.com/repos/huggingface/datasets/issues/530
IC_kwDODunzps44noUt
https://api.github.com/repos/huggingface/datasets/issues/comments/949912877/reactions
2021-10-22T19:38:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/949912877
https://avatars.githubusercontent.com/u/25391924?v=4
https://api.github.com/users/mahalaxminanya/events{/privacy}
https://api.github.com/users/mahalaxminanya/followers
https://api.github.com/users/mahalaxminanya/following{/other_user}
https://api.github.com/users/mahalaxminanya/gists{/gist_id}
https://github.com/mahalaxminanya
mahalaxminanya
MDQ6VXNlcjI1MzkxOTI0
https://api.github.com/users/mahalaxminanya/orgs
https://api.github.com/users/mahalaxminanya/received_events
https://api.github.com/users/mahalaxminanya/repos
https://api.github.com/users/mahalaxminanya/starred{/owner}{/repo}
https://api.github.com/users/mahalaxminanya/subscriptions
User
https://api.github.com/users/mahalaxminanya
public
I think it's better if it's clear whether the returned tensor is ragged or not when the type is set to tensorflow.
Previously it was a tensor (not ragged) if numpy could stack the output (which can change depending on the batch of example you take), which make things difficult to handle, as it may sometimes return a ragged tensor and sometimes not.

Therefore I reverted this behavior to always return a ragged tensor as we used to do.