[Tensorflow] Use something else than `from_tensor_slices()`
MEMBER
I guess we can use `tf.data.Dataset.from_generator` instead. I'll give it a try.
2020-05-27T15:48:35Z
https://github.com/huggingface/datasets/issues/193#issuecomment-634753711
https://api.github.com/repos/huggingface/datasets/issues/193
MDEyOklzc3VlQ29tbWVudDYzNDc1MzcxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/634753711/reactions
2020-05-27T15:48:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634753711
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Is `tf.data.Dataset.from_generator` working on TPU ?
2020-05-27T23:27:26Z
https://github.com/huggingface/datasets/issues/193#issuecomment-634996689
https://api.github.com/repos/huggingface/datasets/issues/193
MDEyOklzc3VlQ29tbWVudDYzNDk5NjY4OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/634996689/reactions
2020-05-27T23:27:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634996689
https://avatars.githubusercontent.com/u/43774355?v=4
https://api.github.com/users/astariul/events{/privacy}
https://api.github.com/users/astariul/followers
https://api.github.com/users/astariul/following{/other_user}
https://api.github.com/users/astariul/gists{/gist_id}
https://github.com/astariul
astariul
MDQ6VXNlcjQzNzc0MzU1
https://api.github.com/users/astariul/orgs
https://api.github.com/users/astariul/received_events
https://api.github.com/users/astariul/repos
https://api.github.com/users/astariul/starred{/owner}{/repo}
https://api.github.com/users/astariul/subscriptions
User
https://api.github.com/users/astariul
public
NONE
`from_generator` is not working on TPU, I met the following error :

```
File "/usr/local/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/usr/.venv/bart/lib/python3.6/site-packages/tensorflow_core/python/eager/context.py", line 1900, in execution_mode
    executor_new.wait()
  File "/home/usr/.venv/bart/lib/python3.6/site-packages/tensorflow_core/python/eager/executor.py", line 67, in wait
    pywrap_tensorflow.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.NotFoundError: No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node {{node PyFunc}}
        .  Registered:  <no registered kernels>

         [[PyFunc]]
```

---

@lhoestq It seems you merged some changes that allow lazy-loading. **Can you give an example of how to use ?** Maybe the Colab notebook should be updated with this method as well.
2020-06-02T23:44:37Z
https://github.com/huggingface/datasets/issues/193#issuecomment-637864441
https://api.github.com/repos/huggingface/datasets/issues/193
MDEyOklzc3VlQ29tbWVudDYzNzg2NDQ0MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/637864441/reactions
2020-06-02T23:49:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/637864441
https://avatars.githubusercontent.com/u/43774355?v=4
https://api.github.com/users/astariul/events{/privacy}
https://api.github.com/users/astariul/followers
https://api.github.com/users/astariul/following{/other_user}
https://api.github.com/users/astariul/gists{/gist_id}
https://github.com/astariul
astariul
MDQ6VXNlcjQzNzc0MzU1
https://api.github.com/users/astariul/orgs
https://api.github.com/users/astariul/received_events
https://api.github.com/users/astariul/repos
https://api.github.com/users/astariul/starred{/owner}{/repo}
https://api.github.com/users/astariul/subscriptions
User
https://api.github.com/users/astariul
public
MEMBER
Could you send me the code you used to run create the dataset using `.from_generator` ? What version of tensorflow are you using ?
2020-06-03T08:44:24Z
https://github.com/huggingface/datasets/issues/193#issuecomment-638053534
https://api.github.com/repos/huggingface/datasets/issues/193
MDEyOklzc3VlQ29tbWVudDYzODA1MzUzNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/638053534/reactions
2020-06-03T08:44:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/638053534
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I'm using TF2.2

Here is my code :
```
import nlp
from transformers import BartTokenizer

tokenizer = BartTokenizer.from_pretrained('bart-large')

def encode(sample):
    article_inputs = tokenizer.encode_plus(sample["article"], max_length=tokenizer.model_max_length, pad_to_max_length=True)
    summary_inputs = tokenizer.encode_plus(sample["highlights"], max_length=tokenizer.model_max_length, pad_to_max_length=True)

    article_inputs.update({"lm_labels": summary_inputs['input_ids']})
    return article_inputs

cnn_dm = nlp.load_dataset('cnn_dailymail', '3.0.0', split='test')
cnn_dm = cnn_dm.map(encode)

def gen():
    for sample in cnn_dm:
        s = {}
        s['input_ids'] = sample['input_ids']
        s['attention_mask'] = sample['attention_mask']
        s['lm_labels'] = sample['lm_labels']
        yield s

dataset = tf.data.Dataset.from_generator(gen, output_types={k: tf.int32 for k in ['input_ids', 'attention_mask', 'lm_labels']}, output_shapes={k: tf.TensorShape([tokenizer.model_max_length]) for k in ['input_ids', 'attention_mask', 'lm_labels']}
```
2020-06-03T08:56:43Z
https://github.com/huggingface/datasets/issues/193#issuecomment-638060951
https://api.github.com/repos/huggingface/datasets/issues/193
MDEyOklzc3VlQ29tbWVudDYzODA2MDk1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/638060951/reactions
2020-06-03T08:56:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/638060951
https://avatars.githubusercontent.com/u/43774355?v=4
https://api.github.com/users/astariul/events{/privacy}
https://api.github.com/users/astariul/followers
https://api.github.com/users/astariul/following{/other_user}
https://api.github.com/users/astariul/gists{/gist_id}
https://github.com/astariul
astariul
MDQ6VXNlcjQzNzc0MzU1
https://api.github.com/users/astariul/orgs
https://api.github.com/users/astariul/received_events
https://api.github.com/users/astariul/repos
https://api.github.com/users/astariul/starred{/owner}{/repo}
https://api.github.com/users/astariul/subscriptions
User
https://api.github.com/users/astariul
public
MEMBER
Apparently we'll have to wait for the next tensorflow release to use `.from_generator` and TPU. See https://github.com/tensorflow/tensorflow/issues/34346#issuecomment-598262489
2020-06-03T12:40:47Z
https://github.com/huggingface/datasets/issues/193#issuecomment-638171032
https://api.github.com/repos/huggingface/datasets/issues/193
MDEyOklzc3VlQ29tbWVudDYzODE3MTAzMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/638171032/reactions
2020-06-03T12:40:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/638171032
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Fixed by https://github.com/huggingface/datasets/pull/339
2020-10-27T15:28:11Z
https://github.com/huggingface/datasets/issues/193#issuecomment-717323309
https://api.github.com/repos/huggingface/datasets/issues/193
MDEyOklzc3VlQ29tbWVudDcxNzMyMzMwOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/717323309/reactions
2020-10-27T15:28:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/717323309
https://avatars.githubusercontent.com/u/10469459?v=4
https://api.github.com/users/yjernite/events{/privacy}
https://api.github.com/users/yjernite/followers
https://api.github.com/users/yjernite/following{/other_user}
https://api.github.com/users/yjernite/gists{/gist_id}
https://github.com/yjernite
yjernite
MDQ6VXNlcjEwNDY5NDU5
https://api.github.com/users/yjernite/orgs
https://api.github.com/users/yjernite/received_events
https://api.github.com/users/yjernite/repos
https://api.github.com/users/yjernite/starred{/owner}{/repo}
https://api.github.com/users/yjernite/subscriptions
User
https://api.github.com/users/yjernite
public
In the example notebook, the TF Dataset is built using `from_tensor_slices()` :

```python
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'start_positions', 'end_positions']
train_tf_dataset.set_format(type='tensorflow', columns=columns)
features = {x: train_tf_dataset[x] for x in columns[:3]} 
labels = {"output_1": train_tf_dataset["start_positions"]}
labels["output_2"] = train_tf_dataset["end_positions"]
tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)
```

But according to [official tensorflow documentation](https://www.tensorflow.org/guide/data#consuming_numpy_arrays), this will load the entire dataset to memory.

**This defeats one purpose of this library, which is lazy loading.**

Is there any other way to load the `nlp` dataset into TF dataset lazily ?

---

For example, is it possible to use [Arrow dataset](https://www.tensorflow.org/io/api_docs/python/tfio/arrow/ArrowDataset) ? If yes, is there any code example ?