Add dataset.export() to TFRecords
MEMBER
Really cool @jarednielsen !
Do you think we can make it work with dataset with nested features like `squad` ?

I just did a PR to fix `.set_format` for datasets with nested features, but as soon as it's merged we could try to make the conversion work on a dataset like `squad`.
2020-07-06T11:30:30Z
https://github.com/huggingface/datasets/pull/339#issuecomment-654177418
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY1NDE3NzQxOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/654177418/reactions
2020-07-06T11:30:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/654177418
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
For datasets with nested features we have two aspects to take into account:
1) There can be nested dict of features. What is done in tensorflow_datasets to make things work is to flatten the dictionaries to end up with one single dictionary. A dict like `{"column1": {"subfeature": ...}}` is converted to `{"column1/subfeature":...}`
2) There can be ragged tensors, i.e. lists of objects with non-fixed shapes. For example in squad there are often multiple possible answers per question. What is done in tensorflow_datasets to make things work is to concatenate everything and add ragged attributes (cf serialization code [here](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/example_serializer.py))
2020-07-06T12:16:37Z
https://github.com/huggingface/datasets/pull/339#issuecomment-654198387
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY1NDE5ODM4Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/654198387/reactions
2020-07-06T12:17:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/654198387
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Note that we have `flatten` method in `ArrowDataset`
2020-07-06T20:28:57Z
https://github.com/huggingface/datasets/pull/339#issuecomment-654449143
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY1NDQ0OTE0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/654449143/reactions
2020-07-06T20:28:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/654449143
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
I added support for nested dictionaries. A few more design decisions popped up:

_Should we serialize from NumPy arrays or from tf.Tensors?_
- The [tfds example serializer](url) works from NumPy arrays.
- Calling `dset.set_format("tensorflow")` makes `__getitem__` return a tf.Tensor. So serializing from NumPy arrays would mean calling `dset.export()` before setting the format, which is confusing.
- NumPy arrays can be serialized as their underlying datatype (int, float), while tf.Tensors must be converted to strings before serialization. This adds another step when serializing and deserializing, and removes the static-typing advantages of the TFRecord format.

I think we should export directly from the underlying NumPy arrays into TFRecords, rather than using an intermediate step of tf.Tensor.

_Should we serialize lists of dictionaries?_
- The test_format_nested() test creates a list of dictionaries: https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/tests/test_arrow_dataset.py#L278-L288
- This is difficult to serialize effectively, and I'm not aware of any dataset that has this format. SQuAD has a dictionary of lists, such as the `answers` key. Is this necessary?
2020-07-06T20:33:56Z
https://github.com/huggingface/datasets/pull/339#issuecomment-654451296
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY1NDQ1MTI5Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/654451296/reactions
2020-07-06T20:33:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/654451296
https://avatars.githubusercontent.com/u/4564897?v=4
https://api.github.com/users/jarednielsen/events{/privacy}
https://api.github.com/users/jarednielsen/followers
https://api.github.com/users/jarednielsen/following{/other_user}
https://api.github.com/users/jarednielsen/gists{/gist_id}
https://github.com/jarednielsen
jarednielsen
MDQ6VXNlcjQ1NjQ4OTc=
https://api.github.com/users/jarednielsen/orgs
https://api.github.com/users/jarednielsen/received_events
https://api.github.com/users/jarednielsen/repos
https://api.github.com/users/jarednielsen/starred{/owner}{/repo}
https://api.github.com/users/jarednielsen/subscriptions
User
https://api.github.com/users/jarednielsen
public
CONTRIBUTOR
Thanks @thomwolf, used dset.flatten() to simplify. That handles the case of nested dictionaries, and then lists can be read into a tf.io.RaggedFeature in the case of something like squad answers.
2020-07-06T20:47:46Z
https://github.com/huggingface/datasets/pull/339#issuecomment-654457252
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY1NDQ1NzI1Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/654457252/reactions
2020-07-06T20:47:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/654457252
https://avatars.githubusercontent.com/u/4564897?v=4
https://api.github.com/users/jarednielsen/events{/privacy}
https://api.github.com/users/jarednielsen/followers
https://api.github.com/users/jarednielsen/following{/other_user}
https://api.github.com/users/jarednielsen/gists{/gist_id}
https://github.com/jarednielsen
jarednielsen
MDQ6VXNlcjQ1NjQ4OTc=
https://api.github.com/users/jarednielsen/orgs
https://api.github.com/users/jarednielsen/received_events
https://api.github.com/users/jarednielsen/repos
https://api.github.com/users/jarednielsen/starred{/owner}{/repo}
https://api.github.com/users/jarednielsen/subscriptions
User
https://api.github.com/users/jarednielsen
public
MEMBER
@jarednielsen I just checked and indeed we don't have lists of dicts, we can just focus on the squad format as a reference then :) I'll change the test to remove this format that's not supposed to happen
2020-07-07T16:35:38Z
https://github.com/huggingface/datasets/pull/339#issuecomment-654982515
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY1NDk4MjUxNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/654982515/reactions
2020-07-07T16:35:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/654982515
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Actually I realised that `flatten` also handles nested things like pyarrow's list<struct> so it's fine :D 
This is so cool !

Could you also add a test with a squad-like dataset ? As soon as we have that I think we'll be good to merge @jarednielsen  :)
Good job !
2020-07-13T14:32:53Z
https://github.com/huggingface/datasets/pull/339#issuecomment-657596502
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY1NzU5NjUwMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/657596502/reactions
2020-07-13T14:32:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/657596502
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Great, done! I think this could be a great canonical way to generate a dataset.
2020-07-14T04:18:00Z
https://github.com/huggingface/datasets/pull/339#issuecomment-657956681
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY1Nzk1NjY4MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/657956681/reactions
2020-07-14T04:18:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/657956681
https://avatars.githubusercontent.com/u/4564897?v=4
https://api.github.com/users/jarednielsen/events{/privacy}
https://api.github.com/users/jarednielsen/followers
https://api.github.com/users/jarednielsen/following{/other_user}
https://api.github.com/users/jarednielsen/gists{/gist_id}
https://github.com/jarednielsen
jarednielsen
MDQ6VXNlcjQ1NjQ4OTc=
https://api.github.com/users/jarednielsen/orgs
https://api.github.com/users/jarednielsen/received_events
https://api.github.com/users/jarednielsen/repos
https://api.github.com/users/jarednielsen/starred{/owner}{/repo}
https://api.github.com/users/jarednielsen/subscriptions
User
https://api.github.com/users/jarednielsen
public
CONTRIBUTOR
I tried to match the format of Dataset.sort() and Dataset.shuffle() with the docstring. What difference are you referring to specifically?
2020-07-15T15:48:03Z
https://github.com/huggingface/datasets/pull/339#issuecomment-658845246
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY1ODg0NTI0Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/658845246/reactions
2020-07-15T15:48:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/658845246
https://avatars.githubusercontent.com/u/4564897?v=4
https://api.github.com/users/jarednielsen/events{/privacy}
https://api.github.com/users/jarednielsen/followers
https://api.github.com/users/jarednielsen/following{/other_user}
https://api.github.com/users/jarednielsen/gists{/gist_id}
https://github.com/jarednielsen
jarednielsen
MDQ6VXNlcjQ1NjQ4OTc=
https://api.github.com/users/jarednielsen/orgs
https://api.github.com/users/jarednielsen/received_events
https://api.github.com/users/jarednielsen/repos
https://api.github.com/users/jarednielsen/starred{/owner}{/repo}
https://api.github.com/users/jarednielsen/subscriptions
User
https://api.github.com/users/jarednielsen
public
MEMBER
Oh my bad they're fine actually (I was thinking of the backticks that we don't use in the docstrings of the transformers repo for argument names)
2020-07-15T16:10:25Z
https://github.com/huggingface/datasets/pull/339#issuecomment-658855835
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY1ODg1NTgzNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/658855835/reactions
2020-07-15T16:10:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/658855835
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
One final thing: now that we have a brand new documentation, could you just add `export` to the list of documented methods in [docs/source/package_reference/main_classes.rst](https://github.com/huggingface/nlp/blob/master/docs/source/package_reference/main_classes.rst) (so that it will appear in the docs [here](https://huggingface.co/nlp/package_reference/main_classes.html)) ?

2020-07-16T12:14:01Z
https://github.com/huggingface/datasets/pull/339#issuecomment-659371303
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY1OTM3MTMwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/659371303/reactions
2020-07-16T12:14:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/659371303
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Done
2020-07-17T00:25:59Z
https://github.com/huggingface/datasets/pull/339#issuecomment-659752590
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY1OTc1MjU5MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/659752590/reactions
2020-07-17T00:25:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/659752590
https://avatars.githubusercontent.com/u/4564897?v=4
https://api.github.com/users/jarednielsen/events{/privacy}
https://api.github.com/users/jarednielsen/followers
https://api.github.com/users/jarednielsen/following{/other_user}
https://api.github.com/users/jarednielsen/gists{/gist_id}
https://github.com/jarednielsen
jarednielsen
MDQ6VXNlcjQ1NjQ4OTc=
https://api.github.com/users/jarednielsen/orgs
https://api.github.com/users/jarednielsen/received_events
https://api.github.com/users/jarednielsen/repos
https://api.github.com/users/jarednielsen/starred{/owner}{/repo}
https://api.github.com/users/jarednielsen/subscriptions
User
https://api.github.com/users/jarednielsen
public
MEMBER
Cool thanks :)
2020-07-17T17:21:46Z
https://github.com/huggingface/datasets/pull/339#issuecomment-660237589
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY2MDIzNzU4OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/660237589/reactions
2020-07-17T17:21:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/660237589
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Since #403 (it just got merged), we return python objects and not numpy arrays anymore (unless format="numpy" is specified).
Do you think it can break the export method ? Could you try to rebase from master to run the CI to make sure it's fine ?
2020-07-17T17:33:50Z
https://github.com/huggingface/datasets/pull/339#issuecomment-660244573
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY2MDI0NDU3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/660244573/reactions
2020-07-17T17:33:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/660244573
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Good catch. I fixed it up so it works with the new format. By the way, when dset.format == "numpy", it now returns single items (like `0`) as a 0-dimensional NumPy array. Not sure if that is desired.
2020-07-17T17:59:51Z
https://github.com/huggingface/datasets/pull/339#issuecomment-660257899
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY2MDI1Nzg5OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/660257899/reactions
2020-07-17T17:59:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/660257899
https://avatars.githubusercontent.com/u/4564897?v=4
https://api.github.com/users/jarednielsen/events{/privacy}
https://api.github.com/users/jarednielsen/followers
https://api.github.com/users/jarednielsen/following{/other_user}
https://api.github.com/users/jarednielsen/gists{/gist_id}
https://github.com/jarednielsen
jarednielsen
MDQ6VXNlcjQ1NjQ4OTc=
https://api.github.com/users/jarednielsen/orgs
https://api.github.com/users/jarednielsen/received_events
https://api.github.com/users/jarednielsen/repos
https://api.github.com/users/jarednielsen/starred{/owner}{/repo}
https://api.github.com/users/jarednielsen/subscriptions
User
https://api.github.com/users/jarednielsen
public
MEMBER
I played a little bit with the code and it works quite well :)

I found two cases for which it doesn't work though:
- if the features dict depth is > 2 (ex: wikisql), because `flatten` only flattens the first level of nesting (it can be fixed by calling `flatten` several times in a row, see [here](https://issues.apache.org/jira/browse/ARROW-4090))
- Or if there are 2d features (ex: wikisql, `table.rows` is a sequence of sequences of strings), because tf.train.Features only support 1-d lists. That's why tensorflow-datasets flattens these 2-d features to 1-d and adds ragged features that are the shapes of the arrays, so that they can be reconstructed.

I think we can ignore the 2d stuff right now (some work is being done in #363 ), but I'd like to see the `flatten` issue fixed soon

2020-07-20T14:28:12Z
https://github.com/huggingface/datasets/pull/339#issuecomment-661074658
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY2MTA3NDY1OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/661074658/reactions
2020-07-20T14:28:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/661074658
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
That seems like a bug in `pyarrow`, or at least in `flatten()`. Looks like it should be a separate PR.
2020-07-20T22:43:21Z
https://github.com/huggingface/datasets/pull/339#issuecomment-661398577
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY2MTM5ODU3Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/661398577/reactions
2020-07-20T22:43:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/661398577
https://avatars.githubusercontent.com/u/4564897?v=4
https://api.github.com/users/jarednielsen/events{/privacy}
https://api.github.com/users/jarednielsen/followers
https://api.github.com/users/jarednielsen/following{/other_user}
https://api.github.com/users/jarednielsen/gists{/gist_id}
https://github.com/jarednielsen
jarednielsen
MDQ6VXNlcjQ1NjQ4OTc=
https://api.github.com/users/jarednielsen/orgs
https://api.github.com/users/jarednielsen/received_events
https://api.github.com/users/jarednielsen/repos
https://api.github.com/users/jarednielsen/starred{/owner}{/repo}
https://api.github.com/users/jarednielsen/subscriptions
User
https://api.github.com/users/jarednielsen
public
MEMBER
I made `.flatten` work on our side (it calls pyarrow's flatten several times until it's really flat).

The only datasets that won't work are those with lists of lists of features, which is a rare case. Hopefully we can make this work with the multi-dimensional arrays changes we're also doing.

I think we can merge now :) cc @thomwolf 
2020-07-21T08:30:39Z
https://github.com/huggingface/datasets/pull/339#issuecomment-661715542
https://api.github.com/repos/huggingface/datasets/issues/339
MDEyOklzc3VlQ29tbWVudDY2MTcxNTU0Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/661715542/reactions
2020-07-21T08:30:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/661715542
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Fixes https://github.com/huggingface/nlp/issues/337

Some design decisions:

- Simplified the function API to not handle sharding. It writes the entire dataset as a single TFRecord file. This simplifies the function logic and users can use other functions (`select`, `shard`, etc) to handle custom sharding or splitting.
- Use `from_generator()` instead of `from_tensor_slices()` to address the memory issues discussed in https://github.com/huggingface/nlp/issues/315 and https://github.com/huggingface/nlp/issues/193.
- Performs introspection using the values from `dataset.set_format()` to identify the TF datatypes. Currently it supports string, float, and int. If this should be extended for other datatypes, let me know.
- There are quite a few helper functions required within the `export()` method. If these are better placed in a utils file somewhere, let me know.

Also, I noticed that 
```python
dataset = dataset.select(indices)
dataset.set_format("tensorflow")
# dataset._format_type is "tensorflow"
```
gives a different output than
```python
dataset.set_format("tensorflow")
dataset = dataset.select(indices)
# dataset._format_type is None
```
The latter loses the format of its parent dataset. Is there interest in making `set_format` a functional method that returns itself (can be chained), and that derived datasets maintain the format of their parent?