Add `key` type and duplicates verification with hashing
CONTRIBUTOR
@lhoestq The tests for key type and duplicate keys have been added and verified successfully.
After generating with an intentionally faulty `mnist` script, when there is an incompatible key type, it shows:

```
Downloading and preparing dataset mnist/mnist (download: 11.06 MiB, generated: 60.62 MiB, post-processed: Unknown size, total: 71.67 MiB) to C:\Users\nikhil\.cache\huggingface\datasets\mnist\mnist\1.0.0\5064c25e57a1678f700d2dc798ffe8a6d519405cca7d33670fffda477857a994...
0 examples [00:00, ? examples/s]2021-04-26 02:50:03.703836: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll

FAILURE TO GENERATE DATASET: Invalid key type detected
Found Key [0, 0] of type <class 'list'>
Keys should be either str, int or bytes type
```

In the case of duplicate keys, it now gives:

```
Downloading and preparing dataset mnist/mnist (download: 11.06 MiB, generated: 60.62 MiB, post-processed: Unknown size, total: 71.67 MiB) to C:\Users\nikhil\.cache\huggingface\datasets\mnist\mnist\1.0.0\5064c25e57a1678f700d2dc798ffe8a6d519405cca7d33670fffda477857a994...
0 examples [00:00, ? examples/s]2021-04-26 02:53:13.498579: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "f:\datasets\datasets-1\src\datasets\load.py", line 746, in load_dataset
    builder_instance.download_and_prepare(
  File "f:\datasets\datasets-1\src\datasets\builder.py", line 587, in download_and_prepare
    self._download_and_prepare(
  File "f:\datasets\datasets-1\src\datasets\builder.py", line 665, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "f:\datasets\datasets-1\src\datasets\builder.py", line 1002, in _prepare_split
    writer.write(example, key)
  File "f:\datasets\datasets-1\src\datasets\arrow_writer.py", line 321, in write
    self.check_duplicates()
  File "f:\datasets\datasets-1\src\datasets\arrow_writer.py", line 331, in check_duplicates
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 234467
Keys should be unique and deterministic in nature
```
Please let me know if this is what we wanted to implement. Thanks!
2021-04-25T21:24:28Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-826392347
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgyNjM5MjM0Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/826392347/reactions
2021-04-25T21:24:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826392347
https://avatars.githubusercontent.com/u/42388668?v=4
https://api.github.com/users/NikhilBartwal/events{/privacy}
https://api.github.com/users/NikhilBartwal/followers
https://api.github.com/users/NikhilBartwal/following{/other_user}
https://api.github.com/users/NikhilBartwal/gists{/gist_id}
https://github.com/NikhilBartwal
NikhilBartwal
MDQ6VXNlcjQyMzg4NjY4
https://api.github.com/users/NikhilBartwal/orgs
https://api.github.com/users/NikhilBartwal/received_events
https://api.github.com/users/NikhilBartwal/repos
https://api.github.com/users/NikhilBartwal/starred{/owner}{/repo}
https://api.github.com/users/NikhilBartwal/subscriptions
User
https://api.github.com/users/NikhilBartwal
public
MEMBER
This looks pretty cool !
We can make focus on the GeneratorBasedBuilder for now yes.

Do you think we could make the ArrowWriter not look for duplicates by default ?
This way we can just enable duplicate detections when instantiating the writer in the GeneratorBasedBuilder for now.
2021-04-26T16:07:59Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-826960741
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgyNjk2MDc0MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/826960741/reactions
2021-04-26T16:07:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826960741
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thank you @lhoestq



> Do you think we could make the ArrowWriter not look for duplicates by default ?

We can definitely do that by including a `check_duplicates` argument while instantiating `ArrowWriter()`. 

However, since only `GeneratorBasedBuilder` uses the `write()` function (which includes the detection code) and the others like `ArrowBasedBuilder` use `write_table()` which remains as it was (without duplicate detection). I don't think it would be necessary.

Nonetheless, doing this would require just some small changes. Please let me know your thoughts on this. Thanks!
2021-04-26T17:03:31Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-827001079
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgyNzAwMTA3OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/827001079/reactions
2021-04-26T17:03:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/827001079
https://avatars.githubusercontent.com/u/42388668?v=4
https://api.github.com/users/NikhilBartwal/events{/privacy}
https://api.github.com/users/NikhilBartwal/followers
https://api.github.com/users/NikhilBartwal/following{/other_user}
https://api.github.com/users/NikhilBartwal/gists{/gist_id}
https://github.com/NikhilBartwal
NikhilBartwal
MDQ6VXNlcjQyMzg4NjY4
https://api.github.com/users/NikhilBartwal/orgs
https://api.github.com/users/NikhilBartwal/received_events
https://api.github.com/users/NikhilBartwal/repos
https://api.github.com/users/NikhilBartwal/starred{/owner}{/repo}
https://api.github.com/users/NikhilBartwal/subscriptions
User
https://api.github.com/users/NikhilBartwal
public
MEMBER
I like the idea of having the duplicate detection optional for other uses of the ArrowWriter.
This class is the main tool to write python data in arrow format so I'd expect it to be flexible.
That's why I think by default it shouldn't require users to provide keys or do any duplicates detection.

An alternative would be to subclass the writer to include duplicates detection in another class.

Both options are fine for me, let me know what you think !
2021-04-27T15:37:21Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-827706050
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgyNzcwNjA1MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/827706050/reactions
2021-04-27T15:37:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/827706050
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> This class is the main tool to write python data in arrow format so I'd expect it to be flexible.
> That's why I think by default it shouldn't require users to provide keys or do any duplicates detection.

Well, that makes sense as the writer can indeed be used for other purposes as well.

> We can definitely do that by including a `check_duplicates` argument while instantiating `ArrowWriter()`.

I think that this would be the simplest and the more efficient option for achieving this as subclassing the writer only for this would lead to unnecessary complexity and code duplication (in case of `writer()`). 

I will be adding the changes soon. Thanks for the feedback @lhoestq!
2021-04-27T21:13:58Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-827935172
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgyNzkzNTE3Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/827935172/reactions
2021-04-27T21:13:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/827935172
https://avatars.githubusercontent.com/u/42388668?v=4
https://api.github.com/users/NikhilBartwal/events{/privacy}
https://api.github.com/users/NikhilBartwal/followers
https://api.github.com/users/NikhilBartwal/following{/other_user}
https://api.github.com/users/NikhilBartwal/gists{/gist_id}
https://github.com/NikhilBartwal
NikhilBartwal
MDQ6VXNlcjQyMzg4NjY4
https://api.github.com/users/NikhilBartwal/orgs
https://api.github.com/users/NikhilBartwal/received_events
https://api.github.com/users/NikhilBartwal/repos
https://api.github.com/users/NikhilBartwal/starred{/owner}{/repo}
https://api.github.com/users/NikhilBartwal/subscriptions
User
https://api.github.com/users/NikhilBartwal
public
CONTRIBUTOR
@lhoestq I have pushed the final changes just now. 
Now, the keys and duplicate checking will be necessary only when the `ArrowWriter` is initialized with `check_duplicates=True` specifically (in this case, for `GeneratorBasedBuilders`)

Let me know if this is what was required. Thanks!
2021-04-28T20:03:48Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-828740931
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgyODc0MDkzMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/828740931/reactions
2021-04-28T20:03:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/828740931
https://avatars.githubusercontent.com/u/42388668?v=4
https://api.github.com/users/NikhilBartwal/events{/privacy}
https://api.github.com/users/NikhilBartwal/followers
https://api.github.com/users/NikhilBartwal/following{/other_user}
https://api.github.com/users/NikhilBartwal/gists{/gist_id}
https://github.com/NikhilBartwal
NikhilBartwal
MDQ6VXNlcjQyMzg4NjY4
https://api.github.com/users/NikhilBartwal/orgs
https://api.github.com/users/NikhilBartwal/received_events
https://api.github.com/users/NikhilBartwal/repos
https://api.github.com/users/NikhilBartwal/starred{/owner}{/repo}
https://api.github.com/users/NikhilBartwal/subscriptions
User
https://api.github.com/users/NikhilBartwal
public
CONTRIBUTOR
@lhoestq Thanks for the feedback! I will be adding the tests for the same very soon. 

However, I'm not sure as to what exactly is causing the `segmentation fault` in the failing CI tests. It seems to be something from `test_concatenation_table_cast` from `test_table.py`, but I'm not sure as to what exactly. Would be great if you could help. Thanks!
2021-05-03T19:31:59Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-831482498
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgzMTQ4MjQ5OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/831482498/reactions
2021-05-03T19:31:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/831482498
https://avatars.githubusercontent.com/u/42388668?v=4
https://api.github.com/users/NikhilBartwal/events{/privacy}
https://api.github.com/users/NikhilBartwal/followers
https://api.github.com/users/NikhilBartwal/following{/other_user}
https://api.github.com/users/NikhilBartwal/gists{/gist_id}
https://github.com/NikhilBartwal
NikhilBartwal
MDQ6VXNlcjQyMzg4NjY4
https://api.github.com/users/NikhilBartwal/orgs
https://api.github.com/users/NikhilBartwal/received_events
https://api.github.com/users/NikhilBartwal/repos
https://api.github.com/users/NikhilBartwal/starred{/owner}{/repo}
https://api.github.com/users/NikhilBartwal/subscriptions
User
https://api.github.com/users/NikhilBartwal
public
MEMBER
You can merge master into your branch to fix this issue.
Basically pyarrow 4.0.0 has a segfault issue (which has now been resolved on the master branch of pyarrow).
So until 4.0.1 comes out we changed to using `pyarrow<4.0.0` recently.
2021-05-03T19:51:53Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-831493623
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgzMTQ5MzYyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/831493623/reactions
2021-05-03T19:51:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/831493623
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq Thanks for the help with the CI failures. Apologies for the multiple merge commits. My local repo got messy while merging which led to this.
Will be pushing the commit for the tests soon!
2021-05-03T21:51:46Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-831558290
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgzMTU1ODI5MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/831558290/reactions
2021-05-03T21:51:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/831558290
https://avatars.githubusercontent.com/u/42388668?v=4
https://api.github.com/users/NikhilBartwal/events{/privacy}
https://api.github.com/users/NikhilBartwal/followers
https://api.github.com/users/NikhilBartwal/following{/other_user}
https://api.github.com/users/NikhilBartwal/gists{/gist_id}
https://github.com/NikhilBartwal
NikhilBartwal
MDQ6VXNlcjQyMzg4NjY4
https://api.github.com/users/NikhilBartwal/orgs
https://api.github.com/users/NikhilBartwal/received_events
https://api.github.com/users/NikhilBartwal/repos
https://api.github.com/users/NikhilBartwal/starred{/owner}{/repo}
https://api.github.com/users/NikhilBartwal/subscriptions
User
https://api.github.com/users/NikhilBartwal
public
CONTRIBUTOR
Hey @lhoestq, I've just added the required tests for checking key duplicates and invalid key data types.
I think we have caught a nice little issue as 27 datasets are currently using non-unique keys (hence, the failing tests: All these datasets are giving `DuplicateKeysError` during testing). 
These datasets were not detected earlier as there was no key checking when `num_examples < writer_batch_size` due to which they passed the dummy data generation test. This bug was fixed by adding the test to `writer.finalize()` method as well for  checking any leftover examples from batches. 

I'd like to make changes to the faulty datasets' scripts. However, I was wondering if I should do that in this PR itself or open a new PR as this might get messy in the same PR. Let me know your thoughts on this. Thanks!
2021-05-05T20:41:03Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-832992676
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgzMjk5MjY3Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/832992676/reactions
2021-05-05T21:03:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/832992676
https://avatars.githubusercontent.com/u/42388668?v=4
https://api.github.com/users/NikhilBartwal/events{/privacy}
https://api.github.com/users/NikhilBartwal/followers
https://api.github.com/users/NikhilBartwal/following{/other_user}
https://api.github.com/users/NikhilBartwal/gists{/gist_id}
https://github.com/NikhilBartwal
NikhilBartwal
MDQ6VXNlcjQyMzg4NjY4
https://api.github.com/users/NikhilBartwal/orgs
https://api.github.com/users/NikhilBartwal/received_events
https://api.github.com/users/NikhilBartwal/repos
https://api.github.com/users/NikhilBartwal/starred{/owner}{/repo}
https://api.github.com/users/NikhilBartwal/subscriptions
User
https://api.github.com/users/NikhilBartwal
public
MEMBER
Hi ! Once https://github.com/huggingface/datasets/pull/2333 is merged, feel free to merge master into your branch to fix the CI :)
2021-05-07T15:28:49Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-834524115
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgzNDUyNDExNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/834524115/reactions
2021-05-07T15:28:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/834524115
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Thanks a lot for the help @lhoestq. Besides merging the new changes, I guess this PR is completed for now :)
2021-05-07T15:47:17Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-834552069
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgzNDU1MjA2OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/834552069/reactions
2021-05-07T15:47:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/834552069
https://avatars.githubusercontent.com/u/42388668?v=4
https://api.github.com/users/NikhilBartwal/events{/privacy}
https://api.github.com/users/NikhilBartwal/followers
https://api.github.com/users/NikhilBartwal/following{/other_user}
https://api.github.com/users/NikhilBartwal/gists{/gist_id}
https://github.com/NikhilBartwal
NikhilBartwal
MDQ6VXNlcjQyMzg4NjY4
https://api.github.com/users/NikhilBartwal/orgs
https://api.github.com/users/NikhilBartwal/received_events
https://api.github.com/users/NikhilBartwal/repos
https://api.github.com/users/NikhilBartwal/starred{/owner}{/repo}
https://api.github.com/users/NikhilBartwal/subscriptions
User
https://api.github.com/users/NikhilBartwal
public
MEMBER
I just merged the PR, feel free to merge `master` into your branch. It should fix most most of the CI issues. If there are some left we can fix them in this PR :)
2021-05-07T15:57:45Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-834566775
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgzNDU2Njc3NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/834566775/reactions
2021-05-07T15:57:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/834566775
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq Looks like the PR is completed now. Thanks for helping me out so much in this :)
2021-05-07T16:18:33Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-834587078
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgzNDU4NzA3OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/834587078/reactions
2021-05-07T16:18:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/834587078
https://avatars.githubusercontent.com/u/42388668?v=4
https://api.github.com/users/NikhilBartwal/events{/privacy}
https://api.github.com/users/NikhilBartwal/followers
https://api.github.com/users/NikhilBartwal/following{/other_user}
https://api.github.com/users/NikhilBartwal/gists{/gist_id}
https://github.com/NikhilBartwal
NikhilBartwal
MDQ6VXNlcjQyMzg4NjY4
https://api.github.com/users/NikhilBartwal/orgs
https://api.github.com/users/NikhilBartwal/received_events
https://api.github.com/users/NikhilBartwal/repos
https://api.github.com/users/NikhilBartwal/starred{/owner}{/repo}
https://api.github.com/users/NikhilBartwal/subscriptions
User
https://api.github.com/users/NikhilBartwal
public
CONTRIBUTOR
Hey @lhoestq, I've added the test and corrected the Cl errors as well. Do let me know if this requires any change. Thanks!
2021-05-07T18:06:46Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-834662932
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgzNDY2MjkzMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/834662932/reactions
2021-05-07T18:06:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/834662932
https://avatars.githubusercontent.com/u/42388668?v=4
https://api.github.com/users/NikhilBartwal/events{/privacy}
https://api.github.com/users/NikhilBartwal/followers
https://api.github.com/users/NikhilBartwal/following{/other_user}
https://api.github.com/users/NikhilBartwal/gists{/gist_id}
https://github.com/NikhilBartwal
NikhilBartwal
MDQ6VXNlcjQyMzg4NjY4
https://api.github.com/users/NikhilBartwal/orgs
https://api.github.com/users/NikhilBartwal/received_events
https://api.github.com/users/NikhilBartwal/repos
https://api.github.com/users/NikhilBartwal/starred{/owner}{/repo}
https://api.github.com/users/NikhilBartwal/subscriptions
User
https://api.github.com/users/NikhilBartwal
public
MEMBER
Merging. I'll update the comment on the master branch (for some reason I can edit files on this branch)
2021-05-10T17:31:18Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-837012770
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgzNzAxMjc3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/837012770/reactions
2021-05-10T17:31:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/837012770
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq Thank you for the help and feedback. Feels great to contribute!
2021-05-10T18:04:37Z
https://github.com/huggingface/datasets/pull/2245#issuecomment-837064155
https://api.github.com/repos/huggingface/datasets/issues/2245
MDEyOklzc3VlQ29tbWVudDgzNzA2NDE1NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/837064155/reactions
2021-05-10T18:04:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/837064155
https://avatars.githubusercontent.com/u/42388668?v=4
https://api.github.com/users/NikhilBartwal/events{/privacy}
https://api.github.com/users/NikhilBartwal/followers
https://api.github.com/users/NikhilBartwal/following{/other_user}
https://api.github.com/users/NikhilBartwal/gists{/gist_id}
https://github.com/NikhilBartwal
NikhilBartwal
MDQ6VXNlcjQyMzg4NjY4
https://api.github.com/users/NikhilBartwal/orgs
https://api.github.com/users/NikhilBartwal/received_events
https://api.github.com/users/NikhilBartwal/repos
https://api.github.com/users/NikhilBartwal/starred{/owner}{/repo}
https://api.github.com/users/NikhilBartwal/subscriptions
User
https://api.github.com/users/NikhilBartwal
public
Closes #2230 
There is currently no verification for the data type and the uniqueness of the keys yielded by the `dataset_builder`.
This PR is currently a work in progress with the following goals:

-  [x] Adding `hash_salt` to `ArrowWriter` so that the keys belonging to different splits have different hash
-  [x] Add `key` arrtibute to `ArrowWriter.write()` for hashing
-  [x] Add a hashing class which takes an input key of certain type (`str`/`int`/anything convertible to string) and produces a 128-bit hash using `hashlib.md5`
-  [x] Creating a function giving a custom error message when non-unique keys are found 
   **[This will take care of type-checking for keys]**
-  [x] Checking for duplicate keys in `writer.write()` for each batch

[**NOTE**: This PR is currently concerned with `GeneratorBasedBuilder` only, for simplification. A subsequent PR will be made in future for `ArrowBasedBuilder`]

@lhoestq Thank you for the feedback. It would be great to have your guidance on this!