DuplicatedKeysError on personal dataset
MEMBER
Hi ! In your dataset script you must be yielding examples like
```python
for line in file:
    ...
    yield key, {...}
```

Since `datasets` 1.7.0 we enforce the keys to be unique.
However it looks like your examples generator creates duplicate keys: at least two examples have key 0.

You can fix that by making sure that your keys are unique.

For example if you use a counter to define the key of each example, make sure that your counter is not reset to 0 in during examples generation (between two open files for examples).

Let me know if you have other questions :)
2021-06-02T15:03:14Z
https://github.com/huggingface/datasets/issues/2441#issuecomment-853104767
https://api.github.com/repos/huggingface/datasets/issues/2441
MDEyOklzc3VlQ29tbWVudDg1MzEwNDc2Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/853104767/reactions
2021-06-02T15:03:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/853104767
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Yup, I indeed was generating duplicate keys. Fixed it and now it's working.
2021-06-04T23:50:03Z
https://github.com/huggingface/datasets/issues/2441#issuecomment-855133695
https://api.github.com/repos/huggingface/datasets/issues/2441
MDEyOklzc3VlQ29tbWVudDg1NTEzMzY5NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/855133695/reactions
2021-06-04T23:50:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/855133695
https://avatars.githubusercontent.com/u/22605313?v=4
https://api.github.com/users/lucaguarro/events{/privacy}
https://api.github.com/users/lucaguarro/followers
https://api.github.com/users/lucaguarro/following{/other_user}
https://api.github.com/users/lucaguarro/gists{/gist_id}
https://github.com/lucaguarro
lucaguarro
MDQ6VXNlcjIyNjA1MzEz
https://api.github.com/users/lucaguarro/orgs
https://api.github.com/users/lucaguarro/received_events
https://api.github.com/users/lucaguarro/repos
https://api.github.com/users/lucaguarro/starred{/owner}{/repo}
https://api.github.com/users/lucaguarro/subscriptions
User
https://api.github.com/users/lucaguarro
public
## Describe the bug
Ever since today, I have been getting a DuplicatedKeysError while trying to load my dataset from my own script.
Error returned when running this line: `dataset = load_dataset('/content/drive/MyDrive/Thesis/Datasets/book_preprocessing/goodreads_maharjan_trimmed_and_nered/goodreadsnered.py')`
Note that my script was working fine with earlier versions of the Datasets library. Cannot say with 100% certainty if I have been doing something wrong with my dataset script this whole time or if this is simply a bug with the new version of datasets.

## Steps to reproduce the bug
I cannot provide code to reproduce the error as I am working with my own dataset. I can however provide my script if requested.

## Expected results
For my data to be loaded.

## Actual results
**DuplicatedKeysError** exception is raised
```
Downloading and preparing dataset good_reads_practice_dataset/main_domain (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/good_reads_practice_dataset/main_domain/1.1.0/64ff7c3fee2693afdddea75002eb6887d4fedc3d812ae3622128c8504ab21655...

---------------------------------------------------------------------------

DuplicatedKeysError                       Traceback (most recent call last)

<ipython-input-6-c342ea0dae9d> in <module>()
----> 1 dataset = load_dataset('/content/drive/MyDrive/Thesis/Datasets/book_preprocessing/goodreads_maharjan_trimmed_and_nered/goodreadsnered.py')

5 frames

/usr/local/lib/python3.7/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, **config_kwargs)
    749         try_from_hf_gcs=try_from_hf_gcs,
    750         base_path=base_path,
--> 751         use_auth_token=use_auth_token,
    752     )
    753 

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    573                     if not downloaded_from_gcs:
    574                         self._download_and_prepare(
--> 575                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    576                         )
    577                     # Sync info

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    650             try:
    651                 # Prepare split will record examples associated to the split
--> 652                 self._prepare_split(split_generator, **prepare_split_kwargs)
    653             except OSError as e:
    654                 raise OSError(

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _prepare_split(self, split_generator)
    990                     writer.write(example, key)
    991             finally:
--> 992                 num_examples, num_bytes = writer.finalize()
    993 
    994         split_generator.split_info.num_examples = num_examples

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in finalize(self, close_stream)
    407         # In case current_examples < writer_batch_size, but user uses finalize()
    408         if self._check_duplicates:
--> 409             self.check_duplicate_keys()
    410             # Re-intializing to empty list for next batch
    411             self.hkey_record = []

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in check_duplicate_keys(self)
    347         for hash, key in self.hkey_record:
    348             if hash in tmp_record:
--> 349                 raise DuplicatedKeysError(key)
    350             else:
    351                 tmp_record.add(hash)

DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 0
Keys should be unique and deterministic in nature
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.7.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.9
- PyArrow version: 3.0.0
