NonMatchingChecksumError when checksum is None
MEMBER
Hi @jxmorris12, thanks for reporting.

The objective of `verify_checksums` is to check that both checksums are equal. Therefore if one is None and the other is non-None, they are not equal, and the function accordingly raises a NonMatchingChecksumError. That behavior is expected.

The question is: how did you generate the expected checksum? Normally, it should not be None. To properly generate it (it is contained in the `dataset_infos.json` file), you should have runned: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md
```shell
datasets-cli test <your-dataset-folder> --save_infos --all_configs
```

On the other hand, you should take into account that the generation of this file is NOT mandatory for personal/community datasets (we only require it for "canonical" datasets, i.e., datasets added to our library GitHub repository: https://github.com/huggingface/datasets/tree/master/datasets). Therefore, other option would be just to delete the `dataset_infos.json` file. If that file is not present, the function `verify_checksums` is not executed.

Finally, you can circumvent the `verify_checksums` function by passing `ignore_verifications=True` to `load_dataset`:
```python
load_dataset(..., ignore_verifications=True)
``` 
2022-03-08T06:45:18Z
https://github.com/huggingface/datasets/issues/3848#issuecomment-1061461344
https://api.github.com/repos/huggingface/datasets/issues/3848
IC_kwDODunzps4_RJ1g
https://api.github.com/repos/huggingface/datasets/issues/comments/1061461344/reactions
2022-03-08T06:45:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1061461344
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
Thanks @albertvillanova!

That's fine. I did run that command when I was adding a new dataset. Maybe because the command crashed in the middle, the checksum wasn't stored properly. I don't know where the bug is happening. But either (i) `verify_checksums` should properly handle this edge case, where the passed checksum is None or (ii) the `datasets-cli test` shouldn't generate a corrupted dataset_infos.json file.

Just a more high-level thing, I was trying to follow the instructions for adding a dataset in the CONTRIBUTING.md, so if running that command isn't even necessary, that should probably be mentioned in the document, right? But that's somewhat of a moot point, since something isn't working quite right internally if I was able to get into this corrupted state in the first place, just by following those instructions.
2022-03-08T17:00:03Z
https://github.com/huggingface/datasets/issues/3848#issuecomment-1061996589
https://api.github.com/repos/huggingface/datasets/issues/3848
IC_kwDODunzps4_TMgt
https://api.github.com/repos/huggingface/datasets/issues/comments/1061996589/reactions
2022-03-08T17:00:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1061996589
https://avatars.githubusercontent.com/u/13238952?v=4
https://api.github.com/users/jxmorris12/events{/privacy}
https://api.github.com/users/jxmorris12/followers
https://api.github.com/users/jxmorris12/following{/other_user}
https://api.github.com/users/jxmorris12/gists{/gist_id}
https://github.com/jxmorris12
jxmorris12
MDQ6VXNlcjEzMjM4OTUy
https://api.github.com/users/jxmorris12/orgs
https://api.github.com/users/jxmorris12/received_events
https://api.github.com/users/jxmorris12/repos
https://api.github.com/users/jxmorris12/starred{/owner}{/repo}
https://api.github.com/users/jxmorris12/subscriptions
User
https://api.github.com/users/jxmorris12
public
MEMBER
Hi @jxmorris12,

Definitely, your `dataset_infos.json` was corrupted (and wrongly contains expected None checksum). 

While we further investigate how this can happen and fix it, feel free to delete your `dataset_infos.json` file and recreate it with:
```shell
datasets-cli test <your-dataset-folder> --save_infos --all_configs
```

Also note that `verify_checksum` is working as expected: if it receives a None and and a non-None checksums as input pair, it must raise an exception: they are not equal. That is not a bug.
2022-03-09T08:25:00Z
https://github.com/huggingface/datasets/issues/3848#issuecomment-1062668221
https://api.github.com/repos/huggingface/datasets/issues/3848
IC_kwDODunzps4_Vwe9
https://api.github.com/repos/huggingface/datasets/issues/comments/1062668221/reactions
2022-03-09T08:38:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1062668221
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
At a higher level, also note that we are preparing the release of `datasets` version 2.0, and some docs are being updated...

In order to add a dataset, I think the most updated instructions are in our official documentation pages: https://huggingface.co/docs/datasets/share
2022-03-09T08:34:09Z
https://github.com/huggingface/datasets/issues/3848#issuecomment-1062675577
https://api.github.com/repos/huggingface/datasets/issues/3848
IC_kwDODunzps4_VyR5
https://api.github.com/repos/huggingface/datasets/issues/comments/1062675577/reactions
2022-03-09T08:34:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1062675577
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
Thanks for the info. Maybe you can update the contributing.md if it's not up-to-date.
2022-03-09T15:12:34Z
https://github.com/huggingface/datasets/issues/3848#issuecomment-1063023112
https://api.github.com/repos/huggingface/datasets/issues/3848
IC_kwDODunzps4_XHII
https://api.github.com/repos/huggingface/datasets/issues/comments/1063023112/reactions
2022-03-09T15:12:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1063023112
https://avatars.githubusercontent.com/u/13238952?v=4
https://api.github.com/users/jxmorris12/events{/privacy}
https://api.github.com/users/jxmorris12/followers
https://api.github.com/users/jxmorris12/following{/other_user}
https://api.github.com/users/jxmorris12/gists{/gist_id}
https://github.com/jxmorris12
jxmorris12
MDQ6VXNlcjEzMjM4OTUy
https://api.github.com/users/jxmorris12/orgs
https://api.github.com/users/jxmorris12/received_events
https://api.github.com/users/jxmorris12/repos
https://api.github.com/users/jxmorris12/starred{/owner}{/repo}
https://api.github.com/users/jxmorris12/subscriptions
User
https://api.github.com/users/jxmorris12
public
MEMBER
Hi @jxmorris12, we have discovered the bug why `None` checksums wrongly appeared when generating the `dataset_infos.json` file:
- #3892

The fix will be accessible once this PR merged. And we are planning to do our 2.0 release today.

We are also working on updating all our docs for our release today.
2022-03-15T07:23:28Z
https://github.com/huggingface/datasets/issues/3848#issuecomment-1067652324
https://api.github.com/repos/huggingface/datasets/issues/3848
IC_kwDODunzps4_oxTk
https://api.github.com/repos/huggingface/datasets/issues/comments/1067652324/reactions
2022-03-15T07:23:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1067652324
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
Thanks @albertvillanova - congrats on the release!
2022-03-15T14:37:26Z
https://github.com/huggingface/datasets/issues/3848#issuecomment-1068061941
https://api.github.com/repos/huggingface/datasets/issues/3848
IC_kwDODunzps4_qVT1
https://api.github.com/repos/huggingface/datasets/issues/comments/1068061941/reactions
2022-03-15T14:37:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1068061941
https://avatars.githubusercontent.com/u/13238952?v=4
https://api.github.com/users/jxmorris12/events{/privacy}
https://api.github.com/users/jxmorris12/followers
https://api.github.com/users/jxmorris12/following{/other_user}
https://api.github.com/users/jxmorris12/gists{/gist_id}
https://github.com/jxmorris12
jxmorris12
MDQ6VXNlcjEzMjM4OTUy
https://api.github.com/users/jxmorris12/orgs
https://api.github.com/users/jxmorris12/received_events
https://api.github.com/users/jxmorris12/repos
https://api.github.com/users/jxmorris12/starred{/owner}{/repo}
https://api.github.com/users/jxmorris12/subscriptions
User
https://api.github.com/users/jxmorris12
public
I ran into the following error when adding a new dataset:

```bash
expected_checksums = {'https://adversarialglue.github.io/dataset/dev.zip': {'checksum': None, 'num_bytes': 40662}}
recorded_checksums = {'https://adversarialglue.github.io/dataset/dev.zip': {'checksum': 'efb4cbd3aa4a87bfaffc310ae951981cc0a36c6c71c6425dd74e5b55f2f325c9', 'num_bytes': 40662}}
verification_name = 'dataset source files'

    def verify_checksums(expected_checksums: Optional[dict], recorded_checksums: dict, verification_name=None):
        if expected_checksums is None:
            logger.info("Unable to verify checksums.")
            return
        if len(set(expected_checksums) - set(recorded_checksums)) > 0:
            raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
        if len(set(recorded_checksums) - set(expected_checksums)) > 0:
            raise UnexpectedDownloadedFile(str(set(recorded_checksums) - set(expected_checksums)))
        bad_urls = [url for url in expected_checksums if expected_checksums[url] != recorded_checksums[url]]
        for_verification_name = " for " + verification_name if verification_name is not None else ""
        if len(bad_urls) > 0:
            error_msg = "Checksums didn't match" + for_verification_name + ":\n"
>           raise NonMatchingChecksumError(error_msg + str(bad_urls))
E           datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
E           ['https://adversarialglue.github.io/dataset/dev.zip']

src/datasets/utils/info_utils.py:40: NonMatchingChecksumError
```

## Expected results
The dataset downloads correctly, and there is no error.

## Actual results
Datasets library is looking for a checksum of None, and it gets a non-None checksum, and throws an error. This is clearly a bug.