Fix checksum verification
MEMBER
Hi ! If I'm not mistaken both `expected_checksums[url]` and `recorded_checksums[url]` are dictionaries with keys "checksum" and "num_bytes". So we need to check whether `expected_checksums[url] != recorded_checksums[url]` (or simply `expected_checksums[url]["checksum"] != recorded_checksums[url]["checksum"]`)

But in your fix you're checking `expected_checksums[url] != recorded_checksums[url]['checksum']`.

So I think it's fine to keep this as is
2023-01-05T15:24:22Z
https://github.com/huggingface/datasets/pull/5396#issuecomment-1372359961
https://api.github.com/repos/huggingface/datasets/issues/5396
IC_kwDODunzps5RzI0Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1372359961/reactions
2023-01-05T15:24:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1372359961
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
No, the issue is that there is comparison of sclar value and dictionary.
2023-01-05T18:43:35Z
https://github.com/huggingface/datasets/pull/5396#issuecomment-1372597620
https://api.github.com/repos/huggingface/datasets/issues/5396
IC_kwDODunzps5R0C10
https://api.github.com/repos/huggingface/datasets/issues/comments/1372597620/reactions
2023-01-05T18:43:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1372597620
https://avatars.githubusercontent.com/u/9336514?v=4
https://api.github.com/users/daskol/events{/privacy}
https://api.github.com/users/daskol/followers
https://api.github.com/users/daskol/following{/other_user}
https://api.github.com/users/daskol/gists{/gist_id}
https://github.com/daskol
daskol
MDQ6VXNlcjkzMzY1MTQ=
https://api.github.com/users/daskol/orgs
https://api.github.com/users/daskol/received_events
https://api.github.com/users/daskol/repos
https://api.github.com/users/daskol/starred{/owner}{/repo}
https://api.github.com/users/daskol/subscriptions
User
https://api.github.com/users/daskol
public
CONTRIBUTOR
Acording to [`DatasetInfo`][1], we need specify a dictionary which maps a URL to a checksum as follows.

```python
CHECKSUMS = {
    URL: 'a5dc6bf63ea088ade6e98594bfa386f45211c38b2a3db3dd11b33bd530f3c481',
}

class FancyDataset:
    def _info(self):
        return DatasetInfo(..., download_checksums=CHECKSUMS)
```

However, `load_dataset` fails with this checksum definition.

[1]: https://github.com/huggingface/datasets/blob/main/src/datasets/info.py#L124-L125
2023-01-05T19:08:19Z
https://github.com/huggingface/datasets/pull/5396#issuecomment-1372621512
https://api.github.com/repos/huggingface/datasets/issues/5396
IC_kwDODunzps5R0IrI
https://api.github.com/repos/huggingface/datasets/issues/comments/1372621512/reactions
2023-01-05T19:08:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1372621512
https://avatars.githubusercontent.com/u/9336514?v=4
https://api.github.com/users/daskol/events{/privacy}
https://api.github.com/users/daskol/followers
https://api.github.com/users/daskol/following{/other_user}
https://api.github.com/users/daskol/gists{/gist_id}
https://github.com/daskol
daskol
MDQ6VXNlcjkzMzY1MTQ=
https://api.github.com/users/daskol/orgs
https://api.github.com/users/daskol/received_events
https://api.github.com/users/daskol/repos
https://api.github.com/users/daskol/starred{/owner}{/repo}
https://api.github.com/users/daskol/subscriptions
User
https://api.github.com/users/daskol
public
MEMBER
I think it has to be formatted like this right now. Maybe the DatasetInfo doc is unclear and we can improve it
```python
CHECKSUMS = {
    URL: {"checksum": checksum, "num_bytes": num_bytes},
}
```
2023-01-13T16:01:15Z
https://github.com/huggingface/datasets/pull/5396#issuecomment-1382056022
https://api.github.com/repos/huggingface/datasets/issues/5396
IC_kwDODunzps5SYIBW
https://api.github.com/repos/huggingface/datasets/issues/comments/1382056022/reactions
2023-01-13T16:01:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1382056022
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Right. I am not sure that this is a correct way to do it. People usually calculate sha256, md5, or whatever else but not size in bytes. Also, people use only some of checksum algorithms. This means that comparing dictionaries in `verify_checksums` is too strict (requires equality of all items) and raises compatibility issues in the future. Another issue is that a comparison of dictionaries assumes type constraints which imply type equality. 

Since almost noone uses checksums as far as I known, my PR suggests a minimal change to mitigate these issues except support of a specific checksum algorithm which is a separated feature and should be contributed in a separate PRs from my perspective.
2023-01-13T23:20:08Z
https://github.com/huggingface/datasets/pull/5396#issuecomment-1382578631
https://api.github.com/repos/huggingface/datasets/issues/5396
IC_kwDODunzps5SaHnH
https://api.github.com/repos/huggingface/datasets/issues/comments/1382578631/reactions
2023-01-13T23:20:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1382578631
https://avatars.githubusercontent.com/u/9336514?v=4
https://api.github.com/users/daskol/events{/privacy}
https://api.github.com/users/daskol/followers
https://api.github.com/users/daskol/following{/other_user}
https://api.github.com/users/daskol/gists{/gist_id}
https://github.com/daskol
daskol
MDQ6VXNlcjkzMzY1MTQ=
https://api.github.com/users/daskol/orgs
https://api.github.com/users/daskol/received_events
https://api.github.com/users/daskol/repos
https://api.github.com/users/daskol/starred{/owner}{/repo}
https://api.github.com/users/daskol/subscriptions
User
https://api.github.com/users/daskol
public
MEMBER
Applying this change will break the verification code, since the `expected_checksums` is a dict with those two keys.
2023-01-25T15:49:54Z
https://github.com/huggingface/datasets/pull/5396#issuecomment-1403829608
https://api.github.com/repos/huggingface/datasets/issues/5396
IC_kwDODunzps5TrL1o
https://api.github.com/repos/huggingface/datasets/issues/comments/1403829608/reactions
2023-01-25T15:49:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1403829608
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5396). All of your documentation changes will be reflected on that endpoint.
2023-01-25T15:54:00Z
https://github.com/huggingface/datasets/pull/5396#issuecomment-1403835207
https://api.github.com/repos/huggingface/datasets/issues/5396
IC_kwDODunzps5TrNNH
https://api.github.com/repos/huggingface/datasets/issues/comments/1403835207/reactions
2023-01-25T15:54:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1403835207
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
Expected checksum was verified against checksum dict (not checksum).