"Checksums didn't match for dataset source files" error while loading openwebtext dataset
MEMBER
Hi try, to provide more information please.

Example code in a colab to reproduce the error, details on what you are trying to do and what you were expected and details on your environment (OS, PyPi packages version).
2020-10-12T11:49:09Z
https://github.com/huggingface/datasets/issues/726#issuecomment-707071715
https://api.github.com/repos/huggingface/datasets/issues/726
MDEyOklzc3VlQ29tbWVudDcwNzA3MTcxNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/707071715/reactions
2020-10-12T11:49:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/707071715
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
> Hi try, to provide more information please.
> 
> Example code in a colab to reproduce the error, details on what you are trying to do and what you were expected and details on your environment (OS, PyPi packages version).

I have update the description, sorry for the incomplete issue by mistake.
2020-10-12T11:55:54Z
https://github.com/huggingface/datasets/issues/726#issuecomment-707074740
https://api.github.com/repos/huggingface/datasets/issues/726
MDEyOklzc3VlQ29tbWVudDcwNzA3NDc0MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/707074740/reactions
2020-10-12T11:55:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/707074740
https://avatars.githubusercontent.com/u/16469472?v=4
https://api.github.com/users/SparkJiao/events{/privacy}
https://api.github.com/users/SparkJiao/followers
https://api.github.com/users/SparkJiao/following{/other_user}
https://api.github.com/users/SparkJiao/gists{/gist_id}
https://github.com/SparkJiao
SparkJiao
MDQ6VXNlcjE2NDY5NDcy
https://api.github.com/users/SparkJiao/orgs
https://api.github.com/users/SparkJiao/received_events
https://api.github.com/users/SparkJiao/repos
https://api.github.com/users/SparkJiao/starred{/owner}{/repo}
https://api.github.com/users/SparkJiao/subscriptions
User
https://api.github.com/users/SparkJiao
public
NONE
Hi, I have manually downloaded the compressed dataset `openwebtext.tar.xz' and use the following command to preprocess the examples:
```
>>> dataset = load_dataset('/home/admin/workspace/datasets/datasets-master/datasets-master/datasets/openwebtext', data_dir='/home/admin/workspace/datasets')
Using custom data configuration default
Downloading and preparing dataset openwebtext/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/admin/.cache/huggingface/datasets/openwebtext/default/0.0.0/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02...
Dataset openwebtext downloaded and prepared to /home/admin/.cache/huggingface/datasets/openwebtext/default/0.0.0/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02. Subsequent calls will reuse this data.
>>> len(dataset['train'])
74571
>>>
```
The size of the pre-processed example file is only 354MB, however the processed bookcorpus dataset is 4.6g. Are there any problems?
2020-10-12T12:33:42Z
https://github.com/huggingface/datasets/issues/726#issuecomment-707092478
https://api.github.com/repos/huggingface/datasets/issues/726
MDEyOklzc3VlQ29tbWVudDcwNzA5MjQ3OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/707092478/reactions
2020-10-12T12:34:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/707092478
https://avatars.githubusercontent.com/u/16469472?v=4
https://api.github.com/users/SparkJiao/events{/privacy}
https://api.github.com/users/SparkJiao/followers
https://api.github.com/users/SparkJiao/following{/other_user}
https://api.github.com/users/SparkJiao/gists{/gist_id}
https://github.com/SparkJiao
SparkJiao
MDQ6VXNlcjE2NDY5NDcy
https://api.github.com/users/SparkJiao/orgs
https://api.github.com/users/SparkJiao/received_events
https://api.github.com/users/SparkJiao/repos
https://api.github.com/users/SparkJiao/starred{/owner}{/repo}
https://api.github.com/users/SparkJiao/subscriptions
User
https://api.github.com/users/SparkJiao
public
NONE
NonMatchingChecksumError: Checksums didn't match for dataset source files:

i got this issue when i try to work on my own datasets kindly tell me, from where i can get checksums of train and dev file in my github repo
2021-04-27T06:56:50Z
https://github.com/huggingface/datasets/issues/726#issuecomment-827362700
https://api.github.com/repos/huggingface/datasets/issues/726
MDEyOklzc3VlQ29tbWVudDgyNzM2MjcwMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/827362700/reactions
2021-04-27T06:56:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/827362700
https://avatars.githubusercontent.com/u/50129957?v=4
https://api.github.com/users/Muhammadharun786/events{/privacy}
https://api.github.com/users/Muhammadharun786/followers
https://api.github.com/users/Muhammadharun786/following{/other_user}
https://api.github.com/users/Muhammadharun786/gists{/gist_id}
https://github.com/Muhammadharun786
Muhammadharun786
MDQ6VXNlcjUwMTI5OTU3
https://api.github.com/users/Muhammadharun786/orgs
https://api.github.com/users/Muhammadharun786/received_events
https://api.github.com/users/Muhammadharun786/repos
https://api.github.com/users/Muhammadharun786/starred{/owner}{/repo}
https://api.github.com/users/Muhammadharun786/subscriptions
User
https://api.github.com/users/Muhammadharun786
public
NONE
Hi,  I got the similar issue for xnli dataset while working on colab with python3.7.  

`nlp.load_dataset(path = 'xnli')`

The above command resulted in following issue : 
```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://www.nyu.edu/projects/bowman/xnli/XNLI-1.0.zip']
```

Any idea how to fix this  ?
2021-09-01T08:43:11Z
https://github.com/huggingface/datasets/issues/726#issuecomment-910071435
https://api.github.com/repos/huggingface/datasets/issues/726
IC_kwDODunzps42PpaL
https://api.github.com/repos/huggingface/datasets/issues/comments/910071435/reactions
2021-09-01T08:43:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/910071435
https://avatars.githubusercontent.com/u/29161745?v=4
https://api.github.com/users/tanvidadu/events{/privacy}
https://api.github.com/users/tanvidadu/followers
https://api.github.com/users/tanvidadu/following{/other_user}
https://api.github.com/users/tanvidadu/gists{/gist_id}
https://github.com/tanvidadu
tanvidadu
MDQ6VXNlcjI5MTYxNzQ1
https://api.github.com/users/tanvidadu/orgs
https://api.github.com/users/tanvidadu/received_events
https://api.github.com/users/tanvidadu/repos
https://api.github.com/users/tanvidadu/starred{/owner}{/repo}
https://api.github.com/users/tanvidadu/subscriptions
User
https://api.github.com/users/tanvidadu
public
NONE
Did anyone figure out how to fix this error?
2021-10-10T01:52:21Z
https://github.com/huggingface/datasets/issues/726#issuecomment-939390857
https://api.github.com/repos/huggingface/datasets/issues/726
IC_kwDODunzps43_feJ
https://api.github.com/repos/huggingface/datasets/issues/comments/939390857/reactions
2021-10-10T01:52:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/939390857
https://avatars.githubusercontent.com/u/8942987?v=4
https://api.github.com/users/RylanSchaeffer/events{/privacy}
https://api.github.com/users/RylanSchaeffer/followers
https://api.github.com/users/RylanSchaeffer/following{/other_user}
https://api.github.com/users/RylanSchaeffer/gists{/gist_id}
https://github.com/RylanSchaeffer
RylanSchaeffer
MDQ6VXNlcjg5NDI5ODc=
https://api.github.com/users/RylanSchaeffer/orgs
https://api.github.com/users/RylanSchaeffer/received_events
https://api.github.com/users/RylanSchaeffer/repos
https://api.github.com/users/RylanSchaeffer/starred{/owner}{/repo}
https://api.github.com/users/RylanSchaeffer/subscriptions
User
https://api.github.com/users/RylanSchaeffer
public
MEMBER
Fixed by:
- #2857
2022-02-15T10:38:57Z
https://github.com/huggingface/datasets/issues/726#issuecomment-1040116307
https://api.github.com/repos/huggingface/datasets/issues/726
IC_kwDODunzps49_upT
https://api.github.com/repos/huggingface/datasets/issues/comments/1040116307/reactions
2022-02-15T10:38:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1040116307
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Says fixed but I'm still getting it. 

command:

 dataset = load_dataset("ted_talks_iwslt", language_pair=("en", "es"), year="2014",download_mode="force_redownload")

got:

Using custom data configuration en_es_2014-35a2d3350a0f9823
Downloading and preparing dataset ted_talks_iwslt/en_es_2014 (download: 2.15 KiB, generated: Unknown size, post-processed: Unknown size, total: 2.15 KiB) to /home/ken/.cache/huggingface/datasets/ted_talks_iwslt/en_es_2014-35a2d3350a0f9823/1.1.0/43935b3fe470c753a023642e1f54b068c590847f9928bd3f2ec99f15702ad6a6...
Downloading:
2.21k/? [00:00<00:00, 141kB/s]

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?id=1Cz1Un9p8Xn9IpEMMrg2kXSDt0dnjxc4z&export=download']
2022-02-17T17:50:04Z
https://github.com/huggingface/datasets/issues/726#issuecomment-1043238995
https://api.github.com/repos/huggingface/datasets/issues/726
IC_kwDODunzps4-LpBT
https://api.github.com/repos/huggingface/datasets/issues/comments/1043238995/reactions
2022-02-17T17:53:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1043238995
https://avatars.githubusercontent.com/u/5727476?v=4
https://api.github.com/users/OtwellResearch/events{/privacy}
https://api.github.com/users/OtwellResearch/followers
https://api.github.com/users/OtwellResearch/following{/other_user}
https://api.github.com/users/OtwellResearch/gists{/gist_id}
https://github.com/OtwellResearch
OtwellResearch
MDQ6VXNlcjU3Mjc0NzY=
https://api.github.com/users/OtwellResearch/orgs
https://api.github.com/users/OtwellResearch/received_events
https://api.github.com/users/OtwellResearch/repos
https://api.github.com/users/OtwellResearch/starred{/owner}{/repo}
https://api.github.com/users/OtwellResearch/subscriptions
User
https://api.github.com/users/OtwellResearch
public
Hi,
I have encountered this problem during loading the openwebtext dataset:
```
>>> dataset = load_dataset('openwebtext')
Downloading and preparing dataset openwebtext/plain_text (download: 12.00 GiB, generated: 37.04 GiB, post-processed: Unknown size, total: 49.03 GiB) to /home/admin/.cache/huggingface/datasets/openwebtext/plain_text/1.0.0/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/admin/workspace/anaconda3/envs/torch1.6-py3.7/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/home/admin/workspace/anaconda3/envs/torch1.6-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/admin/workspace/anaconda3/envs/torch1.6-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 536, in _download_and_prepare
    self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
  File "/home/admin/workspace/anaconda3/envs/torch1.6-py3.7/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 39, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://zenodo.org/record/3834942/files/openwebtext.tar.xz']
```
I think this problem is caused because the released dataset has changed. Or I should download the dataset manually?

Sorry for release the unfinised issue by mistake.