OpenWebText: NonMatchingSplitsSizesError
MEMBER
Thanks for reporting, I'm updating the verifications metadata
2021-08-31T14:38:51Z
https://github.com/huggingface/datasets/issues/2839#issuecomment-909299590
https://api.github.com/repos/huggingface/datasets/issues/2839
IC_kwDODunzps42Ms-G
https://api.github.com/repos/huggingface/datasets/issues/comments/909299590/reactions
2021-08-31T14:38:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/909299590
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I just regenerated the verifications metadata and noticed that nothing changed: the data file is fine (the checksum didn't change), and the number of examples is still 8013769. Not sure how you managed to get 7982430 examples.

Can you try to delete your cache ( by default at `~/.cache/huggingface/datasets`) and try again please ?
Also, on which platform are you (linux/macos/windows) ?
2021-08-31T17:12:45Z
https://github.com/huggingface/datasets/issues/2839#issuecomment-909426974
https://api.github.com/repos/huggingface/datasets/issues/2839
IC_kwDODunzps42NMEe
https://api.github.com/repos/huggingface/datasets/issues/comments/909426974/reactions
2021-08-31T17:14:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/909426974
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
I'll try without deleting the whole cache (we have large datasets already stored). I was under the impression that `download_mode="force_redownload"` would bypass cache.
Sorry plateform should be linux (Redhat version 8.1)
2021-09-06T17:09:15Z
https://github.com/huggingface/datasets/issues/2839#issuecomment-913776807
https://api.github.com/repos/huggingface/datasets/issues/2839
IC_kwDODunzps42dyCn
https://api.github.com/repos/huggingface/datasets/issues/comments/913776807/reactions
2021-09-06T20:34:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/913776807
https://avatars.githubusercontent.com/u/24695242?v=4
https://api.github.com/users/thomasw21/events{/privacy}
https://api.github.com/users/thomasw21/followers
https://api.github.com/users/thomasw21/following{/other_user}
https://api.github.com/users/thomasw21/gists{/gist_id}
https://github.com/thomasw21
thomasw21
MDQ6VXNlcjI0Njk1MjQy
https://api.github.com/users/thomasw21/orgs
https://api.github.com/users/thomasw21/received_events
https://api.github.com/users/thomasw21/repos
https://api.github.com/users/thomasw21/starred{/owner}{/repo}
https://api.github.com/users/thomasw21/subscriptions
User
https://api.github.com/users/thomasw21
public
MEMBER
Hi @thomasw21 , are you still having this issue after clearing your cache ?
2021-09-21T13:55:10Z
https://github.com/huggingface/datasets/issues/2839#issuecomment-924013920
https://api.github.com/repos/huggingface/datasets/issues/2839
IC_kwDODunzps43E1Vg
https://api.github.com/repos/huggingface/datasets/issues/comments/924013920/reactions
2021-09-21T13:55:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/924013920
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Sorry I haven't had time to work on this. I'll close and re-open if I can't figure out why I'm having this issue. Thanks for taking a look !
2021-09-21T14:09:43Z
https://github.com/huggingface/datasets/issues/2839#issuecomment-924027224
https://api.github.com/repos/huggingface/datasets/issues/2839
IC_kwDODunzps43E4lY
https://api.github.com/repos/huggingface/datasets/issues/comments/924027224/reactions
2021-09-21T14:09:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/924027224
https://avatars.githubusercontent.com/u/24695242?v=4
https://api.github.com/users/thomasw21/events{/privacy}
https://api.github.com/users/thomasw21/followers
https://api.github.com/users/thomasw21/following{/other_user}
https://api.github.com/users/thomasw21/gists{/gist_id}
https://github.com/thomasw21
thomasw21
MDQ6VXNlcjI0Njk1MjQy
https://api.github.com/users/thomasw21/orgs
https://api.github.com/users/thomasw21/received_events
https://api.github.com/users/thomasw21/repos
https://api.github.com/users/thomasw21/starred{/owner}{/repo}
https://api.github.com/users/thomasw21/subscriptions
User
https://api.github.com/users/thomasw21
public
## Describe the bug

When downloading `openwebtext`, I'm getting:
```
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=39769494896, num_examples=8013769, dataset_name='openwebtext'), 'recorded': SplitInfo(name='train', num_bytes=39611023912, num_examples=7982430, dataset_name='openwebtext')}]
```

I suspect that the file we download from has changed since the size doesn't look like to match with documentation

`Downloading:   0%|          | 0.00/12.9G [00:00<?, ?B/s]` This suggest the total size is 12.9GB, whereas the one documented mentions `Size of downloaded dataset files: 12283.35 MB`.

## Steps to reproduce the bug
```python
from datasets import load_dataset

load_dataset("openwebtext", download_mode="force_redownload")
```

## Expected results

Loading is successful

## Actual results

Loading throws above error.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.10.2
- Platform: linux (Redhat version 8.1)
- Python version: 3.8
- PyArrow version: 4.0.1
