Loading wikipedia 20200501.en throws pyarrow related error
NONE
That's how I loaded the dataset
```python
from datasets import load_dataset
ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache')
```
2021-03-30T21:13:58Z
https://github.com/huggingface/datasets/issues/2144#issuecomment-810581946
https://api.github.com/repos/huggingface/datasets/issues/2144
MDEyOklzc3VlQ29tbWVudDgxMDU4MTk0Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/810581946/reactions
2021-03-30T21:13:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/810581946
https://avatars.githubusercontent.com/u/26637405?v=4
https://api.github.com/users/TomPyonsuke/events{/privacy}
https://api.github.com/users/TomPyonsuke/followers
https://api.github.com/users/TomPyonsuke/following{/other_user}
https://api.github.com/users/TomPyonsuke/gists{/gist_id}
https://github.com/TomPyonsuke
TomPyonsuke
MDQ6VXNlcjI2NjM3NDA1
https://api.github.com/users/TomPyonsuke/orgs
https://api.github.com/users/TomPyonsuke/received_events
https://api.github.com/users/TomPyonsuke/repos
https://api.github.com/users/TomPyonsuke/starred{/owner}{/repo}
https://api.github.com/users/TomPyonsuke/subscriptions
User
https://api.github.com/users/TomPyonsuke
public
MEMBER
Hi ! It looks like the arrow file in the folder
`/usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931` is corrupted.

Can you take a look and check that it's 18.3GB ?

If not, then maybe you need to redownload it:
```python
from datasets import load_dataset
ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache', download_mode="force_redownload")
```
2021-03-31T13:16:38Z
https://github.com/huggingface/datasets/issues/2144#issuecomment-811059883
https://api.github.com/repos/huggingface/datasets/issues/2144
MDEyOklzc3VlQ29tbWVudDgxMTA1OTg4Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/811059883/reactions
2021-03-31T13:16:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811059883
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> Hi ! It looks like the arrow file in the folder
> `/usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931` is corrupted.
> 
> Can you take a look and check that it's 18.3GB ?
> 
> If not, then maybe you need to redownload it:
> 
> ```python
> from datasets import load_dataset
> ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache', download_mode="force_redownload")
> ```

Hi Ihoestq, thanks for the reply! Actually i think my issue is i couldn't download the dataset beyond 10.7G. It feels like the whole dataset is split into different volumes and after the first one was downloaded it crashed before proceeding to the next one. I did try 'force_redownload' mode but still got the same issue.
2021-03-31T16:31:49Z
https://github.com/huggingface/datasets/issues/2144#issuecomment-811217389
https://api.github.com/repos/huggingface/datasets/issues/2144
MDEyOklzc3VlQ29tbWVudDgxMTIxNzM4OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/811217389/reactions
2021-03-31T16:31:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811217389
https://avatars.githubusercontent.com/u/26637405?v=4
https://api.github.com/users/TomPyonsuke/events{/privacy}
https://api.github.com/users/TomPyonsuke/followers
https://api.github.com/users/TomPyonsuke/following{/other_user}
https://api.github.com/users/TomPyonsuke/gists{/gist_id}
https://github.com/TomPyonsuke
TomPyonsuke
MDQ6VXNlcjI2NjM3NDA1
https://api.github.com/users/TomPyonsuke/orgs
https://api.github.com/users/TomPyonsuke/received_events
https://api.github.com/users/TomPyonsuke/repos
https://api.github.com/users/TomPyonsuke/starred{/owner}{/repo}
https://api.github.com/users/TomPyonsuke/subscriptions
User
https://api.github.com/users/TomPyonsuke
public
MEMBER
I just tried on my side and got no issues.
When downloading the dataset again, did it crash at 10.7GB as well ?
2021-03-31T17:34:39Z
https://github.com/huggingface/datasets/issues/2144#issuecomment-811277314
https://api.github.com/repos/huggingface/datasets/issues/2144
MDEyOklzc3VlQ29tbWVudDgxMTI3NzMxNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/811277314/reactions
2021-03-31T17:34:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811277314
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> I just tried on my side and got no issues.
> When downloading the dataset again, did it crash at 10.7GB as well ?

Yes i have tried it multiple times on different machines. I am wondering if you could share the screenshot of your dependency versions and i will try to make them the same as yours?
2021-03-31T21:02:34Z
https://github.com/huggingface/datasets/issues/2144#issuecomment-811463318
https://api.github.com/repos/huggingface/datasets/issues/2144
MDEyOklzc3VlQ29tbWVudDgxMTQ2MzMxOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/811463318/reactions
2021-03-31T21:02:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811463318
https://avatars.githubusercontent.com/u/26637405?v=4
https://api.github.com/users/TomPyonsuke/events{/privacy}
https://api.github.com/users/TomPyonsuke/followers
https://api.github.com/users/TomPyonsuke/following{/other_user}
https://api.github.com/users/TomPyonsuke/gists{/gist_id}
https://github.com/TomPyonsuke
TomPyonsuke
MDQ6VXNlcjI2NjM3NDA1
https://api.github.com/users/TomPyonsuke/orgs
https://api.github.com/users/TomPyonsuke/received_events
https://api.github.com/users/TomPyonsuke/repos
https://api.github.com/users/TomPyonsuke/starred{/owner}{/repo}
https://api.github.com/users/TomPyonsuke/subscriptions
User
https://api.github.com/users/TomPyonsuke
public
MEMBER
I tried using `datasets` from `master` on macos with python 3.7.2
I also have `requests==2.23.0` and `tqdm==4.45.0`.
2021-04-01T09:21:17Z
https://github.com/huggingface/datasets/issues/2144#issuecomment-811777163
https://api.github.com/repos/huggingface/datasets/issues/2144
MDEyOklzc3VlQ29tbWVudDgxMTc3NzE2Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/811777163/reactions
2021-04-01T09:21:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811777163
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
**Problem description**
I am getting the following error when trying to load wikipedia/20200501.en dataset.

**Error log**
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, post-processed: Unknown size, total: 34.06 GiB) to /usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931...
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.6k/14.6k [00:00<00:00, 5.41MB/s]
Downloading:  59%|███████████████████████████████████████████████████████████████████████████████████████▊                                                              | 10.7G/18.3G [11:30<08:08, 15.5MB/s]
Dataset wikipedia downloaded and prepared to /usr/local/workspace/NAS_NLP/cache/wikipedia/20200501.en/1.0.0/50aa706aa417bb77d910ad61211cc672c0ef3e0f224225a5e0a18277ade8b931. Subsequent calls will reuse this data.
Traceback (most recent call last):
  File "load_wiki.py", line 2, in <module>
    ds = load_dataset('wikipedia', '20200501.en', cache_dir='/usr/local/workspace/NAS_NLP/cache')
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 751, in load_dataset
    ds = builder_instance.as_dataset(split=split, ignore_verifications=ignore_verifications, in_memory=keep_in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 746, in as_dataset
    map_tuple=True,
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 204, in map_nested
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 204, in <listcomp>
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/py_utils.py", line 142, in _single_map_nested
    return function(data_struct)
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 763, in _build_single_dataset
    in_memory=in_memory,
  File "/usr/local/lib/python3.6/dist-packages/datasets/builder.py", line 835, in _as_dataset
    in_memory=in_memory,
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 215, in read
    return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 236, in read_files
    pa_table = self._read_files(files, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 171, in _read_files
    pa_table: pa.Table = self._get_dataset_from_filename(f_dict, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 302, in _get_dataset_from_filename
    pa_table = ArrowReader.read_table(filename, in_memory=in_memory)
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 324, in read_table
    pa_table = f.read_all()
  File "pyarrow/ipc.pxi", line 544, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: Expected to be able to read 9176784 bytes for message body, got 4918712

**Detailed version info**
datasets==1.5.0
  - dataclasses [required: Any, installed: 0.8]
  - dill [required: Any, installed: 0.3.3]
  - fsspec [required: Any, installed: 0.8.7]
    - importlib-metadata [required: Any, installed: 1.7.0]
      - zipp [required: >=0.5, installed: 3.1.0]
  - huggingface-hub [required: <0.1.0, installed: 0.0.7]
    - filelock [required: Any, installed: 3.0.12]
    - importlib-metadata [required: Any, installed: 1.7.0]
      - zipp [required: >=0.5, installed: 3.1.0]
    - requests [required: Any, installed: 2.24.0]
      - certifi [required: >=2017.4.17, installed: 2020.6.20]
      - chardet [required: >=3.0.2,<4, installed: 3.0.4]
      - idna [required: >=2.5,<3, installed: 2.6]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
    - tqdm [required: Any, installed: 4.49.0]
  - importlib-metadata [required: Any, installed: 1.7.0]
    - zipp [required: >=0.5, installed: 3.1.0]
  - multiprocess [required: Any, installed: 0.70.11.1]
    - dill [required: >=0.3.3, installed: 0.3.3]
  - numpy [required: >=1.17, installed: 1.17.0]
  - pandas [required: Any, installed: 1.1.5]
    - numpy [required: >=1.15.4, installed: 1.17.0]
    - python-dateutil [required: >=2.7.3, installed: 2.8.0]
      - six [required: >=1.5, installed: 1.15.0]
    - pytz [required: >=2017.2, installed: 2020.1]
  - pyarrow [required: >=0.17.1, installed: 3.0.0]
    - numpy [required: >=1.16.6, installed: 1.17.0]
  - requests [required: >=2.19.0, installed: 2.24.0]
    - certifi [required: >=2017.4.17, installed: 2020.6.20]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.6]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
  - tqdm [required: >=4.27,<4.50.0, installed: 4.49.0]
  - xxhash [required: Any, installed: 2.0.0]
