MissingBeamOptions for Wikipedia 20200501.en
MEMBER
Fixed. Could you try again @mitchellgordon95 ?
It was due a file not being updated on S3.

We need to make sure all the datasets scripts get updated properly @julien-c 
2020-07-17T08:17:39Z
https://github.com/huggingface/datasets/issues/407#issuecomment-659952030
https://api.github.com/repos/huggingface/datasets/issues/407
MDEyOklzc3VlQ29tbWVudDY1OTk1MjAzMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/659952030/reactions
2020-07-17T08:17:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/659952030
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Works for me! Thanks.
2020-07-17T14:24:28Z
https://github.com/huggingface/datasets/issues/407#issuecomment-660135554
https://api.github.com/repos/huggingface/datasets/issues/407
MDEyOklzc3VlQ29tbWVudDY2MDEzNTU1NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/660135554/reactions
2020-07-17T14:24:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/660135554
https://avatars.githubusercontent.com/u/7490438?v=4
https://api.github.com/users/mitchellgordon95/events{/privacy}
https://api.github.com/users/mitchellgordon95/followers
https://api.github.com/users/mitchellgordon95/following{/other_user}
https://api.github.com/users/mitchellgordon95/gists{/gist_id}
https://github.com/mitchellgordon95
mitchellgordon95
MDQ6VXNlcjc0OTA0Mzg=
https://api.github.com/users/mitchellgordon95/orgs
https://api.github.com/users/mitchellgordon95/received_events
https://api.github.com/users/mitchellgordon95/repos
https://api.github.com/users/mitchellgordon95/starred{/owner}{/repo}
https://api.github.com/users/mitchellgordon95/subscriptions
User
https://api.github.com/users/mitchellgordon95
public
NONE
I found the same issue with almost any language other than English. (For English, it works). Will someone need to update the file on S3 again?
2021-01-12T00:52:40Z
https://github.com/huggingface/datasets/issues/407#issuecomment-758320361
https://api.github.com/repos/huggingface/datasets/issues/407
MDEyOklzc3VlQ29tbWVudDc1ODMyMDM2MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/758320361/reactions
2021-01-12T00:52:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/758320361
https://avatars.githubusercontent.com/u/51496332?v=4
https://api.github.com/users/xchenlp/events{/privacy}
https://api.github.com/users/xchenlp/followers
https://api.github.com/users/xchenlp/following{/other_user}
https://api.github.com/users/xchenlp/gists{/gist_id}
https://github.com/xchenlp
xchenlp
MDQ6VXNlcjUxNDk2MzMy
https://api.github.com/users/xchenlp/orgs
https://api.github.com/users/xchenlp/received_events
https://api.github.com/users/xchenlp/repos
https://api.github.com/users/xchenlp/starred{/owner}{/repo}
https://api.github.com/users/xchenlp/subscriptions
User
https://api.github.com/users/xchenlp
public
MEMBER
This is because only some languages are already preprocessed (en, de, fr, it) and stored on our google storage.
We plan to have a systematic way to preprocess more wikipedia languages in the future.

For the other languages you have to process them on your side using apache beam. That's why the lib asks for a Beam runner.
2021-01-12T11:41:16Z
https://github.com/huggingface/datasets/issues/407#issuecomment-758600885
https://api.github.com/repos/huggingface/datasets/issues/407
MDEyOklzc3VlQ29tbWVudDc1ODYwMDg4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/758600885/reactions
2021-01-12T11:41:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/758600885
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
There may or may not be a regression for the pre-processed Wikipedia dataset. This was working fine 10 commits ago (without having Apache Beam available):

```
nlp.load_dataset('wikipedia', "20200501.en", split='train')
```

And now, having pulled master, I get:

```
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, total: 34.06 GiB) to /home/hltcoe/mgordon/.cache/huggingface/datasets/wikipedia/20200501.en/1.0.0/76b0b2747b679bb0ee7a1621e50e5a6378477add0c662668a324a5bc07d516dd...
Traceback (most recent call last):
  File "scripts/download.py", line 11, in <module>
    fire.Fire(download_pretrain)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "scripts/download.py", line 6, in download_pretrain
    nlp.load_dataset('wikipedia', "20200501.en", split='train')
  File "/exp/mgordon/nlp/src/nlp/load.py", line 534, in load_dataset
    save_infos=save_infos,
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 460, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 870, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
nlp.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, S
park, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
        `load_dataset('wikipedia', '20200501.en', beam_runner='DirectRunner')`
```