not being able to get wikipedia es language
NONE
@lhoestq  I really appreciate if you could help me providiing processed datasets, I do not really have access to enough resources to run the apache-beam and need to run the codes on these datasets. Only en/de/fr currently works, but I need all the languages more or less. thanks 
2021-03-05T08:34:42Z
https://github.com/huggingface/datasets/issues/1994#issuecomment-791257070
https://api.github.com/repos/huggingface/datasets/issues/1994
MDEyOklzc3VlQ29tbWVudDc5MTI1NzA3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/791257070/reactions
2021-03-05T08:45:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791257070
https://avatars.githubusercontent.com/u/79165106?v=4
https://api.github.com/users/dorost1234/events{/privacy}
https://api.github.com/users/dorost1234/followers
https://api.github.com/users/dorost1234/following{/other_user}
https://api.github.com/users/dorost1234/gists{/gist_id}
https://github.com/dorost1234
dorost1234
MDQ6VXNlcjc5MTY1MTA2
https://api.github.com/users/dorost1234/orgs
https://api.github.com/users/dorost1234/received_events
https://api.github.com/users/dorost1234/repos
https://api.github.com/users/dorost1234/starred{/owner}{/repo}
https://api.github.com/users/dorost1234/subscriptions
User
https://api.github.com/users/dorost1234
public
CONTRIBUTOR
Hi @dorost1234, I think I can help you a little. I’ve processed some Wikipedia datasets (Spanish inclusive) using the HF/datasets library during recent research.

@lhoestq Could you help me to upload these preprocessed datasets to Huggingface's repositories? To be more precise, I've built datasets from the following languages using the 20201201 dumps: Spanish, Portuguese, Russian, French, Japanese, Chinese, and Turkish. Process these datasets have high costs that most of the community can't afford. I think these preprocessed datasets I have could be helpful for someone without access to high-resource machines to process Wikipedia's dumps like @dorost1234


2021-03-05T23:54:55Z
https://github.com/huggingface/datasets/issues/1994#issuecomment-791798195
https://api.github.com/repos/huggingface/datasets/issues/1994
MDEyOklzc3VlQ29tbWVudDc5MTc5ODE5NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/791798195/reactions
2021-03-05T23:54:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791798195
https://avatars.githubusercontent.com/u/5097052?v=4
https://api.github.com/users/jonatasgrosman/events{/privacy}
https://api.github.com/users/jonatasgrosman/followers
https://api.github.com/users/jonatasgrosman/following{/other_user}
https://api.github.com/users/jonatasgrosman/gists{/gist_id}
https://github.com/jonatasgrosman
jonatasgrosman
MDQ6VXNlcjUwOTcwNTI=
https://api.github.com/users/jonatasgrosman/orgs
https://api.github.com/users/jonatasgrosman/received_events
https://api.github.com/users/jonatasgrosman/repos
https://api.github.com/users/jonatasgrosman/starred{/owner}{/repo}
https://api.github.com/users/jonatasgrosman/subscriptions
User
https://api.github.com/users/jonatasgrosman
public
NONE
Thank you so much @jonatasgrosman , I greatly appreciate your help with them. 
Yes, I unfortunately does not have access to a good resource and need it for my
research. I greatly appreciate @lhoestq  your help with uploading the processed datasets in huggingface datasets. This would be really helpful for some users like me with not access to high-memory GPU resources.

thank you both so much again.

On Sat, Mar 6, 2021 at 12:55 AM Jonatas Grosman <notifications@github.com>
wrote:

> Hi @dorost1234 <https://github.com/dorost1234>, I think I can help you a
> little. I’ve processed some Wikipedia datasets (Spanish inclusive) using
> the HF/datasets library during recent research.
>
> @lhoestq <https://github.com/lhoestq> Could you help me to upload these
> preprocessed datasets to Huggingface's repositories? To be more precise,
> I've built datasets from the following languages using the 20201201 dumps:
> Spanish, Portuguese, Russian, French, Japanese, Chinese, and Turkish.
> Process these datasets have high costs that most of the community can't
> afford. I think these preprocessed datasets I have could be helpful for
> someone without access to high-resource machines to process Wikipedia's
> dumps like @dorost1234 <https://github.com/dorost1234>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/1994#issuecomment-791798195>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMWMK5GFJFU3ACCJFUDTCFVNZANCNFSM4YUZIF4A>
> .
>

2021-03-06T10:22:33Z
https://github.com/huggingface/datasets/issues/1994#issuecomment-791908156
https://api.github.com/repos/huggingface/datasets/issues/1994
MDEyOklzc3VlQ29tbWVudDc5MTkwODE1Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/791908156/reactions
2021-03-06T10:52:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791908156
https://avatars.githubusercontent.com/u/79165106?v=4
https://api.github.com/users/dorost1234/events{/privacy}
https://api.github.com/users/dorost1234/followers
https://api.github.com/users/dorost1234/following{/other_user}
https://api.github.com/users/dorost1234/gists{/gist_id}
https://github.com/dorost1234
dorost1234
MDQ6VXNlcjc5MTY1MTA2
https://api.github.com/users/dorost1234/orgs
https://api.github.com/users/dorost1234/received_events
https://api.github.com/users/dorost1234/repos
https://api.github.com/users/dorost1234/starred{/owner}{/repo}
https://api.github.com/users/dorost1234/subscriptions
User
https://api.github.com/users/dorost1234
public
CONTRIBUTOR
Hi @dorost1234, so sorry, but looking at my files here, I figure out that I've preprocessed files using the HF/datasets for all the languages previously listed by me (Portuguese, Russian, French, Japanese, Chinese, and Turkish) except the Spanish (on my tests I've used the [wikicorpus](https://www.cs.upc.edu/~nlp/wikicorpus/) instead).

Only with the Spanish Wikipedia's dump, I had the same `KeyError: '000nbsp'` problem already reported here https://github.com/huggingface/datasets/issues/577

So nowadays, even with access to a high resource machine, you couldn't be able to get Wikipedia's Spanish data using the HF/datasets :(





2021-03-06T16:35:55Z
https://github.com/huggingface/datasets/issues/1994#issuecomment-791985546
https://api.github.com/repos/huggingface/datasets/issues/1994
MDEyOklzc3VlQ29tbWVudDc5MTk4NTU0Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/791985546/reactions
2021-03-06T16:35:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/791985546
https://avatars.githubusercontent.com/u/5097052?v=4
https://api.github.com/users/jonatasgrosman/events{/privacy}
https://api.github.com/users/jonatasgrosman/followers
https://api.github.com/users/jonatasgrosman/following{/other_user}
https://api.github.com/users/jonatasgrosman/gists{/gist_id}
https://github.com/jonatasgrosman
jonatasgrosman
MDQ6VXNlcjUwOTcwNTI=
https://api.github.com/users/jonatasgrosman/orgs
https://api.github.com/users/jonatasgrosman/received_events
https://api.github.com/users/jonatasgrosman/repos
https://api.github.com/users/jonatasgrosman/starred{/owner}{/repo}
https://api.github.com/users/jonatasgrosman/subscriptions
User
https://api.github.com/users/jonatasgrosman
public
NONE
Thanks a lot for the information and help. This would be great to have
these datasets.
@lhoestq <https://github.com/lhoestq>  Do you know a way I could get
smaller amount of these data like 1 GBtype of each language to deal with
computatioanl requirements? thanks

On Sat, Mar 6, 2021 at 5:36 PM Jonatas Grosman <notifications@github.com>
wrote:

> Hi @dorost1234 <https://github.com/dorost1234>, so sorry, but looking at
> my files here, I figure out that I've preprocessed files using the
> HF/datasets for all the languages previously listed by me (Portuguese,
> Russian, French, Japanese, Chinese, and Turkish) except the Spanish (on my
> tests I've used the wikicorpus <https://www.cs.upc.edu/~nlp/wikicorpus/>
> instead).
>
> Only with the Spanish Wikipedia's dump, I had the same KeyError: '000nbsp'
> problem already reported here #577
> <https://github.com/huggingface/datasets/issues/577>
>
> So nowadays, even with access to a high resource machine, you couldn't be
> able to get Wikipedia's Spanish data using the HF/datasets :(
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/1994#issuecomment-791985546>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMWMO7WOHWLOROPD6Q3TCJKXPANCNFSM4YUZIF4A>
> .
>

2021-03-06T22:53:21Z
https://github.com/huggingface/datasets/issues/1994#issuecomment-792097889
https://api.github.com/repos/huggingface/datasets/issues/1994
MDEyOklzc3VlQ29tbWVudDc5MjA5Nzg4OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/792097889/reactions
2021-03-06T22:53:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/792097889
https://avatars.githubusercontent.com/u/79165106?v=4
https://api.github.com/users/dorost1234/events{/privacy}
https://api.github.com/users/dorost1234/followers
https://api.github.com/users/dorost1234/following{/other_user}
https://api.github.com/users/dorost1234/gists{/gist_id}
https://github.com/dorost1234
dorost1234
MDQ6VXNlcjc5MTY1MTA2
https://api.github.com/users/dorost1234/orgs
https://api.github.com/users/dorost1234/received_events
https://api.github.com/users/dorost1234/repos
https://api.github.com/users/dorost1234/starred{/owner}{/repo}
https://api.github.com/users/dorost1234/subscriptions
User
https://api.github.com/users/dorost1234
public
MEMBER
Hi ! As mentioned above the Spanish configuration have parsing issues from `mwparserfromhell`. I haven't tested with the latest `mwparserfromhell` >=0.6 though. Which version of `mwparserfromhell` are you using ?

> @lhoestq Could you help me to upload these preprocessed datasets to Huggingface's repositories? To be more precise, I've built datasets from the following languages using the 20201201 dumps: Spanish, Portuguese, Russian, French, Japanese, Chinese, and Turkish. Process these datasets have high costs that most of the community can't afford. I think these preprocessed datasets I have could be helpful for someone without access to high-resource machines to process Wikipedia's dumps like @dorost1234

That would be awesome ! Feel free to ping me on slack so we can put the processed wikipedia files on google storage with the other ones we've already preprocessed.

> Do you know a way I could get smaller amount of these data like 1 GBtype of each language to deal with computatioanl requirements? thanks

I'd suggest to copy the [wikipedia.py](https://github.com/huggingface/datasets/blob/master/datasets/wikipedia/wikipedia.py) to a new script `custom_wikipedia.py` and modify it to only download and process only a subset of the raw data files.
You can for example replace [this line](https://github.com/huggingface/datasets/blob/64e59fc45ca2134218b3e42e83fddddbe840ff74/datasets/wikipedia/wikipedia.py#L446) by:
```python
            if total_bytes >= (1 << 30):  # stop if the total amount of data is >= 1GB
                break
            else:
                xml_urls.append(_base_url(lang) + fname)
```

Then you can load your custom wikipedia dataset with
```python
load_dataset("path/to/my/custom_wikipedia.py", f"{date}.{language}")
```
2021-03-08T14:55:00Z
https://github.com/huggingface/datasets/issues/1994#issuecomment-792809370
https://api.github.com/repos/huggingface/datasets/issues/1994
MDEyOklzc3VlQ29tbWVudDc5MjgwOTM3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/792809370/reactions
2021-03-08T14:55:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/792809370
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi @lhoestq!

> Hi ! As mentioned above the Spanish configuration have parsing issues from mwparserfromhell. I haven't tested with the latest mwparserfromhell >=0.6 though. Which version of mwparserfromhell are you using ?

I'm using the latest mwparserfromhell version (0.6)

> That would be awesome ! Feel free to ping me on slack so we can put the processed wikipedia files on google storage with the other ones we've already preprocessed.

I'll ping you there 👍 
2021-03-08T19:18:53Z
https://github.com/huggingface/datasets/issues/1994#issuecomment-793006978
https://api.github.com/repos/huggingface/datasets/issues/1994
MDEyOklzc3VlQ29tbWVudDc5MzAwNjk3OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/793006978/reactions
2021-03-08T19:18:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/793006978
https://avatars.githubusercontent.com/u/5097052?v=4
https://api.github.com/users/jonatasgrosman/events{/privacy}
https://api.github.com/users/jonatasgrosman/followers
https://api.github.com/users/jonatasgrosman/following{/other_user}
https://api.github.com/users/jonatasgrosman/gists{/gist_id}
https://github.com/jonatasgrosman
jonatasgrosman
MDQ6VXNlcjUwOTcwNTI=
https://api.github.com/users/jonatasgrosman/orgs
https://api.github.com/users/jonatasgrosman/received_events
https://api.github.com/users/jonatasgrosman/repos
https://api.github.com/users/jonatasgrosman/starred{/owner}{/repo}
https://api.github.com/users/jonatasgrosman/subscriptions
User
https://api.github.com/users/jonatasgrosman
public
NONE
Thank you so much @jonatasgrosman  and @lhoestq  this would be a great help. I am really thankful to you both and to wonderful Huggingface dataset library allowing us to train models at scale.
2021-03-11T20:46:03Z
https://github.com/huggingface/datasets/issues/1994#issuecomment-797037468
https://api.github.com/repos/huggingface/datasets/issues/1994
MDEyOklzc3VlQ29tbWVudDc5NzAzNzQ2OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/797037468/reactions
2021-03-11T20:46:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797037468
https://avatars.githubusercontent.com/u/79165106?v=4
https://api.github.com/users/dorost1234/events{/privacy}
https://api.github.com/users/dorost1234/followers
https://api.github.com/users/dorost1234/following{/other_user}
https://api.github.com/users/dorost1234/gists{/gist_id}
https://github.com/dorost1234
dorost1234
MDQ6VXNlcjc5MTY1MTA2
https://api.github.com/users/dorost1234/orgs
https://api.github.com/users/dorost1234/received_events
https://api.github.com/users/dorost1234/repos
https://api.github.com/users/dorost1234/starred{/owner}{/repo}
https://api.github.com/users/dorost1234/subscriptions
User
https://api.github.com/users/dorost1234
public
Hi
I am trying to run a code with wikipedia of config 20200501.es, getting:

Traceback (most recent call last):
  File "run_mlm_t5.py", line 608, in <module>
    main()
  File "run_mlm_t5.py", line 359, in main
    datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name)
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/load.py", line 612, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 527, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/dara/libs/anaconda3/envs/success432/lib/python3.7/site-packages/datasets-1.2.1-py3.7.egg/datasets/builder.py", line 1050, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
datasets.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). 
Example of usage: 
	`load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

thanks @lhoestq  for any suggestion/help 