MemoryError when loading German Wikipedia
MEMBER
Hi !

As you noticed, "big" datasets like Wikipedia require apache beam to be processed.
However users usually don't have an apache beam runtime available (spark, dataflow, etc.) so our goal for this library is to also make available processed versions of these datasets, so that users can just download and use them right away.

This is the case for english and french wikipedia right now: we've processed them ourselves and now they are available from our google storage. However we've not processed the german one (yet).
2020-06-17T15:22:02Z
https://github.com/huggingface/datasets/issues/278#issuecomment-645441945
https://api.github.com/repos/huggingface/datasets/issues/278
MDEyOklzc3VlQ29tbWVudDY0NTQ0MTk0NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/645441945/reactions
2020-06-17T15:22:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/645441945
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi @lhoestq 

Thank you for your quick reply. I thought this might be the case, that the processing was done for some languages and not for others. Is there any set timeline for when other languages (German, Italian) will be processed?

Given enough memory, is it possible to process the data ourselves by specifying the `beam_runner`?
2020-06-17T15:33:03Z
https://github.com/huggingface/datasets/issues/278#issuecomment-645448646
https://api.github.com/repos/huggingface/datasets/issues/278
MDEyOklzc3VlQ29tbWVudDY0NTQ0ODY0Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/645448646/reactions
2020-06-17T15:33:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/645448646
https://avatars.githubusercontent.com/u/4698028?v=4
https://api.github.com/users/gregburman/events{/privacy}
https://api.github.com/users/gregburman/followers
https://api.github.com/users/gregburman/following{/other_user}
https://api.github.com/users/gregburman/gists{/gist_id}
https://github.com/gregburman
gregburman
MDQ6VXNlcjQ2OTgwMjg=
https://api.github.com/users/gregburman/orgs
https://api.github.com/users/gregburman/received_events
https://api.github.com/users/gregburman/repos
https://api.github.com/users/gregburman/starred{/owner}{/repo}
https://api.github.com/users/gregburman/subscriptions
User
https://api.github.com/users/gregburman
public
MEMBER
Adding them is definitely in our short term objectives. I'll be working on this early next week :)

Although if you have an apache beam runtime feel free to specify the beam runner. You can find more info [here](https://github.com/huggingface/nlp/blob/master/docs/beam_dataset.md) on how to make it work on Dataflow but you can adapt it for Spark or any other beam runtime (by changing the `runner`).

However if you don't have a beam runtime and even if you have enough memory, I discourage you to use the `DirectRunner` on the german or italian wikipedia. According to Apache Beam documentation it was made for testing purposes and therefore it is memory-inefficient.
2020-06-17T16:31:31Z
https://github.com/huggingface/datasets/issues/278#issuecomment-645481263
https://api.github.com/repos/huggingface/datasets/issues/278
MDEyOklzc3VlQ29tbWVudDY0NTQ4MTI2Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/645481263/reactions
2020-06-17T16:31:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/645481263
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
German is [almost] done @gregburman
2020-06-18T10:16:52Z
https://github.com/huggingface/datasets/issues/278#issuecomment-645925115
https://api.github.com/repos/huggingface/datasets/issues/278
MDEyOklzc3VlQ29tbWVudDY0NTkyNTExNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/645925115/reactions
2020-06-18T10:21:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/645925115
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I added the German and the Italian Wikipedia to our google cloud storage:
First update the `nlp` package to 0.3.0:
```bash
pip install nlp --upgrade
```
and then
```python
from nlp import load_dataset
wiki_de = load_dataset("wikipedia", "20200501.de")
wiki_it = load_dataset("wikipedia", "20200501.it")
```
The datasets are downloaded and directly ready to use (no processing).
2020-06-19T09:41:11Z
https://github.com/huggingface/datasets/issues/278#issuecomment-646541039
https://api.github.com/repos/huggingface/datasets/issues/278
MDEyOklzc3VlQ29tbWVudDY0NjU0MTAzOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/646541039/reactions
2020-06-19T09:41:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/646541039
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi @lhoestq 

Wow, thanks so much, that's **really** incredible! I was considering looking at creating my own Beam Dataset, as per the doc you linked, but instead opted to process the data myself using `wikiextractor`. However, now that this is available, I'll definitely switch across and use it.

Thanks so much for the incredible work, this really helps out our team considerably!

Have a great (and well-deserved ;) weekend ahead!

P.S. I'm not sure if I should close the issue here - if so I'm happy to do so.
2020-06-19T12:49:02Z
https://github.com/huggingface/datasets/issues/278#issuecomment-646617487
https://api.github.com/repos/huggingface/datasets/issues/278
MDEyOklzc3VlQ29tbWVudDY0NjYxNzQ4Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/646617487/reactions
2020-06-19T12:49:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/646617487
https://avatars.githubusercontent.com/u/4698028?v=4
https://api.github.com/users/gregburman/events{/privacy}
https://api.github.com/users/gregburman/followers
https://api.github.com/users/gregburman/following{/other_user}
https://api.github.com/users/gregburman/gists{/gist_id}
https://github.com/gregburman
gregburman
MDQ6VXNlcjQ2OTgwMjg=
https://api.github.com/users/gregburman/orgs
https://api.github.com/users/gregburman/received_events
https://api.github.com/users/gregburman/repos
https://api.github.com/users/gregburman/starred{/owner}{/repo}
https://api.github.com/users/gregburman/subscriptions
User
https://api.github.com/users/gregburman
public
MEMBER
Thanks for your message, glad I could help :)
Closing this one.
2020-06-19T12:53:02Z
https://github.com/huggingface/datasets/issues/278#issuecomment-646619225
https://api.github.com/repos/huggingface/datasets/issues/278
MDEyOklzc3VlQ29tbWVudDY0NjYxOTIyNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/646619225/reactions
2020-06-19T12:53:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/646619225
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Hi, first off let me say thank you for all the awesome work you're doing at Hugging Face across all your projects (NLP, Transformers, Tokenizers) - they're all amazing contributions to us working with NLP models :)

I'm trying to download the German Wikipedia dataset as follows:

```
wiki = nlp.load_dataset("wikipedia", "20200501.de", split="train")
```

However, when I do so, I get the following error:

```
Downloading and preparing dataset wikipedia/20200501.de (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/ubuntu/.cache/huggingface/datasets/wikipedia/20200501.de/1.0.0...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/anaconda3/envs/albert/lib/python3.7/site-packages/nlp/load.py", line 520, in load_dataset
    save_infos=save_infos,
  File "/home/ubuntu/anaconda3/envs/albert/lib/python3.7/site-packages/nlp/builder.py", line 433, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/ubuntu/anaconda3/envs/albert/lib/python3.7/site-packages/nlp/builder.py", line 824, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
nlp.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). 
Example of usage: 
	`load_dataset('wikipedia', '20200501.de', beam_runner='DirectRunner')`
```

So, following on from the example usage at the bottom, I tried specifying `beam_runner='DirectRunner`, however when I do this after about 20 min after the data has all downloaded, I get a `MemoryError` as warned.

This isn't an issue for the English or French Wikipedia datasets (I've tried both), as neither seem to require that `beam_runner` be specified. Can you please clarify why this is an issue for the German dataset?

My nlp version is 0.2.1.

Thank you!