ERROR:root:mwparserfromhell
MEMBER
It looks like it comes from `mwparserfromhell`.

Would it be possible to get the bad `section` that causes this issue ? The `section` string is from `datasets/wikipedia.py:L548` ? You could just add a `try` statement and print the section if the line `section_text.append(section.strip_code().strip())` crashes.

It will help us know if we have to fix it on our side or if it is a `mwparserfromhell` issue.
2020-06-30T16:05:09Z
https://github.com/huggingface/datasets/issues/321#issuecomment-651891543
https://api.github.com/repos/huggingface/datasets/issues/321
MDEyOklzc3VlQ29tbWVudDY1MTg5MTU0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/651891543/reactions
2020-06-30T16:05:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/651891543
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi, 

Thank you for you answer.
I have try to print the bad section using `try` and `except`, but it is a bit weird as the error seems to appear 3 times for instance, but the two first error does not print anything (as if the function did not go in the `except` part).
For the third one, I got that (I haven't display the entire text) :

> error : ==== Parque nacional Cajas ====
> {{AP|Parque nacional Cajas}}
> [[Archivo:Ecuador cajas national park.jpg|thumb|left|300px|Laguna del Cajas]]
> El parque nacional Cajas está situado en los [[Cordillera de los Andes|Andes]], al sur del [[Ecuador]], en la provincia de [[Provincia de Azuay|Azuay]], a 33
>  [[km]] al noroccidente de la ciudad de [[Cuenca (Ecuador)|Cuenca]]. Los accesos más comunes al parque inician todos en Cuenca: Desde allí, la vía Cuenca-Mol
> leturo atraviesa en Control de [[Surocucho]] en poco más de 30 minutos de viaje; más adelante, esta misma carretera pasa a orillas de la laguna La Toreadora donde están el Centro Administrativo y de Información del parque. Siguiendo de largo hacia [[Molleturo]], por esta vía se conoce el sector norte del Cajas y se serpentea entre varias lagunas mayores y menores.
> Para acceder al parque desde la costa, la vía Molleturo-Cuenca es también la mejor opción.

How can I display the link instead of the text ? I suppose it will help you more 
2020-07-03T14:07:42Z
https://github.com/huggingface/datasets/issues/321#issuecomment-653565361
https://api.github.com/repos/huggingface/datasets/issues/321
MDEyOklzc3VlQ29tbWVudDY1MzU2NTM2MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/653565361/reactions
2020-07-03T14:07:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/653565361
https://avatars.githubusercontent.com/u/26505641?v=4
https://api.github.com/users/Shiro-LK/events{/privacy}
https://api.github.com/users/Shiro-LK/followers
https://api.github.com/users/Shiro-LK/following{/other_user}
https://api.github.com/users/Shiro-LK/gists{/gist_id}
https://github.com/Shiro-LK
Shiro-LK
MDQ6VXNlcjI2NTA1NjQx
https://api.github.com/users/Shiro-LK/orgs
https://api.github.com/users/Shiro-LK/received_events
https://api.github.com/users/Shiro-LK/repos
https://api.github.com/users/Shiro-LK/starred{/owner}{/repo}
https://api.github.com/users/Shiro-LK/subscriptions
User
https://api.github.com/users/Shiro-LK
public
MEMBER
The error appears several times as Apache Beam retries to process examples up to 4 times irc.

I just tried to run this text into `mwparserfromhell` but it worked without the issue.

I used this code (from the `wikipedia.py` script):
```python
import mwparserfromhell as parser
import re
import six

raw_content = r"""==== Parque nacional Cajas ====
{{AP|Parque nacional Cajas}}
[[Archivo:Ecuador cajas national park.jpg|thumb|left|300px|Laguna del Cajas]]
El parque nacional Cajas está situado en los [[Cordillera de los Andes|Andes]], al sur del [[Ecuador]], en la provincia de [[Provincia de Azuay|Azuay]], a 33
[[km]] al noroccidente de la ciudad de [[Cuenca (Ecuador)|Cuenca]]. Los accesos más comunes al parque inician todos en Cuenca: Desde allí, la vía Cuenca-Mol
leturo atraviesa en Control de [[Surocucho]] en poco más de 30 minutos de viaje; más adelante, esta misma carretera pasa a orillas de la laguna La Toreadora donde están el Centro Administrativo y de Información del parque. Siguiendo de largo hacia [[Molleturo]], por esta vía se conoce el sector norte del Cajas y se serpentea entre varias lagunas mayores y menores.
"""

wikicode = parser.parse(raw_content)

# Filters for references, tables, and file/image links.
re_rm_wikilink = re.compile("^(?:File|Image|Media):", flags=re.IGNORECASE | re.UNICODE)

def rm_wikilink(obj):
    return bool(re_rm_wikilink.match(six.text_type(obj.title)))

def rm_tag(obj):
    return six.text_type(obj.tag) in {"ref", "table"}

def rm_template(obj):
    return obj.name.lower() in {"reflist", "notelist", "notelist-ua", "notelist-lr", "notelist-ur", "notelist-lg"}

def try_remove_obj(obj, section):
    try:
        section.remove(obj)
    except ValueError:
        # For unknown reasons, objects are sometimes not found.
        pass

section_text = []
for section in wikicode.get_sections(flat=True, include_lead=True, include_headings=True):
    for obj in section.ifilter_wikilinks(matches=rm_wikilink, recursive=True):
        try_remove_obj(obj, section)
    for obj in section.ifilter_templates(matches=rm_template, recursive=True):
        try_remove_obj(obj, section)
    for obj in section.ifilter_tags(matches=rm_tag, recursive=True):
        try_remove_obj(obj, section)

    section_text.append(section.strip_code().strip())
```
2020-07-03T14:23:01Z
https://github.com/huggingface/datasets/issues/321#issuecomment-653571796
https://api.github.com/repos/huggingface/datasets/issues/321
MDEyOklzc3VlQ29tbWVudDY1MzU3MTc5Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/653571796/reactions
2020-07-03T14:23:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/653571796
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Not sure why we're having this issue. Maybe could you get also the file that's causing that ?
2020-07-03T14:28:33Z
https://github.com/huggingface/datasets/issues/321#issuecomment-653574119
https://api.github.com/repos/huggingface/datasets/issues/321
MDEyOklzc3VlQ29tbWVudDY1MzU3NDExOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/653574119/reactions
2020-07-03T14:28:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/653574119
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
thanks for your answer.
How can I know which file is causing the issue ? 
I am trying to load the spanish wikipedia data. 
2020-07-03T16:16:21Z
https://github.com/huggingface/datasets/issues/321#issuecomment-653612279
https://api.github.com/repos/huggingface/datasets/issues/321
MDEyOklzc3VlQ29tbWVudDY1MzYxMjI3OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/653612279/reactions
2020-07-03T16:16:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/653612279
https://avatars.githubusercontent.com/u/26505641?v=4
https://api.github.com/users/Shiro-LK/events{/privacy}
https://api.github.com/users/Shiro-LK/followers
https://api.github.com/users/Shiro-LK/following{/other_user}
https://api.github.com/users/Shiro-LK/gists{/gist_id}
https://github.com/Shiro-LK
Shiro-LK
MDQ6VXNlcjI2NTA1NjQx
https://api.github.com/users/Shiro-LK/orgs
https://api.github.com/users/Shiro-LK/received_events
https://api.github.com/users/Shiro-LK/repos
https://api.github.com/users/Shiro-LK/starred{/owner}{/repo}
https://api.github.com/users/Shiro-LK/subscriptions
User
https://api.github.com/users/Shiro-LK
public
MEMBER
Because of the way Apache Beam works we indeed don't have access to the file name at this point in the code.
We'll have to use some tricks I think :p 

You can append `filepath` to `title` in `wikipedia.py:L512` for example. [[EDIT: it's L494 my bad]]
Then just do `try:...except:` on the  call of `_parse_and_clean_wikicode` L500 I guess.

Thanks for diving into this ! I tried it myself but I run out of memory on my laptop
As soon as we have the name of the file it should be easier to find what's wrong.
2020-07-03T16:27:29Z
https://github.com/huggingface/datasets/issues/321#issuecomment-653615411
https://api.github.com/repos/huggingface/datasets/issues/321
MDEyOklzc3VlQ29tbWVudDY1MzYxNTQxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/653615411/reactions
2020-07-06T16:33:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/653615411
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks for your help.

I tried to print the "title" of the document inside the` except (mwparserfromhell.parser.ParserError) as e`,the title displayed was : "Campeonato Mundial de futsal de la AMF 2015". (Wikipedia ES) Is it what you were looking for ?
2020-07-06T16:30:58Z
https://github.com/huggingface/datasets/issues/321#issuecomment-654340822
https://api.github.com/repos/huggingface/datasets/issues/321
MDEyOklzc3VlQ29tbWVudDY1NDM0MDgyMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/654340822/reactions
2020-07-06T16:30:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/654340822
https://avatars.githubusercontent.com/u/26505641?v=4
https://api.github.com/users/Shiro-LK/events{/privacy}
https://api.github.com/users/Shiro-LK/followers
https://api.github.com/users/Shiro-LK/following{/other_user}
https://api.github.com/users/Shiro-LK/gists{/gist_id}
https://github.com/Shiro-LK
Shiro-LK
MDQ6VXNlcjI2NTA1NjQx
https://api.github.com/users/Shiro-LK/orgs
https://api.github.com/users/Shiro-LK/received_events
https://api.github.com/users/Shiro-LK/repos
https://api.github.com/users/Shiro-LK/starred{/owner}{/repo}
https://api.github.com/users/Shiro-LK/subscriptions
User
https://api.github.com/users/Shiro-LK
public
MEMBER
Thanks a lot @Shiro-LK !

I was able to reproduce the issue. It comes from [this table on wikipedia](https://es.wikipedia.org/wiki/Campeonato_Mundial_de_futsal_de_la_AMF_2015#Clasificados) that can't be parsed.

The file in which the problem occurs comes from the wikipedia dumps, and it can be downloaded [here](https://dumps.wikimedia.org/eswiki/20200501/eswiki-20200501-pages-articles-multistream6.xml-p6424816p7924815.bz2)

Parsing the file this way raises the parsing issue:

```python
import mwparserfromhell as parser
from tqdm.auto import tqdm
import bz2
import six
import logging
import codecs
import xml.etree.cElementTree as etree

filepath = "path/to/eswiki-20200501-pages-articles-multistream6.xml-p6424816p7924815.bz2"

def _extract_content(filepath):
    """Extracts article content from a single WikiMedia XML file."""
    logging.info("generating examples from = %s", filepath)
    with open(filepath, "rb") as f:
        f = bz2.BZ2File(filename=f)
        if six.PY3:
            # Workaround due to:
            # https://github.com/tensorflow/tensorflow/issues/33563
            utf_f = codecs.getreader("utf-8")(f)
        else:
            utf_f = f
        # To clear root, to free-up more memory than just `elem.clear()`.
        context = etree.iterparse(utf_f, events=("end",))
        context = iter(context)
        unused_event, root = next(context)
        for unused_event, elem in tqdm(context, total=949087):
            if not elem.tag.endswith("page"):
                continue
            namespace = elem.tag[:-4]
            title = elem.find("./{0}title".format(namespace)).text
            ns = elem.find("./{0}ns".format(namespace)).text
            id_ = elem.find("./{0}id".format(namespace)).text
            # Filter pages that are not in the "main" namespace.
            if ns != "0":
                root.clear()
                continue
            raw_content = elem.find("./{0}revision/{0}text".format(namespace)).text
            root.clear()

            if "Campeonato Mundial de futsal de la AMF 2015" in title:
                yield (id_, title, raw_content)

for id_, title, raw_content in _extract_content(filepath):
    wikicode = parser.parse(raw_content)
```

The copied the raw content that can't be parsed [here](https://pastebin.com/raw/ZbmevLyH).

The minimal code to reproduce is:
```python
import mwparserfromhell as parser
import requests

raw_content = requests.get("https://pastebin.com/raw/ZbmevLyH").content.decode("utf-8")
wikicode = parser.parse(raw_content)

```

I will create an issue on mwparserfromhell's repo to see if we can fix that

2020-07-06T18:22:55Z
https://github.com/huggingface/datasets/issues/321#issuecomment-654393833
https://api.github.com/repos/huggingface/datasets/issues/321
MDEyOklzc3VlQ29tbWVudDY1NDM5MzgzMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/654393833/reactions
2020-07-06T18:24:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/654393833
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
This going to be fixed in the next `mwparserfromhell` release :)
2020-07-23T16:28:33Z
https://github.com/huggingface/datasets/issues/321#issuecomment-663104781
https://api.github.com/repos/huggingface/datasets/issues/321
MDEyOklzc3VlQ29tbWVudDY2MzEwNDc4MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/663104781/reactions
2020-07-23T16:28:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/663104781
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Fixed in `mwparserfromhell` version 0.6.
2022-02-14T15:21:46Z
https://github.com/huggingface/datasets/issues/321#issuecomment-1039212456
https://api.github.com/repos/huggingface/datasets/issues/321
IC_kwDODunzps498R-o
https://api.github.com/repos/huggingface/datasets/issues/comments/1039212456/reactions
2022-02-14T15:21:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1039212456
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was : 
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`

