wikipedia dataset incomplete
MEMBER
Hi !
From what pickle file fo you get this ?
I guess you mean the dataset loaded using `load_dataset` ?
2021-01-21T15:27:30Z
https://github.com/huggingface/datasets/issues/1759#issuecomment-764721531
https://api.github.com/repos/huggingface/datasets/issues/1759
MDEyOklzc3VlQ29tbWVudDc2NDcyMTUzMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/764721531/reactions
2021-01-21T15:27:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/764721531
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
yes sorry, I used the `load_dataset`function and saved the data to a pickle file so I don't always have to reload it and are able to work offline. 
2021-01-21T17:13:51Z
https://github.com/huggingface/datasets/issues/1759#issuecomment-764802499
https://api.github.com/repos/huggingface/datasets/issues/1759
MDEyOklzc3VlQ29tbWVudDc2NDgwMjQ5OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/764802499/reactions
2021-01-21T17:15:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/764802499
https://avatars.githubusercontent.com/u/19912393?v=4
https://api.github.com/users/ChrisDelClea/events{/privacy}
https://api.github.com/users/ChrisDelClea/followers
https://api.github.com/users/ChrisDelClea/following{/other_user}
https://api.github.com/users/ChrisDelClea/gists{/gist_id}
https://github.com/ChrisDelClea
ChrisDelClea
MDQ6VXNlcjE5OTEyMzkz
https://api.github.com/users/ChrisDelClea/orgs
https://api.github.com/users/ChrisDelClea/received_events
https://api.github.com/users/ChrisDelClea/repos
https://api.github.com/users/ChrisDelClea/starred{/owner}{/repo}
https://api.github.com/users/ChrisDelClea/subscriptions
User
https://api.github.com/users/ChrisDelClea
public
MEMBER
The wikipedia articles are processed using the `mwparserfromhell` library. Even if it works well in most cases, such issues can happen unfortunately. You can find the repo here: https://github.com/earwig/mwparserfromhell

There also exist other datasets based on wikipedia that were processed differently (and are often cleaner) such as `wiki40b`.


2021-01-21T17:19:08Z
https://github.com/huggingface/datasets/issues/1759#issuecomment-764806183
https://api.github.com/repos/huggingface/datasets/issues/1759
MDEyOklzc3VlQ29tbWVudDc2NDgwNjE4Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/764806183/reactions
2021-01-21T17:22:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/764806183
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
ok great. Thank you, @lhoestq. 
2021-01-21T17:21:06Z
https://github.com/huggingface/datasets/issues/1759#issuecomment-764807463
https://api.github.com/repos/huggingface/datasets/issues/1759
MDEyOklzc3VlQ29tbWVudDc2NDgwNzQ2Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/764807463/reactions
2021-01-21T17:21:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/764807463
https://avatars.githubusercontent.com/u/19912393?v=4
https://api.github.com/users/ChrisDelClea/events{/privacy}
https://api.github.com/users/ChrisDelClea/followers
https://api.github.com/users/ChrisDelClea/following{/other_user}
https://api.github.com/users/ChrisDelClea/gists{/gist_id}
https://github.com/ChrisDelClea
ChrisDelClea
MDQ6VXNlcjE5OTEyMzkz
https://api.github.com/users/ChrisDelClea/orgs
https://api.github.com/users/ChrisDelClea/received_events
https://api.github.com/users/ChrisDelClea/repos
https://api.github.com/users/ChrisDelClea/starred{/owner}{/repo}
https://api.github.com/users/ChrisDelClea/subscriptions
User
https://api.github.com/users/ChrisDelClea
public
Hey guys,

I am using the https://github.com/huggingface/datasets/tree/master/datasets/wikipedia dataset.
Unfortunately, I found out that there is an incompleteness for the German dataset.
 For reasons unknown to me, the number of inhabitants has been removed from many pages:
Thorey-sur-Ouche has 128 inhabitants according to the webpage (https://de.wikipedia.org/wiki/Thorey-sur-Ouche).
The pickle file however shows: französische Gemeinde mit  Einwohnern (Stand).
 Is it possible to fix this?

Best regards 
Chris
