Improve Wikipedia loading script
CONTRIBUTOR
Thanks! See https://public.paws.wmcloud.org/User:Isaac_(WMF)/HuggingFace%20Wikipedia%20Processing.ipynb for more implementation details / some data around the overhead induced by adding the extra preprocessing steps (stripping link prefixes and magic words)
2021-12-07T20:25:51Z
https://github.com/huggingface/datasets/issues/3400#issuecomment-988240507
https://api.github.com/repos/huggingface/datasets/issues/3400
IC_kwDODunzps4651p7
https://api.github.com/repos/huggingface/datasets/issues/comments/988240507/reactions
2021-12-07T20:25:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/988240507
https://avatars.githubusercontent.com/u/45494522?v=4
https://api.github.com/users/geohci/events{/privacy}
https://api.github.com/users/geohci/followers
https://api.github.com/users/geohci/following{/other_user}
https://api.github.com/users/geohci/gists{/gist_id}
https://github.com/geohci
geohci
MDQ6VXNlcjQ1NDk0NTIy
https://api.github.com/users/geohci/orgs
https://api.github.com/users/geohci/received_events
https://api.github.com/users/geohci/repos
https://api.github.com/users/geohci/starred{/owner}{/repo}
https://api.github.com/users/geohci/subscriptions
User
https://api.github.com/users/geohci
public
MEMBER
Closed by:
- #3435
2022-03-22T16:52:28Z
https://github.com/huggingface/datasets/issues/3400#issuecomment-1075387696
https://api.github.com/repos/huggingface/datasets/issues/3400
IC_kwDODunzps5AGR0w
https://api.github.com/repos/huggingface/datasets/issues/comments/1075387696/reactions
2022-03-22T16:52:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1075387696
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
As reported by @geohci, the "wikipedia" processing/loading script could be improved by some additional small suggested processing functions:
- _extract_content(filepath):
  - Replace .startswith("#redirect") with more structured approach: if elem.find(f"./{namespace}redirect") is None: continue
- _parse_and_clean_wikicode(raw_content, parser):
  - Remove rm_template from cleaning -- this is redundant with .strip_code() from mwparserformhell
  - Build a language-specific list of namespace prefixes to filter out per below get_namespace_prefixes
  - Optional: strip prefixes like categories -- e.g., Category:Towns in Tianjin becomes Towns in Tianjin
  - Optional: strip magic words



