wikipedia.py generator that extracts XML doesn't release memory
MEMBER
Hi @miyamonz 
Thanks for investigating this issue, good job !
It would be awesome to integrate your fix in the library, could you open a pull request ?
2021-03-11T14:54:49Z
https://github.com/huggingface/datasets/issues/2031#issuecomment-796795057
https://api.github.com/repos/huggingface/datasets/issues/2031
MDEyOklzc3VlQ29tbWVudDc5Njc5NTA1Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/796795057/reactions
2021-03-11T14:54:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/796795057
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
OK! I'll send it later.
2021-03-12T08:21:26Z
https://github.com/huggingface/datasets/issues/2031#issuecomment-797319949
https://api.github.com/repos/huggingface/datasets/issues/2031
MDEyOklzc3VlQ29tbWVudDc5NzMxOTk0OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/797319949/reactions
2021-03-12T08:21:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797319949
https://avatars.githubusercontent.com/u/6331508?v=4
https://api.github.com/users/miyamonz/events{/privacy}
https://api.github.com/users/miyamonz/followers
https://api.github.com/users/miyamonz/following{/other_user}
https://api.github.com/users/miyamonz/gists{/gist_id}
https://github.com/miyamonz
miyamonz
MDQ6VXNlcjYzMzE1MDg=
https://api.github.com/users/miyamonz/orgs
https://api.github.com/users/miyamonz/received_events
https://api.github.com/users/miyamonz/repos
https://api.github.com/users/miyamonz/starred{/owner}{/repo}
https://api.github.com/users/miyamonz/subscriptions
User
https://api.github.com/users/miyamonz
public
I tried downloading Japanese wikipedia, but it always failed because of out of memory maybe.

I found that the generator function that extracts XML data in wikipedia.py doesn't release memory in the loop.

https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L464-L502

`root.clear()` intend to clear memory, but it doesn't.
https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L490
https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L494
I replaced them with `elem.clear()`, then it seems to work correctly.

here is the notebook to reproduce it.
https://gist.github.com/miyamonz/dc06117302b6e85fa51cbf46dde6bb51#file-xtract_content-ipynb