Use gc.collect only when needed to avoid slow downs
MEMBER
I continue thinking that the origin of the issue has to do with tqdm (and not with Arrow): this issue only arises for version 4.50.0 (and later) of tqdm, not for previous versions of tqdm.

My guess is that tqdm made a change from version 4.50.0 that does not properly release the iterable. 
2021-06-11T19:31:07Z
https://github.com/huggingface/datasets/pull/2483#issuecomment-859795432
https://api.github.com/repos/huggingface/datasets/issues/2483
MDEyOklzc3VlQ29tbWVudDg1OTc5NTQzMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/859795432/reactions
2021-06-11T19:31:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/859795432
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
FR
2021-06-18T19:25:06Z
https://github.com/huggingface/datasets/pull/2483#issuecomment-864232451
https://api.github.com/repos/huggingface/datasets/issues/2483
MDEyOklzc3VlQ29tbWVudDg2NDIzMjQ1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/864232451/reactions
2021-06-18T19:25:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/864232451
https://avatars.githubusercontent.com/u/71971234?v=4
https://api.github.com/users/aDrIaNo34500/events{/privacy}
https://api.github.com/users/aDrIaNo34500/followers
https://api.github.com/users/aDrIaNo34500/following{/other_user}
https://api.github.com/users/aDrIaNo34500/gists{/gist_id}
https://github.com/aDrIaNo34500
aDrIaNo34500
MDQ6VXNlcjcxOTcxMjM0
https://api.github.com/users/aDrIaNo34500/orgs
https://api.github.com/users/aDrIaNo34500/received_events
https://api.github.com/users/aDrIaNo34500/repos
https://api.github.com/users/aDrIaNo34500/starred{/owner}{/repo}
https://api.github.com/users/aDrIaNo34500/subscriptions
User
https://api.github.com/users/aDrIaNo34500
public
In https://github.com/huggingface/datasets/commit/42320a110d9d072703814e1f630a0d90d626a1e6 we added a call to gc.collect to resolve some issues on windows (see https://github.com/huggingface/datasets/pull/2482)

However calling gc.collect too often causes significant slow downs (the CI run time doubled).
So I just moved the gc.collect call to the exact place where it's actually needed: when post-processing a dataset