Fix memory issue in multiprocessing: Don't pickle table index
MEMBER
The code quality check is going to be fixed by #2265 
2021-04-26T09:36:26Z
https://github.com/huggingface/datasets/pull/2264#issuecomment-826678590
https://api.github.com/repos/huggingface/datasets/issues/2264
MDEyOklzc3VlQ29tbWVudDgyNjY3ODU5MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/826678590/reactions
2021-04-26T09:36:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826678590
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
The memory issue didn't come from `self.__dict__.copy()` but from the fact that this dict contains `_batches` which has all the batches of the table in it.
Therefore for a MemoryMappedTable all the data in `_batches` were copied in memory when pickling and this is the issue.
2021-04-26T09:56:18Z
https://github.com/huggingface/datasets/pull/2264#issuecomment-826694278
https://api.github.com/repos/huggingface/datasets/issues/2264
MDEyOklzc3VlQ29tbWVudDgyNjY5NDI3OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/826694278/reactions
2021-04-26T09:56:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826694278
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I'm still investigating why we didn't catch this issue in the tests.
This test should have caught it but didn't:

https://github.com/huggingface/datasets/blob/3db67f5ff6cbf807b129d2b4d1107af27623b608/tests/test_table.py#L350-L353
2021-04-26T09:58:18Z
https://github.com/huggingface/datasets/pull/2264#issuecomment-826695735
https://api.github.com/repos/huggingface/datasets/issues/2264
MDEyOklzc3VlQ29tbWVudDgyNjY5NTczNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/826695735/reactions
2021-04-26T09:58:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826695735
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I'll focus on the patch release and fix the test in another PR after the release
2021-04-26T10:07:19Z
https://github.com/huggingface/datasets/pull/2264#issuecomment-826702770
https://api.github.com/repos/huggingface/datasets/issues/2264
MDEyOklzc3VlQ29tbWVudDgyNjcwMjc3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/826702770/reactions
2021-04-26T10:07:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826702770
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Yes, I think it is better that way...
2021-04-26T10:30:28Z
https://github.com/huggingface/datasets/pull/2264#issuecomment-826719431
https://api.github.com/repos/huggingface/datasets/issues/2264
MDEyOklzc3VlQ29tbWVudDgyNjcxOTQzMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/826719431/reactions
2021-04-26T10:30:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826719431
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
The table index is currently being pickled when doing multiprocessing, which brings all the record batches of the dataset in memory.

I fixed that by not pickling the index attributes. Therefore each process has to rebuild the index when unpickling the table.

Fix issue #2256

We'll do a patch release asap !