Don't use take on dataset table in pyarrow 1.0.x
MEMBER
I tried lower batch sizes and it didn't accelerate filter (quite the opposite actually).
The slow-down also appears for pyarrow 0.17.1 for some reason, not sure it comes from these changes
2020-09-19T16:23:25Z
https://github.com/huggingface/datasets/pull/645#issuecomment-695328022
https://api.github.com/repos/huggingface/datasets/issues/645
MDEyOklzc3VlQ29tbWVudDY5NTMyODAyMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/695328022/reactions
2020-09-19T16:23:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/695328022
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I just checked the benchmarks of other PRs and some of them had 300s (!!) for filter. This needs some investigation..
2020-09-19T16:32:46Z
https://github.com/huggingface/datasets/pull/645#issuecomment-695329044
https://api.github.com/repos/huggingface/datasets/issues/645
MDEyOklzc3VlQ29tbWVudDY5NTMyOTA0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/695329044/reactions
2020-09-19T16:45:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/695329044
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Merging this one since it's not the cause of the the slow down
2020-09-19T16:46:26Z
https://github.com/huggingface/datasets/pull/645#issuecomment-695330549
https://api.github.com/repos/huggingface/datasets/issues/645
MDEyOklzc3VlQ29tbWVudDY5NTMzMDU0OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/695330549/reactions
2020-09-19T16:46:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/695330549
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq What might be the reason for the slowdown? When I was training large batchsize, the slowdown was obvious, and my original assumption was to increase write_batch_size
2023-09-19T07:59:19Z
https://github.com/huggingface/datasets/pull/645#issuecomment-1725013152
https://api.github.com/repos/huggingface/datasets/issues/645
IC_kwDODunzps5m0Zyg
https://api.github.com/repos/huggingface/datasets/issues/comments/1725013152/reactions
2023-09-19T07:59:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1725013152
https://avatars.githubusercontent.com/u/51043929?v=4
https://api.github.com/users/aihao2000/events{/privacy}
https://api.github.com/users/aihao2000/followers
https://api.github.com/users/aihao2000/following{/other_user}
https://api.github.com/users/aihao2000/gists{/gist_id}
https://github.com/aihao2000
aihao2000
MDQ6VXNlcjUxMDQzOTI5
https://api.github.com/users/aihao2000/orgs
https://api.github.com/users/aihao2000/received_events
https://api.github.com/users/aihao2000/repos
https://api.github.com/users/aihao2000/starred{/owner}{/repo}
https://api.github.com/users/aihao2000/subscriptions
User
https://api.github.com/users/aihao2000
public
Fix #615 