Fix ArrowWriter closes stream at exit
MEMBER
Oh nice thanks for adding the context manager ! All the streams and RecordBatchWriter will be properly closed now. Hopefully this gives a better experience on windows on which it's super important to close stuff.

Not sure about the error, it looks like a process crashed silently.
Let me take a look
2021-03-02T14:38:23Z
https://github.com/huggingface/datasets/pull/1971#issuecomment-788954800
https://api.github.com/repos/huggingface/datasets/issues/1971
MDEyOklzc3VlQ29tbWVudDc4ODk1NDgwMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/788954800/reactions
2021-03-02T14:38:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/788954800
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> Hopefully this gives a better experience on windows on which it's super important to close stuff.

Exactly! On Windows, you got:
> PermissionError: [WinError 32] The process cannot access the file because it is being used by another process

when trying to access the unclosed `stream` file, e.g. by `with incomplete_dir(self._cache_dir) as tmp_data_dir`: `shutil.rmtree(tmp_dir)`

The reason is: https://docs.python.org/3/library/os.html#os.remove

> On Windows, attempting to remove a file that is in use causes an exception to be raised; on Unix, the directory entry is removed but the storage allocated to the file is not made available until the original file is no longer in use.



2021-03-02T14:54:32Z
https://github.com/huggingface/datasets/pull/1971#issuecomment-788965939
https://api.github.com/repos/huggingface/datasets/issues/1971
MDEyOklzc3VlQ29tbWVudDc4ODk2NTkzOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/788965939/reactions
2021-03-02T14:55:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/788965939
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
The test passes on my windows. This was probably a circleCI issue. I re-ran the circleCI tests
2021-03-02T14:55:40Z
https://github.com/huggingface/datasets/pull/1971#issuecomment-788966713
https://api.github.com/repos/huggingface/datasets/issues/1971
MDEyOklzc3VlQ29tbWVudDc4ODk2NjcxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/788966713/reactions
2021-03-02T14:55:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/788966713
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
NICE! It passed!
2021-03-02T14:56:29Z
https://github.com/huggingface/datasets/pull/1971#issuecomment-788967340
https://api.github.com/repos/huggingface/datasets/issues/1971
MDEyOklzc3VlQ29tbWVudDc4ODk2NzM0MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/788967340/reactions
2021-03-02T14:56:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/788967340
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Maybe you can merge master into this branch and check the CI before merging ?
2021-03-10T10:31:41Z
https://github.com/huggingface/datasets/pull/1971#issuecomment-795216684
https://api.github.com/repos/huggingface/datasets/issues/1971
MDEyOklzc3VlQ29tbWVudDc5NTIxNjY4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/795216684/reactions
2021-03-10T10:31:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/795216684
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq done! ;)
2021-03-10T16:04:26Z
https://github.com/huggingface/datasets/pull/1971#issuecomment-795658681
https://api.github.com/repos/huggingface/datasets/issues/1971
MDEyOklzc3VlQ29tbWVudDc5NTY1ODY4MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/795658681/reactions
2021-03-10T16:04:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/795658681
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Thanks ! merging
2021-03-10T16:36:50Z
https://github.com/huggingface/datasets/pull/1971#issuecomment-795704076
https://api.github.com/repos/huggingface/datasets/issues/1971
MDEyOklzc3VlQ29tbWVudDc5NTcwNDA3Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/795704076/reactions
2021-03-10T16:36:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/795704076
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Current implementation of ArrowWriter does not properly release the `stream` resource (by closing it) if its `finalize()` method is not called and/or an Exception is raised before/during the call to its `finalize()` method.

Therefore, ArrowWriter should be used as a context manager that properly closes its `stream` resource at exit.