Support streaming in size estimation function in `push_to_hub`
COLLABORATOR
would this allow to include the size in the dataset info without downloading the files? related to https://github.com/huggingface/datasets/pull/3670
2022-02-16T14:11:23Z
https://github.com/huggingface/datasets/pull/3732#issuecomment-1041531203
https://api.github.com/repos/huggingface/datasets/issues/3732
IC_kwDODunzps4-FIFD
https://api.github.com/repos/huggingface/datasets/issues/comments/1041531203/reactions
2022-02-16T14:11:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1041531203
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
COLLABORATOR
@severo I don't think so. We could use this to get `info.download_checksums["num_bytes"]`, but we must process the files to get the rest of the size info. 
2022-02-16T17:27:05Z
https://github.com/huggingface/datasets/pull/3732#issuecomment-1041908938
https://api.github.com/repos/huggingface/datasets/issues/3732
IC_kwDODunzps4-GkTK
https://api.github.com/repos/huggingface/datasets/issues/comments/1041908938/reactions
2022-02-16T17:27:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1041908938
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
This PR adds the streamable version of `os.path.getsize` (`fsspec` can return `None`, so we fall back to `fs.open` to make it more robust) to account for possible streamable paths in the nested `extra_nbytes_visitor` function inside `push_to_hub`.