Support remote data files
MEMBER
@lhoestq maybe we could also use (if available) the ETag of the remote file in `create_config_id`?
2021-07-09T14:09:26Z
https://github.com/huggingface/datasets/pull/2616#issuecomment-877217238
https://api.github.com/repos/huggingface/datasets/issues/2616
MDEyOklzc3VlQ29tbWVudDg3NzIxNzIzOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/877217238/reactions
2021-07-09T14:09:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/877217238
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
> @lhoestq maybe we could also use (if available) the ETag of the remote file in `create_config_id`?

Sure ! We can get the ETag with
```python
headers = get_authentication_headers_for_url(url, use_auth_token=use_auth_token)  # auth for private repos
etag = http_head(url, headers=headers).headers.get("ETag")
```

Since the computation of the `config_id` is done in the `DatasetBuilder.__init__`, then this means that we need to add a new parameter `use_auth_token` in `DatasetBuilder.__init__`

Does that sound good ? We can add this in a following PR
2021-07-09T16:11:47Z
https://github.com/huggingface/datasets/pull/2616#issuecomment-877297016
https://api.github.com/repos/huggingface/datasets/issues/2616
MDEyOklzc3VlQ29tbWVudDg3NzI5NzAxNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/877297016/reactions
2021-07-09T16:12:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/877297016
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Add support for (streaming) remote data files:

```python
data_files = f"https://huggingface.co/datasets/{repo_id}/resolve/main/{relative_file_path}"
ds = load_dataset("json", split="train", data_files=data_files, streaming=True)
```

cc: @thomwolf 