Fix streaming for servers not supporting HTTP range requests
COLLABORATOR
Does it mean that huge files might end up being downloaded? It would go against the purpose of streaming, I think. At least, this fallback should be an option that could be disabled
2022-02-08T17:12:35Z
https://github.com/huggingface/datasets/pull/3689#issuecomment-1032858345
https://api.github.com/repos/huggingface/datasets/issues/3689
IC_kwDODunzps49kCrp
https://api.github.com/repos/huggingface/datasets/issues/comments/1032858345/reactions
2022-02-08T17:12:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1032858345
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
Yes, it is against the purpose of streaming, but streaming is not possible if the server does not allow HTTP range requests.

We have two options: either we download the file or we throw an error.
2022-02-08T17:19:13Z
https://github.com/huggingface/datasets/pull/3689#issuecomment-1032864812
https://api.github.com/repos/huggingface/datasets/issues/3689
IC_kwDODunzps49kEQs
https://api.github.com/repos/huggingface/datasets/issues/comments/1032864812/reactions
2022-02-08T17:19:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1032864812
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
COLLABORATOR
I think we simply cannot fallback to downloading the file if streaming fails without the user being aware of it. Some options: 
- make the fallback optional (using an env var? or a function param)
- use the fallback only if the dataset size is under some threshold (provided we have the data in the DatasetInfo) -> it's the option I use in `datasets-preview-backend` ([here](https://github.com/huggingface/datasets-preview-backend/blob/48ac19e49c19809763e8d640986bf2c3d792faed/src/datasets_preview_backend/models/typed_row.py#L40) and [here](https://github.com/huggingface/datasets-preview-backend/blob/aa86c5493b275c9e2dbae7dab7bd469da5773a41/src/datasets_preview_backend/models/split.py#L31-L37))
- throw an exception and let the user decide what to do

2022-02-08T18:12:33Z
https://github.com/huggingface/datasets/pull/3689#issuecomment-1032914992
https://api.github.com/repos/huggingface/datasets/issues/3689
IC_kwDODunzps49kQgw
https://api.github.com/repos/huggingface/datasets/issues/comments/1032914992/reactions
2022-02-08T18:12:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1032914992
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
IMO in general we should throw an exception and ask the user to not use streaming mode in that case.

Your second point is also interesting but I feel like it could be confusing for users sometimes: it doesn't feel natural that the streaming-ability should depend on the size of the file.
2022-02-08T19:43:03Z
https://github.com/huggingface/datasets/pull/3689#issuecomment-1032994318
https://api.github.com/repos/huggingface/datasets/issues/3689
IC_kwDODunzps49kj4O
https://api.github.com/repos/huggingface/datasets/issues/comments/1032994318/reactions
2022-02-08T19:43:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1032994318
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Sure, I think we should just throw an exception

2022-02-08T20:25:39Z
https://github.com/huggingface/datasets/pull/3689#issuecomment-1033029876
https://api.github.com/repos/huggingface/datasets/issues/3689
IC_kwDODunzps49ksj0
https://api.github.com/repos/huggingface/datasets/issues/comments/1033029876/reactions
2022-02-08T20:25:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1033029876
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
Current behavior is already throwing an Exception:
```
ValueError: Cannot seek streaming HTTP file
```

We could customize the exception class and/or the exception message.
2022-02-09T06:45:12Z
https://github.com/huggingface/datasets/pull/3689#issuecomment-1033405133
https://api.github.com/repos/huggingface/datasets/issues/3689
IC_kwDODunzps49mILN
https://api.github.com/repos/huggingface/datasets/issues/comments/1033405133/reactions
2022-02-09T06:45:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1033405133
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
COLLABORATOR
I'm not sure we really need to change anything. I opened the issue https://github.com/huggingface/datasets/issues/3677  because discovery was streamable and is not anymore (according to my test suite in https://github.com/huggingface/datasets-preview-backend): I was not sure if it was due to some regression in the library, or to some change in the dataset itself.
2022-02-09T07:19:35Z
https://github.com/huggingface/datasets/pull/3689#issuecomment-1033425165
https://api.github.com/repos/huggingface/datasets/issues/3689
IC_kwDODunzps49mNEN
https://api.github.com/repos/huggingface/datasets/issues/comments/1033425165/reactions
2022-02-09T07:19:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1033425165
https://avatars.githubusercontent.com/u/1676121?v=4
https://api.github.com/users/severo/events{/privacy}
https://api.github.com/users/severo/followers
https://api.github.com/users/severo/following{/other_user}
https://api.github.com/users/severo/gists{/gist_id}
https://github.com/severo
severo
MDQ6VXNlcjE2NzYxMjE=
https://api.github.com/users/severo/orgs
https://api.github.com/users/severo/received_events
https://api.github.com/users/severo/repos
https://api.github.com/users/severo/starred{/owner}{/repo}
https://api.github.com/users/severo/subscriptions
User
https://api.github.com/users/severo
public
MEMBER
I'm wondering why it worked before and it is no longer working...
2022-02-09T07:45:07Z
https://github.com/huggingface/datasets/pull/3689#issuecomment-1033440625
https://api.github.com/repos/huggingface/datasets/issues/3689
IC_kwDODunzps49mQ1x
https://api.github.com/repos/huggingface/datasets/issues/comments/1033440625/reactions
2022-02-09T07:45:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1033440625
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
> We could customize the exception class and/or the exception message.

Yup a message that says that the host doesn't support streaming because it doesn't support HTTP Range requests would be useful !
2022-02-09T23:08:44Z
https://github.com/huggingface/datasets/pull/3689#issuecomment-1034293857
https://api.github.com/repos/huggingface/datasets/issues/3689
IC_kwDODunzps49phJh
https://api.github.com/repos/huggingface/datasets/issues/comments/1034293857/reactions
2022-02-09T23:08:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1034293857
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
DONE, @lhoestq. 
2022-02-10T08:16:51Z
https://github.com/huggingface/datasets/pull/3689#issuecomment-1034614841
https://api.github.com/repos/huggingface/datasets/issues/3689
IC_kwDODunzps49qvg5
https://api.github.com/repos/huggingface/datasets/issues/comments/1034614841/reactions
2022-02-10T08:16:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1034614841
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Some servers do not support HTTP range requests, whereas this is required to stream some file formats (like ZIP).

~~This PR implements a workaround for those cases, by download the files locally in a temporary directory (cleaned up by the OS once the process is finished).~~

This PR raises custom error explaining that streaming is not possible because data host server does not support HTTP range requests.

Fix #3677.