Fix text loader to split only on universal newlines
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3910). All of your documentation changes will be reflected on that endpoint.
2022-03-14T16:01:46Z
https://github.com/huggingface/datasets/pull/3910#issuecomment-1066996384
https://api.github.com/repos/huggingface/datasets/issues/3910
IC_kwDODunzps4_mRKg
https://api.github.com/repos/huggingface/datasets/issues/comments/1066996384/reactions
2022-03-14T16:01:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1066996384
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
Looks like the test needs to be updated for windows ^^'
2022-03-14T16:54:16Z
https://github.com/huggingface/datasets/pull/3910#issuecomment-1067058649
https://api.github.com/repos/huggingface/datasets/issues/3910
IC_kwDODunzps4_mgXZ
https://api.github.com/repos/huggingface/datasets/issues/comments/1067058649/reactions
2022-03-14T16:54:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1067058649
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I don't think this is the same issue as in https://github.com/oscar-corpus/corpus/issues/18, where the OSCAR metadata has line offsets that use only `\n` as the newline marker to count lines, not `\r\n` or `\r`.

It looks like the OSCAR data loader is opening the data files with `gzip.open` directly and I don't think this text loader is used, but I'm not familiar with a lot of `datasets` internals so I could be mistaken?
2022-03-15T08:30:44Z
https://github.com/huggingface/datasets/pull/3910#issuecomment-1067700393
https://api.github.com/repos/huggingface/datasets/issues/3910
IC_kwDODunzps4_o9Cp
https://api.github.com/repos/huggingface/datasets/issues/comments/1067700393/reactions
2022-03-15T08:30:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1067700393
https://avatars.githubusercontent.com/u/5794899?v=4
https://api.github.com/users/adrianeboyd/events{/privacy}
https://api.github.com/users/adrianeboyd/followers
https://api.github.com/users/adrianeboyd/following{/other_user}
https://api.github.com/users/adrianeboyd/gists{/gist_id}
https://github.com/adrianeboyd
adrianeboyd
MDQ6VXNlcjU3OTQ4OTk=
https://api.github.com/users/adrianeboyd/orgs
https://api.github.com/users/adrianeboyd/received_events
https://api.github.com/users/adrianeboyd/repos
https://api.github.com/users/adrianeboyd/starred{/owner}{/repo}
https://api.github.com/users/adrianeboyd/subscriptions
User
https://api.github.com/users/adrianeboyd
public
MEMBER
You are right @adrianeboyd.

This PR fixes #3729.

Additionally, this PR is somehow related to the OSCAR issue. However, the OSCAR issue have multiple root causes: one is the offset initialization (as you pointed out); other is similar to this case: Unicode newlines are not properly handled.

I will make a change proposal for OSCAR this afternoon.
2022-03-15T14:40:38Z
https://github.com/huggingface/datasets/pull/3910#issuecomment-1068065316
https://api.github.com/repos/huggingface/datasets/issues/3910
IC_kwDODunzps4_qWIk
https://api.github.com/repos/huggingface/datasets/issues/comments/1068065316/reactions
2022-03-15T14:40:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1068065316
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
@lhoestq I'm working on fixing the Windows tests on my Windows machine...
2022-03-15T14:41:31Z
https://github.com/huggingface/datasets/pull/3910#issuecomment-1068066226
https://api.github.com/repos/huggingface/datasets/issues/3910
IC_kwDODunzps4_qWWy
https://api.github.com/repos/huggingface/datasets/issues/comments/1068066226/reactions
2022-03-15T14:41:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1068066226
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I finally changed the approach in order to avoid having "\r\n" and "\r" line breaks in Python `str` read from files on Windows/old Macintosh machines.
2022-03-15T16:05:47Z
https://github.com/huggingface/datasets/pull/3910#issuecomment-1068166234
https://api.github.com/repos/huggingface/datasets/issues/3910
IC_kwDODunzps4_quxa
https://api.github.com/repos/huggingface/datasets/issues/comments/1068166234/reactions
2022-03-15T16:05:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1068166234
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Currently, `text` loader breaks on a superset of universal newlines, which also contains Unicode line boundaries. See: https://docs.python.org/3/library/stdtypes.html#str.splitlines

However, the expected behavior is to get the lines splitted only on universal newlines: "\n", "\r\n" and "\r".

See: oscar-corpus/corpus#18

Fix #3729.