[Feature Request] Add optional parameter in text loading script to preserve linebreaks
MEMBER
Hi ! Thanks for your message.
Indeed it's a free feature we can add and that can be useful.
If you want to contribute, feel free to open a PR to add it to the text dataset script :)
2020-11-20T17:41:59Z
https://github.com/huggingface/datasets/issues/870#issuecomment-731307589
https://api.github.com/repos/huggingface/datasets/issues/870
MDEyOklzc3VlQ29tbWVudDczMTMwNzU4OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/731307589/reactions
2020-11-20T17:41:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/731307589
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Resolved via #1913.
2022-06-01T15:25:52Z
https://github.com/huggingface/datasets/issues/870#issuecomment-1143754302
https://api.github.com/repos/huggingface/datasets/issues/870
IC_kwDODunzps5ELE4-
https://api.github.com/repos/huggingface/datasets/issues/comments/1143754302/reactions
2022-06-01T15:25:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1143754302
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
I'm working on a project about rhyming verse using phonetic poetry and song lyrics, and line breaks are a vital part of the data. 

I recently switched over to use the datasets library when my various corpora grew larger than my computer's memory. And so far, it is SO great. 

But the first time I processed all of my data into a dataset, I hadn't realized the text loader script was processing the source files line-by-line and stripping off the newlines. 

Once I caught the issue, I made my own data loader by modifying one line in the default text loader (changing `batch = batch.splitlines()` to `batch = batch.splitlines(True)` inside `_generate_tables`). And so I'm all set as far as my project is concerned.

But if my use case is more general, it seems like it'd be pretty trivial to add a kwarg to the default text loader called keeplinebreaks or something, which would default to False and get passed to `splitlines()`. 