Add keep_linebreaks parameter to text loader
CONTRIBUTOR
Just so I understand how it can be used in practice, do you have an example showing how to load a text dataset with this option?
2021-02-19T16:47:07Z
https://github.com/huggingface/datasets/pull/1913#issuecomment-782196183
https://api.github.com/repos/huggingface/datasets/issues/1913
MDEyOklzc3VlQ29tbWVudDc4MjE5NjE4Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/782196183/reactions
2021-02-19T16:47:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/782196183
https://avatars.githubusercontent.com/u/35901082?v=4
https://api.github.com/users/sgugger/events{/privacy}
https://api.github.com/users/sgugger/followers
https://api.github.com/users/sgugger/following{/other_user}
https://api.github.com/users/sgugger/gists{/gist_id}
https://github.com/sgugger
sgugger
MDQ6VXNlcjM1OTAxMDgy
https://api.github.com/users/sgugger/orgs
https://api.github.com/users/sgugger/received_events
https://api.github.com/users/sgugger/repos
https://api.github.com/users/sgugger/starred{/owner}{/repo}
https://api.github.com/users/sgugger/subscriptions
User
https://api.github.com/users/sgugger
public
MEMBER
Sure ! Here is an example:
```python
from datasets import load_dataset

load_dataset("text", keep_linebreaks=True, data_files=...)
```

I'll update the documentation to explain this
2021-02-19T17:45:53Z
https://github.com/huggingface/datasets/pull/1913#issuecomment-782231155
https://api.github.com/repos/huggingface/datasets/issues/1913
MDEyOklzc3VlQ29tbWVudDc4MjIzMTE1NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/782231155/reactions
2021-02-19T17:45:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/782231155
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Perfect!
2021-02-19T18:33:37Z
https://github.com/huggingface/datasets/pull/1913#issuecomment-782260871
https://api.github.com/repos/huggingface/datasets/issues/1913
MDEyOklzc3VlQ29tbWVudDc4MjI2MDg3MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/782260871/reactions
2021-02-19T18:33:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/782260871
https://avatars.githubusercontent.com/u/35901082?v=4
https://api.github.com/users/sgugger/events{/privacy}
https://api.github.com/users/sgugger/followers
https://api.github.com/users/sgugger/following{/other_user}
https://api.github.com/users/sgugger/gists{/gist_id}
https://github.com/sgugger
sgugger
MDQ6VXNlcjM1OTAxMDgy
https://api.github.com/users/sgugger/orgs
https://api.github.com/users/sgugger/received_events
https://api.github.com/users/sgugger/repos
https://api.github.com/users/sgugger/starred{/owner}{/repo}
https://api.github.com/users/sgugger/subscriptions
User
https://api.github.com/users/sgugger
public
As asked in #870 and https://github.com/huggingface/transformers/issues/10269 there should be a parameter to keep the linebreaks when loading a text dataset.
cc @sgugger @jncasey