Text builder with custom separator line boundaries
NONE
Gently pinging @lhoestq
2022-03-02T14:51:43Z
https://github.com/huggingface/datasets/issues/3804#issuecomment-1057015404
https://api.github.com/repos/huggingface/datasets/issues/3804
IC_kwDODunzps4_AMZs
https://api.github.com/repos/huggingface/datasets/issues/comments/1057015404/reactions
2022-03-02T14:51:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1057015404
https://avatars.githubusercontent.com/u/18630848?v=4
https://api.github.com/users/cronoik/events{/privacy}
https://api.github.com/users/cronoik/followers
https://api.github.com/users/cronoik/following{/other_user}
https://api.github.com/users/cronoik/gists{/gist_id}
https://github.com/cronoik
cronoik
MDQ6VXNlcjE4NjMwODQ4
https://api.github.com/users/cronoik/orgs
https://api.github.com/users/cronoik/received_events
https://api.github.com/users/cronoik/repos
https://api.github.com/users/cronoik/starred{/owner}{/repo}
https://api.github.com/users/cronoik/subscriptions
User
https://api.github.com/users/cronoik
public
MEMBER
Hi ! Interresting :)

Could you give more details on what kind of separators you would like to use instead ?
2022-03-02T14:56:01Z
https://github.com/huggingface/datasets/issues/3804#issuecomment-1057021836
https://api.github.com/repos/huggingface/datasets/issues/3804
IC_kwDODunzps4_AN-M
https://api.github.com/repos/huggingface/datasets/issues/comments/1057021836/reactions
2022-03-02T14:56:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1057021836
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
In my case, I just want to use `\n` but not `U+2028`.
2022-03-02T15:07:29Z
https://github.com/huggingface/datasets/issues/3804#issuecomment-1057036803
https://api.github.com/repos/huggingface/datasets/issues/3804
IC_kwDODunzps4_ARoD
https://api.github.com/repos/huggingface/datasets/issues/comments/1057036803/reactions
2022-03-02T15:07:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1057036803
https://avatars.githubusercontent.com/u/18630848?v=4
https://api.github.com/users/cronoik/events{/privacy}
https://api.github.com/users/cronoik/followers
https://api.github.com/users/cronoik/following{/other_user}
https://api.github.com/users/cronoik/gists{/gist_id}
https://github.com/cronoik
cronoik
MDQ6VXNlcjE4NjMwODQ4
https://api.github.com/users/cronoik/orgs
https://api.github.com/users/cronoik/received_events
https://api.github.com/users/cronoik/repos
https://api.github.com/users/cronoik/starred{/owner}{/repo}
https://api.github.com/users/cronoik/subscriptions
User
https://api.github.com/users/cronoik
public
MEMBER
Ok I see, maybe there can be a `sep` parameter to allow users to specify what line/paragraph separator they'd like to use
2022-03-02T16:16:57Z
https://github.com/huggingface/datasets/issues/3804#issuecomment-1057108432
https://api.github.com/repos/huggingface/datasets/issues/3804
IC_kwDODunzps4_AjHQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1057108432/reactions
2022-03-02T16:16:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1057108432
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Related to:
- #3729 
- #3910
2022-03-03T14:38:40Z
https://github.com/huggingface/datasets/issues/3804#issuecomment-1058107386
https://api.github.com/repos/huggingface/datasets/issues/3804
IC_kwDODunzps4_EW_6
https://api.github.com/repos/huggingface/datasets/issues/comments/1058107386/reactions
2022-03-16T15:50:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1058107386
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Thanks for requesting this enhancement. We have recently found a somehow related issue with another dataset:
- #3704

Let me make a PR proposal.
2022-03-16T15:53:58Z
https://github.com/huggingface/datasets/issues/3804#issuecomment-1069287512
https://api.github.com/repos/huggingface/datasets/issues/3804
IC_kwDODunzps4_vAhY
https://api.github.com/repos/huggingface/datasets/issues/comments/1069287512/reactions
2022-03-16T15:53:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1069287512
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
**Is your feature request related to a problem? Please describe.**
The current [Text](https://github.com/huggingface/datasets/blob/207be676bffe9d164740a41a883af6125edef135/src/datasets/packaged_modules/text/text.py#L23) builder implementation splits texts with `splitlines()` which splits the text on several line boundaries. Not all of them are always wanted. 

**Describe the solution you'd like**
```python
                if self.config.sample_by == "line":
                    batch_idx = 0
                    while True:
                        batch = f.read(self.config.chunksize)
                        if not batch:
                            break
                        batch += f.readline()  # finish current line
                        if self.config.custom_newline is None: 
                            batch = batch.splitlines(keepends=self.config.keep_linebreaks)
                        else:
                            batch = batch.split(self.config.custom_newline)[:-1]
                        pa_table = pa.Table.from_arrays([pa.array(batch)], schema=schema)
                        # Uncomment for debugging (will print the Arrow table size and elements)
                        # logger.warning(f"pa_table: {pa_table} num rows: {pa_table.num_rows}")
                        # logger.warning('\n'.join(str(pa_table.slice(i, 1).to_pydict()) for i in range(pa_table.num_rows)))
                        yield (file_idx, batch_idx), pa_table
                        batch_idx += 1
```

**A clear and concise description of what you want to happen.**
Creating the dataset rows with a subset of the `splitlines()` line boundaries.