Extend text to support yielding lines, paragraphs or documents
MEMBER
The parameter can also be named `split_by` with values "line", "paragraph" or "document" (no 's' at the end)
2021-12-17T13:35:18Z
https://github.com/huggingface/datasets/pull/3442#issuecomment-996727299
https://api.github.com/repos/huggingface/datasets/issues/3442
IC_kwDODunzps47aNoD
https://api.github.com/repos/huggingface/datasets/issues/comments/996727299/reactions
2021-12-17T13:37:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/996727299
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> The parameter can also be named `split_by` with values "line", "paragraph" or "document" (no 's' at the end)

@lhoestq @mariosasko I would avoid the term `split` in this context and keep it only for "train", "validation" and "test" splits.
- https://huggingface.co/docs/datasets/process.html#split
  > datasets.Dataset.train_test_split() creates train and test splits, if your dataset doesn’t already have them.
- https://huggingface.co/docs/datasets/process.html#process-multiple-splits
  > Many datasets have splits that you can process simultaneously with datasets.DatasetDict.map().

Please note that in the documentation, one of the terms more frequently used in this context is **"row"**:
- https://huggingface.co/docs/datasets/access.html#features-and-columns
  > A dataset is a table of rows and typed columns.

  > Return the number of rows and columns with the following standard attributes:
  > dataset.num_columns
  > 4
  > dataset.num_rows
  > 3668

- https://huggingface.co/docs/datasets/access.html#rows-slices-batches-and-columns
  > Get several rows of your dataset at a time with slice notation or a list of indices:
- https://huggingface.co/docs/datasets/process.html#map
  > This function can even create new rows and columns.

Other of the terms more frequently used in the docs (in the code as well) is **"example"**:
- https://huggingface.co/docs/datasets/process.html#map
  > It allows you to apply a processing function to each example in a dataset, independently or in batches.
- https://huggingface.co/docs/datasets/process.html#batch-processing
  > datasets.Dataset.map() also supports working with batches of examples.
- https://huggingface.co/docs/datasets/process.html#split-long-examples
  > When your examples are too long, you may want to split them
- https://huggingface.co/docs/datasets/process.html#data-augmentation
  > With batch processing, you can even augment your dataset with additional examples.

Less frequently used: **"item"**:
- https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.add_item
  > Add item to Dataset.

Other term used in the docs (although less frequently) is **"sample"**. The advantage of this word is that it is also a verb, so we can use the parameter: "sample_by" (if you insist on using a verb instead of a noun).

In summary, these proposals:
- config.row
- config.example
- config.item
- config.sample
- config.sample_by
2021-12-17T17:47:31Z
https://github.com/huggingface/datasets/pull/3442#issuecomment-996906997
https://api.github.com/repos/huggingface/datasets/issues/3442
IC_kwDODunzps47a5f1
https://api.github.com/repos/huggingface/datasets/issues/comments/996906997/reactions
2021-12-17T17:50:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/996906997
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I like `sample_by`. Another idea I had was `separate_by`.

It could also be `sampling`, `sampling_method`, `separation_method`.

Not a big fan of the proposed nouns alone since they are very generic, that's why I tried to have something more specific.

I also agree that we actually should avoid `split` to avoid any confusion
2021-12-20T11:49:21Z
https://github.com/huggingface/datasets/pull/3442#issuecomment-997854071
https://api.github.com/repos/huggingface/datasets/issues/3442
IC_kwDODunzps47egt3
https://api.github.com/repos/huggingface/datasets/issues/comments/997854071/reactions
2021-12-20T11:50:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/997854071
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
COLLABORATOR
Thanks for the analysis of the used terms. I also like `sample_by` (`separate_by` is good too).
2021-12-20T12:25:29Z
https://github.com/huggingface/datasets/pull/3442#issuecomment-997879170
https://api.github.com/repos/huggingface/datasets/issues/3442
IC_kwDODunzps47em2C
https://api.github.com/repos/huggingface/datasets/issues/comments/997879170/reactions
2021-12-20T12:25:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/997879170
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Thank you !! :D 
2021-12-20T16:59:10Z
https://github.com/huggingface/datasets/pull/3442#issuecomment-998106103
https://api.github.com/repos/huggingface/datasets/issues/3442
IC_kwDODunzps47feP3
https://api.github.com/repos/huggingface/datasets/issues/comments/998106103/reactions
2021-12-20T16:59:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/998106103
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Add `config.row` option to `text` module to allow yielding lines (default, current case), paragraphs or documents.

Feel free to comment on the name of the config parameter `row`:
- Currently, the docs state datasets are made of rows and columns
- Other names I considered: `example`, `item`