documentation missing how to split a dataset
NONE
this seems to work for my specific problem:

`self.train_ds, self.test_ds, self.val_ds = map(_prepare_ds, ('train', 'test[:25%]+test[50%:75%]', 'test[75%:]'))`
2020-06-10T22:17:12Z
https://github.com/huggingface/datasets/issues/259#issuecomment-642296434
https://api.github.com/repos/huggingface/datasets/issues/259
MDEyOklzc3VlQ29tbWVudDY0MjI5NjQzNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/642296434/reactions
2020-06-10T22:17:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/642296434
https://avatars.githubusercontent.com/u/2873355?v=4
https://api.github.com/users/fotisj/events{/privacy}
https://api.github.com/users/fotisj/followers
https://api.github.com/users/fotisj/following{/other_user}
https://api.github.com/users/fotisj/gists{/gist_id}
https://github.com/fotisj
fotisj
MDQ6VXNlcjI4NzMzNTU=
https://api.github.com/users/fotisj/orgs
https://api.github.com/users/fotisj/received_events
https://api.github.com/users/fotisj/repos
https://api.github.com/users/fotisj/starred{/owner}{/repo}
https://api.github.com/users/fotisj/subscriptions
User
https://api.github.com/users/fotisj
public
MEMBER
Currently you can indeed split a dataset using `ds_test = nlp.load_dataset('imdb, split='test[:5000]')` (works also with percentages).

However right now we don't have a way to shuffle a dataset but we are thinking about it in the discussion in #166. Feel free to share your thoughts about it.

One trick that you can do until we have a better solution is to shuffle and split the indices of your dataset:
```python
import nlp
from sklearn.model_selection import train_test_split

imdb = nlp.load_dataset('imbd', split='test')
test_indices, val_indices = train_test_split(range(len(imdb)))
```

and then to iterate each split:
```python
for i in test_indices:
    example = imdb[i]
   ...
```

2020-06-11T09:37:08Z
https://github.com/huggingface/datasets/issues/259#issuecomment-642533977
https://api.github.com/repos/huggingface/datasets/issues/259
MDEyOklzc3VlQ29tbWVudDY0MjUzMzk3Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/642533977/reactions
2020-06-11T09:37:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/642533977
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I added a small guide [here](https://github.com/huggingface/nlp/tree/master/docs/splits.md) that explains how to split a dataset. It is very similar to the tensorflow datasets guide, as we kept the same logic.
2020-06-15T16:56:36Z
https://github.com/huggingface/datasets/issues/259#issuecomment-644252329
https://api.github.com/repos/huggingface/datasets/issues/259
MDEyOklzc3VlQ29tbWVudDY0NDI1MjMyOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/644252329/reactions
2020-06-15T16:56:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/644252329
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks a lot, the new explanation is very helpful!

About using train_test_split from sklearn: I stumbled across the [same error message as this user ](https://github.com/huggingface/nlp/issues/147 )and thought it can't be used at the moment in this context. Will check it out again.

One of the problems is how to shuffle very large datasets, which don't fit into the memory. Well, one strategy could be shuffling data in sections. But in a case where the data is sorted by the labels you have to swap larger sections first. 

2020-06-15T17:50:04Z
https://github.com/huggingface/datasets/issues/259#issuecomment-644278446
https://api.github.com/repos/huggingface/datasets/issues/259
MDEyOklzc3VlQ29tbWVudDY0NDI3ODQ0Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/644278446/reactions
2020-06-15T17:50:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/644278446
https://avatars.githubusercontent.com/u/2873355?v=4
https://api.github.com/users/fotisj/events{/privacy}
https://api.github.com/users/fotisj/followers
https://api.github.com/users/fotisj/following{/other_user}
https://api.github.com/users/fotisj/gists{/gist_id}
https://github.com/fotisj
fotisj
MDQ6VXNlcjI4NzMzNTU=
https://api.github.com/users/fotisj/orgs
https://api.github.com/users/fotisj/received_events
https://api.github.com/users/fotisj/repos
https://api.github.com/users/fotisj/starred{/owner}{/repo}
https://api.github.com/users/fotisj/subscriptions
User
https://api.github.com/users/fotisj
public
MEMBER
We added a way to shuffle datasets (shuffle the indices and then reorder to make a new dataset).
You can do `shuffled_dset = dataset.shuffle(seed=my_seed)`. It shuffles the whole dataset.
There is also `dataset.train_test_split()` which if very handy (with the same signature as sklearn).

Closing this issue as we added the docs for splits and tools to split datasets. Thanks again for your feedback !
2020-06-18T22:20:24Z
https://github.com/huggingface/datasets/issues/259#issuecomment-646334187
https://api.github.com/repos/huggingface/datasets/issues/259
MDEyOklzc3VlQ29tbWVudDY0NjMzNDE4Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/646334187/reactions
2020-06-18T22:20:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/646334187
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
https://huggingface.co/docs/datasets/v1.0.1/package_reference/builder_classes.html#datasets.Split still links to https://github.com/huggingface/datasets/tree/main/docs/splits.md which is a 404

2023-03-13T21:30:16Z
https://github.com/huggingface/datasets/issues/259#issuecomment-1466990491
https://api.github.com/repos/huggingface/datasets/issues/259
IC_kwDODunzps5XcH-b
https://api.github.com/repos/huggingface/datasets/issues/comments/1466990491/reactions
2023-03-13T21:30:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1466990491
https://avatars.githubusercontent.com/u/11773823?v=4
https://api.github.com/users/scottfleming/events{/privacy}
https://api.github.com/users/scottfleming/followers
https://api.github.com/users/scottfleming/following{/other_user}
https://api.github.com/users/scottfleming/gists{/gist_id}
https://github.com/scottfleming
scottfleming
MDQ6VXNlcjExNzczODIz
https://api.github.com/users/scottfleming/orgs
https://api.github.com/users/scottfleming/received_events
https://api.github.com/users/scottfleming/repos
https://api.github.com/users/scottfleming/starred{/owner}{/repo}
https://api.github.com/users/scottfleming/subscriptions
User
https://api.github.com/users/scottfleming
public
MEMBER
The updated documentation doesn't link to this anymore: https://huggingface.co/docs/datasets/v2.10.0/en/package_reference/builder_classes#datasets.Split
2023-03-14T13:56:06Z
https://github.com/huggingface/datasets/issues/259#issuecomment-1468150061
https://api.github.com/repos/huggingface/datasets/issues/259
IC_kwDODunzps5XgjEt
https://api.github.com/repos/huggingface/datasets/issues/comments/1468150061/reactions
2023-03-14T13:56:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1468150061
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
I am trying to understand how to split a dataset ( as arrow_dataset). 
I know I can do something like this to access a split which is already in the original dataset : 

`ds_test = nlp.load_dataset('imdb, split='test') `

But how can I split ds_test into a test and a validation set (without reading the data into memory and keeping the arrow_dataset as container)?
I guess it has something to do with the module split :-) but there is no real documentation in the code but only a reference to a longer description: 

> See the  [guide on splits](https://github.com/huggingface/nlp/tree/master/docs/splits.md)  for more information.

But the guide seems to be missing.

To clarify: I know that this has been modelled after the dataset of tensorflow and that some of the documentation there can be used [like this one](https://www.tensorflow.org/datasets/splits). But to come back to the example above: I cannot simply split the testset doing this: 
`ds_test = nlp.load_dataset('imdb, split='test'[:5000]) `
`ds_val = nlp.load_dataset('imdb, split='test'[5000:])`

because the imdb test data is sorted by class (probably not a good idea anyway)
