Workflow for Tabular data
CONTRIBUTOR
I use below to load a dataset:
```
dataset = datasets.load_dataset("scikit-learn/auto-mpg")
df = pd.DataFrame(dataset["train"])
```
TBH as said, tabular folk split their own dataset, they sometimes have two splits, sometimes three. Maybe somehow avoiding it for tabular datasets might be good for later. (it's just UX improvement) 
2022-07-01T09:57:46Z
https://github.com/huggingface/datasets/issues/4456#issuecomment-1172167780
https://api.github.com/repos/huggingface/datasets/issues/4456
IC_kwDODunzps5F3dxk
https://api.github.com/repos/huggingface/datasets/issues/comments/1172167780/reactions
2022-07-01T09:57:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1172167780
https://avatars.githubusercontent.com/u/53175384?v=4
https://api.github.com/users/merveenoyan/events{/privacy}
https://api.github.com/users/merveenoyan/followers
https://api.github.com/users/merveenoyan/following{/other_user}
https://api.github.com/users/merveenoyan/gists{/gist_id}
https://github.com/merveenoyan
merveenoyan
MDQ6VXNlcjUzMTc1Mzg0
https://api.github.com/users/merveenoyan/orgs
https://api.github.com/users/merveenoyan/received_events
https://api.github.com/users/merveenoyan/repos
https://api.github.com/users/merveenoyan/starred{/owner}{/repo}
https://api.github.com/users/merveenoyan/subscriptions
User
https://api.github.com/users/merveenoyan
public
NONE
is very slow batch access of a dataset (tabular, csv) with many columns to be expected?
2023-02-02T23:46:07Z
https://github.com/huggingface/datasets/issues/4456#issuecomment-1414519442
https://api.github.com/repos/huggingface/datasets/issues/4456
IC_kwDODunzps5UT9qS
https://api.github.com/repos/huggingface/datasets/issues/comments/1414519442/reactions
2023-02-02T23:46:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1414519442
https://avatars.githubusercontent.com/u/28735574?v=4
https://api.github.com/users/wconnell/events{/privacy}
https://api.github.com/users/wconnell/followers
https://api.github.com/users/wconnell/following{/other_user}
https://api.github.com/users/wconnell/gists{/gist_id}
https://github.com/wconnell
wconnell
MDQ6VXNlcjI4NzM1NTc0
https://api.github.com/users/wconnell/orgs
https://api.github.com/users/wconnell/received_events
https://api.github.com/users/wconnell/repos
https://api.github.com/users/wconnell/starred{/owner}{/repo}
https://api.github.com/users/wconnell/subscriptions
User
https://api.github.com/users/wconnell
public
MEMBER
Define "many" ? x)
2023-02-03T10:52:05Z
https://github.com/huggingface/datasets/issues/4456#issuecomment-1415692524
https://api.github.com/repos/huggingface/datasets/issues/4456
IC_kwDODunzps5UYcDs
https://api.github.com/repos/huggingface/datasets/issues/comments/1415692524/reactions
2023-02-03T10:52:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1415692524
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
~20k! I was surprised batch loading with as few as 32 samples was really slow. I was speculating the columnar format was the cause -- or do you see good performance with this approx size of tabular data?
2023-02-03T16:31:05Z
https://github.com/huggingface/datasets/issues/4456#issuecomment-1416108659
https://api.github.com/repos/huggingface/datasets/issues/4456
IC_kwDODunzps5UaBpz
https://api.github.com/repos/huggingface/datasets/issues/comments/1416108659/reactions
2023-02-03T16:31:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1416108659
https://avatars.githubusercontent.com/u/28735574?v=4
https://api.github.com/users/wconnell/events{/privacy}
https://api.github.com/users/wconnell/followers
https://api.github.com/users/wconnell/following{/other_user}
https://api.github.com/users/wconnell/gists{/gist_id}
https://github.com/wconnell
wconnell
MDQ6VXNlcjI4NzM1NTc0
https://api.github.com/users/wconnell/orgs
https://api.github.com/users/wconnell/received_events
https://api.github.com/users/wconnell/repos
https://api.github.com/users/wconnell/starred{/owner}{/repo}
https://api.github.com/users/wconnell/subscriptions
User
https://api.github.com/users/wconnell
public
MEMBER
20k can be a lot for a columnar format but maybe we can optimize a few things.

It would be cool to profile the code to see if there's an unoptimized part of the code that slows everything down.

(it's also possible to kill the job when it accesses the batch, it often gives you the traceback at the location where the code was running)
2023-02-03T16:43:19Z
https://github.com/huggingface/datasets/issues/4456#issuecomment-1416122849
https://api.github.com/repos/huggingface/datasets/issues/4456
IC_kwDODunzps5UaFHh
https://api.github.com/repos/huggingface/datasets/issues/comments/1416122849/reactions
2023-02-03T16:43:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1416122849
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
FWIW I've worked with tabular data with 540k columns.
2023-03-03T10:37:57Z
https://github.com/huggingface/datasets/issues/4456#issuecomment-1453318926
https://api.github.com/repos/huggingface/datasets/issues/4456
IC_kwDODunzps5Wn-MO
https://api.github.com/repos/huggingface/datasets/issues/comments/1453318926/reactions
2023-03-03T10:37:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1453318926
https://avatars.githubusercontent.com/u/1663864?v=4
https://api.github.com/users/adrinjalali/events{/privacy}
https://api.github.com/users/adrinjalali/followers
https://api.github.com/users/adrinjalali/following{/other_user}
https://api.github.com/users/adrinjalali/gists{/gist_id}
https://github.com/adrinjalali
adrinjalali
MDQ6VXNlcjE2NjM4NjQ=
https://api.github.com/users/adrinjalali/orgs
https://api.github.com/users/adrinjalali/received_events
https://api.github.com/users/adrinjalali/repos
https://api.github.com/users/adrinjalali/starred{/owner}{/repo}
https://api.github.com/users/adrinjalali/subscriptions
User
https://api.github.com/users/adrinjalali
public
NONE
thats awesome, whats your secret? would love to see an example!
2023-03-03T16:33:42Z
https://github.com/huggingface/datasets/issues/4456#issuecomment-1453790562
https://api.github.com/repos/huggingface/datasets/issues/4456
IC_kwDODunzps5WpxVi
https://api.github.com/repos/huggingface/datasets/issues/comments/1453790562/reactions
2023-03-03T16:33:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1453790562
https://avatars.githubusercontent.com/u/28735574?v=4
https://api.github.com/users/wconnell/events{/privacy}
https://api.github.com/users/wconnell/followers
https://api.github.com/users/wconnell/following{/other_user}
https://api.github.com/users/wconnell/gists{/gist_id}
https://github.com/wconnell
wconnell
MDQ6VXNlcjI4NzM1NTc0
https://api.github.com/users/wconnell/orgs
https://api.github.com/users/wconnell/received_events
https://api.github.com/users/wconnell/repos
https://api.github.com/users/wconnell/starred{/owner}{/repo}
https://api.github.com/users/wconnell/subscriptions
User
https://api.github.com/users/wconnell
public
NONE
@wconnell I'm not sure what you mean by my secret, I load them into a numpy array 😁 

An example dataset is [here](https://portal.gdc.cancer.gov/repository?facetTab=files&filters=%7B%22content%22%3A%5B%7B%22content%22%3A%7B%22field%22%3A%22cases.project.project_id%22%2C%22value%22%3A%5B%22TCGA-CESC%22%5D%7D%2C%22op%22%3A%22in%22%7D%2C%7B%22content%22%3A%7B%22field%22%3A%22files.data_category%22%2C%22value%22%3A%5B%22DNA%20Methylation%22%5D%7D%2C%22op%22%3A%22in%22%7D%5D%2C%22op%22%3A%22and%22%7D&searchTableTab=files) which is a dataset of DNA methylation reads. This dataset is about 950 rows and 450k columns. 
2023-03-06T08:53:55Z
https://github.com/huggingface/datasets/issues/4456#issuecomment-1455726407
https://api.github.com/repos/huggingface/datasets/issues/4456
IC_kwDODunzps5WxJ9H
https://api.github.com/repos/huggingface/datasets/issues/comments/1455726407/reactions
2023-03-06T08:53:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1455726407
https://avatars.githubusercontent.com/u/1663864?v=4
https://api.github.com/users/adrinjalali/events{/privacy}
https://api.github.com/users/adrinjalali/followers
https://api.github.com/users/adrinjalali/following{/other_user}
https://api.github.com/users/adrinjalali/gists{/gist_id}
https://github.com/adrinjalali
adrinjalali
MDQ6VXNlcjE2NjM4NjQ=
https://api.github.com/users/adrinjalali/orgs
https://api.github.com/users/adrinjalali/received_events
https://api.github.com/users/adrinjalali/repos
https://api.github.com/users/adrinjalali/starred{/owner}{/repo}
https://api.github.com/users/adrinjalali/subscriptions
User
https://api.github.com/users/adrinjalali
public
Tabular data are treated very differently than data for NLP, audio, vision, etc. and therefore the worflow for tabular data in `datasets` is not ideal.

For example for tabular data, it is common to use pandas/spark/dask to process the data, and then load the data into X and y (X is an array of features and y an array of labels), then train_test_split and finally feed the data to a machine learning model.

In `datasets` the workflow is different: we use load_dataset, then map, then train_test_split (if we only have a train split) and we end up with columnar dataset splits, not formatted as X and y.

Right now, it is already possible to convert a dataset from and to pandas, but there are still many things that could improve the workflow for tabular data:
- be able to load the data into X and y
- be able to load a dataset from the output of spark or dask (as far as I know it's usually csv or parquet files on S3/GCS/HDFS etc.)
- support "unsplit" datasets explicitly, instead of putting everything in "train" by default

cc @adrinjalali @merveenoyan feel free to complete/correct this :)

Feel free to also share ideas of APIs that would be super intuitive in your opinion !