[Question] Create Apache Arrow dataset from raw text file
MEMBER
We store every dataset in the Arrow format. This is convenient as it supports nested types and memory mapping. If you are curious feel free to check the [pyarrow documentation](https://arrow.apache.org/docs/python/)

You can use this library to load your covid papers by creating a dataset script. You can find inspiration from the ones we've already written in `/datasets`. Here is a link to the steps to [add a dataset](https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset)
2020-05-26T14:50:29Z
https://github.com/huggingface/datasets/issues/192#issuecomment-634074166
https://api.github.com/repos/huggingface/datasets/issues/192
MDEyOklzc3VlQ29tbWVudDYzNDA3NDE2Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/634074166/reactions
2020-05-26T14:50:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/634074166
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hello @mrm8488 and @lhoestq 

Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?

Thanks :)
2020-07-07T17:05:51Z
https://github.com/huggingface/datasets/issues/192#issuecomment-654999980
https://api.github.com/repos/huggingface/datasets/issues/192
MDEyOklzc3VlQ29tbWVudDY1NDk5OTk4MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/654999980/reactions
2020-07-07T19:37:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/654999980
https://avatars.githubusercontent.com/u/41843953?v=4
https://api.github.com/users/IbtihalFerwana/events{/privacy}
https://api.github.com/users/IbtihalFerwana/followers
https://api.github.com/users/IbtihalFerwana/following{/other_user}
https://api.github.com/users/IbtihalFerwana/gists{/gist_id}
https://github.com/IbtihalFerwana
IbtihalFerwana
MDQ6VXNlcjQxODQzOTUz
https://api.github.com/users/IbtihalFerwana/orgs
https://api.github.com/users/IbtihalFerwana/received_events
https://api.github.com/users/IbtihalFerwana/repos
https://api.github.com/users/IbtihalFerwana/starred{/owner}{/repo}
https://api.github.com/users/IbtihalFerwana/subscriptions
User
https://api.github.com/users/IbtihalFerwana
public
MEMBER
> Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?

Sure, to get a dataset in arrow format you can either:
- [load from local files (txt, json, csv)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-local-files)
- OR [load from python data (dict, pandas)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-in-memory-data)
- OR [create your own dataset script](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#using-a-custom-dataset-loading-script)

2020-07-23T16:40:33Z
https://github.com/huggingface/datasets/issues/192#issuecomment-663110742
https://api.github.com/repos/huggingface/datasets/issues/192
MDEyOklzc3VlQ29tbWVudDY2MzExMDc0Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/663110742/reactions
2020-07-23T16:40:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/663110742
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
> > Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?
> 
> Sure, to get a dataset in arrow format you can either:
> 
>     * [load from local files (txt, json, csv)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-local-files)
> 
>     * OR [load from python data (dict, pandas)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-in-memory-data)
> 
>     * OR [create your own dataset script](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#using-a-custom-dataset-loading-script)

Links were broken. 

Updated links provided as below
- [load from local files (txt, json, csv)](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-or-remote-files)
- [load from python data (dict, pandas)](https://huggingface.co/docs/datasets/loading_datasets.html#from-in-memory-data)
- [create your own dataset script](https://huggingface.co/docs/datasets/loading_datasets.html#using-a-custom-dataset-loading-script)

2021-12-18T01:45:34Z
https://github.com/huggingface/datasets/issues/192#issuecomment-997121184
https://api.github.com/repos/huggingface/datasets/issues/192
IC_kwDODunzps47btyg
https://api.github.com/repos/huggingface/datasets/issues/comments/997121184/reactions
2021-12-18T01:45:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/997121184
https://avatars.githubusercontent.com/u/33477318?v=4
https://api.github.com/users/codenamewei/events{/privacy}
https://api.github.com/users/codenamewei/followers
https://api.github.com/users/codenamewei/following{/other_user}
https://api.github.com/users/codenamewei/gists{/gist_id}
https://github.com/codenamewei
codenamewei
MDQ6VXNlcjMzNDc3MzE4
https://api.github.com/users/codenamewei/orgs
https://api.github.com/users/codenamewei/received_events
https://api.github.com/users/codenamewei/repos
https://api.github.com/users/codenamewei/starred{/owner}{/repo}
https://api.github.com/users/codenamewei/subscriptions
User
https://api.github.com/users/codenamewei
public
Hi guys, I have gathered and preprocessed about 2GB of COVID papers from CORD dataset @ Kggle. I have seen you have a text dataset as "Crime and punishment" in Apache arrow format. Do you have any script to do it from a raw txt file (preprocessed as for BERT like) or any guide?
Is the worth of send it to you and add it to the NLP library?
Thanks, Manu
