Large dataset in Squad2-format
MEMBER
I'm pretty sure you can get some inspiration from the squad_v2 script. It looks like the dataset is quite big so it will take some time for the users to generate it, but it should be reasonable.

Also you are saying that you are still making the dataset grow in size right ?
It's probably good practice to let the users do their training/evaluations with the exact same version of the dataset.
We allow for each dataset to specify a version (ex: 1.0.0) and increment this number every time there are new samples in the dataset for example. Does it look like a good solution for you ? Or would you rather have one final version with the full dataset ?
2020-06-30T15:43:12Z
https://github.com/huggingface/datasets/issues/326#issuecomment-651877517
https://api.github.com/repos/huggingface/datasets/issues/326
MDEyOklzc3VlQ29tbWVudDY1MTg3NzUxNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/651877517/reactions
2020-06-30T15:43:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/651877517
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
It would also be good if there is any possibility for versioning, I think this way is much better than the dynamic way.
If you mean that part to put the tiles into one is the generation it would take up to 15-20 minutes on home computer hardware.
Are there any compression or optimization algorithms while generating the dataset ?
Otherwise the hardware limit is around 32 GB ram at the moment.
If everything works well we will add some more gigabytes of data in future what would make it pretty memory costly.
2020-06-30T16:38:02Z
https://github.com/huggingface/datasets/issues/326#issuecomment-651909910
https://api.github.com/repos/huggingface/datasets/issues/326
MDEyOklzc3VlQ29tbWVudDY1MTkwOTkxMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/651909910/reactions
2020-06-30T16:38:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/651909910
https://avatars.githubusercontent.com/u/47894090?v=4
https://api.github.com/users/flozi00/events{/privacy}
https://api.github.com/users/flozi00/followers
https://api.github.com/users/flozi00/following{/other_user}
https://api.github.com/users/flozi00/gists{/gist_id}
https://github.com/flozi00
flozi00
MDQ6VXNlcjQ3ODk0MDkw
https://api.github.com/users/flozi00/orgs
https://api.github.com/users/flozi00/received_events
https://api.github.com/users/flozi00/repos
https://api.github.com/users/flozi00/starred{/owner}{/repo}
https://api.github.com/users/flozi00/subscriptions
User
https://api.github.com/users/flozi00
public
MEMBER
15-20 minutes is fine !
Also there's no RAM limitations as we save to disk every 1000 elements while generating the dataset by default.
After generation, the dataset is ready to use with (again) no RAM limitations as we do memory-mapping.
2020-06-30T16:41:34Z
https://github.com/huggingface/datasets/issues/326#issuecomment-651911828
https://api.github.com/repos/huggingface/datasets/issues/326
MDEyOklzc3VlQ29tbWVudDY1MTkxMTgyOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/651911828/reactions
2020-06-30T16:41:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/651911828
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Wow, that sounds pretty cool.
Actually I have the problem of running out of memory while tokenization on our local machine.
That wouldn't happen again, would it ?
2020-06-30T16:44:04Z
https://github.com/huggingface/datasets/issues/326#issuecomment-651913212
https://api.github.com/repos/huggingface/datasets/issues/326
MDEyOklzc3VlQ29tbWVudDY1MTkxMzIxMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/651913212/reactions
2020-06-30T16:44:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/651913212
https://avatars.githubusercontent.com/u/47894090?v=4
https://api.github.com/users/flozi00/events{/privacy}
https://api.github.com/users/flozi00/followers
https://api.github.com/users/flozi00/following{/other_user}
https://api.github.com/users/flozi00/gists{/gist_id}
https://github.com/flozi00
flozi00
MDQ6VXNlcjQ3ODk0MDkw
https://api.github.com/users/flozi00/orgs
https://api.github.com/users/flozi00/received_events
https://api.github.com/users/flozi00/repos
https://api.github.com/users/flozi00/starred{/owner}{/repo}
https://api.github.com/users/flozi00/subscriptions
User
https://api.github.com/users/flozi00
public
MEMBER
You can do the tokenization step using `my_tokenized_dataset = my_dataset.map(my_tokenize_function)` that writes the tokenized texts on disk as well. And then `my_tokenized_dataset` will be a memory-mapped dataset too, so you should be fine :)
2020-06-30T16:48:45Z
https://github.com/huggingface/datasets/issues/326#issuecomment-651915704
https://api.github.com/repos/huggingface/datasets/issues/326
MDEyOklzc3VlQ29tbWVudDY1MTkxNTcwNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/651915704/reactions
2020-06-30T16:48:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/651915704
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Does it have an affect to the trainings speed ?
2020-06-30T17:37:35Z
https://github.com/huggingface/datasets/issues/326#issuecomment-651941021
https://api.github.com/repos/huggingface/datasets/issues/326
MDEyOklzc3VlQ29tbWVudDY1MTk0MTAyMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/651941021/reactions
2020-06-30T17:40:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/651941021
https://avatars.githubusercontent.com/u/47894090?v=4
https://api.github.com/users/flozi00/events{/privacy}
https://api.github.com/users/flozi00/followers
https://api.github.com/users/flozi00/following{/other_user}
https://api.github.com/users/flozi00/gists{/gist_id}
https://github.com/flozi00
flozi00
MDQ6VXNlcjQ3ODk0MDkw
https://api.github.com/users/flozi00/orgs
https://api.github.com/users/flozi00/received_events
https://api.github.com/users/flozi00/repos
https://api.github.com/users/flozi00/starred{/owner}{/repo}
https://api.github.com/users/flozi00/subscriptions
User
https://api.github.com/users/flozi00
public
MEMBER
In your training loop, loading the tokenized texts is going to be fast and pretty much negligible compared to a forward pass. You shouldn't expect any slow down.
2020-07-02T13:45:43Z
https://github.com/huggingface/datasets/issues/326#issuecomment-653015067
https://api.github.com/repos/huggingface/datasets/issues/326
MDEyOklzc3VlQ29tbWVudDY1MzAxNTA2Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/653015067/reactions
2020-07-02T13:45:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/653015067
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Closing this one. Feel free to re-open if you have other questions
2020-07-09T09:01:50Z
https://github.com/huggingface/datasets/issues/326#issuecomment-656005573
https://api.github.com/repos/huggingface/datasets/issues/326
MDEyOklzc3VlQ29tbWVudDY1NjAwNTU3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/656005573/reactions
2020-07-09T09:01:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656005573
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
At the moment we are building an large question answering dataset and think about sharing it with the huggingface community.
Caused the computing power we splitted it into multiple tiles, but they are all in the same format.
Right now the most important facts about are this:
- Contexts: 1.047.671
- questions: 1.677.732
- Answers: 6.742.406
- unanswerable: 377.398

It is already cleaned

<pre><code>
train_data = [
    {
        'context': "this is the context",
        'qas': [
            {
                'id': "00002",
                'is_impossible': False,
                'question': "whats is this",
                'answers': [
                    {
                        'text': "answer",
                        'answer_start': 0
                    }
                ]
            },
            {
                'id': "00003",
                'is_impossible': False,
                'question': "question2",
                'answers': [
                    {
                        'text': "answer2",
                        'answer_start': 1
                    }
                ]
            }
        ]
    }
]
</code></pre>

Cause it is growing every day we are thinking about an structure like this:
We host an Json file, containing all the download links and the script can load it dynamically.
At the moment it is around ~20GB

Any advice how to handle this, or an ready to use template ?