Youtube caption corrections
CONTRIBUTOR
Sorry about forgetting flake8.
Rather than use up the circleci resources on a new push with only formatting changes, I will wait to push until the results from all tests finish and/or any feedback comes in... probably tomorrow for me.
2020-12-09T06:09:28Z
https://github.com/huggingface/datasets/pull/1357#issuecomment-741554251
https://api.github.com/repos/huggingface/datasets/issues/1357
MDEyOklzc3VlQ29tbWVudDc0MTU1NDI1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/741554251/reactions
2020-12-09T06:09:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/741554251
https://avatars.githubusercontent.com/u/21292059?v=4
https://api.github.com/users/2dot71mily/events{/privacy}
https://api.github.com/users/2dot71mily/followers
https://api.github.com/users/2dot71mily/following{/other_user}
https://api.github.com/users/2dot71mily/gists{/gist_id}
https://github.com/2dot71mily
2dot71mily
MDQ6VXNlcjIxMjkyMDU5
https://api.github.com/users/2dot71mily/orgs
https://api.github.com/users/2dot71mily/received_events
https://api.github.com/users/2dot71mily/repos
https://api.github.com/users/2dot71mily/starred{/owner}{/repo}
https://api.github.com/users/2dot71mily/subscriptions
User
https://api.github.com/users/2dot71mily
public
CONTRIBUTOR

So... my normal work is with mercurial and seem to have clearly forked this up using git... :(

What I did is after calling:
```
git fetch upstream
git rebase upstream/master
```

I then I attempt to pull in my most recent changes UI commit changes based on @lhoestq's feedback with:
```
git pull
```  
... which I now suspect undid the above fetch and rebase. Will look into fixing later today when I have more time. Sorry!

2020-12-10T17:33:08Z
https://github.com/huggingface/datasets/pull/1357#issuecomment-742673923
https://api.github.com/repos/huggingface/datasets/issues/1357
MDEyOklzc3VlQ29tbWVudDc0MjY3MzkyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/742673923/reactions
2020-12-10T17:33:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/742673923
https://avatars.githubusercontent.com/u/21292059?v=4
https://api.github.com/users/2dot71mily/events{/privacy}
https://api.github.com/users/2dot71mily/followers
https://api.github.com/users/2dot71mily/following{/other_user}
https://api.github.com/users/2dot71mily/gists{/gist_id}
https://github.com/2dot71mily
2dot71mily
MDQ6VXNlcjIxMjkyMDU5
https://api.github.com/users/2dot71mily/orgs
https://api.github.com/users/2dot71mily/received_events
https://api.github.com/users/2dot71mily/repos
https://api.github.com/users/2dot71mily/starred{/owner}{/repo}
https://api.github.com/users/2dot71mily/subscriptions
User
https://api.github.com/users/2dot71mily
public
CONTRIBUTOR
My dummy data seems quite large as a single row is composed of tokens/labels for an entire youtube video, with at least one row required for each file, which in this case 1 file per 13 youtube channels.

To make it smaller I passed `--n_lines 1` to reduce about 5x.

I then manually reduced size of the particularly long youtube lectures to get the size to about 30KB. However, after recompressing into a zip, and running dummy data test I got the following error:
`FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_youtube_caption_corrections - OSError: Cannot find data file. `, despite file being there, which I haven't had a chance yet to debug.
2020-12-10T20:55:07Z
https://github.com/huggingface/datasets/pull/1357#issuecomment-742793221
https://api.github.com/repos/huggingface/datasets/issues/1357
MDEyOklzc3VlQ29tbWVudDc0Mjc5MzIyMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/742793221/reactions
2020-12-10T20:55:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/742793221
https://avatars.githubusercontent.com/u/21292059?v=4
https://api.github.com/users/2dot71mily/events{/privacy}
https://api.github.com/users/2dot71mily/followers
https://api.github.com/users/2dot71mily/following{/other_user}
https://api.github.com/users/2dot71mily/gists{/gist_id}
https://github.com/2dot71mily
2dot71mily
MDQ6VXNlcjIxMjkyMDU5
https://api.github.com/users/2dot71mily/orgs
https://api.github.com/users/2dot71mily/received_events
https://api.github.com/users/2dot71mily/repos
https://api.github.com/users/2dot71mily/starred{/owner}{/repo}
https://api.github.com/users/2dot71mily/subscriptions
User
https://api.github.com/users/2dot71mily
public
CONTRIBUTOR
I wrote a small script to generate a smaller json file for the dummy_data, with the hope that I could resolve the pytest error noted above (in case related to a manual typo I could have introduce), however the test contains to fail locally... here's to hoping it can pass on remote!
2020-12-11T04:35:07Z
https://github.com/huggingface/datasets/pull/1357#issuecomment-742962666
https://api.github.com/repos/huggingface/datasets/issues/1357
MDEyOklzc3VlQ29tbWVudDc0Mjk2MjY2Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/742962666/reactions
2020-12-11T04:35:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/742962666
https://avatars.githubusercontent.com/u/21292059?v=4
https://api.github.com/users/2dot71mily/events{/privacy}
https://api.github.com/users/2dot71mily/followers
https://api.github.com/users/2dot71mily/following{/other_user}
https://api.github.com/users/2dot71mily/gists{/gist_id}
https://github.com/2dot71mily
2dot71mily
MDQ6VXNlcjIxMjkyMDU5
https://api.github.com/users/2dot71mily/orgs
https://api.github.com/users/2dot71mily/received_events
https://api.github.com/users/2dot71mily/repos
https://api.github.com/users/2dot71mily/starred{/owner}{/repo}
https://api.github.com/users/2dot71mily/subscriptions
User
https://api.github.com/users/2dot71mily
public
CONTRIBUTOR
Sorry for delayed comments here. Last commit made two changes:
- Increased the valency of the labels from just True/False to more categories to describe the various types of diffs encountered. This required some rewrite of the README
- Reduced the number of remote files to be downloaded from 13 to 4, by combining all 13 of the channel-specific files together, and the splitting them up in a way to meet Github file size requirements. This also reduces size of the dummy-data.
2020-12-14T05:39:28Z
https://github.com/huggingface/datasets/pull/1357#issuecomment-744181985
https://api.github.com/repos/huggingface/datasets/issues/1357
MDEyOklzc3VlQ29tbWVudDc0NDE4MTk4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/744181985/reactions
2020-12-14T05:39:28Z
https://api.github.com/repos/huggingface/datasets/issues/comments/744181985
https://avatars.githubusercontent.com/u/21292059?v=4
https://api.github.com/users/2dot71mily/events{/privacy}
https://api.github.com/users/2dot71mily/followers
https://api.github.com/users/2dot71mily/following{/other_user}
https://api.github.com/users/2dot71mily/gists{/gist_id}
https://github.com/2dot71mily
2dot71mily
MDQ6VXNlcjIxMjkyMDU5
https://api.github.com/users/2dot71mily/orgs
https://api.github.com/users/2dot71mily/received_events
https://api.github.com/users/2dot71mily/repos
https://api.github.com/users/2dot71mily/starred{/owner}{/repo}
https://api.github.com/users/2dot71mily/subscriptions
User
https://api.github.com/users/2dot71mily
public
CONTRIBUTOR
@lhoestq, thank you for the great feedback, especially given how busy you guys are now! 

I checked out GitHub release tags and looks cool. I have added the version tag to the url, instead of the commit sha as originally suggested, with the hope that it serves the same purpose of pinning the content to this url. Please let me know if I have misunderstood.

In regard to dynamically changing the number of files downloaded by first downloading a JSON listing the files, I love that idea. But I am a little confused, as I was thinking that any changes to the dataset itself would require a new PR with an updated `dataset_infos.json`, e.g. `num_examples` would increase. 

If the purpose of this is not to permit dynamic (without a PR needed) growth of the number of files, but instead to provide stability to the consumers of the dataset, maybe I continued use the release tags, maintaining access to old releases could serve this purpose? I am still learning about these release tags...  
2020-12-15T16:50:32Z
https://github.com/huggingface/datasets/pull/1357#issuecomment-745420151
https://api.github.com/repos/huggingface/datasets/issues/1357
MDEyOklzc3VlQ29tbWVudDc0NTQyMDE1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/745420151/reactions
2020-12-15T16:50:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/745420151
https://avatars.githubusercontent.com/u/21292059?v=4
https://api.github.com/users/2dot71mily/events{/privacy}
https://api.github.com/users/2dot71mily/followers
https://api.github.com/users/2dot71mily/following{/other_user}
https://api.github.com/users/2dot71mily/gists{/gist_id}
https://github.com/2dot71mily
2dot71mily
MDQ6VXNlcjIxMjkyMDU5
https://api.github.com/users/2dot71mily/orgs
https://api.github.com/users/2dot71mily/received_events
https://api.github.com/users/2dot71mily/repos
https://api.github.com/users/2dot71mily/starred{/owner}{/repo}
https://api.github.com/users/2dot71mily/subscriptions
User
https://api.github.com/users/2dot71mily
public
MEMBER
For dynamic datasets, i.e. datasets that evolve over time, we support custom configurations: they are configurations that are not part of the BUILDER_CONFIGS or in the dataset_infos.json

For example for wikipedia, you can use the latest wiki dump by specifying `date=` inside `load_dataset()`. A configuration is created on the fly for this date and is used to build the dataset using the latest data.

Therefore we don't need to have PRs to update the script for each wikipedia release.

One downside though is that we don't have metadata in advance such as the size of the dataset.

I think this could be a nice addition for the youtube caption dataset in the future to be have a system of releases and be able to load the version we want easily. What do you think ?
2020-12-15T16:57:24Z
https://github.com/huggingface/datasets/pull/1357#issuecomment-745424545
https://api.github.com/repos/huggingface/datasets/issues/1357
MDEyOklzc3VlQ29tbWVudDc0NTQyNDU0NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/745424545/reactions
2020-12-15T17:00:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/745424545
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR




> For dynamic datasets, i.e. datasets that evolve over time, we support custom configurations: they are configurations that are not part of the BUILDER_CONFIGS or in the dataset_infos.json
> 
 
> I think this could be a nice addition for the youtube caption dataset in the future to be have a system of releases and be able to load the version we want easily. What do you think ?

Thank you for the suggestion! This sounds great! I will take a look at the some datasets that do this, and would love to give it a try in the future, if I continue to grow the captions dataset in a meaningful way. 

Appreciate all the help on this. It has been a really great experience for me. :)
2020-12-15T17:28:02Z
https://github.com/huggingface/datasets/pull/1357#issuecomment-745443279
https://api.github.com/repos/huggingface/datasets/issues/1357
MDEyOklzc3VlQ29tbWVudDc0NTQ0MzI3OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/745443279/reactions
2020-12-15T17:28:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/745443279
https://avatars.githubusercontent.com/u/21292059?v=4
https://api.github.com/users/2dot71mily/events{/privacy}
https://api.github.com/users/2dot71mily/followers
https://api.github.com/users/2dot71mily/following{/other_user}
https://api.github.com/users/2dot71mily/gists{/gist_id}
https://github.com/2dot71mily
2dot71mily
MDQ6VXNlcjIxMjkyMDU5
https://api.github.com/users/2dot71mily/orgs
https://api.github.com/users/2dot71mily/received_events
https://api.github.com/users/2dot71mily/repos
https://api.github.com/users/2dot71mily/starred{/owner}{/repo}
https://api.github.com/users/2dot71mily/subscriptions
User
https://api.github.com/users/2dot71mily
public
CONTRIBUTOR
Excited to merge! And sorry to be such a github n00b, but from what I've quickly read, I don't 'Close pull request', but rather the next steps are action taken on your end... Please let me know if there is some action to be taken at my end first. :/
2020-12-15T17:56:03Z
https://github.com/huggingface/datasets/pull/1357#issuecomment-745460610
https://api.github.com/repos/huggingface/datasets/issues/1357
MDEyOklzc3VlQ29tbWVudDc0NTQ2MDYxMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/745460610/reactions
2020-12-15T17:56:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/745460610
https://avatars.githubusercontent.com/u/21292059?v=4
https://api.github.com/users/2dot71mily/events{/privacy}
https://api.github.com/users/2dot71mily/followers
https://api.github.com/users/2dot71mily/following{/other_user}
https://api.github.com/users/2dot71mily/gists{/gist_id}
https://github.com/2dot71mily
2dot71mily
MDQ6VXNlcjIxMjkyMDU5
https://api.github.com/users/2dot71mily/orgs
https://api.github.com/users/2dot71mily/received_events
https://api.github.com/users/2dot71mily/repos
https://api.github.com/users/2dot71mily/starred{/owner}{/repo}
https://api.github.com/users/2dot71mily/subscriptions
User
https://api.github.com/users/2dot71mily
public
MEMBER
Alright merging this one then :) 
2020-12-15T18:12:52Z
https://github.com/huggingface/datasets/pull/1357#issuecomment-745470180
https://api.github.com/repos/huggingface/datasets/issues/1357
MDEyOklzc3VlQ29tbWVudDc0NTQ3MDE4MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/745470180/reactions
2020-12-15T18:12:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/745470180
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
This PR adds a new dataset of YouTube captions, error and corrections. This dataset was created in just the last week, as inspired by this sprint!