PG-19
MEMBER
Sounds good! Do you want to give it a try?
2020-06-16T06:46:25Z
https://github.com/huggingface/datasets/issues/274#issuecomment-644566626
https://api.github.com/repos/huggingface/datasets/issues/274
MDEyOklzc3VlQ29tbWVudDY0NDU2NjYyNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/644566626/reactions
2020-06-16T06:46:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/644566626
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
Ok, I'll see if I can figure it out tomorrow!
2020-06-16T23:26:25Z
https://github.com/huggingface/datasets/issues/274#issuecomment-645061293
https://api.github.com/repos/huggingface/datasets/issues/274
MDEyOklzc3VlQ29tbWVudDY0NTA2MTI5Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/645061293/reactions
2020-06-16T23:26:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/645061293
https://avatars.githubusercontent.com/u/108653?v=4
https://api.github.com/users/lucidrains/events{/privacy}
https://api.github.com/users/lucidrains/followers
https://api.github.com/users/lucidrains/following{/other_user}
https://api.github.com/users/lucidrains/gists{/gist_id}
https://github.com/lucidrains
lucidrains
MDQ6VXNlcjEwODY1Mw==
https://api.github.com/users/lucidrains/orgs
https://api.github.com/users/lucidrains/received_events
https://api.github.com/users/lucidrains/repos
https://api.github.com/users/lucidrains/starred{/owner}{/repo}
https://api.github.com/users/lucidrains/subscriptions
User
https://api.github.com/users/lucidrains
public
CONTRIBUTOR
Got around to this today, and so far so good, I'm able to download and load pg19 locally. However, I think there may be an issue with the dummy data, and testing in general.

The problem lies in the fact that each book from pg19 actually resides as its own text file in a google cloud folder that denotes the split, where the book id is the name of the text file. https://console.cloud.google.com/storage/browser/deepmind-gutenberg/train/ I don't believe there's anywhere else (even in the supplied metadata), where the mapping of id -> split can be found.

Therefore I end up making a network call `tf.io.gfile.listdir` to get all the files within each of the split directories. https://github.com/lucidrains/nlp/commit/adbacbd85decc80db2347d0882e7dab4faa6fd03#diff-cece8f166a85dd927caf574ba303d39bR78

Does this network call need to be eventually stubbed out for testing?
2020-06-22T22:57:06Z
https://github.com/huggingface/datasets/issues/274#issuecomment-647809544
https://api.github.com/repos/huggingface/datasets/issues/274
MDEyOklzc3VlQ29tbWVudDY0NzgwOTU0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/647809544/reactions
2020-06-22T22:57:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/647809544
https://avatars.githubusercontent.com/u/108653?v=4
https://api.github.com/users/lucidrains/events{/privacy}
https://api.github.com/users/lucidrains/followers
https://api.github.com/users/lucidrains/following{/other_user}
https://api.github.com/users/lucidrains/gists{/gist_id}
https://github.com/lucidrains
lucidrains
MDQ6VXNlcjEwODY1Mw==
https://api.github.com/users/lucidrains/orgs
https://api.github.com/users/lucidrains/received_events
https://api.github.com/users/lucidrains/repos
https://api.github.com/users/lucidrains/starred{/owner}{/repo}
https://api.github.com/users/lucidrains/subscriptions
User
https://api.github.com/users/lucidrains
public
CONTRIBUTOR
Ohh nevermind, I think I can use `download_custom` here with `listdir` as the custom function. Ok, I'll keep trying to make the dummy data work!
2020-06-23T20:24:53Z
https://github.com/huggingface/datasets/issues/274#issuecomment-648398301
https://api.github.com/repos/huggingface/datasets/issues/274
MDEyOklzc3VlQ29tbWVudDY0ODM5ODMwMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/648398301/reactions
2020-06-23T20:24:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/648398301
https://avatars.githubusercontent.com/u/108653?v=4
https://api.github.com/users/lucidrains/events{/privacy}
https://api.github.com/users/lucidrains/followers
https://api.github.com/users/lucidrains/following{/other_user}
https://api.github.com/users/lucidrains/gists{/gist_id}
https://github.com/lucidrains
lucidrains
MDQ6VXNlcjEwODY1Mw==
https://api.github.com/users/lucidrains/orgs
https://api.github.com/users/lucidrains/received_events
https://api.github.com/users/lucidrains/repos
https://api.github.com/users/lucidrains/starred{/owner}{/repo}
https://api.github.com/users/lucidrains/subscriptions
User
https://api.github.com/users/lucidrains
public
Hi, and thanks for all your open-sourced work, as always!

I was wondering if you would be open to adding PG-19 to your collection of datasets. https://github.com/deepmind/pg19 It is often used for benchmarking long-range language modeling.