Narrativeqa (with full text)
MEMBER
I took a look at the dummy data creation for this dataset.

Maybe it didn't work on your side might be because `master.zip` and `narrativeqa_full_text.zip` are supposed to be directories and not acutal zip files in the dummy data folder.

I managed to make it work with this `dummy_data.zip` file:
https://drive.google.com/file/d/1G9ZHAjelazNApbFI0ep2dnSAWklXgGMd/view?usp=sharing
2020-08-13T17:07:54Z
https://github.com/huggingface/datasets/pull/499#issuecomment-673597647
https://api.github.com/repos/huggingface/datasets/issues/499
MDEyOklzc3VlQ29tbWVudDY3MzU5NzY0Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/673597647/reactions
2020-08-13T17:07:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/673597647
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq Hmmm wasn't that. Must have been something else I missed.

Have committed your working version though now.
2020-08-13T17:18:29Z
https://github.com/huggingface/datasets/pull/499#issuecomment-673602936
https://api.github.com/repos/huggingface/datasets/issues/499
MDEyOklzc3VlQ29tbWVudDY3MzYwMjkzNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/673602936/reactions
2020-08-13T17:18:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/673602936
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
MEMBER
Ok thanks.
Could you rebase from master to fix the CI please ?
2020-08-14T13:16:32Z
https://github.com/huggingface/datasets/pull/499#issuecomment-674069697
https://api.github.com/repos/huggingface/datasets/issues/499
MDEyOklzc3VlQ29tbWVudDY3NDA2OTY5Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/674069697/reactions
2020-08-14T13:16:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/674069697
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Hi @ghomasHudson, did you get the chance to add the test split and regenerate the dataset_infos.json file  ?
2020-08-28T09:03:12Z
https://github.com/huggingface/datasets/pull/499#issuecomment-682414221
https://api.github.com/repos/huggingface/datasets/issues/499
MDEyOklzc3VlQ29tbWVudDY4MjQxNDIyMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/682414221/reactions
2020-08-28T09:03:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/682414221
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> Hi @ghomasHudson, did you get the chance to add the test split and regenerate the dataset_infos.json file ?

Have added the test set code but getting an OverflowError when trying to regen the dataset_infos.json:

---
OverflowError: There was an overflow in the <class 'pyarrow.lib.StructArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

---

2020-09-03T20:33:31Z
https://github.com/huggingface/datasets/pull/499#issuecomment-686750670
https://api.github.com/repos/huggingface/datasets/issues/499
MDEyOklzc3VlQ29tbWVudDY4Njc1MDY3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/686750670/reactions
2020-09-03T20:33:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/686750670
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
MEMBER
Thanks for reporting @ghomasHudson , I'll look into it
2020-09-07T11:39:08Z
https://github.com/huggingface/datasets/pull/499#issuecomment-688269322
https://api.github.com/repos/huggingface/datasets/issues/499
MDEyOklzc3VlQ29tbWVudDY4ODI2OTMyMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/688269322/reactions
2020-09-07T11:39:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688269322
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
It looks like it's an issue with Pyarrow.
By changing the `DEFAULT_MAX_BATCH_SIZE` to 1000 instead of 10 000 in `arrow_writer.py` I was able to run the command.

Basically it seems that is an Arrow StructArray has more than 1-2GB of data, then it shuffles some of its content.
I can't find any issue on Apache Arrow's JIRA about this problem. It will require more investigation.

Maybe we can simply automatically decrease the writer's batch size when this happens. We can just check if the arrow array is more than a certain amount of bytes. 
2020-09-09T17:18:59Z
https://github.com/huggingface/datasets/pull/499#issuecomment-689704329
https://api.github.com/repos/huggingface/datasets/issues/499
MDEyOklzc3VlQ29tbWVudDY4OTcwNDMyOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/689704329/reactions
2020-09-09T17:18:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/689704329
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq I've finally got round to regenerating the `dataset_infos.json` for this and adding all 3 splits. I've done this and updated for the new version of datasets.

The CI tests still aren't passing though (they pass on my machine). `test_load_dataset_narrativeqa` seems to fail but I have no idea how. Would appreciate if you have any ideas - would be great to finally finish this one!
2020-12-07T11:32:42Z
https://github.com/huggingface/datasets/pull/499#issuecomment-739861273
https://api.github.com/repos/huggingface/datasets/issues/499
MDEyOklzc3VlQ29tbWVudDczOTg2MTI3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/739861273/reactions
2020-12-07T13:04:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/739861273
https://avatars.githubusercontent.com/u/13795113?v=4
https://api.github.com/users/ghomasHudson/events{/privacy}
https://api.github.com/users/ghomasHudson/followers
https://api.github.com/users/ghomasHudson/following{/other_user}
https://api.github.com/users/ghomasHudson/gists{/gist_id}
https://github.com/ghomasHudson
ghomasHudson
MDQ6VXNlcjEzNzk1MTEz
https://api.github.com/users/ghomasHudson/orgs
https://api.github.com/users/ghomasHudson/received_events
https://api.github.com/users/ghomasHudson/repos
https://api.github.com/users/ghomasHudson/starred{/owner}{/repo}
https://api.github.com/users/ghomasHudson/subscriptions
User
https://api.github.com/users/ghomasHudson
public
MEMBER
The dummy data test fails, apparently it's because no examples are yielded for the dummy data.

Also it looks like the PR now show changes in many other files than the ones for NarrativeQA, could you create another branch and another PR please ?

Feel free to ping me on the new PR so we can fi the dummy data together
2020-12-09T09:19:36Z
https://github.com/huggingface/datasets/pull/499#issuecomment-741643791
https://api.github.com/repos/huggingface/datasets/issues/499
MDEyOklzc3VlQ29tbWVudDc0MTY0Mzc5MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/741643791/reactions
2020-12-09T09:19:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/741643791
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Following the uploading of the full text data in #309, I've added the full text to the narrativeqa dataset.

Few notes:
- Had some encoding issues using the default `open` so am using `open(encoding="latin-1"...` which seems to fix it. Looks fine.
-  Can't get the dummy data to work. Currently putting stuff at: 
    ```
    dummy
    |---- 0.0.0
            |- dummy_data.zip
                |-master.zip
                |   |- narrativeqa-master
                |       |- documents.csv
                |       |- qaps.csv
                |       |- third_party ......
                | 
                | - narrativeqa_full_text.zip
                |       | - 001.content
                |       | - ....
    ```
    Not sure what I'm messing up here (probably something obvious).