Segmentation fault when loading local JSON dataset as of #372
NONE
I've seen this sort of thing before -- it might help to delete the directory -- I've also noticed that there is an error with the json Dataloader for any data I've tried to load. I've replaced it with this, which skips over the data feature population step:


```python
import os

import pyarrow.json as paj

import nlp as hf_nlp

from nlp import DatasetInfo, BuilderConfig, SplitGenerator, Split, utils
from nlp.arrow_writer import ArrowWriter


class JSONDatasetBuilder(hf_nlp.ArrowBasedBuilder):
    BUILDER_CONFIG_CLASS = BuilderConfig

    def _info(self):
        return DatasetInfo()

    def _split_generators(self, dl_manager):
        """ We handle string, list and dicts in datafiles
        """
        if isinstance(self.config.data_files, (str, list, tuple)):
            files = self.config.data_files
            if isinstance(files, str):
                files = [files]
            return [SplitGenerator(name=Split.TRAIN, gen_kwargs={"files": files})]
        splits = []
        for split_name in [Split.TRAIN, Split.VALIDATION, Split.TEST]:
            if split_name in self.config.data_files:
                files = self.config.data_files[split_name]
                if isinstance(files, str):
                    files = [files]
                splits.append(SplitGenerator(name=split_name, gen_kwargs={"files": files}))
        return splits

    def _prepare_split(self, split_generator):
        fname = "{}-{}.arrow".format(self.name, split_generator.name)
        fpath = os.path.join(self._cache_dir, fname)

        writer = ArrowWriter(path=fpath)

        generator = self._generate_tables(**split_generator.gen_kwargs)
        for key, table in utils.tqdm(generator, unit=" tables", leave=False):
            writer.write_table(table)
        num_examples, num_bytes = writer.finalize()

        split_generator.split_info.num_examples = num_examples
        split_generator.split_info.num_bytes = num_bytes

    def _generate_tables(self, files):
        for i, file in enumerate(files):
            pa_table = paj.read_json(
                file
            )
            yield i, pa_table

```
2020-07-10T20:43:21Z
https://github.com/huggingface/datasets/issues/373#issuecomment-656881212
https://api.github.com/repos/huggingface/datasets/issues/373
MDEyOklzc3VlQ29tbWVudDY1Njg4MTIxMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/656881212/reactions
2020-07-10T20:43:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656881212
https://avatars.githubusercontent.com/u/2000204?v=4
https://api.github.com/users/timothyjlaurent/events{/privacy}
https://api.github.com/users/timothyjlaurent/followers
https://api.github.com/users/timothyjlaurent/following{/other_user}
https://api.github.com/users/timothyjlaurent/gists{/gist_id}
https://github.com/timothyjlaurent
timothyjlaurent
MDQ6VXNlcjIwMDAyMDQ=
https://api.github.com/users/timothyjlaurent/orgs
https://api.github.com/users/timothyjlaurent/received_events
https://api.github.com/users/timothyjlaurent/repos
https://api.github.com/users/timothyjlaurent/starred{/owner}{/repo}
https://api.github.com/users/timothyjlaurent/subscriptions
User
https://api.github.com/users/timothyjlaurent
public
CONTRIBUTOR
Yes, deleting the directory solves the error whenever I try to rerun.

By replacing the json-loader, you mean the cached file in my `site-packages` directory? e.g. `/home/XXX/.cache/lib/python3.7/site-packages/nlp/datasets/json/(...)/json.py` 

When I was testing this out before the #372 PR was merged I had issues installing it properly locally. Since the `json.py` script was downloaded instead of actually using the one provided in the local install. Manually updating that file seemed to solve it, but it didn't seem like a proper solution. Especially when having to run this on a remote compute cluster with no access to that directory.
2020-07-10T21:07:57Z
https://github.com/huggingface/datasets/issues/373#issuecomment-656891373
https://api.github.com/repos/huggingface/datasets/issues/373
MDEyOklzc3VlQ29tbWVudDY1Njg5MTM3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/656891373/reactions
2020-07-10T21:09:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656891373
https://avatars.githubusercontent.com/u/24683907?v=4
https://api.github.com/users/vegarab/events{/privacy}
https://api.github.com/users/vegarab/followers
https://api.github.com/users/vegarab/following{/other_user}
https://api.github.com/users/vegarab/gists{/gist_id}
https://github.com/vegarab
vegarab
MDQ6VXNlcjI0NjgzOTA3
https://api.github.com/users/vegarab/orgs
https://api.github.com/users/vegarab/received_events
https://api.github.com/users/vegarab/repos
https://api.github.com/users/vegarab/starred{/owner}{/repo}
https://api.github.com/users/vegarab/subscriptions
User
https://api.github.com/users/vegarab
public
MEMBER
I see, diving in the JSON file for SQuAD it's a pretty complex structure.

The best solution for you, if you have a dataset really similar to SQuAD would be to copy and modify the SQuAD data processing script. We will probably add soon an option to be able to specify file path to use instead of the automatic URL encoded in the script but in the meantime you can:
- copy the [squad script](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py)  in a new script for your dataset
- in the new script replace [these `urls_to_download `](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py#L99-L102) by `urls_to_download=self.config.data_files`
- load the dataset with `dataset = load_dataset('path/to/your/new/script', data_files={nlp.Split.TRAIN: "./datasets/train-v2.0.json"})`

This way you can reuse all the processing logic of the SQuAD loading script.
2020-07-10T21:39:15Z
https://github.com/huggingface/datasets/issues/373#issuecomment-656903437
https://api.github.com/repos/huggingface/datasets/issues/373
MDEyOklzc3VlQ29tbWVudDY1NjkwMzQzNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/656903437/reactions
2020-07-10T21:39:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656903437
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
This seems like a more sensible solution! Thanks, @thomwolf. It's been a little daunting to understand what these scripts actually do, due to the level of abstraction and central documentation.

Am I correct in assuming that the `_generate_examples()` function is the actual procedure for how the data is loaded from file? Meaning that essentially with a file containing another format, that is the only function that requires re-implementation? I'm working with a lot of datasets that, due to licensing and privacy, cannot be published. As this library is so neatly integrated with the transformers library and gives easy access to public sets such as SQUAD and increased performance, it is very neat to be able to load my private sets as well. As of now, I have just been working on scripts for translating all my data into the SQUAD-format before using the json script, but I see that it might not be necessary after all. 
2020-07-10T22:02:09Z
https://github.com/huggingface/datasets/issues/373#issuecomment-656911701
https://api.github.com/repos/huggingface/datasets/issues/373
MDEyOklzc3VlQ29tbWVudDY1NjkxMTcwMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/656911701/reactions
2020-07-10T22:02:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656911701
https://avatars.githubusercontent.com/u/24683907?v=4
https://api.github.com/users/vegarab/events{/privacy}
https://api.github.com/users/vegarab/followers
https://api.github.com/users/vegarab/following{/other_user}
https://api.github.com/users/vegarab/gists{/gist_id}
https://github.com/vegarab
vegarab
MDQ6VXNlcjI0NjgzOTA3
https://api.github.com/users/vegarab/orgs
https://api.github.com/users/vegarab/received_events
https://api.github.com/users/vegarab/repos
https://api.github.com/users/vegarab/starred{/owner}{/repo}
https://api.github.com/users/vegarab/subscriptions
User
https://api.github.com/users/vegarab
public
MEMBER
Yes `_generate_examples()` is the main entry point. If you change the shape of the returned dictionary you also need to update the `features` in the `_info`.

I'm currently writing the doc so it should be easier soon to use the library and know how to add your datasets.

2020-07-11T00:32:08Z
https://github.com/huggingface/datasets/issues/373#issuecomment-656949898
https://api.github.com/repos/huggingface/datasets/issues/373
MDEyOklzc3VlQ29tbWVudDY1Njk0OTg5OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/656949898/reactions
2020-07-11T00:32:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/656949898
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Could you try to update pyarrow to >=0.17.0 @vegarab ?
I don't have any segmentation fault with my version of pyarrow (0.17.1)

I tested with
```python
import nlp
s = nlp.load_dataset("json", data_files="train-v2.0.json", field="data", split="train")
s[0]
# {'title': 'Normans', 'paragraphs': [{'qas': [{'question': 'In what country is Normandy located?', 'id':...
```
2020-07-16T16:09:25Z
https://github.com/huggingface/datasets/issues/373#issuecomment-659513615
https://api.github.com/repos/huggingface/datasets/issues/373
MDEyOklzc3VlQ29tbWVudDY1OTUxMzYxNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/659513615/reactions
2020-07-16T16:09:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/659513615
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Also if you want to have your own dataset script, we now have a new documentation !
See here:
https://huggingface.co/nlp/add_dataset.html
2020-07-16T16:11:08Z
https://github.com/huggingface/datasets/issues/373#issuecomment-659514632
https://api.github.com/repos/huggingface/datasets/issues/373
MDEyOklzc3VlQ29tbWVudDY1OTUxNDYzMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/659514632/reactions
2020-07-16T16:11:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/659514632
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq 
For some reason, I am not able to reproduce the segmentation fault, on pyarrow==0.16.0. Using the exact same environment and file.

Anyhow, I discovered that pyarrow>=0.17.0 is required to read in a JSON file where the pandas structs contain lists. Otherwise, pyarrow complains when attempting to cast the struct:
```py
import nlp
>>> s = nlp.load_dataset("json", data_files="datasets/train-v2.0.json", field="data", split="train")
Using custom data configuration default
>>> s[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 558, in __getitem__
    format_kwargs=self._format_kwargs,
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 498, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pandas().to_dict("list"))
  File "pyarrow/array.pxi", line 559, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 1367, in pyarrow.lib.Table._to_pandas
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 766, in table_to_blockmanager
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 1101, in _table_to_blocks
    list(extension_columns.keys()))
  File "pyarrow/table.pxi", line 881, in pyarrow.lib.table_to_blocks
  File "pyarrow/error.pxi", line 105, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Not implemented type for Arrow list to pandas: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>
>>> s
Dataset(schema: {'title': 'string', 'paragraphs': 'list<item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>>'}, num_rows: 35)
```

Upgrading to >=0.17.0 provides the same dataset structure, but accessing the records is possible without the same exception. 


2020-07-20T10:32:15Z
https://github.com/huggingface/datasets/issues/373#issuecomment-660946252
https://api.github.com/repos/huggingface/datasets/issues/373
MDEyOklzc3VlQ29tbWVudDY2MDk0NjI1Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/660946252/reactions
2020-07-20T10:32:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/660946252
https://avatars.githubusercontent.com/u/24683907?v=4
https://api.github.com/users/vegarab/events{/privacy}
https://api.github.com/users/vegarab/followers
https://api.github.com/users/vegarab/following{/other_user}
https://api.github.com/users/vegarab/gists{/gist_id}
https://github.com/vegarab
vegarab
MDQ6VXNlcjI0NjgzOTA3
https://api.github.com/users/vegarab/orgs
https://api.github.com/users/vegarab/received_events
https://api.github.com/users/vegarab/repos
https://api.github.com/users/vegarab/starred{/owner}{/repo}
https://api.github.com/users/vegarab/subscriptions
User
https://api.github.com/users/vegarab
public
CONTRIBUTOR
Very happy to see some extended documentation! 
2020-07-20T10:33:31Z
https://github.com/huggingface/datasets/issues/373#issuecomment-660946817
https://api.github.com/repos/huggingface/datasets/issues/373
MDEyOklzc3VlQ29tbWVudDY2MDk0NjgxNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/660946817/reactions
2020-07-20T10:33:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/660946817
https://avatars.githubusercontent.com/u/24683907?v=4
https://api.github.com/users/vegarab/events{/privacy}
https://api.github.com/users/vegarab/followers
https://api.github.com/users/vegarab/following{/other_user}
https://api.github.com/users/vegarab/gists{/gist_id}
https://github.com/vegarab
vegarab
MDQ6VXNlcjI0NjgzOTA3
https://api.github.com/users/vegarab/orgs
https://api.github.com/users/vegarab/received_events
https://api.github.com/users/vegarab/repos
https://api.github.com/users/vegarab/starred{/owner}{/repo}
https://api.github.com/users/vegarab/subscriptions
User
https://api.github.com/users/vegarab
public
CONTRIBUTOR
#376 seems to be reporting the same issue as mentioned above. 
2020-07-20T14:52:25Z
https://github.com/huggingface/datasets/issues/373#issuecomment-661089266
https://api.github.com/repos/huggingface/datasets/issues/373
MDEyOklzc3VlQ29tbWVudDY2MTA4OTI2Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/661089266/reactions
2020-07-20T14:52:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/661089266
https://avatars.githubusercontent.com/u/24683907?v=4
https://api.github.com/users/vegarab/events{/privacy}
https://api.github.com/users/vegarab/followers
https://api.github.com/users/vegarab/following{/other_user}
https://api.github.com/users/vegarab/gists{/gist_id}
https://github.com/vegarab
vegarab
MDQ6VXNlcjI0NjgzOTA3
https://api.github.com/users/vegarab/orgs
https://api.github.com/users/vegarab/received_events
https://api.github.com/users/vegarab/repos
https://api.github.com/users/vegarab/starred{/owner}{/repo}
https://api.github.com/users/vegarab/subscriptions
User
https://api.github.com/users/vegarab
public
NONE
This issue helped me a lot, thanks.
Hope this issue will be fixed soon.
2020-12-15T07:27:20Z
https://github.com/huggingface/datasets/issues/373#issuecomment-745108278
https://api.github.com/repos/huggingface/datasets/issues/373
MDEyOklzc3VlQ29tbWVudDc0NTEwODI3OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/745108278/reactions
2020-12-15T07:27:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/745108278
https://avatars.githubusercontent.com/u/8568937?v=4
https://api.github.com/users/HarborZeng/events{/privacy}
https://api.github.com/users/HarborZeng/followers
https://api.github.com/users/HarborZeng/following{/other_user}
https://api.github.com/users/HarborZeng/gists{/gist_id}
https://github.com/HarborZeng
HarborZeng
MDQ6VXNlcjg1Njg5Mzc=
https://api.github.com/users/HarborZeng/orgs
https://api.github.com/users/HarborZeng/received_events
https://api.github.com/users/HarborZeng/repos
https://api.github.com/users/HarborZeng/starred{/owner}{/repo}
https://api.github.com/users/HarborZeng/subscriptions
User
https://api.github.com/users/HarborZeng
public
The last issue was closed (#369) once the #372 update was merged. However, I'm still not able to load a SQuAD formatted JSON file. Instead of the previously recorded pyarrow error, I now get a segmentation fault. 

```
dataset = nlp.load_dataset('json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
```
causes
```
Using custom data configuration default
Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/XXX/.cache/huggingface/datasets/json/default/0.0.0...
0 tables [00:00, ? tables/s]Segmentation fault (core dumped)
```
where `./datasets/train-v2.0.json` is downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/.
This is consistent with other SQuAD-formatted JSON files.

When attempting to load the dataset again, I get the following:
```
Using custom data configuration default
Traceback (most recent call last):
  File "dataloader.py", line 6, in <module>
    'json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 382, in download_and_prepare
    with incomplete_dir(self._cache_dir) as tmp_data_dir:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 368, in incomplete_dir
    os.makedirs(tmp_dir)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/XXX/.cache/huggingface/datasets/json/default/0.0.0.incomplete'
```

(Not sure if you wanted this in the previous issue #369 or not as it was closed.)