Implement Dataset from JSON and JSON Lines
MEMBER
Thanks @lhoestq. I was trying to follow @thomwolf suggestion about integrating that script but as `from_json` method...
> Note that I don't think this is necessary a breaking change, we can still keep the old scripts around

Do you think there is a better way of doing it?

I was trying to implement more or less the same logic as in the script, but I confess I assumed the target was in-memory only...
2021-02-25T14:19:25Z
https://github.com/huggingface/datasets/pull/1943#issuecomment-785928685
https://api.github.com/repos/huggingface/datasets/issues/1943
MDEyOklzc3VlQ29tbWVudDc4NTkyODY4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/785928685/reactions
2021-02-25T14:19:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/785928685
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Basically, I was trying to reimplement `Json(datasets.ArrowBasedBuilder)._generate_tables`, and no writing to arrow file (I assumed only in-memory usage). I started with the first "else" clause... 

I was planning to remove my `_cast_table_to_info_features` and use `paj.read_json(parse_options=...)` instead (like in the script).
2021-02-25T14:33:36Z
https://github.com/huggingface/datasets/pull/1943#issuecomment-785938439
https://api.github.com/repos/huggingface/datasets/issues/1943
MDEyOklzc3VlQ29tbWVudDc4NTkzODQzOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/785938439/reactions
2021-02-25T14:36:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/785938439
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
@lhoestq I am wondering why `keep_in_memory` has no effect for JSON...
2021-03-09T16:21:59Z
https://github.com/huggingface/datasets/pull/1943#issuecomment-794114825
https://api.github.com/repos/huggingface/datasets/issues/1943
MDEyOklzc3VlQ29tbWVudDc5NDExNDgyNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/794114825/reactions
2021-03-09T16:21:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/794114825
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
What's the issue exactly ? Apparently it's correctly passed to as_dataset so I don't find the issue
2021-03-09T16:24:07Z
https://github.com/huggingface/datasets/pull/1943#issuecomment-794117394
https://api.github.com/repos/huggingface/datasets/issues/1943
MDEyOklzc3VlQ29tbWVudDc5NDExNzM5NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/794117394/reactions
2021-03-09T16:24:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/794117394
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Nevermind @lhoestq, I found where the problem was in my code... I push!
2021-03-09T16:38:52Z
https://github.com/huggingface/datasets/pull/1943#issuecomment-794136966
https://api.github.com/repos/huggingface/datasets/issues/1943
MDEyOklzc3VlQ29tbWVudDc5NDEzNjk2Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/794136966/reactions
2021-03-09T16:38:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/794136966
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
<s>merging master into this branch should fix the CI issue :)</s>

Oops I didn't refresh the page sorry ^^'

Looks all good !
2021-03-15T09:17:38Z
https://github.com/huggingface/datasets/pull/1943#issuecomment-799255143
https://api.github.com/repos/huggingface/datasets/issues/1943
MDEyOklzc3VlQ29tbWVudDc5OTI1NTE0Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/799255143/reactions
2021-03-15T09:27:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/799255143
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Good job ! I think we can merge after the last changes regarding the error message and the docstring above :)
2021-03-16T10:56:01Z
https://github.com/huggingface/datasets/pull/1943#issuecomment-800158624
https://api.github.com/repos/huggingface/datasets/issues/1943
MDEyOklzc3VlQ29tbWVudDgwMDE1ODYyNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/800158624/reactions
2021-03-16T10:56:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800158624
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq Done! And I have also added some tests for the `field` parameter.
2021-03-16T12:02:05Z
https://github.com/huggingface/datasets/pull/1943#issuecomment-800201732
https://api.github.com/repos/huggingface/datasets/issues/1943
MDEyOklzc3VlQ29tbWVudDgwMDIwMTczMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/800201732/reactions
2021-03-16T12:02:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800201732
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Let me add some more tests for dict of lists JSON file, please.
2021-03-16T13:19:23Z
https://github.com/huggingface/datasets/pull/1943#issuecomment-800251205
https://api.github.com/repos/huggingface/datasets/issues/1943
MDEyOklzc3VlQ29tbWVudDgwMDI1MTIwNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/800251205/reactions
2021-03-16T13:19:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800251205
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
@lhoestq done! ;)
2021-03-16T14:05:34Z
https://github.com/huggingface/datasets/pull/1943#issuecomment-800286050
https://api.github.com/repos/huggingface/datasets/issues/1943
MDEyOklzc3VlQ29tbWVudDgwMDI4NjA1MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/800286050/reactions
2021-03-16T14:05:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800286050
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
We can merge. Additional work will be done in another PR. ;)
2021-03-17T10:56:44Z
https://github.com/huggingface/datasets/pull/1943#issuecomment-800988619
https://api.github.com/repos/huggingface/datasets/issues/1943
MDEyOklzc3VlQ29tbWVudDgwMDk4ODYxOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/800988619/reactions
2021-03-17T10:56:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800988619
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Implement `Dataset.from_jsonl`.