[arrow_dataset.py] add new filter function
MEMBER
I agree that a `.filter` method would be VERY useful and appreciated. I'm not a big fan of using `flatten_nested` as it completely breaks down the structure of the example and it may create bugs. Right now I think it may not work for nested structures. Maybe there's a simpler way that we've not figured out yet.
2020-05-28T17:09:38Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635479281
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTQ3OTI4MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/635479281/reactions
2020-05-28T17:09:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635479281
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Instead of flattening everything and rebuilding the example, maybe we can try to access the examples like this:
```python
for i in range(num_examples):
    example = map_nested(lambda x: x[i], batch)
    # ... then test to keep it or not
```
2020-05-28T17:17:50Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635483410
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTQ4MzQxMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/635483410/reactions
2020-05-28T17:17:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635483410
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> Instead of flattening everything and rebuilding the example, maybe we can try to access the examples like this:
> 
> ```python
> for i in range(num_examples):
>     example = map_nested(lambda x: x[i], batch)
>     # ... then test to keep it or not
> ```

Awesome I'll check it out :-) 
2020-05-28T17:48:58Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635499507
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTQ5OTUwNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/635499507/reactions
2020-05-28T17:48:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635499507
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
> Instead of flattening everything and rebuilding the example, maybe we can try to access the examples like this:
> 
> ```python
> for i in range(num_examples):
>     example = map_nested(lambda x: x[i], batch)
>     # ... then test to keep it or not
> ```

Awesome this function is definitely much nicer!
2020-05-28T18:07:44Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635509385
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTUwOTM4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/635509385/reactions
2020-05-28T18:07:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635509385
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
Actually I just realized that `map_nested` might not work either as it applies the function at the very last list of the structure. However we can imagine that a single example has also a list in its structure:
```python
one_example = {
    "title": "blabla",
    "paragraphs": [
        "p1", "p2", ...
    ]
}
```
2020-05-28T18:08:34Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635509806
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTUwOTgwNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/635509806/reactions
2020-05-28T18:12:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635509806
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
We'll probably have to take into account the `dset._data.schema` to extract the examples from the batch.
2020-05-28T18:12:27Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635511730
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTUxMTczMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/635511730/reactions
2020-05-28T18:12:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635511730
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> Actually I just realized that `map_nested` might not work either as it applies the function at the very last list of the structure. However we can imagine that a single example has also a list in its structure:
> 
> ```python
> one_example = {
>     "title": "blabla",
>     "paragraphs": [
>         "p1", "p2", ...
>     ]
> }
> ```

They both work. I'm using it on trivia_qa which is pretty nested. If you use the option `dict_only=True` I think it's fine.
2020-05-28T18:21:11Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635516277
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTUxNjI3Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/635516277/reactions
2020-05-28T18:21:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635516277
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
> We'll probably have to take into account the `dset._data.schema` to extract the examples from the batch.

Why? 
2020-05-28T18:27:48Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635519491
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTUxOTQ5MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/635519491/reactions
2020-05-28T18:27:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635519491
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
Actually it's fine. I guess this is going to be yet another thing to be unit-tested just to make sure ^^
2020-05-28T18:45:53Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635528598
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTUyODU5OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/635528598/reactions
2020-05-28T18:45:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635528598
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Yes, I will need to add tests and documentation! 
@thomwolf - would a function like this be ok? It abstracts `.map()` a bit which might be hard to understand. 
2020-05-28T20:24:47Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635587711
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTU4NzcxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/635587711/reactions
2020-05-28T20:24:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635587711
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
MEMBER
I tried on some datasets with nested structure and it works fine ! Great work :D 

2020-05-29T09:49:46Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635882914
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTg4MjkxNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/635882914/reactions
2020-05-29T09:49:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635882914
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Awesome :-), I will add documentation and some simple unittests
2020-05-29T10:07:22Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635890698
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTg5MDY5OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/635890698/reactions
2020-05-29T10:07:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635890698
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
CONTRIBUTOR
Ok merging!
2020-05-29T11:32:16Z
https://github.com/huggingface/datasets/pull/214#issuecomment-635923544
https://api.github.com/repos/huggingface/datasets/issues/214
MDEyOklzc3VlQ29tbWVudDYzNTkyMzU0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/635923544/reactions
2020-05-29T11:32:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/635923544
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
The `.map()` function is super useful, but can IMO a bit tedious when filtering certain examples.
I think, filtering out examples is also a very common operation people would like to perform on datasets.

This PR is a proposal to add a `.filter()` function in the same spirit than the `.map()` function.

Here is a sample code you can play around with:

```python
ds = nlp.load_dataset("squad", split="validation[:10%]")


def remove_under_idx_5(example, idx):
    return idx < 5


def only_keep_examples_with_is_in_context(example):
    return "is" in example["context"]


result_keep_only_first_5 = ds.filter(remove_under_idx_5, with_indices=True, load_from_cache_file=False)
result_keep_examples_with_is_in_context = ds.filter(only_keep_examples_with_is_in_context, load_from_cache_file=False)

print("Original number of examples: {}".format(len(ds)))
print("First five examples number of examples: {}".format(len(result_keep_only_first_5)))
print("Is in context examples number of examples: {}".format(len(result_keep_examples_with_is_in_context)))
```