Apply utf-8 encoding to all datasets
MEMBER
Not sure why the AWS test is failing - perhaps I made too many concurrent CI builds 😢. Can someone please rerun the CI to check the error is not on my end?
2020-08-06T21:55:31Z
https://github.com/huggingface/datasets/pull/481#issuecomment-670211178
https://api.github.com/repos/huggingface/datasets/issues/481
MDEyOklzc3VlQ29tbWVudDY3MDIxMTE3OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/670211178/reactions
2020-08-06T21:55:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/670211178
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
I pushed an improved docstring and the unit tests now pass, which suggests the previous failure on AWS was simply a timeout error. 

For some reason the docs are now failing to build, but does not seem related to my changes:
```
Warning, treated as error:
/home/circleci/nlp/src/nlp/dataset_dict.py:docstring of nlp.DatasetDict.filter:27:Inline interpreted text or phrase reference start-string without end-string.
make: *** [Makefile:20: html] Error 2
```

Any ideas what's going wrong?
2020-08-08T12:42:49Z
https://github.com/huggingface/datasets/pull/481#issuecomment-670924803
https://api.github.com/repos/huggingface/datasets/issues/481
MDEyOklzc3VlQ29tbWVudDY3MDkyNDgwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/670924803/reactions
2020-08-08T16:03:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/670924803
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
The build_doc fail has been fixed on master.
It was due to the latest update of sphinx that has some issues, so I pinned the previous version for now.
2020-08-12T09:32:42Z
https://github.com/huggingface/datasets/pull/481#issuecomment-672765659
https://api.github.com/repos/huggingface/datasets/issues/481
MDEyOklzc3VlQ29tbWVudDY3Mjc2NTY1OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/672765659/reactions
2020-08-12T09:32:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/672765659
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I noticed that you also changed the Apache Beam `open` to also use utf-8. However it doesn't have an `encoding` parameter.
Therefore you should ignore lines like

```python
beam.io.filesystems.FileSystems.open(filepath)
```

I guess you could add a rule to your regex to only include the `open` call that have a space right before it.
2020-08-12T09:36:57Z
https://github.com/huggingface/datasets/pull/481#issuecomment-672767454
https://api.github.com/repos/huggingface/datasets/issues/481
MDEyOklzc3VlQ29tbWVudDY3Mjc2NzQ1NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/672767454/reactions
2020-08-12T09:37:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/672767454
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Good catch @lhoestq! Your suggestion to match on `open(...)` with a whitespace was a great idea - it allowed me to simplify the regexp considerably 😄.

I fixed the Apache Beam false positives and also caught a few problems in `json.load()`, e.g.
```python
relation_name_map = json.load(open(rel_info), encoding='utf-8')
```

I've tested that the new regexp doesn't reintroduce these false positives, so I think the PR is ready for another review.
2020-08-12T20:12:09Z
https://github.com/huggingface/datasets/pull/481#issuecomment-673086552
https://api.github.com/repos/huggingface/datasets/issues/481
MDEyOklzc3VlQ29tbWVudDY3MzA4NjU1Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/673086552/reactions
2020-08-12T20:12:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/673086552
https://avatars.githubusercontent.com/u/26859204?v=4
https://api.github.com/users/lewtun/events{/privacy}
https://api.github.com/users/lewtun/followers
https://api.github.com/users/lewtun/following{/other_user}
https://api.github.com/users/lewtun/gists{/gist_id}
https://github.com/lewtun
lewtun
MDQ6VXNlcjI2ODU5MjA0
https://api.github.com/users/lewtun/orgs
https://api.github.com/users/lewtun/received_events
https://api.github.com/users/lewtun/repos
https://api.github.com/users/lewtun/starred{/owner}{/repo}
https://api.github.com/users/lewtun/subscriptions
User
https://api.github.com/users/lewtun
public
MEMBER
Ok to merge this @lhoestq ?
2020-08-20T07:48:08Z
https://github.com/huggingface/datasets/pull/481#issuecomment-677434911
https://api.github.com/repos/huggingface/datasets/issues/481
MDEyOklzc3VlQ29tbWVudDY3NzQzNDkxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/677434911/reactions
2020-08-20T07:48:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/677434911
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
## Description
This PR applies utf-8 encoding for all instances of `with open(...) as f` to all Python files in `datasets/`. As suggested by @thomwolf in #468 , we use regular expressions and the following function

```python
def apply_encoding_on_file_open(filepath: str):
    """Apply UTF-8 encoding for all instances where a non-binary file is opened."""
    
    with open(filepath, 'r', encoding='utf-8') as input_file:
        regexp = re.compile(r"(?!.*\b(?:encoding|rb|w|wb|w+|wb+|ab|ab+)\b)(?<=\s)(open)\((.*)\)")
        input_text = input_file.read()
        match = regexp.search(input_text)
        
        if match:
            output = regexp.sub(lambda m: m.group()[:-1]+', encoding="utf-8")', input_text)
            with open(filepath, 'w', encoding='utf-8') as output_file:
                output_file.write(output)
```

to perform the replacement. 

Note:

1. I excluded all _**binary files**_ from the search since it's possible some objects are opened for which the encoding doesn't make sense. Please correct me if I'm wrong and I'll tweak the regexp accordingly
2. There were two edge cases where the regexp failed (e.g. two `open` instances on a single line), but I decided to just fix these manually in the interest of time.
3. I only applied the replacement to files in `datasets/`. Let me know if this should be extended to other places like `metrics/`
4. I have implemented a unit test that should catch missing encodings in future CI runs

Closes #468  and possibly #347 