'cp950' codec error from load_dataset('xtreme', 'tydiqa')
CONTRIBUTOR
This is probably a Windows issue, we need to specify the encoding when `load_dataset()` reads the original CSV file.
Try to find the `open()` statement called by `load_dataset()` and add an `encoding='utf-8'` parameter.
See issues #242 and #307 
2020-07-07T21:38:36Z
https://github.com/huggingface/datasets/issues/347#issuecomment-655149685
https://api.github.com/repos/huggingface/datasets/issues/347
MDEyOklzc3VlQ29tbWVudDY1NTE0OTY4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/655149685/reactions
2020-07-07T21:38:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/655149685
https://avatars.githubusercontent.com/u/15801338?v=4
https://api.github.com/users/patpizio/events{/privacy}
https://api.github.com/users/patpizio/followers
https://api.github.com/users/patpizio/following{/other_user}
https://api.github.com/users/patpizio/gists{/gist_id}
https://github.com/patpizio
patpizio
MDQ6VXNlcjE1ODAxMzM4
https://api.github.com/users/patpizio/orgs
https://api.github.com/users/patpizio/received_events
https://api.github.com/users/patpizio/repos
https://api.github.com/users/patpizio/starred{/owner}{/repo}
https://api.github.com/users/patpizio/subscriptions
User
https://api.github.com/users/patpizio
public
MEMBER
It should be in `xtreme.py:L755`:
```python
        if self.config.name == "tydiqa" or self.config.name.startswith("MLQA") or self.config.name == "SQuAD":
            with open(filepath) as f:
                data = json.load(f)
```

Could you try to add the encoding parameter:
```python
open(filepath, encoding='utf-8')
```
2020-07-09T08:49:07Z
https://github.com/huggingface/datasets/issues/347#issuecomment-655998861
https://api.github.com/repos/huggingface/datasets/issues/347
MDEyOklzc3VlQ29tbWVudDY1NTk5ODg2MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/655998861/reactions
2020-07-09T08:49:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/655998861
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Hello @jerryIsHere :) Did it work ?
If so we may change the dataset script to force the utf-8 encoding
2020-07-13T10:01:04Z
https://github.com/huggingface/datasets/issues/347#issuecomment-657460703
https://api.github.com/repos/huggingface/datasets/issues/347
MDEyOklzc3VlQ29tbWVudDY1NzQ2MDcwMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/657460703/reactions
2020-07-13T10:01:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/657460703
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq sorry for being that late, I found 4 copy of xtreme.py. I did the changes as what has been told to all of them.
The problem is not solved
2020-07-13T10:35:40Z
https://github.com/huggingface/datasets/issues/347#issuecomment-657480130
https://api.github.com/repos/huggingface/datasets/issues/347
MDEyOklzc3VlQ29tbWVudDY1NzQ4MDEzMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/657480130/reactions
2020-07-13T10:35:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/657480130
https://avatars.githubusercontent.com/u/50871412?v=4
https://api.github.com/users/cosmeowpawlitan/events{/privacy}
https://api.github.com/users/cosmeowpawlitan/followers
https://api.github.com/users/cosmeowpawlitan/following{/other_user}
https://api.github.com/users/cosmeowpawlitan/gists{/gist_id}
https://github.com/cosmeowpawlitan
cosmeowpawlitan
MDQ6VXNlcjUwODcxNDEy
https://api.github.com/users/cosmeowpawlitan/orgs
https://api.github.com/users/cosmeowpawlitan/received_events
https://api.github.com/users/cosmeowpawlitan/repos
https://api.github.com/users/cosmeowpawlitan/starred{/owner}{/repo}
https://api.github.com/users/cosmeowpawlitan/subscriptions
User
https://api.github.com/users/cosmeowpawlitan
public
MEMBER
Could you provide a better error message so that we can make sure it comes from the opening of the `tydiqa`'s json files ?

2020-07-16T13:25:17Z
https://github.com/huggingface/datasets/issues/347#issuecomment-659409784
https://api.github.com/repos/huggingface/datasets/issues/347
MDEyOklzc3VlQ29tbWVudDY1OTQwOTc4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/659409784/reactions
2020-07-16T13:25:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/659409784
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
@lhoestq 
The error message is same as before:
Exception has occurred: UnicodeDecodeError
'cp950' codec can't decode byte 0xe2 in position 111: illegal multibyte sequence
  File "D:\python\test\test.py", line 3, in <module>
    dataset = load_dataset('xtreme', 'tydiqa')

![image](https://user-images.githubusercontent.com/50871412/87748794-7c216880-c829-11ea-94f0-7caeacb4d865.png)

I said that I found 4 copy of xtreme.py and add the 「, encoding='utf-8'」 parameter to the open() function
these python script was found under this directory
C:\Users\USER\AppData\Local\Programs\Python\Python37\Lib\site-packages\nlp\datasets\xtreme

2020-07-17T04:31:52Z
https://github.com/huggingface/datasets/issues/347#issuecomment-659841614
https://api.github.com/repos/huggingface/datasets/issues/347
MDEyOklzc3VlQ29tbWVudDY1OTg0MTYxNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/659841614/reactions
2020-07-17T04:31:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/659841614
https://avatars.githubusercontent.com/u/50871412?v=4
https://api.github.com/users/cosmeowpawlitan/events{/privacy}
https://api.github.com/users/cosmeowpawlitan/followers
https://api.github.com/users/cosmeowpawlitan/following{/other_user}
https://api.github.com/users/cosmeowpawlitan/gists{/gist_id}
https://github.com/cosmeowpawlitan
cosmeowpawlitan
MDQ6VXNlcjUwODcxNDEy
https://api.github.com/users/cosmeowpawlitan/orgs
https://api.github.com/users/cosmeowpawlitan/received_events
https://api.github.com/users/cosmeowpawlitan/repos
https://api.github.com/users/cosmeowpawlitan/starred{/owner}{/repo}
https://api.github.com/users/cosmeowpawlitan/subscriptions
User
https://api.github.com/users/cosmeowpawlitan
public
CONTRIBUTOR
Hi there !
I encountered the same issue with the IMDB dataset on windows. It threw an error about charmap not being able to decode a symbol during the first time I tried to download it. I checked on a remote linux machine I have, and it can't be reproduced.
I added ```encoding='UTF-8'``` to both lines that have ```open``` in ```imdb.py```  (108 and 114) and it worked for me.
Thank you !
2020-07-21T13:10:42Z
https://github.com/huggingface/datasets/issues/347#issuecomment-661851838
https://api.github.com/repos/huggingface/datasets/issues/347
MDEyOklzc3VlQ29tbWVudDY2MTg1MTgzOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/661851838/reactions
2020-07-21T13:10:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/661851838
https://avatars.githubusercontent.com/u/25091538?v=4
https://api.github.com/users/ghazi-f/events{/privacy}
https://api.github.com/users/ghazi-f/followers
https://api.github.com/users/ghazi-f/following{/other_user}
https://api.github.com/users/ghazi-f/gists{/gist_id}
https://github.com/ghazi-f
ghazi-f
MDQ6VXNlcjI1MDkxNTM4
https://api.github.com/users/ghazi-f/orgs
https://api.github.com/users/ghazi-f/received_events
https://api.github.com/users/ghazi-f/repos
https://api.github.com/users/ghazi-f/starred{/owner}{/repo}
https://api.github.com/users/ghazi-f/subscriptions
User
https://api.github.com/users/ghazi-f
public
MEMBER
> Hi there !
> I encountered the same issue with the IMDB dataset on windows. It threw an error about charmap not being able to decode a symbol during the first time I tried to download it. I checked on a remote linux machine I have, and it can't be reproduced.
> I added `encoding='UTF-8'` to both lines that have `open` in `imdb.py` (108 and 114) and it worked for me.
> Thank you !

Hello !
Glad you managed to fix this issue on your side.
Do you mind opening a PR for IMDB ?
2020-07-21T13:12:51Z
https://github.com/huggingface/datasets/issues/347#issuecomment-661853035
https://api.github.com/repos/huggingface/datasets/issues/347
MDEyOklzc3VlQ29tbWVudDY2MTg1MzAzNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/661853035/reactions
2020-07-21T13:13:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/661853035
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> This is probably a Windows issue, we need to specify the encoding when `load_dataset()` reads the original CSV file.
> Try to find the `open()` statement called by `load_dataset()` and add an `encoding='utf-8'` parameter.
> See issues #242 and #307

Sorry for not responding for about a month.
I have just found that it is necessary to change / add the environment variable as what was told in #242.
Everything works after I add the new environment variable and restart my PC.

I think the encoding issue for windows isn't limited to the open() function call specific to few dataset, but actually in the entire library, depends on the machine / os you use.
2020-08-23T07:57:17Z
https://github.com/huggingface/datasets/issues/347#issuecomment-678742731
https://api.github.com/repos/huggingface/datasets/issues/347
MDEyOklzc3VlQ29tbWVudDY3ODc0MjczMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/678742731/reactions
2020-08-23T07:57:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/678742731
https://avatars.githubusercontent.com/u/50871412?v=4
https://api.github.com/users/cosmeowpawlitan/events{/privacy}
https://api.github.com/users/cosmeowpawlitan/followers
https://api.github.com/users/cosmeowpawlitan/following{/other_user}
https://api.github.com/users/cosmeowpawlitan/gists{/gist_id}
https://github.com/cosmeowpawlitan
cosmeowpawlitan
MDQ6VXNlcjUwODcxNDEy
https://api.github.com/users/cosmeowpawlitan/orgs
https://api.github.com/users/cosmeowpawlitan/received_events
https://api.github.com/users/cosmeowpawlitan/repos
https://api.github.com/users/cosmeowpawlitan/starred{/owner}{/repo}
https://api.github.com/users/cosmeowpawlitan/subscriptions
User
https://api.github.com/users/cosmeowpawlitan
public
MEMBER
Since #481 we shouldn't have other issues with encodings as they need to be set to "utf-8" be default.

Closing this one, but feel free to re-open if you gave other questions
2020-09-07T14:51:45Z
https://github.com/huggingface/datasets/issues/347#issuecomment-688374044
https://api.github.com/repos/huggingface/datasets/issues/347
MDEyOklzc3VlQ29tbWVudDY4ODM3NDA0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/688374044/reactions
2020-09-07T14:51:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688374044
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly
