strange datasets from OSCAR corpus
MEMBER
Hi ! Thanks for reporting
cc @pjox is this an issue from the data ?

Anyway we should at least mention that OSCAR could contain such contents in the dataset card, you're totally right @jerryIsHere 
2021-05-31T18:55:33Z
https://github.com/huggingface/datasets/issues/2396#issuecomment-851636671
https://api.github.com/repos/huggingface/datasets/issues/2396
MDEyOklzc3VlQ29tbWVudDg1MTYzNjY3MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/851636671/reactions
2021-05-31T18:55:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/851636671
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi @jerryIsHere , sorry for the late response! Sadly this is normal, the problem comes form fasttext's classifier which we used to create the original corpus. In general the classifier is not really capable of properly recognizing Yue Chineese so the file ends un being just noise from Common Crawl. Some of these problems with OSCAR were already discussed [here](https://arxiv.org/pdf/2103.12028.pdf) but we are working on explicitly documenting the problems by language on our website. In fact, could please you open an issue on [our repo](https://github.com/oscar-corpus/oscar-website/issues) as well so that we can track it?
2021-06-17T13:42:15Z
https://github.com/huggingface/datasets/issues/2396#issuecomment-863250113
https://api.github.com/repos/huggingface/datasets/issues/2396
MDEyOklzc3VlQ29tbWVudDg2MzI1MDExMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/863250113/reactions
2021-06-17T13:42:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/863250113
https://avatars.githubusercontent.com/u/635220?v=4
https://api.github.com/users/pjox/events{/privacy}
https://api.github.com/users/pjox/followers
https://api.github.com/users/pjox/following{/other_user}
https://api.github.com/users/pjox/gists{/gist_id}
https://github.com/pjox
pjox
MDQ6VXNlcjYzNTIyMA==
https://api.github.com/users/pjox/orgs
https://api.github.com/users/pjox/received_events
https://api.github.com/users/pjox/repos
https://api.github.com/users/pjox/starred{/owner}{/repo}
https://api.github.com/users/pjox/subscriptions
User
https://api.github.com/users/pjox
public
![image](https://user-images.githubusercontent.com/50871412/119260850-4f876b80-bc07-11eb-8894-124302600643.png)
![image](https://user-images.githubusercontent.com/50871412/119260875-675eef80-bc07-11eb-9da4-ee27567054ac.png)
From the [official site ](https://oscar-corpus.com/), the Yue Chinese dataset should have 2.2KB data.
7 training instances is obviously not a right number.
As I can read Yue Chinese, I call tell the last instance is definitely not something that would appear on Common Crawl.
And even if you don't read Yue Chinese, you can tell the first six instance are problematic.
(It is embarrassing, as the 7 training instances look exactly like something from a pornographic novel or flitting messages in a chat of a dating app)
It might not be the problem of the huggingface/datasets implementation, because when I tried to download the dataset from the official site, I found out that the zip file is corrupted.
I will try to inform the host of OSCAR corpus later.
Awy a remake about this dataset in huggingface/datasets is needed, perhaps after the host of the dataset fixes the issue.

> Hi @jerryIsHere , sorry for the late response! Sadly this is normal, the problem comes form fasttext's classifier which we used to create the original corpus. In general the classifier is not really capable of properly recognizing Yue Chineese so the file ends un being just noise from Common Crawl. Some of these problems with OSCAR were already discussed [here](https://arxiv.org/pdf/2103.12028.pdf) but we are working on explicitly documenting the problems by language on our website. In fact, could please you open an issue on [our repo](https://github.com/oscar-corpus/oscar-website/issues) as well so that we can track it?

Thanks a lot, the new post is here:
https://github.com/oscar-corpus/oscar-website/issues/11