News_commentary Dataset Translation Pairs are of Incorrect Language Specified Pairs
COLLABORATOR
These ranges seem to be valid English. Closing.
2022-10-05T17:13:49Z
https://github.com/huggingface/datasets/issues/2398#issuecomment-1268708857
https://api.github.com/repos/huggingface/datasets/issues/2398
IC_kwDODunzps5LnvX5
https://api.github.com/repos/huggingface/datasets/issues/comments/1268708857/reactions
2022-10-05T17:13:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1268708857
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
I used load_dataset to load the news_commentary dataset for "ar-en" translation pairs but found translations from Arabic to Hindi.  

```
train_ds = load_dataset("news_commentary", "ar-en", split='train[:98%]')
val_ds = load_dataset("news_commentary", "ar-en", split='train[98%:]')

# filtering out examples that are not ar-en translations but ar-hi
val_ds = val_ds.filter(lambda example, indice: indice not in chain(range(1312,1327) ,range(1384,1399), range(1030,1042)), with_indices=True)
```

* I'm fairly new to using datasets so I might be doing something wrong