Make tests run faster
MEMBER
LOL, I was also working on something similar 😅. I'm gonna have a look!!!
2021-04-26T16:07:11Z
https://github.com/huggingface/datasets/pull/2266#issuecomment-826960121
https://api.github.com/repos/huggingface/datasets/issues/2266
MDEyOklzc3VlQ29tbWVudDgyNjk2MDEyMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/826960121/reactions
2021-04-26T16:07:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/826960121
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Sorry I didn't know you were also working on it ^^'
And yes I 100% agree with you on the points you mentioned. We should definitely improve the coverage. It would be nice to have a clearer separation to know which tests in the suite are unit tests and which ones are integration tests

2021-04-26T17:49:27Z
https://github.com/huggingface/datasets/pull/2266#issuecomment-827029935
https://api.github.com/repos/huggingface/datasets/issues/2266
MDEyOklzc3VlQ29tbWVudDgyNzAyOTkzNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/827029935/reactions
2021-04-26T17:49:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/827029935
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Never mind: we both noticed tests can be improved. More PRs to come... 😉 

According to the literature, unit tests are those that test a behavior unit, isolated from the other components and must be very fast: for me, this last requirement implies that they must be performed completely _in memory_.

As opposed, integration tests are those which also test interactions with _external_ components, like web services, databases, file system, etc.

The problem I see is that our code is still too coupled and it is difficult to isolate components for testing. Therefore, I would suggest acting iteratively, by refactoring to decouple components and then implement unit tests for each component in isolation.
2021-04-27T09:10:35Z
https://github.com/huggingface/datasets/pull/2266#issuecomment-827450038
https://api.github.com/repos/huggingface/datasets/issues/2266
MDEyOklzc3VlQ29tbWVudDgyNzQ1MDAzOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/827450038/reactions
2021-04-27T09:13:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/827450038
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
From 7min to 2min to run pytest.
Ideally we should keep the whole CI run time below 10min.

In this PR I removed the remote tests that were never used.
I also replaced nested parametrized tests with unit tests.
This makes me think that we could still add more high level tests to check for a few combinations of parameters (but not all of them since there are too many of them).
Let me know what you think

Finally in another PR we can also separate in two circleci jobs:
- the tests of the code code of the lib
- the tests of the all the dataset/metric scripts.