Specify split post processing + Add post processing resources downloading
MEMBER
I was using a hack in `wiki_dpr` to download the index from GCS even for the configurations without the embeddings.
However as GCS is something internal, I changed the logic to add a download step for indexes directly in the dataset script, using the `DownloadManager`.

This change was directly linked to the changes I did to take into account the split name in the post processing, so I included this change in this PR too.

To summarize:

Dataset builders can now implement
- `_post_processing_resources(split)`: return a dict `resource_name -> resource_file_name`. It defines the additional resources such as indexes or arrow files that you need in post processing
- `_download_post_processing_resources(split, resource_name, dl_manager))`: if some resources can be downloaded, you can use the download_manager to download them
- `_post_process(dataset, resources_path)`: (main function for post processing) given a dataset, you can apply dataset transforms or add indexes. For resources that have been downloaded, you can load them. For the others, you can generate and save them. The paths to load/save resources are in `resources_path` which is a dictionary `resource_name -> resource_path`

About the CI:
```
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::AWSDatasetTest::test_load_dataset_wiki_dpr
```
It fails because I changed the input of post processing functions (to include the split name)
2020-07-28T09:21:48Z
https://github.com/huggingface/datasets/pull/431#issuecomment-664913614
https://api.github.com/repos/huggingface/datasets/issues/431
MDEyOklzc3VlQ29tbWVudDY2NDkxMzYxNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/664913614/reactions
2020-07-28T09:21:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/664913614
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
I started to add metadata in the DatasetInfo.
Note that because there are new fields, **ALL the dataset_info[s].json generated after these changes won't be loadable from older versions of the lib**

Right now it looks like this:
```json
        "post_processing_resources_checksums": {
            "train": {
                "embeddings_index": {
                    "num_bytes": 30720045,
                    "checksum": "b04fb4f4f3ab83b9d1b9f6f9eb236f1c04a9fd61bef7cee16b12df8ac911766a"
                }
            }
        },
        "post_processing_size": 30720045,
```
2020-07-30T10:53:25Z
https://github.com/huggingface/datasets/pull/431#issuecomment-666295508
https://api.github.com/repos/huggingface/datasets/issues/431
MDEyOklzc3VlQ29tbWVudDY2NjI5NTUwOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/666295508/reactions
2020-07-30T10:54:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/666295508
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Good point. Should we anticipate already that we may add other fields in the future and change the code to support the addition of new fields without breaking backward compatibility in the future?
2020-07-30T11:05:43Z
https://github.com/huggingface/datasets/pull/431#issuecomment-666300805
https://api.github.com/repos/huggingface/datasets/issues/431
MDEyOklzc3VlQ29tbWVudDY2NjMwMDgwNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/666300805/reactions
2020-07-30T11:05:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/666300805
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
I added:
- post processing features (inside a PostProcessedInfo object)
- backward compatibility for dataset info
- post processing tests (as_dataset and download_and_prepare) for map (change features), select (change number of elements) and add_faiss_index (add indexes)
And I fixed a bug in `map` that I found thanks to the new tests

Now I just have to move `post_processing_resources_checksums` to  PostProcessedInfo as well and everything should be good :)
Edit: done
2020-07-30T15:19:57Z
https://github.com/huggingface/datasets/pull/431#issuecomment-666448915
https://api.github.com/repos/huggingface/datasets/issues/431
MDEyOklzc3VlQ29tbWVudDY2NjQ0ODkxNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/666448915/reactions
2020-07-30T17:38:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/666448915
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Previously if you tried to do

```python
from nlp import load_dataset
wiki = load_dataset("wiki_dpr", "psgs_w100_with_nq_embeddings", split="train[:100]", with_index=True)
```
Then you'd get an error `Index size should match Dataset size...`
This was because it was trying to use the full index (21M elements).

To fix that I made it so post processing resources can be named according to the split.

I'm going to add tests on post processing too.

Note that the CI will fail as I added a new argument in `_post_processing_resources`: the AWS version of wiki_dpr fails, and there's also an error telling that it is not synced (it'll be synced once it's merged):
```
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::AWSDatasetTest::test_load_dataset_wiki_dpr
FAILED tests/test_hf_gcp.py::TestDatasetSynced::test_script_synced_with_s3_wiki_dpr
```

EDIT: I did a change to ignore the script hash to locate the arrow files on GCS, so I removed the sync test. It was there just because of the hash logic for files on GCS