caching in map causes same result to be returned for train, validation and test
CONTRIBUTOR
Hi @dpressel, 

thanks for posting your issue! Can you maybe add a complete code snippet that we can copy paste to reproduce the error? For example, I'm not sure where the variable `train_set` comes from in your code and it seems like you are loading multiple datasets at once?  
2020-05-18T21:19:52Z
https://github.com/huggingface/datasets/issues/160#issuecomment-630441107
https://api.github.com/repos/huggingface/datasets/issues/160
MDEyOklzc3VlQ29tbWVudDYzMDQ0MTEwNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/630441107/reactions
2020-05-18T21:19:52Z
https://api.github.com/repos/huggingface/datasets/issues/comments/630441107
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
NONE
Hi, the full example was listed in the PR above, but here is the exact link:

https://github.com/dpressel/mead-baseline/blob/3c1aa3ca062cb23f303ca98ac40b6652b37ee971/api-examples/layers-classify-hf-datasets.py

The problem is coming from
```
            if cache_file_name is None:
                # we create a unique hash from the function, current dataset file and the mapping args
                cache_kwargs = {
                    "with_indices": with_indices,
                    "batched": batched,
                    "batch_size": batch_size,
                    "remove_columns": remove_columns,
                    "keep_in_memory": keep_in_memory,
                    "load_from_cache_file": load_from_cache_file,
                    "cache_file_name": cache_file_name,
                    "writer_batch_size": writer_batch_size,
                    "arrow_schema": arrow_schema,
                    "disable_nullable": disable_nullable,
                }
                cache_file_name = self._get_cache_file_path(function, cache_kwargs)
```
The cached value is always the same, but I was able to change that by just renaming the function each time which seems to fix the issue.
2020-05-18T21:23:15Z
https://github.com/huggingface/datasets/issues/160#issuecomment-630442673
https://api.github.com/repos/huggingface/datasets/issues/160
MDEyOklzc3VlQ29tbWVudDYzMDQ0MjY3Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/630442673/reactions
2020-05-18T21:23:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/630442673
https://avatars.githubusercontent.com/u/247881?v=4
https://api.github.com/users/dpressel/events{/privacy}
https://api.github.com/users/dpressel/followers
https://api.github.com/users/dpressel/following{/other_user}
https://api.github.com/users/dpressel/gists{/gist_id}
https://github.com/dpressel
dpressel
MDQ6VXNlcjI0Nzg4MQ==
https://api.github.com/users/dpressel/orgs
https://api.github.com/users/dpressel/received_events
https://api.github.com/users/dpressel/repos
https://api.github.com/users/dpressel/starred{/owner}{/repo}
https://api.github.com/users/dpressel/subscriptions
User
https://api.github.com/users/dpressel
public
CONTRIBUTOR
Ok, I think @lhoestq has already found a solution :-) Maybe you can chime in @lhoestq 
2020-05-18T21:24:35Z
https://github.com/huggingface/datasets/issues/160#issuecomment-630443220
https://api.github.com/repos/huggingface/datasets/issues/160
MDEyOklzc3VlQ29tbWVudDYzMDQ0MzIyMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/630443220/reactions
2020-05-18T21:24:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/630443220
https://avatars.githubusercontent.com/u/23423619?v=4
https://api.github.com/users/patrickvonplaten/events{/privacy}
https://api.github.com/users/patrickvonplaten/followers
https://api.github.com/users/patrickvonplaten/following{/other_user}
https://api.github.com/users/patrickvonplaten/gists{/gist_id}
https://github.com/patrickvonplaten
patrickvonplaten
MDQ6VXNlcjIzNDIzNjE5
https://api.github.com/users/patrickvonplaten/orgs
https://api.github.com/users/patrickvonplaten/received_events
https://api.github.com/users/patrickvonplaten/repos
https://api.github.com/users/patrickvonplaten/starred{/owner}{/repo}
https://api.github.com/users/patrickvonplaten/subscriptions
User
https://api.github.com/users/patrickvonplaten
public
NONE
This fixed my issue (I think)

https://github.com/dpressel/mead-baseline/commit/48aa8ecde4b307bd3e7dde5fe71e43a1d4769ee1
2020-05-18T21:27:55Z
https://github.com/huggingface/datasets/issues/160#issuecomment-630444748
https://api.github.com/repos/huggingface/datasets/issues/160
MDEyOklzc3VlQ29tbWVudDYzMDQ0NDc0OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/630444748/reactions
2020-05-18T21:27:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/630444748
https://avatars.githubusercontent.com/u/247881?v=4
https://api.github.com/users/dpressel/events{/privacy}
https://api.github.com/users/dpressel/followers
https://api.github.com/users/dpressel/following{/other_user}
https://api.github.com/users/dpressel/gists{/gist_id}
https://github.com/dpressel
dpressel
MDQ6VXNlcjI0Nzg4MQ==
https://api.github.com/users/dpressel/orgs
https://api.github.com/users/dpressel/received_events
https://api.github.com/users/dpressel/repos
https://api.github.com/users/dpressel/starred{/owner}{/repo}
https://api.github.com/users/dpressel/subscriptions
User
https://api.github.com/users/dpressel
public
NONE
> Ok, I think @lhoestq has already found a solution :-) Maybe you can chime in @lhoestq

Oh, awesome!  I see the PR, Ill check it out
2020-05-18T21:29:21Z
https://github.com/huggingface/datasets/issues/160#issuecomment-630445349
https://api.github.com/repos/huggingface/datasets/issues/160
MDEyOklzc3VlQ29tbWVudDYzMDQ0NTM0OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/630445349/reactions
2020-05-18T21:29:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/630445349
https://avatars.githubusercontent.com/u/247881?v=4
https://api.github.com/users/dpressel/events{/privacy}
https://api.github.com/users/dpressel/followers
https://api.github.com/users/dpressel/following{/other_user}
https://api.github.com/users/dpressel/gists{/gist_id}
https://github.com/dpressel
dpressel
MDQ6VXNlcjI0Nzg4MQ==
https://api.github.com/users/dpressel/orgs
https://api.github.com/users/dpressel/received_events
https://api.github.com/users/dpressel/repos
https://api.github.com/users/dpressel/starred{/owner}{/repo}
https://api.github.com/users/dpressel/subscriptions
User
https://api.github.com/users/dpressel
public
MEMBER
The PR should prevent the cache from losing track of the of the dataset type (based on the location of its data). Not sure about your second problem though (cache off).
2020-05-18T21:34:01Z
https://github.com/huggingface/datasets/issues/160#issuecomment-630447240
https://api.github.com/repos/huggingface/datasets/issues/160
MDEyOklzc3VlQ29tbWVudDYzMDQ0NzI0MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/630447240/reactions
2020-05-18T21:34:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/630447240
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Yes, with caching on, it seems to work without the function renaming hack, I mentioned this also in the PR. Thanks!
2020-05-18T21:36:03Z
https://github.com/huggingface/datasets/issues/160#issuecomment-630448106
https://api.github.com/repos/huggingface/datasets/issues/160
MDEyOklzc3VlQ29tbWVudDYzMDQ0ODEwNg==
https://api.github.com/repos/huggingface/datasets/issues/comments/630448106/reactions
2020-05-18T21:36:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/630448106
https://avatars.githubusercontent.com/u/247881?v=4
https://api.github.com/users/dpressel/events{/privacy}
https://api.github.com/users/dpressel/followers
https://api.github.com/users/dpressel/following{/other_user}
https://api.github.com/users/dpressel/gists{/gist_id}
https://github.com/dpressel
dpressel
MDQ6VXNlcjI0Nzg4MQ==
https://api.github.com/users/dpressel/orgs
https://api.github.com/users/dpressel/received_events
https://api.github.com/users/dpressel/repos
https://api.github.com/users/dpressel/starred{/owner}{/repo}
https://api.github.com/users/dpressel/subscriptions
User
https://api.github.com/users/dpressel
public
hello,

I am working on a program that uses the `nlp` library with the `SST2` dataset.

The rough outline of the program is:

```
import nlp as nlp_datasets
...
parser.add_argument('--dataset', help='HuggingFace Datasets id', default=['glue', 'sst2'], nargs='+')
...
dataset = nlp_datasets.load_dataset(*args.dataset)
...
# Create feature vocabs
vocabs = create_vocabs(dataset.values(), vectorizers)
...
# Create a function to vectorize based on vectorizers and vocabs:

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

# factory method to create a `convert_to_features` function based on vocabs
convert_to_features = create_featurizer(vectorizers, vocabs)
train_set = train_set.map(convert_to_features, batched=True)
train_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
train_loader = torch.utils.data.DataLoader(train_set, batch_size=args.batchsz)

valid_set = valid_set.map(convert_to_features, batched=True)
valid_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
valid_loader = torch.utils.data.DataLoader(valid_set, batch_size=args.batchsz)

test_set = test_set.map(convert_to_features, batched=True)
test_set.set_format(type='torch', columns=list(vectorizers.keys()) + ['y', 'lengths'])
test_loader = torch.utils.data.DataLoader(test_set, batch_size=args.batchsz)

print('TS', train_set.num_rows)
print('VS', valid_set.num_rows)
print('ES', test_set.num_rows)

```
Im not sure if Im using it incorrectly, but the results are not what I expect.  Namely, the `.map()`  seems to grab the datset from the cache and then loses track of what the specific dataset is, instead using my training data for all datasets:

```
TS 67349
VS 872
ES 1821
TS 67349
VS 67349
ES 67349
```

The behavior changes if I turn off the caching but then the results fail:

```
train_set = train_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
valid_set = valid_set.map(convert_to_features, batched=True, load_from_cache_file=False)
...
test_set = test_set.map(convert_to_features, batched=True, load_from_cache_file=False)
```

Now I get the right set of features back...
```
TS 67349
VS 872
ES 1821
100%|██████████| 68/68 [00:00<00:00, 92.78it/s]
100%|██████████| 1/1 [00:00<00:00, 75.47it/s]
  0%|          | 0/2 [00:00<?, ?it/s]TS 67349
VS 872
ES 1821
100%|██████████| 2/2 [00:00<00:00, 77.19it/s]
```
but I think its losing track of the original training set:

```
Traceback (most recent call last):
  File "/home/dpressel/dev/work/baseline/api-examples/layers-classify-hf-datasets.py", line 148, in <module>
    for x in train_loader:
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
    output_all_columns=self._output_all_columns,
  File "/home/dpressel/anaconda3/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 294, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pydict())
  File "pyarrow/table.pxi", line 1211, in pyarrow.lib.Table.slice
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 3: In chunk 0: Invalid: Length spanned by list offsets (15859698) larger than values array (length 100000)

Process finished with exit code 1
```

The full-example program (minus the print stmts) is here:
https://github.com/dpressel/mead-baseline/pull/620/files

