ValueError: datasets' indices [1] come from memory and datasets' indices [0] come from disk
MEMBER
Hi ! To help me understand the situation, can you print the values of  `load_from_disk(PATH_DATA_CLS_A)['train']._indices_data_files` and `load_from_disk(PATH_DATA_CLS_B)['train']._indices_data_files` ?
They should both have a path to an arrow file

Also note that from #2025 concatenating datasets will no longer have such restrictions.
2021-03-12T14:55:29Z
https://github.com/huggingface/datasets/issues/2040#issuecomment-797540502
https://api.github.com/repos/huggingface/datasets/issues/2040
MDEyOklzc3VlQ29tbWVudDc5NzU0MDUwMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/797540502/reactions
2021-03-12T14:55:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797540502
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Sure, thanks for the fast reply!

For dataset A: `[{'filename': 'drive/MyDrive/data_target_task/dataset_a/train/cache-4797266bf4db1eb7.arrow'}]`
For dataset B: `[]`

No clue why for B it returns nothing. `PATH_DATA_CLS_B` is exactly the same in `save_to_disk` and `load_from_disk`... Also I can verify that the folder physically exists under 'drive/MyDrive/data_target_task/dataset_b/'
2021-03-12T15:18:05Z
https://github.com/huggingface/datasets/issues/2040#issuecomment-797555913
https://api.github.com/repos/huggingface/datasets/issues/2040
MDEyOklzc3VlQ29tbWVudDc5NzU1NTkxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/797555913/reactions
2021-03-12T15:18:05Z
https://api.github.com/repos/huggingface/datasets/issues/comments/797555913
https://avatars.githubusercontent.com/u/53626067?v=4
https://api.github.com/users/simonschoe/events{/privacy}
https://api.github.com/users/simonschoe/followers
https://api.github.com/users/simonschoe/following{/other_user}
https://api.github.com/users/simonschoe/gists{/gist_id}
https://github.com/simonschoe
simonschoe
MDQ6VXNlcjUzNjI2MDY3
https://api.github.com/users/simonschoe/orgs
https://api.github.com/users/simonschoe/received_events
https://api.github.com/users/simonschoe/repos
https://api.github.com/users/simonschoe/starred{/owner}{/repo}
https://api.github.com/users/simonschoe/subscriptions
User
https://api.github.com/users/simonschoe
public
MEMBER
In the next release you'll be able to concatenate any kinds of dataset (either from memory or from disk).

For now I'd suggest you to flatten the indices of the A and B datasets. This will remove the indices mapping and you will be able to concatenate them. You can flatten the indices with
```python
dataset = dataset.flatten_indices()
```
2021-03-15T10:06:31Z
https://github.com/huggingface/datasets/issues/2040#issuecomment-799291799
https://api.github.com/repos/huggingface/datasets/issues/2040
MDEyOklzc3VlQ29tbWVudDc5OTI5MTc5OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/799291799/reactions
2021-03-15T10:06:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/799291799
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Indeed this works. Not the most elegant solution, but it does the trick. Thanks a lot! 
2021-03-17T17:56:46Z
https://github.com/huggingface/datasets/issues/2040#issuecomment-801291061
https://api.github.com/repos/huggingface/datasets/issues/2040
MDEyOklzc3VlQ29tbWVudDgwMTI5MTA2MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/801291061/reactions
2021-03-17T17:56:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/801291061
https://avatars.githubusercontent.com/u/53626067?v=4
https://api.github.com/users/simonschoe/events{/privacy}
https://api.github.com/users/simonschoe/followers
https://api.github.com/users/simonschoe/following{/other_user}
https://api.github.com/users/simonschoe/gists{/gist_id}
https://github.com/simonschoe
simonschoe
MDQ6VXNlcjUzNjI2MDY3
https://api.github.com/users/simonschoe/orgs
https://api.github.com/users/simonschoe/received_events
https://api.github.com/users/simonschoe/repos
https://api.github.com/users/simonschoe/starred{/owner}{/repo}
https://api.github.com/users/simonschoe/subscriptions
User
https://api.github.com/users/simonschoe
public
Hi there,

I am trying to concat two datasets that I've previously saved to disk via `save_to_disk()` like so (note that both are saved as `DataDict`, `PATH_DATA_CLS_*` are `Path`-objects):
```python
concatenate_datasets([load_from_disk(PATH_DATA_CLS_A)['train'], load_from_disk(PATH_DATA_CLS_B)['train']])
```
Yielding the following error:
```python
ValueError: Datasets' indices should ALL come from memory, or should ALL come from disk.
However datasets' indices [1] come from memory and datasets' indices [0] come from disk.
```
Been trying to solve this for quite some time now. Both `DataDict` have been created by reading in a `csv` via `load_dataset` and subsequently processed using the various `datasets` methods (i.e. filter, map, remove col, rename col). Can't figure out tho...

`load_from_disk(PATH_DATA_CLS_A)['train']` yields:
```python
Dataset({
    features: ['labels', 'text'],
    num_rows: 785
})
```
`load_from_disk(PATH_DATA_CLS_B)['train']` yields:
```python
Dataset({
    features: ['labels', 'text'],
    num_rows: 3341
})
```