Add support for axis in concatenate datasets
MEMBER
@lhoestq I am going to implement the consolidation step you mentioned in #1870.
2021-04-01T14:00:07Z
https://github.com/huggingface/datasets/pull/2151#issuecomment-811928654
https://api.github.com/repos/huggingface/datasets/issues/2151
MDEyOklzc3VlQ29tbWVudDgxMTkyODY1NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/811928654/reactions
2021-04-01T14:00:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811928654
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
@lhoestq I was thinking that the order of the TableBlocks is not relevant, isn't it?

I mean, in order to consolidate _consecutive_ in-memory table blocks, in this case:
```
blocks = [in_memory_1, memory_mapped, in_memory_2]
```
I could reorder the list:
```
blocks = [in_memory_1, in_memory_2, memory_mapped]
```
so that the first 2 can be consolidated into a single one:
```
blocks = [in_memory_3, memory_mapped]
```
2021-04-01T16:21:00Z
https://github.com/huggingface/datasets/pull/2151#issuecomment-812019660
https://api.github.com/repos/huggingface/datasets/issues/2151
MDEyOklzc3VlQ29tbWVudDgxMjAxOTY2MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/812019660/reactions
2021-04-01T16:21:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/812019660
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
I think the order is important, users won't expect the dataset to be "shuffled" when they add a new item
2021-04-06T07:22:17Z
https://github.com/huggingface/datasets/pull/2151#issuecomment-813890439
https://api.github.com/repos/huggingface/datasets/issues/2151
MDEyOklzc3VlQ29tbWVudDgxMzg5MDQzOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/813890439/reactions
2021-04-06T07:22:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/813890439
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> I think the order is important, users won't expect the dataset to be "shuffled" when they add a new item

OK, therefore I leave `_consolidate_blocks` as it is, which currently keeps the order of the blocks (no shuffling).
2021-04-06T07:49:04Z
https://github.com/huggingface/datasets/pull/2151#issuecomment-813907612
https://api.github.com/repos/huggingface/datasets/issues/2151
MDEyOklzc3VlQ29tbWVudDgxMzkwNzYxMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/813907612/reactions
2021-04-06T07:49:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/813907612
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Thank you guys for implementing this. Minor thing I noticed in the [documentation](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.concatenate_datasets): it says "Converts a list of Dataset with **the same schema** into a single Dataset". With the addition of the axis parameter, perhaps this should be reworded, no?
2021-06-23T17:41:02Z
https://github.com/huggingface/datasets/pull/2151#issuecomment-867033781
https://api.github.com/repos/huggingface/datasets/issues/2151
MDEyOklzc3VlQ29tbWVudDg2NzAzMzc4MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/867033781/reactions
2021-06-23T17:41:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/867033781
https://avatars.githubusercontent.com/u/22605313?v=4
https://api.github.com/users/lucaguarro/events{/privacy}
https://api.github.com/users/lucaguarro/followers
https://api.github.com/users/lucaguarro/following{/other_user}
https://api.github.com/users/lucaguarro/gists{/gist_id}
https://github.com/lucaguarro
lucaguarro
MDQ6VXNlcjIyNjA1MzEz
https://api.github.com/users/lucaguarro/orgs
https://api.github.com/users/lucaguarro/received_events
https://api.github.com/users/lucaguarro/repos
https://api.github.com/users/lucaguarro/starred{/owner}{/repo}
https://api.github.com/users/lucaguarro/subscriptions
User
https://api.github.com/users/lucaguarro
public
Add support for `axis` (0 or 1) in `concatenate_datasets`.

Close #853.