concatenate_datasets support axis=0 or 1 ？
MEMBER
Unfortunately `concatenate_datasets` only supports concatenating the rows, while what you want to achieve is concatenate the columns.
Currently to add more columns to a dataset, one must use `map`.
What you can do is somehting like this:
```python
# suppose you have datasets d1, d2, d3
def add_columns(example, index):
    example.update(d2[index])
    example.update(d3[index])
    return example

full_dataset = d1.map(add_columns, with_indices=True)
```
2020-11-16T10:00:00Z
https://github.com/huggingface/datasets/issues/853#issuecomment-727872188
https://api.github.com/repos/huggingface/datasets/issues/853
MDEyOklzc3VlQ29tbWVudDcyNzg3MjE4OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/727872188/reactions
2020-11-16T10:00:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/727872188
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Closing this one, feel free to re-open if you have other questions about this issue
2020-12-23T13:59:53Z
https://github.com/huggingface/datasets/issues/853#issuecomment-750312414
https://api.github.com/repos/huggingface/datasets/issues/853
MDEyOklzc3VlQ29tbWVudDc1MDMxMjQxNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/750312414/reactions
2020-12-23T13:59:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/750312414
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
That's not really difficult to add, though, no?
I think it can be done without copy.
Maybe let's add it to the roadmap?
2020-12-23T14:34:31Z
https://github.com/huggingface/datasets/issues/853#issuecomment-750326523
https://api.github.com/repos/huggingface/datasets/issues/853
MDEyOklzc3VlQ29tbWVudDc1MDMyNjUyMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/750326523/reactions
2020-12-23T14:34:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/750326523
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
Actually it's doable but requires to update the `Dataset._data_files` schema to support this.
I'm re-opening this since we may want to add this in the future
2020-12-23T14:42:58Z
https://github.com/huggingface/datasets/issues/853#issuecomment-750330211
https://api.github.com/repos/huggingface/datasets/issues/853
MDEyOklzc3VlQ29tbWVudDc1MDMzMDIxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/750330211/reactions
2020-12-23T14:44:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/750330211
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hi @lhoestq, I would love to help and add this feature if still needed. My plan is to add an axis variable in the `concatenate_datasets` function in `arrow_dataset.py` and when that is set to 1 concatenate columns instead of rows. 
2021-01-25T09:04:20Z
https://github.com/huggingface/datasets/issues/853#issuecomment-766664528
https://api.github.com/repos/huggingface/datasets/issues/853
MDEyOklzc3VlQ29tbWVudDc2NjY2NDUyOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/766664528/reactions
2021-01-25T09:06:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/766664528
https://avatars.githubusercontent.com/u/35491698?v=4
https://api.github.com/users/KMFODA/events{/privacy}
https://api.github.com/users/KMFODA/followers
https://api.github.com/users/KMFODA/following{/other_user}
https://api.github.com/users/KMFODA/gists{/gist_id}
https://github.com/KMFODA
KMFODA
MDQ6VXNlcjM1NDkxNjk4
https://api.github.com/users/KMFODA/orgs
https://api.github.com/users/KMFODA/received_events
https://api.github.com/users/KMFODA/repos
https://api.github.com/users/KMFODA/starred{/owner}{/repo}
https://api.github.com/users/KMFODA/subscriptions
User
https://api.github.com/users/KMFODA
public
MEMBER
Hi ! I would love to see this feature implemented as well :) Thank you for proposing your help !

Here is a few things about the current implementation:
- A dataset object is a wrapper of one `pyarrow.Table` that contains the data
- Pyarrow offers an API that allows to transform Table objects. For example there are functions like `concat_tables`, `Table.rename_columns`, `Table.add_column` etc.

Therefore adding columns from another dataset is possible thanks to the pyarrow API and in particular `Table.add_column` :) 

However this breaks some features we have regarding pickle. A dataset object can be pickled and unpickled without loading all the data in memory. It is useful for multiprocessing for example. Pickling a dataset object is possible thanks to the `Dataset._data_files` which defines the list of arrow files that will be used to form the final Table (basically all the data from each files are concatenated on axis 0).

Therefore to be able to add columns to a Dataset and still be able to work with it in a multiprocessing setup, we need to extend this last aspect to be able to reconstruct a Table object from multiple arrow files that are combined in both axis 0 and 1. Currently this reconstruction mechanism only supports axis 0.

I'm sure we can figure something out that enables users to add columns from another dataset while keeping the multiprocessing support.
2021-01-25T13:50:18Z
https://github.com/huggingface/datasets/issues/853#issuecomment-766829566
https://api.github.com/repos/huggingface/datasets/issues/853
MDEyOklzc3VlQ29tbWVudDc2NjgyOTU2Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/766829566/reactions
2021-01-25T13:50:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/766829566
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq, we have two Pull Requests to implement:
- Dataset.add_item: #1870
- Dataset.add_column: #2145
which add a single row or column, repectively.

The request here is to implement the concatenation of *multiple* rows/columns. Am I right?

We should agree on the API:
- `concatenate_datasets` with `axis`?
- other Dataset method name?
2021-03-30T15:18:04Z
https://github.com/huggingface/datasets/issues/853#issuecomment-810347707
https://api.github.com/repos/huggingface/datasets/issues/853
MDEyOklzc3VlQ29tbWVudDgxMDM0NzcwNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/810347707/reactions
2021-03-30T15:18:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/810347707
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
For the API, I like `concatenate_datasets` with `axis` personally :)
From a list of `Dataset` objects, it would concatenate them to a new `Dataset` object backed by a `ConcatenationTable`, that is the concatenation of the tables of each input dataset. The concatenation is either on axis=0 (append rows) or on axis=1 (append columns).

Regarding what we need to implement:
The axis=0 is already supported and is the current behavior of `concatenate_datasets`.
Also `add_item` is not needed to implement axis=1 (though it's an awesome addition to this library).

To implement axis=1, we either need `add_column` or a `ConcatenationTable` constructor to concatenate tables horizontally.
I have a preference for using a `ConcatenationTable` constructor because this way we can end up with a `ConcatenationTable` with only 1 additional block per table, while `add_column` would add 1 block per new column.

Maybe we can simply have an equivalent of `ConcatenationTable.from_tables` but for axis=1 ?
`axis` could also be an argument of `ConcatenationTable.from_tables`
2021-03-31T09:38:27Z
https://github.com/huggingface/datasets/issues/853#issuecomment-810927092
https://api.github.com/repos/huggingface/datasets/issues/853
MDEyOklzc3VlQ29tbWVudDgxMDkyNzA5Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/810927092/reactions
2021-03-31T09:45:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/810927092
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
@lhoestq I think I guessed your suggestions in advance... 😉 #2151
2021-04-01T13:55:38Z
https://github.com/huggingface/datasets/issues/853#issuecomment-811925779
https://api.github.com/repos/huggingface/datasets/issues/853
MDEyOklzc3VlQ29tbWVudDgxMTkyNTc3OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/811925779/reactions
2021-04-01T13:55:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811925779
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Cool ! Sorry I missed this one ^^
I'm taking a look ;)
2021-04-01T13:57:51Z
https://github.com/huggingface/datasets/issues/853#issuecomment-811927175
https://api.github.com/repos/huggingface/datasets/issues/853
MDEyOklzc3VlQ29tbWVudDgxMTkyNzE3NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/811927175/reactions
2021-04-01T13:57:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/811927175
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
I want to achieve the following result
![image](https://user-images.githubusercontent.com/12437751/99207426-f0c8db80-27f8-11eb-820a-4d9f7287b742.png)
