Add remove_columns=True
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-04-21T17:39:09Z
https://github.com/huggingface/datasets/pull/4197#issuecomment-1105506036
https://api.github.com/repos/huggingface/datasets/issues/4197
IC_kwDODunzps5B5K70
https://api.github.com/repos/huggingface/datasets/issues/comments/1105506036/reactions
2022-04-22T14:51:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1105506036
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
COLLABORATOR
Any reason why we can't just do `[inputs.copy()]` in this line for in-place operations to not have effects anymore:
https://github.com/huggingface/datasets/blob/bf432011ff9155a5bc16c03956bc63e514baf80d/src/datasets/arrow_dataset.py#L2232.

(in the `batched` case, we can also copy the inputs' values (list objects) to ignore in-place modifications to the inputs' columns)

I think `remove_columns=True` has no meaning, so I'm not a fan of this change.
2022-04-22T13:10:45Z
https://github.com/huggingface/datasets/pull/4197#issuecomment-1106499822
https://api.github.com/repos/huggingface/datasets/issues/4197
IC_kwDODunzps5B89ju
https://api.github.com/repos/huggingface/datasets/issues/comments/1106499822/reactions
2022-04-22T13:10:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1106499822
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
@mariosasko copy does have a cost associated with it ... and plus you'll have to consider `deepcopy` Imagine columnds that are list of list of list of list .... Though I have to agree that `remove_columns=True` doesn't make sense (but, IMO, neither does it in its current use-case as it should refer to `input_columns`) 
2022-04-22T13:12:08Z
https://github.com/huggingface/datasets/pull/4197#issuecomment-1106501213
https://api.github.com/repos/huggingface/datasets/issues/4197
IC_kwDODunzps5B895d
https://api.github.com/repos/huggingface/datasets/issues/comments/1106501213/reactions
2022-04-22T13:26:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1106501213
https://avatars.githubusercontent.com/u/24695242?v=4
https://api.github.com/users/thomasw21/events{/privacy}
https://api.github.com/users/thomasw21/followers
https://api.github.com/users/thomasw21/following{/other_user}
https://api.github.com/users/thomasw21/gists{/gist_id}
https://github.com/thomasw21
thomasw21
MDQ6VXNlcjI0Njk1MjQy
https://api.github.com/users/thomasw21/orgs
https://api.github.com/users/thomasw21/received_events
https://api.github.com/users/thomasw21/repos
https://api.github.com/users/thomasw21/starred{/owner}{/repo}
https://api.github.com/users/thomasw21/subscriptions
User
https://api.github.com/users/thomasw21
public
CONTRIBUTOR
Okay closing this PR for the following reasons:
 - `remove_columns=True` was expected to keep the `.update`-like operator for `.map`. I initially thought it would be a good way to ignore function side effects and only keep output of that function (cf. PR description).
 - expected `remove_columns=True` is a bad API according to @mariosasko and introduces unecessary changes for little gain (strictly equivalent to `remove_columns=dset.column_names`)
2022-04-22T14:45:27Z
https://github.com/huggingface/datasets/pull/4197#issuecomment-1106588265
https://api.github.com/repos/huggingface/datasets/issues/4197
IC_kwDODunzps5B9TJp
https://api.github.com/repos/huggingface/datasets/issues/comments/1106588265/reactions
2022-04-22T14:45:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1106588265
https://avatars.githubusercontent.com/u/24695242?v=4
https://api.github.com/users/thomasw21/events{/privacy}
https://api.github.com/users/thomasw21/followers
https://api.github.com/users/thomasw21/following{/other_user}
https://api.github.com/users/thomasw21/gists{/gist_id}
https://github.com/thomasw21
thomasw21
MDQ6VXNlcjI0Njk1MjQy
https://api.github.com/users/thomasw21/orgs
https://api.github.com/users/thomasw21/received_events
https://api.github.com/users/thomasw21/repos
https://api.github.com/users/thomasw21/starred{/owner}{/repo}
https://api.github.com/users/thomasw21/subscriptions
User
https://api.github.com/users/thomasw21
public
This should fix all the issue we have with in place operations in mapping functions. This is crucial as where we do some weird things like:
```
def apply(batch):
    batch_size = len(batch["id"])
    batch["text"] = ["potato" for _ range(batch_size)]
    return {}

# Columns are: {"id": int}
dset.map(apply, batched=True, remove_columns="text") # crashes because `text` is not in the original columns
dset.map(apply, batched=True) # mapped datasets has `text` column
```

In this PR we suggest to have `remove_columns=True` so that we ignore the input completely, and just use the output to generate mapped dataset. This means that inplace operations won't have any effects anymore.