How to assign new values to Dataset?
COLLABORATOR
Hi! One option is use `map` with a function that overwrites the labels (`dset = dset.map(lamba _: {"label": 0}, features=dset.features`)). Or you can use the `remove_column` + `add_column` combination (`dset = dset.remove_columns("label").add_column("label", [0]*len(data)).cast(dset.features)`, but note that this approach creates an in-memory table for the added column instead of writing to disk, which could be problematic for large datasets.
2022-07-15T16:18:21Z
https://github.com/huggingface/datasets/issues/4684#issuecomment-1185696240
https://api.github.com/repos/huggingface/datasets/issues/4684
IC_kwDODunzps5GrEnw
https://api.github.com/repos/huggingface/datasets/issues/comments/1185696240/reactions
2022-07-15T16:18:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1185696240
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Hi! I tried your proposed solution, but it does not solve my problem unfortunately. I am working with a set of protein sequences that have been tokenized with ESM, but some sequences are longer than `max_length`, they have been truncated in the tokenization. So now I want to truncate my labels as well, but that does not work with a mapping (e.g. `dset.map` as you suggested). Specifically, what I did was the following:

```
def postprocess_tokenize(tokenized_data):
    """
    adjust label lengths if they dont match.
    """
    if len(tokenized_data['input_ids']) < len(tokenized_data['labels']):
        new_labels = tokenized_data['labels'][:len(tokenized_data['input_ids'])]
        tokenized_data["labels"] = new_labels
    return tokenized_data

tokenized_data = tokenized_data.map(postprocess_tokenize, batched=True) # this does not adjust the labels...
```

Any tips on how to do this properly?

More generally, I am wondering why the DataCollator supports padding but does not support truncation? Seems odd to me.

Thanks in advance!
2023-03-20T15:50:41Z
https://github.com/huggingface/datasets/issues/4684#issuecomment-1476488649
https://api.github.com/repos/huggingface/datasets/issues/4684
IC_kwDODunzps5YAW3J
https://api.github.com/repos/huggingface/datasets/issues/comments/1476488649/reactions
2023-03-20T15:50:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1476488649
https://avatars.githubusercontent.com/u/22891452?v=4
https://api.github.com/users/dimiboeckaerts/events{/privacy}
https://api.github.com/users/dimiboeckaerts/followers
https://api.github.com/users/dimiboeckaerts/following{/other_user}
https://api.github.com/users/dimiboeckaerts/gists{/gist_id}
https://github.com/dimiboeckaerts
dimiboeckaerts
MDQ6VXNlcjIyODkxNDUy
https://api.github.com/users/dimiboeckaerts/orgs
https://api.github.com/users/dimiboeckaerts/received_events
https://api.github.com/users/dimiboeckaerts/repos
https://api.github.com/users/dimiboeckaerts/starred{/owner}{/repo}
https://api.github.com/users/dimiboeckaerts/subscriptions
User
https://api.github.com/users/dimiboeckaerts
public
![image](https://user-images.githubusercontent.com/37113676/179149159-bbbda0c8-a661-403c-87ed-dc2b4219cd68.png)

Hi, if I want to change some values of the dataset, or add new columns to it, how can I do it?

For example, I want to change all the labels of the SST2 dataset to `0`:
```python
from datasets import load_dataset
data = load_dataset('glue','sst2')

data['train']['label'] = [0]*len(data)
```

I will get the error:
```
TypeError: 'Dataset' object does not support item assignment
```