UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors
NONE
I have the same issue
2020-10-09T13:56:29Z
https://github.com/huggingface/datasets/issues/616#issuecomment-706195809
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDcwNjE5NTgwOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/706195809/reactions
2020-10-09T13:56:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/706195809
https://avatars.githubusercontent.com/u/8363045?v=4
https://api.github.com/users/sheffier/events{/privacy}
https://api.github.com/users/sheffier/followers
https://api.github.com/users/sheffier/following{/other_user}
https://api.github.com/users/sheffier/gists{/gist_id}
https://github.com/sheffier
sheffier
MDQ6VXNlcjgzNjMwNDU=
https://api.github.com/users/sheffier/orgs
https://api.github.com/users/sheffier/received_events
https://api.github.com/users/sheffier/repos
https://api.github.com/users/sheffier/starred{/owner}{/repo}
https://api.github.com/users/sheffier/subscriptions
User
https://api.github.com/users/sheffier
public
NONE
Same issue here when Trying to load a dataset from disk.
2020-10-30T12:21:15Z
https://github.com/huggingface/datasets/issues/616#issuecomment-719521074
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDcxOTUyMTA3NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/719521074/reactions
2020-10-30T12:21:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/719521074
https://avatars.githubusercontent.com/u/25668035?v=4
https://api.github.com/users/ClaartjeBarkhof/events{/privacy}
https://api.github.com/users/ClaartjeBarkhof/followers
https://api.github.com/users/ClaartjeBarkhof/following{/other_user}
https://api.github.com/users/ClaartjeBarkhof/gists{/gist_id}
https://github.com/ClaartjeBarkhof
ClaartjeBarkhof
MDQ6VXNlcjI1NjY4MDM1
https://api.github.com/users/ClaartjeBarkhof/orgs
https://api.github.com/users/ClaartjeBarkhof/received_events
https://api.github.com/users/ClaartjeBarkhof/repos
https://api.github.com/users/ClaartjeBarkhof/starred{/owner}{/repo}
https://api.github.com/users/ClaartjeBarkhof/subscriptions
User
https://api.github.com/users/ClaartjeBarkhof
public
NONE
I am also experiencing this issue, and don't know if it's affecting my training.
2020-11-14T17:38:46Z
https://github.com/huggingface/datasets/issues/616#issuecomment-727240494
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDcyNzI0MDQ5NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/727240494/reactions
2020-11-14T17:38:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/727240494
https://avatars.githubusercontent.com/u/26638161?v=4
https://api.github.com/users/samuelstevens/events{/privacy}
https://api.github.com/users/samuelstevens/followers
https://api.github.com/users/samuelstevens/following{/other_user}
https://api.github.com/users/samuelstevens/gists{/gist_id}
https://github.com/samuelstevens
samuelstevens
MDQ6VXNlcjI2NjM4MTYx
https://api.github.com/users/samuelstevens/orgs
https://api.github.com/users/samuelstevens/received_events
https://api.github.com/users/samuelstevens/repos
https://api.github.com/users/samuelstevens/starred{/owner}{/repo}
https://api.github.com/users/samuelstevens/subscriptions
User
https://api.github.com/users/samuelstevens
public
NONE
Same here. I hope the dataset is not being modified in-place.
2020-11-20T05:09:47Z
https://github.com/huggingface/datasets/issues/616#issuecomment-730854140
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDczMDg1NDE0MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/730854140/reactions
2020-11-20T05:10:07Z
https://api.github.com/repos/huggingface/datasets/issues/comments/730854140
https://avatars.githubusercontent.com/u/1051900?v=4
https://api.github.com/users/kyoungrok0517/events{/privacy}
https://api.github.com/users/kyoungrok0517/followers
https://api.github.com/users/kyoungrok0517/following{/other_user}
https://api.github.com/users/kyoungrok0517/gists{/gist_id}
https://github.com/kyoungrok0517
kyoungrok0517
MDQ6VXNlcjEwNTE5MDA=
https://api.github.com/users/kyoungrok0517/orgs
https://api.github.com/users/kyoungrok0517/received_events
https://api.github.com/users/kyoungrok0517/repos
https://api.github.com/users/kyoungrok0517/starred{/owner}{/repo}
https://api.github.com/users/kyoungrok0517/subscriptions
User
https://api.github.com/users/kyoungrok0517
public
MEMBER
I think the only way to avoid this warning would be to do a copy of the numpy array before providing it.

This would slow down a bit the iteration over the dataset but maybe it would be safer. We could disable the copy with a flag on the `set_format` command.

In most typical cases of training a NLP model, PyTorch shouldn't modify the input so it's ok to have a non-writable array but I can understand the warning is a bit scary so maybe we could choose the side of non-warning/slower by default and have an option to speedup.

What do you think @lhoestq ? 
2020-11-20T08:05:27Z
https://github.com/huggingface/datasets/issues/616#issuecomment-731008893
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDczMTAwODg5Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/731008893/reactions
2020-11-20T08:05:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/731008893
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
@thomwolf Would it be possible to have the array look writeable, but raise an error if it is actually written to?

I would like to keep my code free of warning, but I also wouldn't like to slow down the program because of unnecessary copy operations.   
2020-11-23T15:52:56Z
https://github.com/huggingface/datasets/issues/616#issuecomment-732247756
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDczMjI0Nzc1Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/732247756/reactions
2020-11-23T15:52:56Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732247756
https://avatars.githubusercontent.com/u/505333?v=4
https://api.github.com/users/AndreasMadsen/events{/privacy}
https://api.github.com/users/AndreasMadsen/followers
https://api.github.com/users/AndreasMadsen/following{/other_user}
https://api.github.com/users/AndreasMadsen/gists{/gist_id}
https://github.com/AndreasMadsen
AndreasMadsen
MDQ6VXNlcjUwNTMzMw==
https://api.github.com/users/AndreasMadsen/orgs
https://api.github.com/users/AndreasMadsen/received_events
https://api.github.com/users/AndreasMadsen/repos
https://api.github.com/users/AndreasMadsen/starred{/owner}{/repo}
https://api.github.com/users/AndreasMadsen/subscriptions
User
https://api.github.com/users/AndreasMadsen
public
MEMBER
@AndreasMadsen probably not I would guess (no free lunch hahah)
2020-11-23T16:40:33Z
https://github.com/huggingface/datasets/issues/616#issuecomment-732279644
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDczMjI3OTY0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/732279644/reactions
2020-11-23T16:40:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732279644
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
@thomwolf Why not? Writable is checked with `arr.flags.writeable`, and writing is done via magic methods.
2020-11-23T21:58:22Z
https://github.com/huggingface/datasets/issues/616#issuecomment-732450059
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDczMjQ1MDA1OQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/732450059/reactions
2020-11-23T21:58:22Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732450059
https://avatars.githubusercontent.com/u/505333?v=4
https://api.github.com/users/AndreasMadsen/events{/privacy}
https://api.github.com/users/AndreasMadsen/followers
https://api.github.com/users/AndreasMadsen/following{/other_user}
https://api.github.com/users/AndreasMadsen/gists{/gist_id}
https://github.com/AndreasMadsen
AndreasMadsen
MDQ6VXNlcjUwNTMzMw==
https://api.github.com/users/AndreasMadsen/orgs
https://api.github.com/users/AndreasMadsen/received_events
https://api.github.com/users/AndreasMadsen/repos
https://api.github.com/users/AndreasMadsen/starred{/owner}{/repo}
https://api.github.com/users/AndreasMadsen/subscriptions
User
https://api.github.com/users/AndreasMadsen
public
MEMBER
Well because I don't know the internal of numpy as well as you I guess hahahah, do you want to try to open a PR proposing a solution?
2020-11-23T22:56:03Z
https://github.com/huggingface/datasets/issues/616#issuecomment-732473898
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDczMjQ3Mzg5OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/732473898/reactions
2020-11-23T22:56:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732473898
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
CONTRIBUTOR
@thomwolf @AndreasMadsen I think this is a terrible idea, n/o, and I am very much against it. Modifying internals of an array in such a hacky way is bound to run into other (user) issues down the line. To users it would not be clear at all what is going on e.g. when they check for write access (which will return True) but then they get a warning that the array is not writeable. That's extremely confusing. 

If your only goal is to get rid of warnings in your code, then you can just use a [simplefilter](https://docs.python.org/3.8/library/warnings.html#temporarily-suppressing-warnings) for UserWarnings in your own code. Changing the code-base in such an intuitive way does not seem like a good way to go and sets a bad precedent, imo. 

(Feel free to disagree, of course.)

IMO a warning can stay (as they can be filtered by users anyway), but it can be clarified why the warning takes place.
2020-11-24T08:09:04Z
https://github.com/huggingface/datasets/issues/616#issuecomment-732728808
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDczMjcyODgwOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/732728808/reactions
2020-11-24T10:14:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732728808
https://avatars.githubusercontent.com/u/2779410?v=4
https://api.github.com/users/BramVanroy/events{/privacy}
https://api.github.com/users/BramVanroy/followers
https://api.github.com/users/BramVanroy/following{/other_user}
https://api.github.com/users/BramVanroy/gists{/gist_id}
https://github.com/BramVanroy
BramVanroy
MDQ6VXNlcjI3Nzk0MTA=
https://api.github.com/users/BramVanroy/orgs
https://api.github.com/users/BramVanroy/received_events
https://api.github.com/users/BramVanroy/repos
https://api.github.com/users/BramVanroy/starred{/owner}{/repo}
https://api.github.com/users/BramVanroy/subscriptions
User
https://api.github.com/users/BramVanroy
public
NONE
> To users it would not be clear at all what is going on e.g. when they check for write access (which will return True) but then they get a warning that the array is not writeable. That's extremely confusing.

Confusion can be resolved with a helpful error message. In this case, that error message can be controlled by huggingface/datasets. The right argument here is that if code depends on `.flags.writable` being truthful (not just for warnings), then it will cause unavoidable errors. Although, I can't imagine such a use-case.

> If your only goal is to get rid of warnings in your code, then you can just use a simplefilter for UserWarnings in your own code. Changing the code-base in such an intuitive way does not seem like a good way to go and sets a bad precedent, imo.

I don't want to ignore all `UserWarnings`, nor all warnings regarding non-writable arrays. Ignoring warnings leads to hard to debug issues.

> IMO a warning can stay (as they can be filtered by users anyway), but it can be clarified why the warning takes place.

Plain use cases should really not generate warnings. It teaches developers to ignore warnings which is a terrible practice.

---

The best solution would be to allow non-writable arrays in `DataLoader`, but that is a PyTorch issue.
2020-11-24T11:09:19Z
https://github.com/huggingface/datasets/issues/616#issuecomment-732836841
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDczMjgzNjg0MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/732836841/reactions
2020-11-24T11:09:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732836841
https://avatars.githubusercontent.com/u/505333?v=4
https://api.github.com/users/AndreasMadsen/events{/privacy}
https://api.github.com/users/AndreasMadsen/followers
https://api.github.com/users/AndreasMadsen/following{/other_user}
https://api.github.com/users/AndreasMadsen/gists{/gist_id}
https://github.com/AndreasMadsen
AndreasMadsen
MDQ6VXNlcjUwNTMzMw==
https://api.github.com/users/AndreasMadsen/orgs
https://api.github.com/users/AndreasMadsen/received_events
https://api.github.com/users/AndreasMadsen/repos
https://api.github.com/users/AndreasMadsen/starred{/owner}{/repo}
https://api.github.com/users/AndreasMadsen/subscriptions
User
https://api.github.com/users/AndreasMadsen
public
CONTRIBUTOR
> The right argument here is that if code depends on `.flags.writable` being truthful (not just for warnings), then it will cause unavoidable errors. Although, I can't imagine such a use-case.

That's exactly the argument in my first sentence. Too often someone "cannot think of a use-case", but you can not foresee the use-cases of a whole research community.
 
> I don't want to ignore all `UserWarnings`, nor all warnings regarding non-writable arrays. Ignoring warnings leads to hard to debug issues.

That's fair.

> Plain use cases should really not generate warnings. It teaches developers to ignore warnings which is a terrible practice.

But this is not a plain use-case (because Pytorch does not support these read-only tensors). Manually setting the flag to writable will solve the issue on the surface but is basically just a hack to compensate for something that is not allowed in another library. 

What about an "ignore_warnings" flag in `set_format` that when True wraps the offending code in a block to ignore userwarnings at that specific step in [_convert_outputs](https://github.com/huggingface/datasets/blob/880c2c76a8223a00c303eab2909371e857113063/src/datasets/arrow_dataset.py#L821)? Something like:

```python
def _convert_outputs(..., ignore_warnings=True):
    ...
    with warnings.catch_warnings():
        if ignore_warnings:
            warnings.simplefilter("ignore", UserWarning)
        return torch.tensor(...)
# continues without warning filter after context manager...
```
2020-11-24T12:32:05Z
https://github.com/huggingface/datasets/issues/616#issuecomment-732945134
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDczMjk0NTEzNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/732945134/reactions
2020-11-25T09:18:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732945134
https://avatars.githubusercontent.com/u/2779410?v=4
https://api.github.com/users/BramVanroy/events{/privacy}
https://api.github.com/users/BramVanroy/followers
https://api.github.com/users/BramVanroy/following{/other_user}
https://api.github.com/users/BramVanroy/gists{/gist_id}
https://github.com/BramVanroy
BramVanroy
MDQ6VXNlcjI3Nzk0MTA=
https://api.github.com/users/BramVanroy/orgs
https://api.github.com/users/BramVanroy/received_events
https://api.github.com/users/BramVanroy/repos
https://api.github.com/users/BramVanroy/starred{/owner}{/repo}
https://api.github.com/users/BramVanroy/subscriptions
User
https://api.github.com/users/BramVanroy
public
NONE
> But this is not a plain use-case (because Pytorch does not support these read-only tensors).

By "plain", I mean the recommended way to use `datasets` with PyTorch according to the `datasets` documentation.
2020-11-26T16:26:35Z
https://github.com/huggingface/datasets/issues/616#issuecomment-734391663
https://api.github.com/repos/huggingface/datasets/issues/616
MDEyOklzc3VlQ29tbWVudDczNDM5MTY2Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/734391663/reactions
2020-11-26T16:26:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/734391663
https://avatars.githubusercontent.com/u/505333?v=4
https://api.github.com/users/AndreasMadsen/events{/privacy}
https://api.github.com/users/AndreasMadsen/followers
https://api.github.com/users/AndreasMadsen/following{/other_user}
https://api.github.com/users/AndreasMadsen/gists{/gist_id}
https://github.com/AndreasMadsen
AndreasMadsen
MDQ6VXNlcjUwNTMzMw==
https://api.github.com/users/AndreasMadsen/orgs
https://api.github.com/users/AndreasMadsen/received_events
https://api.github.com/users/AndreasMadsen/repos
https://api.github.com/users/AndreasMadsen/starred{/owner}{/repo}
https://api.github.com/users/AndreasMadsen/subscriptions
User
https://api.github.com/users/AndreasMadsen
public
NONE
This error is what I see when I run the first lines of the Pytorch Quickstart.  It should also say that it should be ignored and/or how to fix it.   BTW, this is a Pytorch error message -- not a Huggingface error message.   My code runs anyway.
2021-07-22T20:39:36Z
https://github.com/huggingface/datasets/issues/616#issuecomment-885217591
https://api.github.com/repos/huggingface/datasets/issues/616
IC_kwDODunzps40w1k3
https://api.github.com/repos/huggingface/datasets/issues/comments/885217591/reactions
2021-07-22T21:12:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/885217591
https://avatars.githubusercontent.com/u/14264000?v=4
https://api.github.com/users/MikeyBeez/events{/privacy}
https://api.github.com/users/MikeyBeez/followers
https://api.github.com/users/MikeyBeez/following{/other_user}
https://api.github.com/users/MikeyBeez/gists{/gist_id}
https://github.com/MikeyBeez
MikeyBeez
MDQ6VXNlcjE0MjY0MDAw
https://api.github.com/users/MikeyBeez/orgs
https://api.github.com/users/MikeyBeez/received_events
https://api.github.com/users/MikeyBeez/repos
https://api.github.com/users/MikeyBeez/starred{/owner}{/repo}
https://api.github.com/users/MikeyBeez/subscriptions
User
https://api.github.com/users/MikeyBeez
public
I am trying out the library and want to load in pickled data with `from_dict`. In that dict, one column `text` should be tokenized and the other (an embedding vector) should be retained. All other columns should be removed. When I eventually try to set the format for the columns with `set_format` I am getting this strange Userwarning without a stack trace:

> Set __getitem__(key) output type to torch for ['input_ids', 'sembedding'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
> C:\Users\bramv\.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\datasets\arrow_dataset.py:835: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  ..\torch\csrc\utils\tensor_numpy.cpp:141.)
>   return torch.tensor(x, **format_kwargs)

The first one might not be related to the warning, but it is odd that it is shown, too. It is unclear whether that is something that I should do or something that that the program is doing at that moment.

Snippet:
```
    dataset = Dataset.from_dict(torch.load("data/dummy.pt.pt"))
    print(dataset)
    tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
    keys_to_retain = {"input_ids", "sembedding"}
    dataset = dataset.map(lambda example: tokenizer(example["text"], padding='max_length'), batched=True)
    dataset.remove_columns_(set(dataset.column_names) - keys_to_retain)

    dataset.set_format(type="torch", columns=["input_ids", "sembedding"])
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=2)

    print(next(iter(dataloader)))
```

PS: the input type for `remove_columns_` should probably be an Iterable rather than just a List.