PyTorch not available error on SageMaker GPU docker though it is installed 
MEMBER
cc @philschmid 
2021-03-17T11:06:26Z
https://github.com/huggingface/datasets/issues/2068#issuecomment-800994982
https://api.github.com/repos/huggingface/datasets/issues/2068
MDEyOklzc3VlQ29tbWVudDgwMDk5NDk4Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/800994982/reactions
2021-03-17T11:06:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/800994982
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
Hey @sivakhno,

how does your `requirements.txt` look like to install the `datasets` library and which version of it are you running? Can you try to install `datasets>=1.4.0`
2021-03-17T13:47:43Z
https://github.com/huggingface/datasets/issues/2068#issuecomment-801094430
https://api.github.com/repos/huggingface/datasets/issues/2068
MDEyOklzc3VlQ29tbWVudDgwMTA5NDQzMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/801094430/reactions
2021-03-17T13:47:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/801094430
https://avatars.githubusercontent.com/u/32632186?v=4
https://api.github.com/users/philschmid/events{/privacy}
https://api.github.com/users/philschmid/followers
https://api.github.com/users/philschmid/following{/other_user}
https://api.github.com/users/philschmid/gists{/gist_id}
https://github.com/philschmid
philschmid
MDQ6VXNlcjMyNjMyMTg2
https://api.github.com/users/philschmid/orgs
https://api.github.com/users/philschmid/received_events
https://api.github.com/users/philschmid/repos
https://api.github.com/users/philschmid/starred{/owner}{/repo}
https://api.github.com/users/philschmid/subscriptions
User
https://api.github.com/users/philschmid
public
NONE
Hi @philschmid - thanks for suggestion. I am using `datasets==1.4.1`. 
I have also tried using `torch=1.6.0` (docker `763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.6.0-gpu-py3 `), but the error is the same. 
2021-03-17T18:05:12Z
https://github.com/huggingface/datasets/issues/2068#issuecomment-801296915
https://api.github.com/repos/huggingface/datasets/issues/2068
MDEyOklzc3VlQ29tbWVudDgwMTI5NjkxNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/801296915/reactions
2021-03-17T18:05:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/801296915
https://avatars.githubusercontent.com/u/1651457?v=4
https://api.github.com/users/sivakhno/events{/privacy}
https://api.github.com/users/sivakhno/followers
https://api.github.com/users/sivakhno/following{/other_user}
https://api.github.com/users/sivakhno/gists{/gist_id}
https://github.com/sivakhno
sivakhno
MDQ6VXNlcjE2NTE0NTc=
https://api.github.com/users/sivakhno/orgs
https://api.github.com/users/sivakhno/received_events
https://api.github.com/users/sivakhno/repos
https://api.github.com/users/sivakhno/starred{/owner}{/repo}
https://api.github.com/users/sivakhno/subscriptions
User
https://api.github.com/users/sivakhno
public
CONTRIBUTOR
Could paste the code you use the start your training job and the fine-tuning script you run? 
2021-03-17T18:08:08Z
https://github.com/huggingface/datasets/issues/2068#issuecomment-801298807
https://api.github.com/repos/huggingface/datasets/issues/2068
MDEyOklzc3VlQ29tbWVudDgwMTI5ODgwNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/801298807/reactions
2021-03-17T18:08:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/801298807
https://avatars.githubusercontent.com/u/32632186?v=4
https://api.github.com/users/philschmid/events{/privacy}
https://api.github.com/users/philschmid/followers
https://api.github.com/users/philschmid/following{/other_user}
https://api.github.com/users/philschmid/gists{/gist_id}
https://github.com/philschmid
philschmid
MDQ6VXNlcjMyNjMyMTg2
https://api.github.com/users/philschmid/orgs
https://api.github.com/users/philschmid/received_events
https://api.github.com/users/philschmid/repos
https://api.github.com/users/philschmid/starred{/owner}{/repo}
https://api.github.com/users/philschmid/subscriptions
User
https://api.github.com/users/philschmid
public
CONTRIBUTOR
@sivakhno this should be now fixed in `datasets>=1.5.0`. 
2021-03-18T16:23:47Z
https://github.com/huggingface/datasets/issues/2068#issuecomment-802065352
https://api.github.com/repos/huggingface/datasets/issues/2068
MDEyOklzc3VlQ29tbWVudDgwMjA2NTM1Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/802065352/reactions
2021-03-18T16:23:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/802065352
https://avatars.githubusercontent.com/u/32632186?v=4
https://api.github.com/users/philschmid/events{/privacy}
https://api.github.com/users/philschmid/followers
https://api.github.com/users/philschmid/following{/other_user}
https://api.github.com/users/philschmid/gists{/gist_id}
https://github.com/philschmid
philschmid
MDQ6VXNlcjMyNjMyMTg2
https://api.github.com/users/philschmid/orgs
https://api.github.com/users/philschmid/received_events
https://api.github.com/users/philschmid/repos
https://api.github.com/users/philschmid/starred{/owner}{/repo}
https://api.github.com/users/philschmid/subscriptions
User
https://api.github.com/users/philschmid
public
CONTRIBUTOR
@philschmid Recently released tensorflow-macos seems to be missing. 
2021-06-13T16:19:38Z
https://github.com/huggingface/datasets/issues/2068#issuecomment-860236454
https://api.github.com/repos/huggingface/datasets/issues/2068
MDEyOklzc3VlQ29tbWVudDg2MDIzNjQ1NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/860236454/reactions
2021-06-13T16:19:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/860236454
https://avatars.githubusercontent.com/u/12831254?v=4
https://api.github.com/users/slayerjain/events{/privacy}
https://api.github.com/users/slayerjain/followers
https://api.github.com/users/slayerjain/following{/other_user}
https://api.github.com/users/slayerjain/gists{/gist_id}
https://github.com/slayerjain
slayerjain
MDQ6VXNlcjEyODMxMjU0
https://api.github.com/users/slayerjain/orgs
https://api.github.com/users/slayerjain/received_events
https://api.github.com/users/slayerjain/repos
https://api.github.com/users/slayerjain/starred{/owner}{/repo}
https://api.github.com/users/slayerjain/subscriptions
User
https://api.github.com/users/slayerjain
public
CONTRIBUTOR
I've created a PR to add this. 
2021-06-13T16:20:30Z
https://github.com/huggingface/datasets/issues/2068#issuecomment-860236601
https://api.github.com/repos/huggingface/datasets/issues/2068
MDEyOklzc3VlQ29tbWVudDg2MDIzNjYwMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/860236601/reactions
2021-06-13T16:20:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/860236601
https://avatars.githubusercontent.com/u/12831254?v=4
https://api.github.com/users/slayerjain/events{/privacy}
https://api.github.com/users/slayerjain/followers
https://api.github.com/users/slayerjain/following{/other_user}
https://api.github.com/users/slayerjain/gists{/gist_id}
https://github.com/slayerjain
slayerjain
MDQ6VXNlcjEyODMxMjU0
https://api.github.com/users/slayerjain/orgs
https://api.github.com/users/slayerjain/received_events
https://api.github.com/users/slayerjain/repos
https://api.github.com/users/slayerjain/starred{/owner}{/repo}
https://api.github.com/users/slayerjain/subscriptions
User
https://api.github.com/users/slayerjain
public
I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines 

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`. 

Also as a first line in the data loading module I have 

```
import os
os.environ["USE_TF"] = "0" 
os.environ["USE_TORCH"] = "1" 
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks! 

