Loading Data From S3 Path in Sagemaker
MEMBER
This would be a neat feature
2020-11-23T09:42:12Z
https://github.com/huggingface/datasets/issues/878#issuecomment-732044862
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDczMjA0NDg2Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/732044862/reactions
2020-11-23T09:42:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732044862
https://avatars.githubusercontent.com/u/326577?v=4
https://api.github.com/users/julien-c/events{/privacy}
https://api.github.com/users/julien-c/followers
https://api.github.com/users/julien-c/following{/other_user}
https://api.github.com/users/julien-c/gists{/gist_id}
https://github.com/julien-c
julien-c
MDQ6VXNlcjMyNjU3Nw==
https://api.github.com/users/julien-c/orgs
https://api.github.com/users/julien-c/received_events
https://api.github.com/users/julien-c/repos
https://api.github.com/users/julien-c/starred{/owner}{/repo}
https://api.github.com/users/julien-c/subscriptions
User
https://api.github.com/users/julien-c
public
NONE
> neat feature

I dint get these clearly, can you please elaborate like how to work on these 
2020-11-23T09:47:43Z
https://github.com/huggingface/datasets/issues/878#issuecomment-732047782
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDczMjA0Nzc4Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/732047782/reactions
2020-11-23T09:47:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732047782
https://avatars.githubusercontent.com/u/42795522?v=4
https://api.github.com/users/mahesh1amour/events{/privacy}
https://api.github.com/users/mahesh1amour/followers
https://api.github.com/users/mahesh1amour/following{/other_user}
https://api.github.com/users/mahesh1amour/gists{/gist_id}
https://github.com/mahesh1amour
mahesh1amour
MDQ6VXNlcjQyNzk1NTIy
https://api.github.com/users/mahesh1amour/orgs
https://api.github.com/users/mahesh1amour/received_events
https://api.github.com/users/mahesh1amour/repos
https://api.github.com/users/mahesh1amour/starred{/owner}{/repo}
https://api.github.com/users/mahesh1amour/subscriptions
User
https://api.github.com/users/mahesh1amour
public
MEMBER
It could maybe work almost out of the box just by using `cached_path` in the text/csv/json scripts, no?
2020-11-23T10:30:02Z
https://github.com/huggingface/datasets/issues/878#issuecomment-732070809
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDczMjA3MDgwOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/732070809/reactions
2020-11-23T10:30:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732070809
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
Thanks thomwolf and julien-c

I'm still confusion on what you guys said, 

I have solved the problem as follows:

1. read the csv file using pandas from s3 
2. Convert to dictionary key as column name and values as list column data
3. convert it to Dataset using 
`from datasets import Dataset`
`train_dataset = Dataset.from_dict(train_dict)`
2020-11-23T11:36:37Z
https://github.com/huggingface/datasets/issues/878#issuecomment-732106115
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDczMjEwNjExNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/732106115/reactions
2020-11-23T11:36:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732106115
https://avatars.githubusercontent.com/u/42795522?v=4
https://api.github.com/users/mahesh1amour/events{/privacy}
https://api.github.com/users/mahesh1amour/followers
https://api.github.com/users/mahesh1amour/following{/other_user}
https://api.github.com/users/mahesh1amour/gists{/gist_id}
https://github.com/mahesh1amour
mahesh1amour
MDQ6VXNlcjQyNzk1NTIy
https://api.github.com/users/mahesh1amour/orgs
https://api.github.com/users/mahesh1amour/received_events
https://api.github.com/users/mahesh1amour/repos
https://api.github.com/users/mahesh1amour/starred{/owner}{/repo}
https://api.github.com/users/mahesh1amour/subscriptions
User
https://api.github.com/users/mahesh1amour
public
MEMBER
We were brainstorming around your use-case.

Let's keep the issue open for now, I think this is an interesting question to think about.
2020-11-23T11:38:43Z
https://github.com/huggingface/datasets/issues/878#issuecomment-732107317
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDczMjEwNzMxNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/732107317/reactions
2020-11-23T11:38:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732107317
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
> We were brainstorming around your use-case.
> 
> Let's keep the issue open for now, I think this is an interesting question to think about.

Sure thomwolf, Thanks for your concern 
2020-11-23T11:40:46Z
https://github.com/huggingface/datasets/issues/878#issuecomment-732108513
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDczMjEwODUxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/732108513/reactions
2020-11-23T11:40:46Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732108513
https://avatars.githubusercontent.com/u/42795522?v=4
https://api.github.com/users/mahesh1amour/events{/privacy}
https://api.github.com/users/mahesh1amour/followers
https://api.github.com/users/mahesh1amour/following{/other_user}
https://api.github.com/users/mahesh1amour/gists{/gist_id}
https://github.com/mahesh1amour
mahesh1amour
MDQ6VXNlcjQyNzk1NTIy
https://api.github.com/users/mahesh1amour/orgs
https://api.github.com/users/mahesh1amour/received_events
https://api.github.com/users/mahesh1amour/repos
https://api.github.com/users/mahesh1amour/starred{/owner}{/repo}
https://api.github.com/users/mahesh1amour/subscriptions
User
https://api.github.com/users/mahesh1amour
public
MEMBER
I agree it would be cool to have that feature. Also that's good to know that pandas supports this.
For the moment I'd suggest to first download the files locally as thom suggested and then load the dataset by providing paths to the local files
2020-11-24T09:28:36Z
https://github.com/huggingface/datasets/issues/878#issuecomment-732771414
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDczMjc3MTQxNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/732771414/reactions
2020-11-24T09:28:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/732771414
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Don't get

2020-11-24T14:52:59Z
https://github.com/huggingface/datasets/issues/878#issuecomment-733024519
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDczMzAyNDUxOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/733024519/reactions
2020-11-24T14:52:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/733024519
https://avatars.githubusercontent.com/u/74707876?v=4
https://api.github.com/users/jman1973/events{/privacy}
https://api.github.com/users/jman1973/followers
https://api.github.com/users/jman1973/following{/other_user}
https://api.github.com/users/jman1973/gists{/gist_id}
https://github.com/jman1973
jman1973
MDQ6VXNlcjc0NzA3ODc2
https://api.github.com/users/jman1973/orgs
https://api.github.com/users/jman1973/received_events
https://api.github.com/users/jman1973/repos
https://api.github.com/users/jman1973/starred{/owner}{/repo}
https://api.github.com/users/jman1973/subscriptions
User
https://api.github.com/users/jman1973
public
NONE
Any updates on this issue?
I face a similar issue. I have many parquet files in S3 and I would like to train on them. 
To be honest I even face issues with only getting the last layer embedding out of them.
2020-12-21T14:29:51Z
https://github.com/huggingface/datasets/issues/878#issuecomment-749002067
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDc0OTAwMjA2Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/749002067/reactions
2020-12-21T14:29:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749002067
https://avatars.githubusercontent.com/u/10906407?v=4
https://api.github.com/users/dorlavie/events{/privacy}
https://api.github.com/users/dorlavie/followers
https://api.github.com/users/dorlavie/following{/other_user}
https://api.github.com/users/dorlavie/gists{/gist_id}
https://github.com/dorlavie
dorlavie
MDQ6VXNlcjEwOTA2NDA3
https://api.github.com/users/dorlavie/orgs
https://api.github.com/users/dorlavie/received_events
https://api.github.com/users/dorlavie/repos
https://api.github.com/users/dorlavie/starred{/owner}{/repo}
https://api.github.com/users/dorlavie/subscriptions
User
https://api.github.com/users/dorlavie
public
NONE
Hi dorlavie, 
You can find one solution that i have mentioned above, that can help you. 
And there is one more solution also which is downloading files locally

2020-12-21T15:18:23Z
https://github.com/huggingface/datasets/issues/878#issuecomment-749026411
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDc0OTAyNjQxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/749026411/reactions
2020-12-21T15:18:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749026411
https://avatars.githubusercontent.com/u/42795522?v=4
https://api.github.com/users/mahesh1amour/events{/privacy}
https://api.github.com/users/mahesh1amour/followers
https://api.github.com/users/mahesh1amour/following{/other_user}
https://api.github.com/users/mahesh1amour/gists{/gist_id}
https://github.com/mahesh1amour
mahesh1amour
MDQ6VXNlcjQyNzk1NTIy
https://api.github.com/users/mahesh1amour/orgs
https://api.github.com/users/mahesh1amour/received_events
https://api.github.com/users/mahesh1amour/repos
https://api.github.com/users/mahesh1amour/starred{/owner}{/repo}
https://api.github.com/users/mahesh1amour/subscriptions
User
https://api.github.com/users/mahesh1amour
public
NONE
> Hi dorlavie,
> You can find one solution that i have mentioned above, that can help you.
> And there is one more solution also which is downloading files locally

mahesh1amour, thanks for the fast reply

Unfortunately, in my case I can not read with pandas. The dataset is too big (50GB). 
In addition, due to security concerns I am not allowed to save the data locally
2020-12-21T15:30:09Z
https://github.com/huggingface/datasets/issues/878#issuecomment-749033122
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDc0OTAzMzEyMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/749033122/reactions
2020-12-21T15:30:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749033122
https://avatars.githubusercontent.com/u/10906407?v=4
https://api.github.com/users/dorlavie/events{/privacy}
https://api.github.com/users/dorlavie/followers
https://api.github.com/users/dorlavie/following{/other_user}
https://api.github.com/users/dorlavie/gists{/gist_id}
https://github.com/dorlavie
dorlavie
MDQ6VXNlcjEwOTA2NDA3
https://api.github.com/users/dorlavie/orgs
https://api.github.com/users/dorlavie/received_events
https://api.github.com/users/dorlavie/repos
https://api.github.com/users/dorlavie/starred{/owner}{/repo}
https://api.github.com/users/dorlavie/subscriptions
User
https://api.github.com/users/dorlavie
public
CONTRIBUTOR
@dorlavie could use `boto3` to download the data to your local machine and then load it with `dataset`

boto3 example [documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html)
```python
import boto3

s3 = boto3.client('s3')
s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME')
```

datasets example [documentation](https://huggingface.co/docs/datasets/loading_datasets.html)

```python
from datasets import load_dataset
dataset = load_dataset('csv', data_files=['my_file_1.csv', 'my_file_2.csv', 'my_file_3.csv'])
```

2020-12-21T16:11:36Z
https://github.com/huggingface/datasets/issues/878#issuecomment-749053724
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDc0OTA1MzcyNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/749053724/reactions
2020-12-21T16:11:36Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749053724
https://avatars.githubusercontent.com/u/32632186?v=4
https://api.github.com/users/philschmid/events{/privacy}
https://api.github.com/users/philschmid/followers
https://api.github.com/users/philschmid/following{/other_user}
https://api.github.com/users/philschmid/gists{/gist_id}
https://github.com/philschmid
philschmid
MDQ6VXNlcjMyNjMyMTg2
https://api.github.com/users/philschmid/orgs
https://api.github.com/users/philschmid/received_events
https://api.github.com/users/philschmid/repos
https://api.github.com/users/philschmid/starred{/owner}{/repo}
https://api.github.com/users/philschmid/subscriptions
User
https://api.github.com/users/philschmid
public
NONE
Thanks @philschmid for the suggestion.
As I mentioned in the previous comment, due to security issues I can not save the data locally.
I need to read it from S3 and process it directly.

I guess that many other people try to train / fit those models on huge datasets (e.g entire Wiki), what is the best practice in those cases?
2020-12-21T16:38:02Z
https://github.com/huggingface/datasets/issues/878#issuecomment-749066790
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDc0OTA2Njc5MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/749066790/reactions
2020-12-21T16:38:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749066790
https://avatars.githubusercontent.com/u/10906407?v=4
https://api.github.com/users/dorlavie/events{/privacy}
https://api.github.com/users/dorlavie/followers
https://api.github.com/users/dorlavie/following{/other_user}
https://api.github.com/users/dorlavie/gists{/gist_id}
https://github.com/dorlavie
dorlavie
MDQ6VXNlcjEwOTA2NDA3
https://api.github.com/users/dorlavie/orgs
https://api.github.com/users/dorlavie/received_events
https://api.github.com/users/dorlavie/repos
https://api.github.com/users/dorlavie/starred{/owner}{/repo}
https://api.github.com/users/dorlavie/subscriptions
User
https://api.github.com/users/dorlavie
public
MEMBER
If I understand correctly you're not allowed to write data on disk that you downloaded from S3 for example ?
Or is it the use of the `boto3` library that is not allowed in your case ?
2020-12-21T16:49:45Z
https://github.com/huggingface/datasets/issues/878#issuecomment-749074211
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDc0OTA3NDIxMQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/749074211/reactions
2020-12-21T16:49:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749074211
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq yes you are correct.
I am not allowed to save the "raw text" locally - The "raw text" must be saved only on S3.
I am allowed to save the output of any model locally. 
It doesn't matter how I do it boto3/pandas/pyarrow, it is forbidden
2020-12-21T17:03:59Z
https://github.com/huggingface/datasets/issues/878#issuecomment-749085918
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDc0OTA4NTkxOA==
https://api.github.com/repos/huggingface/datasets/issues/comments/749085918/reactions
2020-12-21T17:03:59Z
https://api.github.com/repos/huggingface/datasets/issues/comments/749085918
https://avatars.githubusercontent.com/u/10906407?v=4
https://api.github.com/users/dorlavie/events{/privacy}
https://api.github.com/users/dorlavie/followers
https://api.github.com/users/dorlavie/following{/other_user}
https://api.github.com/users/dorlavie/gists{/gist_id}
https://github.com/dorlavie
dorlavie
MDQ6VXNlcjEwOTA2NDA3
https://api.github.com/users/dorlavie/orgs
https://api.github.com/users/dorlavie/received_events
https://api.github.com/users/dorlavie/repos
https://api.github.com/users/dorlavie/starred{/owner}{/repo}
https://api.github.com/users/dorlavie/subscriptions
User
https://api.github.com/users/dorlavie
public
CONTRIBUTOR
@dorlavie are you using sagemaker for training too? Then you could use S3 URI, for example `s3://my-bucket/my-training-data` and pass it within the `.fit()` function when you start the sagemaker training job. Sagemaker would then download the data from s3 into the training runtime and you could load it from disk

**sagemaker start training job**
```python
pytorch_estimator.fit({'train':'s3://my-bucket/my-training-data','eval':'s3://my-bucket/my-evaluation-data'})
```

**in the train.py script**
```python
from datasets import load_from_disk

train_dataset = load_from_disk(os.environ['SM_CHANNEL_TRAIN'])
```

I have created an example of how to use transformers and datasets with sagemaker. 
https://github.com/philschmid/huggingface-sagemaker-example/tree/main/03_huggingface_sagemaker_trainer_with_data_from_s3

The example contains a jupyter notebook `sagemaker-example.ipynb` and an `src/` folder. The sagemaker-example is a jupyter notebook that is used to create the training job on AWS Sagemaker. The `src/` folder contains the `train.py`, our training script, and `requirements.txt` for additional dependencies.


2020-12-23T09:53:08Z
https://github.com/huggingface/datasets/issues/878#issuecomment-750055807
https://api.github.com/repos/huggingface/datasets/issues/878
MDEyOklzc3VlQ29tbWVudDc1MDA1NTgwNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/750055807/reactions
2020-12-23T09:53:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/750055807
https://avatars.githubusercontent.com/u/32632186?v=4
https://api.github.com/users/philschmid/events{/privacy}
https://api.github.com/users/philschmid/followers
https://api.github.com/users/philschmid/following{/other_user}
https://api.github.com/users/philschmid/gists{/gist_id}
https://github.com/philschmid
philschmid
MDQ6VXNlcjMyNjMyMTg2
https://api.github.com/users/philschmid/orgs
https://api.github.com/users/philschmid/received_events
https://api.github.com/users/philschmid/repos
https://api.github.com/users/philschmid/starred{/owner}{/repo}
https://api.github.com/users/philschmid/subscriptions
User
https://api.github.com/users/philschmid
public
In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'
    
    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`


I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load