Unable to download dataset using Azure Data Lake Gen 2
MEMBER
Hi ! From the `adlfs` docs, there are two filesystems you can use:
> To use the Gen1 filesystem:
> - known_implementations[‘adl’] = {‘class’: ‘adlfs.AzureDatalakeFileSystem’}
> 
> To use the Gen2 filesystem:
>  - known_implementations[‘abfs’] = {‘class’: ‘adlfs.AzureBlobFileSystem’}

If I'm not mistaken you're using the second one - so you should use `abfs://` instead of `adl://`, and also run this at the beginning of your script:
```python
from fsspec.registry import known_implementations
known_implementations['abfs'] = {'class': 'adlfs.AzureDatalakeFileSystem'}
```


2022-10-25T09:54:14Z
https://github.com/huggingface/datasets/issues/5156#issuecomment-1290289415
https://api.github.com/repos/huggingface/datasets/issues/5156
IC_kwDODunzps5M6EEH
https://api.github.com/repos/huggingface/datasets/issues/comments/1290289415/reactions
2022-10-25T09:54:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1290289415
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thank you @lhoestq . Great call.
Using the default class from `known_implementations` dict solved my problem
```
known_implementations[‘abfs’] = {‘class’: ‘adlfs.AzureBlobFileSystem’}
```
I'm closing this issue.
2022-11-17T23:37:08Z
https://github.com/huggingface/datasets/issues/5156#issuecomment-1319351374
https://api.github.com/repos/huggingface/datasets/issues/5156
IC_kwDODunzps5Oo7RO
https://api.github.com/repos/huggingface/datasets/issues/comments/1319351374/reactions
2022-11-17T23:37:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1319351374
https://avatars.githubusercontent.com/u/87379512?v=4
https://api.github.com/users/clarissesimoes/events{/privacy}
https://api.github.com/users/clarissesimoes/followers
https://api.github.com/users/clarissesimoes/following{/other_user}
https://api.github.com/users/clarissesimoes/gists{/gist_id}
https://github.com/clarissesimoes
clarissesimoes
MDQ6VXNlcjg3Mzc5NTEy
https://api.github.com/users/clarissesimoes/orgs
https://api.github.com/users/clarissesimoes/received_events
https://api.github.com/users/clarissesimoes/repos
https://api.github.com/users/clarissesimoes/starred{/owner}{/repo}
https://api.github.com/users/clarissesimoes/subscriptions
User
https://api.github.com/users/clarissesimoes
public
NONE
> Thank you @lhoestq . Great call. Using the default class from `known_implementations` dict solved my problem
> 
> ```
> known_implementations[‘abfs’] = {‘class’: ‘adlfs.AzureBlobFileSystem’}
> ```
> 
> I'm closing this issue.

Hi so here  `Saving serialized datasets

After you have processed your dataset, you can save it to your cloud storage with [Dataset.save_to_disk()](https://huggingface.co/docs/datasets/v2.17.0/en/package_reference/main_classes#datasets.Dataset.save_to_disk):` what is the encoded dataset I have failed to save it 
2024-02-15T09:14:18Z
https://github.com/huggingface/datasets/issues/5156#issuecomment-1945663815
https://api.github.com/repos/huggingface/datasets/issues/5156
IC_kwDODunzps5z-HlH
https://api.github.com/repos/huggingface/datasets/issues/comments/1945663815/reactions
2024-02-15T09:14:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1945663815
https://avatars.githubusercontent.com/u/37067613?v=4
https://api.github.com/users/EricPeter/events{/privacy}
https://api.github.com/users/EricPeter/followers
https://api.github.com/users/EricPeter/following{/other_user}
https://api.github.com/users/EricPeter/gists{/gist_id}
https://github.com/EricPeter
EricPeter
MDQ6VXNlcjM3MDY3NjEz
https://api.github.com/users/EricPeter/orgs
https://api.github.com/users/EricPeter/received_events
https://api.github.com/users/EricPeter/repos
https://api.github.com/users/EricPeter/starred{/owner}{/repo}
https://api.github.com/users/EricPeter/subscriptions
User
https://api.github.com/users/EricPeter
public
MEMBER
Uploading failed ? Did you get an error message ?
2024-02-15T09:48:34Z
https://github.com/huggingface/datasets/issues/5156#issuecomment-1945720443
https://api.github.com/repos/huggingface/datasets/issues/5156
IC_kwDODunzps5z-VZ7
https://api.github.com/repos/huggingface/datasets/issues/comments/1945720443/reactions
2024-02-15T09:48:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1945720443
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
### Describe the bug

When using the DatasetBuilder method with the credentials for the cloud storage Azure Data Lake (adl) Gen2, the following error is showed:
```
Traceback (most recent call last):
  File "download_hf_dataset.py", line 143, in <module>
    main()
  File "download_hf_dataset.py", line 102, in main
    builder.download_and_prepare(save_dir, storage_options=storage_options, max_shard_size="250MB", file_format="parquet")
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/datasets/builder.py", line 671, in download_and_prepare
    fs_token_paths = fsspec.get_fs_token_paths(output_dir, storage_options=storage_options)
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/fsspec/core.py", line 639, in get_fs_token_paths
    fs = cls(**options)
  File "/home/clarisses/miniconda3/envs/hf_datasets_env/lib/python3.8/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'account_name'
```

If I don't pass the storage_options argument (leave it as None), it requires the credentials used in ADL Gen 1:
`TypeError: __init__() missing 3 required positional arguments: 'tenant_id', 'client_id', and 'client_secret'`

Thus, it is not possible to download a dataset from the cloud using Azure Data Lake (adl) Gen2.

### Steps to reproduce the bug

Assuming that you have an account on Azure and at Storage Account that can be used for reproduce:

1. Create a dict with the format to connect to Azure Data Lake Gen 2
```
storage_options = {"account_name": ACCOUNT_NAME, "account_key": ACCOUNT_KEY)  # gen 2 filesystem
```
2. Create a dataset builder for any HF hosted dataset
```
builder = load_dataset_builder(dataset_name)
```
3. Try to download the dataset passing the storage_options as an argument
```
save_dir = 'adl://my_save_dir'
builder.download_and_prepare(save_dir, storage_options=storage_options, max_shard_size="250MB", file_format="parquet")
```

### Expected behavior

Not seeing the error mentioned above and being able to download the dataset to the provided path on ADL

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.5.1