load_dataset('emotion')    UnicodeDecodeError
NONE
I met the same problem, here is my code:
```
from datasets import load_dataset

ds_name = "togethercomputer/RedPajama-Data-1T"
ds = load_dataset(ds_name, download_mode=DownloadMode.FORCE_REDOWNLOAD)
```
And output error is:
```
Traceback (most recent call last):
  File "/home/yatorho/doc/projs/TransformerEngine/local/download_redpajama.py", line 10, in <module>
    ds = load_dataset(ds_name, download_mode=DownloadMode.FORCE_REDOWNLOAD)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 2606, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 2277, in load_dataset_builder
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 1923, in dataset_module_factory
    raise e1 from None
  File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 1875, in dataset_module_factory
    can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read()
                                                                       ^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte
```
My `datasets` version is 2.21.0. Any help here would be appreciated!

2024-09-05T03:06:48Z
https://github.com/huggingface/datasets/issues/6851#issuecomment-2330507609
https://api.github.com/repos/huggingface/datasets/issues/6851
IC_kwDODunzps6K6LlZ
https://api.github.com/repos/huggingface/datasets/issues/comments/2330507609/reactions
2024-09-05T03:06:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2330507609
https://avatars.githubusercontent.com/u/97348748?v=4
https://api.github.com/users/yatorho/events{/privacy}
https://api.github.com/users/yatorho/followers
https://api.github.com/users/yatorho/following{/other_user}
https://api.github.com/users/yatorho/gists{/gist_id}
https://github.com/yatorho
yatorho
U_kgDOBc1sjA
https://api.github.com/users/yatorho/orgs
https://api.github.com/users/yatorho/received_events
https://api.github.com/users/yatorho/repos
https://api.github.com/users/yatorho/starred{/owner}{/repo}
https://api.github.com/users/yatorho/subscriptions
User
https://api.github.com/users/yatorho
public
NONE
> I met the same problem, here is my code:
> 
> ```
> from datasets import load_dataset
> 
> ds_name = "togethercomputer/RedPajama-Data-1T"
> ds = load_dataset(ds_name, download_mode=DownloadMode.FORCE_REDOWNLOAD)
> ```
> 
> And output error is:
> 
> ```
> Traceback (most recent call last):
>   File "/home/yatorho/doc/projs/TransformerEngine/local/download_redpajama.py", line 10, in <module>
>     ds = load_dataset(ds_name, download_mode=DownloadMode.FORCE_REDOWNLOAD)
>          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 2606, in load_dataset
>     builder_instance = load_dataset_builder(
>                        ^^^^^^^^^^^^^^^^^^^^^
>   File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 2277, in load_dataset_builder
>     dataset_module = dataset_module_factory(
>                      ^^^^^^^^^^^^^^^^^^^^^^^
>   File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 1923, in dataset_module_factory
>     raise e1 from None
>   File "/home/yatorho/anaconda3/envs/t24/lib/python3.12/site-packages/datasets/load.py", line 1875, in dataset_module_factory
>     can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read()
>                                                                        ^^^^^^^^
>   File "<frozen codecs>", line 322, in decode
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 1: invalid start byte
> ```
> 
> My `datasets` version is 2.21.0. Any help here would be appreciated!

I passed encoding="utf-16" to the `load_dataset` call and now it works for me.
```
ds = load_dataset(ds_name, download_mode=DownloadMode.FORCE_REDOWNLOAD, encoding="utf-16")
```
2024-09-05T03:11:03Z
https://github.com/huggingface/datasets/issues/6851#issuecomment-2330511091
https://api.github.com/repos/huggingface/datasets/issues/6851
IC_kwDODunzps6K6Mbz
https://api.github.com/repos/huggingface/datasets/issues/comments/2330511091/reactions
2024-09-05T03:11:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2330511091
https://avatars.githubusercontent.com/u/97348748?v=4
https://api.github.com/users/yatorho/events{/privacy}
https://api.github.com/users/yatorho/followers
https://api.github.com/users/yatorho/following{/other_user}
https://api.github.com/users/yatorho/gists{/gist_id}
https://github.com/yatorho
yatorho
U_kgDOBc1sjA
https://api.github.com/users/yatorho/orgs
https://api.github.com/users/yatorho/received_events
https://api.github.com/users/yatorho/repos
https://api.github.com/users/yatorho/starred{/owner}{/repo}
https://api.github.com/users/yatorho/subscriptions
User
https://api.github.com/users/yatorho
public
### Describe the bug

**emotions = load_dataset('emotion')**

_UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte_

### Steps to reproduce the bug

load_dataset('emotion') 

### Expected behavior

succese

### Environment info

py3.10
transformers  4.41.0.dev0
datasets     2.19.0