Problem about wmt17 zh-en dataset
CONTRIBUTOR
Running into the same error with `wmt17/zh-en`, `wmt18/zh-en` and `wmt19/zh-en`.
2022-06-29T18:32:39Z
https://github.com/huggingface/datasets/issues/4575#issuecomment-1170351735
https://api.github.com/repos/huggingface/datasets/issues/4575
IC_kwDODunzps5FwiZ3
https://api.github.com/repos/huggingface/datasets/issues/comments/1170351735/reactions
2022-06-29T18:32:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1170351735
https://avatars.githubusercontent.com/u/16107619?v=4
https://api.github.com/users/VictorSanh/events{/privacy}
https://api.github.com/users/VictorSanh/followers
https://api.github.com/users/VictorSanh/following{/other_user}
https://api.github.com/users/VictorSanh/gists{/gist_id}
https://github.com/VictorSanh
VictorSanh
MDQ6VXNlcjE2MTA3NjE5
https://api.github.com/users/VictorSanh/orgs
https://api.github.com/users/VictorSanh/received_events
https://api.github.com/users/VictorSanh/repos
https://api.github.com/users/VictorSanh/starred{/owner}{/repo}
https://api.github.com/users/VictorSanh/subscriptions
User
https://api.github.com/users/VictorSanh
public
NONE
@albertvillanova @lhoestq Could you take a look at this issue?
2022-08-12T15:40:51Z
https://github.com/huggingface/datasets/issues/4575#issuecomment-1213248893
https://api.github.com/repos/huggingface/datasets/issues/4575
IC_kwDODunzps5IULV9
https://api.github.com/repos/huggingface/datasets/issues/comments/1213248893/reactions
2022-08-12T15:40:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1213248893
https://avatars.githubusercontent.com/u/37647985?v=4
https://api.github.com/users/StevenTang1998/events{/privacy}
https://api.github.com/users/StevenTang1998/followers
https://api.github.com/users/StevenTang1998/following{/other_user}
https://api.github.com/users/StevenTang1998/gists{/gist_id}
https://github.com/StevenTang1998
StevenTang1998
MDQ6VXNlcjM3NjQ3OTg1
https://api.github.com/users/StevenTang1998/orgs
https://api.github.com/users/StevenTang1998/received_events
https://api.github.com/users/StevenTang1998/repos
https://api.github.com/users/StevenTang1998/starred{/owner}{/repo}
https://api.github.com/users/StevenTang1998/subscriptions
User
https://api.github.com/users/StevenTang1998
public
NONE
@winterfell2021 Hi, I wonder where the code you provided should be added. I tried to add them in the `datasets/table.py` in `array_cast` function, however, the 'zh' item is none.
2022-08-13T01:58:47Z
https://github.com/huggingface/datasets/issues/4575#issuecomment-1213631121
https://api.github.com/repos/huggingface/datasets/issues/4575
IC_kwDODunzps5IVoqR
https://api.github.com/repos/huggingface/datasets/issues/comments/1213631121/reactions
2022-08-13T01:58:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1213631121
https://avatars.githubusercontent.com/u/37647985?v=4
https://api.github.com/users/StevenTang1998/events{/privacy}
https://api.github.com/users/StevenTang1998/followers
https://api.github.com/users/StevenTang1998/following{/other_user}
https://api.github.com/users/StevenTang1998/gists{/gist_id}
https://github.com/StevenTang1998
StevenTang1998
MDQ6VXNlcjM3NjQ3OTg1
https://api.github.com/users/StevenTang1998/orgs
https://api.github.com/users/StevenTang1998/received_events
https://api.github.com/users/StevenTang1998/repos
https://api.github.com/users/StevenTang1998/starred{/owner}{/repo}
https://api.github.com/users/StevenTang1998/subscriptions
User
https://api.github.com/users/StevenTang1998
public
NONE
I found some 'zh' item is none while 'c[hn]' is not.
So the code may change to:
```python
if 'c[hn]' in str(array.type):
        py_array = array.to_pylist()
        data_list = []
        for vo in py_array:
            tmp = {
                'en': vo['en'],
            }
            if vo.get('zh'):
                tmp['zh'] = vo['zh']
            else:
                tmp['zh'] = vo['c[hn]']
            data_list.append(tmp)
        array = pa.array(data_list, type=pa.struct([
            pa.field('en', pa.string()),
            pa.field('zh', pa.string()),
        ]))
```
2022-08-13T04:31:30Z
https://github.com/huggingface/datasets/issues/4575#issuecomment-1213653252
https://api.github.com/repos/huggingface/datasets/issues/4575
IC_kwDODunzps5IVuEE
https://api.github.com/repos/huggingface/datasets/issues/comments/1213653252/reactions
2022-08-13T04:31:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1213653252
https://avatars.githubusercontent.com/u/37647985?v=4
https://api.github.com/users/StevenTang1998/events{/privacy}
https://api.github.com/users/StevenTang1998/followers
https://api.github.com/users/StevenTang1998/following{/other_user}
https://api.github.com/users/StevenTang1998/gists{/gist_id}
https://github.com/StevenTang1998
StevenTang1998
MDQ6VXNlcjM3NjQ3OTg1
https://api.github.com/users/StevenTang1998/orgs
https://api.github.com/users/StevenTang1998/received_events
https://api.github.com/users/StevenTang1998/repos
https://api.github.com/users/StevenTang1998/starred{/owner}{/repo}
https://api.github.com/users/StevenTang1998/subscriptions
User
https://api.github.com/users/StevenTang1998
public
MEMBER
I just pushed a fix, we'll do a new release of `datasets` soon to include this fix. In the meantime you can use the fixed dataset by passing `revision="main"` to `load_dataset`
2022-08-23T10:01:02Z
https://github.com/huggingface/datasets/issues/4575#issuecomment-1223844270
https://api.github.com/repos/huggingface/datasets/issues/4575
IC_kwDODunzps5I8mGu
https://api.github.com/repos/huggingface/datasets/issues/comments/1223844270/reactions
2022-08-23T10:01:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1223844270
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
It seems that in subset casia2015, some samples are like `{'c[hn]':'xxx', 'en': 'aa'}`.
So when using `data = load_dataset('wmt17', "zh-en")` to load the wmt17 zh-en dataset, which will raise the exception:
```
Traceback (most recent call last):
  File "train.py", line 78, in <module>
    data = load_dataset(args.dataset, "zh-en")
  File "/usr/local/lib/python3.7/dist-packages/datasets/load.py", line 1684, in load_dataset
    use_auth_token=use_auth_token,
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 705, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 1221, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 793, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 1215, in _prepare_split
    num_examples, num_bytes = writer.finalize()
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py", line 533, in finalize
    self.write_examples_on_file()
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py", line 410, in write_examples_on_file
    self.write_batch(batch_examples=batch_examples)
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py", line 503, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 230, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py", line 198, in __arrow_array__
    out = cast_array_to_feature(out, type, allow_number_to_str=not self.trying_type)
  File "/usr/local/lib/python3.7/dist-packages/datasets/table.py", line 1675, in wrapper
    return func(array, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/datasets/table.py", line 1846, in cast_array_to_feature
    return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
  File "/usr/local/lib/python3.7/dist-packages/datasets/table.py", line 1675, in wrapper
    return func(array, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/datasets/table.py", line 1756, in array_cast
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{pa_type}")
TypeError: Couldn't cast array of type
struct<c[hn]: string, en: string, zh: string>
to
struct<en: string, zh: string>
```

So the solution of this problem is to change the original array manually:
```
if 'c[hn]' in str(array.type):
        py_array = array.to_pylist()
        data_list = []
        for vo in py_array:
            tmp = {
                'en': vo['en'],
            }
            if 'zh' not in vo:
                tmp['zh'] = vo['c[hn]']
            else:
                tmp['zh'] = vo['zh']
            data_list.append(tmp)
        array = pa.array(data_list, type=pa.struct([
            pa.field('en', pa.string()),
            pa.field('zh', pa.string()),
        ]))
```

Therefore, maybe a correct version of original casia2015 file need to be updated