Bug loading `sentence-transformers/parallel-sentences`
CONTRIBUTOR
tagging @nreimers 
2022-10-06T10:48:11Z
https://github.com/huggingface/datasets/issues/5081#issuecomment-1269796053
https://api.github.com/repos/huggingface/datasets/issues/5081
IC_kwDODunzps5Lr4zV
https://api.github.com/repos/huggingface/datasets/issues/comments/1269796053/reactions
2022-10-06T10:48:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1269796053
https://avatars.githubusercontent.com/u/229382?v=4
https://api.github.com/users/PhilipMay/events{/privacy}
https://api.github.com/users/PhilipMay/followers
https://api.github.com/users/PhilipMay/following{/other_user}
https://api.github.com/users/PhilipMay/gists{/gist_id}
https://github.com/PhilipMay
PhilipMay
MDQ6VXNlcjIyOTM4Mg==
https://api.github.com/users/PhilipMay/orgs
https://api.github.com/users/PhilipMay/received_events
https://api.github.com/users/PhilipMay/repos
https://api.github.com/users/PhilipMay/starred{/owner}{/repo}
https://api.github.com/users/PhilipMay/subscriptions
User
https://api.github.com/users/PhilipMay
public
NONE
The dataset is sadly not really compatible to be loaded with `load_dataset`. So far it is better to git clone it and to use the files directly.

A data loading script would be needed to be added to this dataset. But this was too much overhead / not really intuitive how to create it.
2022-10-06T11:46:40Z
https://github.com/huggingface/datasets/issues/5081#issuecomment-1269883672
https://api.github.com/repos/huggingface/datasets/issues/5081
IC_kwDODunzps5LsOMY
https://api.github.com/repos/huggingface/datasets/issues/comments/1269883672/reactions
2022-10-06T11:46:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1269883672
https://avatars.githubusercontent.com/u/10706961?v=4
https://api.github.com/users/nreimers/events{/privacy}
https://api.github.com/users/nreimers/followers
https://api.github.com/users/nreimers/following{/other_user}
https://api.github.com/users/nreimers/gists{/gist_id}
https://github.com/nreimers
nreimers
MDQ6VXNlcjEwNzA2OTYx
https://api.github.com/users/nreimers/orgs
https://api.github.com/users/nreimers/received_events
https://api.github.com/users/nreimers/repos
https://api.github.com/users/nreimers/starred{/owner}{/repo}
https://api.github.com/users/nreimers/subscriptions
User
https://api.github.com/users/nreimers
public
MEMBER
Since the dataset is a bunch of TSVs we should not need a dataset script I think.

By default it tries to load all the TSVs at once, which fails here because they don't all have the same columns (pd.read_csv uses the first line as header by default). But those files have no header ! So, to properly load any TSV file in this repo, one has to pass `names=[...]` for pd.read_csv to know which column names to use.

To fix this situation, we can either do
1. replace the TSVs by TSV with column names
2. OR specify the pd.read_csv kwargs as YAML in the dataset card - and `datasets` would use that by default

WDTY ?
2022-10-10T15:50:39Z
https://github.com/huggingface/datasets/issues/5081#issuecomment-1273515384
https://api.github.com/repos/huggingface/datasets/issues/5081
IC_kwDODunzps5L6E14
https://api.github.com/repos/huggingface/datasets/issues/comments/1273515384/reactions
2022-10-10T15:51:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1273515384
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
There are more issues in the dataset.
To load OpenSubtitles I have to provide this (see `skiprows`):

```python
df_os = pd.read_csv(
    "./parallel-sentences/OpenSubtitles/OpenSubtitles-en-de-train.tsv.gz", 
    sep="\t", 
    quoting=csv.QUOTE_NONE,
    header=None,
    names=["en", "de"],
    skiprows=[540344, 9151700, 10040173, 10040199, 11314673, 11338258, 11869223, 12159297, 12251078, 12303334],
)
```
2022-10-10T15:56:43Z
https://github.com/huggingface/datasets/issues/5081#issuecomment-1273522808
https://api.github.com/repos/huggingface/datasets/issues/5081
IC_kwDODunzps5L6Gp4
https://api.github.com/repos/huggingface/datasets/issues/comments/1273522808/reactions
2022-10-10T15:56:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1273522808
https://avatars.githubusercontent.com/u/229382?v=4
https://api.github.com/users/PhilipMay/events{/privacy}
https://api.github.com/users/PhilipMay/followers
https://api.github.com/users/PhilipMay/following{/other_user}
https://api.github.com/users/PhilipMay/gists{/gist_id}
https://github.com/PhilipMay
PhilipMay
MDQ6VXNlcjIyOTM4Mg==
https://api.github.com/users/PhilipMay/orgs
https://api.github.com/users/PhilipMay/received_events
https://api.github.com/users/PhilipMay/repos
https://api.github.com/users/PhilipMay/starred{/owner}{/repo}
https://api.github.com/users/PhilipMay/subscriptions
User
https://api.github.com/users/PhilipMay
public
MEMBER
What's wrong with those lines exactly ?
Maybe passing `error_bad_lines=False` (and maybe `warn_bad_lines=True`) can be helpful
2022-10-10T16:17:01Z
https://github.com/huggingface/datasets/issues/5081#issuecomment-1273547637
https://api.github.com/repos/huggingface/datasets/issues/5081
IC_kwDODunzps5L6Mt1
https://api.github.com/repos/huggingface/datasets/issues/comments/1273547637/reactions
2022-10-10T16:17:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1273547637
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
CONTRIBUTOR
> What's wrong with those lines exactly ? 

Stuff like this: `ParserError: Error tokenizing data. C error: Expected 2 fields in line 540345, saw 3`


2022-10-10T16:42:07Z
https://github.com/huggingface/datasets/issues/5081#issuecomment-1273575258
https://api.github.com/repos/huggingface/datasets/issues/5081
IC_kwDODunzps5L6Tda
https://api.github.com/repos/huggingface/datasets/issues/comments/1273575258/reactions
2022-10-10T16:42:14Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1273575258
https://avatars.githubusercontent.com/u/229382?v=4
https://api.github.com/users/PhilipMay/events{/privacy}
https://api.github.com/users/PhilipMay/followers
https://api.github.com/users/PhilipMay/following{/other_user}
https://api.github.com/users/PhilipMay/gists{/gist_id}
https://github.com/PhilipMay
PhilipMay
MDQ6VXNlcjIyOTM4Mg==
https://api.github.com/users/PhilipMay/orgs
https://api.github.com/users/PhilipMay/received_events
https://api.github.com/users/PhilipMay/repos
https://api.github.com/users/PhilipMay/starred{/owner}{/repo}
https://api.github.com/users/PhilipMay/subscriptions
User
https://api.github.com/users/PhilipMay
public
CONTRIBUTOR
>  Maybe passing error_bad_lines=False (and maybe warn_bad_lines=True) can be helpful

Yes. That would hide the issue but not solve it.
2022-10-11T06:01:47Z
https://github.com/huggingface/datasets/issues/5081#issuecomment-1274127199
https://api.github.com/repos/huggingface/datasets/issues/5081
IC_kwDODunzps5L8aNf
https://api.github.com/repos/huggingface/datasets/issues/comments/1274127199/reactions
2022-10-11T06:01:47Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1274127199
https://avatars.githubusercontent.com/u/229382?v=4
https://api.github.com/users/PhilipMay/events{/privacy}
https://api.github.com/users/PhilipMay/followers
https://api.github.com/users/PhilipMay/following{/other_user}
https://api.github.com/users/PhilipMay/gists{/gist_id}
https://github.com/PhilipMay
PhilipMay
MDQ6VXNlcjIyOTM4Mg==
https://api.github.com/users/PhilipMay/orgs
https://api.github.com/users/PhilipMay/received_events
https://api.github.com/users/PhilipMay/repos
https://api.github.com/users/PhilipMay/starred{/owner}{/repo}
https://api.github.com/users/PhilipMay/subscriptions
User
https://api.github.com/users/PhilipMay
public
MEMBER
@nreimers WDYT about the two options mentioned above ?
2022-10-11T10:00:48Z
https://github.com/huggingface/datasets/issues/5081#issuecomment-1274441985
https://api.github.com/repos/huggingface/datasets/issues/5081
IC_kwDODunzps5L9nEB
https://api.github.com/repos/huggingface/datasets/issues/comments/1274441985/reactions
2022-10-11T10:00:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1274441985
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("sentence-transformers/parallel-sentences")
```

raises this:

```
/home/phmay/miniconda3/envs/paraphrase-mining/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py:697: FutureWarning: the 'mangle_dupe_cols' keyword is deprecated and will be removed in a future version. Please take steps to stop the use of 'mangle_dupe_cols'
  return pd.read_csv(xopen(filepath_or_buffer, "rb", use_auth_token=use_auth_token), **kwargs)
/home/phmay/miniconda3/envs/paraphrase-mining/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py:697: FutureWarning: the 'mangle_dupe_cols' keyword is deprecated and will be removed in a future version. Please take steps to stop the use of 'mangle_dupe_cols'
  return pd.read_csv(xopen(filepath_or_buffer, "rb", use_auth_token=use_auth_token), **kwargs)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In [4], line 1
----> 1 dataset = load_dataset("sentence-transformers/parallel-sentences", split="train")

File ~/miniconda3/envs/paraphrase-mining/lib/python3.9/site-packages/datasets/load.py:1693, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1690 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   1692 # Download and prepare data
-> 1693 builder_instance.download_and_prepare(
   1694     download_config=download_config,
   1695     download_mode=download_mode,
   1696     ignore_verifications=ignore_verifications,
   1697     try_from_hf_gcs=try_from_hf_gcs,
   1698     use_auth_token=use_auth_token,
   1699 )
   1701 # Build dataset for splits
   1702 keep_in_memory = (
   1703     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1704 )

File ~/miniconda3/envs/paraphrase-mining/lib/python3.9/site-packages/datasets/builder.py:807, in DatasetBuilder.download_and_prepare(self, output_dir, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, storage_options, **download_and_prepare_kwargs)
    801 if not downloaded_from_gcs:
    802     prepare_split_kwargs = {
    803         "file_format": file_format,
    804         "max_shard_size": max_shard_size,
    805         **download_and_prepare_kwargs,
    806     }
--> 807     self._download_and_prepare(
    808         dl_manager=dl_manager,
    809         verify_infos=verify_infos,
    810         **prepare_split_kwargs,
    811         **download_and_prepare_kwargs,
    812     )
    813 # Sync info
    814 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~/miniconda3/envs/paraphrase-mining/lib/python3.9/site-packages/datasets/builder.py:898, in DatasetBuilder._download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    894 split_dict.add(split_generator.split_info)
    896 try:
    897     # Prepare split will record examples associated to the split
--> 898     self._prepare_split(split_generator, **prepare_split_kwargs)
    899 except OSError as e:
    900     raise OSError(
    901         "Cannot find data file. "
    902         + (self.manual_download_instructions or "")
    903         + "\nOriginal error:\n"
    904         + str(e)
    905     ) from None

File ~/miniconda3/envs/paraphrase-mining/lib/python3.9/site-packages/datasets/builder.py:1513, in ArrowBasedBuilder._prepare_split(self, split_generator, file_format, max_shard_size)
   1506             shard_id += 1
   1507             writer = writer_class(
   1508                 features=writer._features,
   1509                 path=fpath.replace("SSSSS", f"{shard_id:05d}"),
   1510                 storage_options=self._fs.storage_options,
   1511                 embed_local_files=embed_local_files,
   1512             )
-> 1513         writer.write_table(table)
   1514 finally:
   1515     num_shards = shard_id + 1

File ~/miniconda3/envs/paraphrase-mining/lib/python3.9/site-packages/datasets/arrow_writer.py:540, in ArrowWriter.write_table(self, pa_table, writer_batch_size)
    538 if self.pa_writer is None:
    539     self._build_writer(inferred_schema=pa_table.schema)
--> 540 pa_table = table_cast(pa_table, self._schema)
    541 if self.embed_local_files:
    542     pa_table = embed_table_storage(pa_table)

File ~/miniconda3/envs/paraphrase-mining/lib/python3.9/site-packages/datasets/table.py:2044, in table_cast(table, schema)
   2032 """Improved version of pa.Table.cast.
   2033 
   2034 It supports casting to feature types stored in the schema metadata.
   (...)
   2041     table (:obj:`pyarrow.Table`): the casted table
   2042 """
   2043 if table.schema != schema:
-> 2044     return cast_table_to_schema(table, schema)
   2045 elif table.schema.metadata != schema.metadata:
   2046     return table.replace_schema_metadata(schema.metadata)

File ~/miniconda3/envs/paraphrase-mining/lib/python3.9/site-packages/datasets/table.py:2005, in cast_table_to_schema(table, schema)
   2003 features = Features.from_arrow_schema(schema)
   2004 if sorted(table.column_names) != sorted(features):
-> 2005     raise ValueError(f"Couldn't cast\n{table.schema}\nto\n{features}\nbecause column names don't match")
   2006 arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
   2007 return pa.Table.from_arrays(arrays, schema=schema)

ValueError: Couldn't cast
Action taken on Parliament's resolutions: see Minutes: string
Následný postup na základě usnesení Parlamentu: viz zápis: string
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 742
to
{'Membership of Parliament: see Minutes': Value(dtype='string', id=None), 'Състав на Парламента: вж. протоколи': Value(dtype='string', id=None)}
because column names don't match
```

## Expected results
no error

## Actual results
error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Linux
- Python version: Python 3.9.13
- PyArrow version: pyarrow               9.0.0
- transformers          4.22.2
- datasets              2.5.2