medical_dialog zh has very slow _generate_examples
MEMBER
Hi @nbroad1881, thanks for reporting.

Let me have a look to try to improve its performance. 
2022-04-08T09:00:48Z
https://github.com/huggingface/datasets/issues/4122#issuecomment-1092623643
https://api.github.com/repos/huggingface/datasets/issues/4122
IC_kwDODunzps5BIB0b
https://api.github.com/repos/huggingface/datasets/issues/comments/1092623643/reactions
2022-04-08T09:00:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1092623643
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
CONTRIBUTOR
Thanks @nbroad1881  for reporting! I don't recall it taking so long. I will also have a look at this. 
@albertvillanova please let me know if I am doing something unnecessary or time consuming.
2022-04-08T09:34:44Z
https://github.com/huggingface/datasets/issues/4122#issuecomment-1092657656
https://api.github.com/repos/huggingface/datasets/issues/4122
IC_kwDODunzps5BIKH4
https://api.github.com/repos/huggingface/datasets/issues/comments/1092657656/reactions
2022-04-08T09:34:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1092657656
https://avatars.githubusercontent.com/u/16264631?v=4
https://api.github.com/users/vrindaprabhu/events{/privacy}
https://api.github.com/users/vrindaprabhu/followers
https://api.github.com/users/vrindaprabhu/following{/other_user}
https://api.github.com/users/vrindaprabhu/gists{/gist_id}
https://github.com/vrindaprabhu
vrindaprabhu
MDQ6VXNlcjE2MjY0NjMx
https://api.github.com/users/vrindaprabhu/orgs
https://api.github.com/users/vrindaprabhu/received_events
https://api.github.com/users/vrindaprabhu/repos
https://api.github.com/users/vrindaprabhu/starred{/owner}{/repo}
https://api.github.com/users/vrindaprabhu/subscriptions
User
https://api.github.com/users/vrindaprabhu
public
MEMBER
Hi @nbroad1881 and @vrindaprabhu,

As a workaround for the performance of the parsing of the raw data files (this could be addressed in a subsequent PR), I have found that there are also processed data files, that do not require parsing. I have added these as new configurations `processed.en` and `processed.zh`:
```python
ds = load_dataset("medical_dialog", "processed.zh")
```
2022-04-08T13:12:38Z
https://github.com/huggingface/datasets/issues/4122#issuecomment-1092845483
https://api.github.com/repos/huggingface/datasets/issues/4122
IC_kwDODunzps5BI3-r
https://api.github.com/repos/huggingface/datasets/issues/comments/1092845483/reactions
2022-04-08T13:12:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1092845483
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
## Describe the bug
After downloading the files from Google Drive, `load_dataset("medical_dialog", "zh", data_dir="./")` takes an unreasonable amount of time. Generating the train/test split for 33% of the dataset takes over 4.5 hours.

## Steps to reproduce the bug
The easiest way I've found to download files from Google Drive is to use `gdown` and use Google Colab because the download speeds will be very high due to the fact that they are both in Google Cloud.

```python
file_ids = [
        "1AnKxGEuzjeQsDHHqL3NqI_aplq2hVL_E",
        "1tt7weAT1SZknzRFyLXOT2fizceUUVRXX",
        "1A64VBbsQ_z8wZ2LDox586JIyyO6mIwWc",
        "1AKntx-ECnrxjB07B6BlVZcFRS4YPTB-J",
        "1xUk8AAua_x27bHUr-vNoAuhEAjTxOvsu",
        "1ezKTfe7BgqVN5o-8Vdtr9iAF0IueCSjP",
        "1tA7bSOxR1RRNqZst8cShzhuNHnayUf7c",
        "1pA3bCFA5nZDhsQutqsJcH3d712giFb0S",
        "1pTLFMdN1A3ro-KYghk4w4sMz6aGaMOdU",
        "1dUSnG0nUPq9TEQyHd6ZWvaxO0OpxVjXD",
        "1UfCH05nuWiIPbDZxQzHHGAHyMh8dmPQH",
]
for i in file_ids:
    url = f"https://drive.google.com/uc?id={i}"
    !gdown $url


from datasets import load_dataset

ds = load_dataset("medical_dialog", "zh", data_dir="./")
```

## Expected results
Faster load time

## Actual results
`Generating train split: 33%: 625519/1921127 [4:31:03<31:39:20, 11.37 examples/s]`

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

@vrindaprabhu , could you take a look at this since you implemented it? I think the `_generate_examples` function might need to be rewritten