load_dataset('json',...) from togethercomputer/RedPajama-Data-1T errors when jsonl rows contains different data fields
MEMBER
Hi ! Feel free to open a discussion at https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T/discussions to ask the file to be fixed (or directly open a PR with the fixed file)

`datasets` expects all the examples to have the same fields
2023-08-18T13:39:34Z
https://github.com/huggingface/datasets/issues/6162#issuecomment-1683937431
https://api.github.com/repos/huggingface/datasets/issues/6162
IC_kwDODunzps5kXtiX
https://api.github.com/repos/huggingface/datasets/issues/comments/1683937431/reactions
2023-08-18T13:39:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1683937431
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq I think the problem is caused by the fact that hugging face datasets writes a copy of data to the local cache using pyarrow. And the data scheme is inferred from the first few data blocks as can be seen [here](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_writer.py#L570). Maybe setting `streaming=True` can workaround this problem. Would you agree with my statement? 
2023-08-18T15:05:09Z
https://github.com/huggingface/datasets/issues/6162#issuecomment-1684053692
https://api.github.com/repos/huggingface/datasets/issues/6162
IC_kwDODunzps5kYJ68
https://api.github.com/repos/huggingface/datasets/issues/comments/1684053692/reactions
2023-08-18T15:05:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1684053692
https://avatars.githubusercontent.com/u/24477841?v=4
https://api.github.com/users/faaany/events{/privacy}
https://api.github.com/users/faaany/followers
https://api.github.com/users/faaany/following{/other_user}
https://api.github.com/users/faaany/gists{/gist_id}
https://github.com/faaany
faaany
MDQ6VXNlcjI0NDc3ODQx
https://api.github.com/users/faaany/orgs
https://api.github.com/users/faaany/received_events
https://api.github.com/users/faaany/repos
https://api.github.com/users/faaany/starred{/owner}{/repo}
https://api.github.com/users/faaany/subscriptions
User
https://api.github.com/users/faaany
public
MEMBER
> @lhoestq I think the problem is caused by the fact that hugging face datasets writes a copy of data to the local cache using pyarrow. And the data scheme is inferred from the first few data blocks as can be seen [here](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_writer.py#L570).

Correct. Therefore any example that doesn't follow the inferred schema will make the code fail.

> Maybe setting streaming=True can workaround this problem. Would you agree with my statement?

You'll meet the same problem but later - when streaming and arriving at the problematic example
2023-08-18T16:12:41Z
https://github.com/huggingface/datasets/issues/6162#issuecomment-1684138688
https://api.github.com/repos/huggingface/datasets/issues/6162
IC_kwDODunzps5kYerA
https://api.github.com/repos/huggingface/datasets/issues/comments/1684138688/reactions
2023-08-18T16:12:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1684138688
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
@lhoestq I just run below test with streaming=True and is not failing at the problematic example
```python
ds = load_dataset('json', data_files='/path_to_local_RedPajamaData/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl', streaming=True)
count = 0
for i in ds['train']:
    count += 1
    print(count)
```

and completes the 262241 samples successfully. It does error our when streaming is not used 
2023-08-18T17:00:35Z
https://github.com/huggingface/datasets/issues/6162#issuecomment-1684192320
https://api.github.com/repos/huggingface/datasets/issues/6162
IC_kwDODunzps5kYrxA
https://api.github.com/repos/huggingface/datasets/issues/comments/1684192320/reactions
2023-08-18T17:00:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1684192320
https://avatars.githubusercontent.com/u/82971690?v=4
https://api.github.com/users/rbrugaro/events{/privacy}
https://api.github.com/users/rbrugaro/followers
https://api.github.com/users/rbrugaro/following{/other_user}
https://api.github.com/users/rbrugaro/gists{/gist_id}
https://github.com/rbrugaro
rbrugaro
MDQ6VXNlcjgyOTcxNjkw
https://api.github.com/users/rbrugaro/orgs
https://api.github.com/users/rbrugaro/received_events
https://api.github.com/users/rbrugaro/repos
https://api.github.com/users/rbrugaro/starred{/owner}{/repo}
https://api.github.com/users/rbrugaro/subscriptions
User
https://api.github.com/users/rbrugaro
public
### Describe the bug

When loading some jsonl from redpajama-data-1T github source [togethercomputer/RedPajama-Data-1T](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T) fails due to one row of the file containing an extra field called **symlink_target: string>**. 
When deleting that line the loading is successful. 

We also tried loading this file with the discrepancy using this function and it is successful
```python
os.environ["RED_PAJAMA_DATA_DIR"] ="/path_to_local_copy_of_RedPajama-Data-1T"
ds = load_dataset('togethercomputer/RedPajama-Data-1T', 'github',cache_dir="/path_to_folder_with_jsonl",streaming=True)['train']
```

### Steps to reproduce the bug

Steps to reproduce the behavior:
1. Load one jsonl from the redpajama-data-1T
```bash 
wget https://data.together.xyz/redpajama-data-1T/v1.0.0/github/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl
```

2.Load dataset will give error:
```python
from datasets import load_dataset
ds = load_dataset('json', data_files='/path_to/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl')
```

_TypeError: Couldn't cast array of type
Struct
<content_hash: string, 
timestamp: string, 
source: string, 
line_count: int64, 
max_line_length: int64,
avg_line_length: double, 
alnum_prop: double, 
repo_name: string, 
id: string, 
size: string, 
binary: bool, 
copies: string, 
ref: string, 
path: string, 
mode: string, 
license: string, 
language: list<item: struct<name: string, bytes: string>>, **symlink_target: string>**
to
{'content_hash': Value(dtype='string', id=None), 
'timestamp': Value(dtype='string', id=None), 
'source': Value(dtype='string', id=None), 
'line_count': Value(dtype='int64', id=None), 
'max_line_length': Value(dtype='int64', id=None), 
'avg_line_length': Value(dtype='float64', id=None), 
'alnum_prop': Value(dtype='float64', id=None), 
'repo_name': Value(dtype='string', id=None), 
'id': Value(dtype='string', id=None), 
'size': Value(dtype='string', id=None), 
'binary': Value(dtype='bool', id=None), 
'copies': Value(dtype='string', id=None), 
'ref': Value(dtype='string', id=None), 
'path': Value(dtype='string', id=None), 
'mode': Value(dtype='string', id=None), 
'license': Value(dtype='string', id=None), 
'language': [{'name': Value(dtype='string', id=None), 'bytes': Value(dtype='string', id=None)}]}_

3. To remove the line causing the problem that includes the **symlink_target: string>** do:
```bash 
sed -i '112252d' filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl
```
4. Rerun the loading function now is succesful:
```python
from datasets import load_dataset
ds = load_dataset('json', data_files='/path_to/filtered_27f05c041a1c401783f90b9415e40e4b.sampled.jsonl')
```

### Expected behavior

Have a clean dataset without discrepancies on the jsonl fields or have the load_dataset('json',...) method not error out. 


### Environment info

- `datasets` version: 2.14.1
- Platform: Linux-4.18.0-425.13.1.el8_7.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3