HF Datasets data access is extremely slow even when in memory
### Describe the bug

Doing a simple `some_dataset[:10]` can take more than a minute.

Profiling it:
<img width="1280" alt="image" src="https://github.com/huggingface/datasets/assets/36224762/e641fb95-ff02-4072-9016-5416a65f75ab">

`some_dataset` is completely in memory with no disk cache.

This is proving fatal to my usage of HF Datasets. Is there a way I can forgo the arrow format and store the dataset as PyTorch tensors so that `_tensorize` is not needed? And is `_consolidate` supposed to take this long?

It's faster to produce the dataset from scratch than to access it from HF Datasets!

### Steps to reproduce the bug

I have uploaded the dataset that causes this problem [here](https://huggingface.co/datasets/NightMachinery/hf_datasets_bug1).

```python
#!/usr/bin/env python3
import sys
import time
import torch
from datasets import load_dataset


def main(dataset_name):
    # Start the timer
    start_time = time.time()

    # Load the dataset from Hugging Face Hub
    dataset = load_dataset(dataset_name)

    # Set the dataset format as torch
    dataset.set_format(type="torch")

    # Perform an identity map
    dataset = dataset.map(lambda example: example, batched=True, batch_size=20)

    # End the timer
    end_time = time.time()

    # Print the time taken
    print(f"Time taken: {end_time - start_time:.2f} seconds")


if __name__ == "__main__":
    dataset_name = "NightMachinery/hf_datasets_bug1"
    print(f"dataset_name: {dataset_name}")
    main(dataset_name)
```

### Expected behavior

_

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3