Saving processed dataset running infinitely
NONE
@thomwolf @lhoestq can you guys please take a look and recommend some solution.
2021-02-19T13:10:43Z
https://github.com/huggingface/datasets/issues/1911#issuecomment-782064683
https://api.github.com/repos/huggingface/datasets/issues/1911
MDEyOklzc3VlQ29tbWVudDc4MjA2NDY4Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/782064683/reactions
2021-02-19T13:10:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/782064683
https://avatars.githubusercontent.com/u/20911334?v=4
https://api.github.com/users/ayubSubhaniya/events{/privacy}
https://api.github.com/users/ayubSubhaniya/followers
https://api.github.com/users/ayubSubhaniya/following{/other_user}
https://api.github.com/users/ayubSubhaniya/gists{/gist_id}
https://github.com/ayubSubhaniya
ayubSubhaniya
MDQ6VXNlcjIwOTExMzM0
https://api.github.com/users/ayubSubhaniya/orgs
https://api.github.com/users/ayubSubhaniya/received_events
https://api.github.com/users/ayubSubhaniya/repos
https://api.github.com/users/ayubSubhaniya/starred{/owner}{/repo}
https://api.github.com/users/ayubSubhaniya/subscriptions
User
https://api.github.com/users/ayubSubhaniya
public
NONE
am suspicious of this thing? what's the purpose of this? pickling and unplickling
`self = pickle.loads(pickle.dumps(self))`

```
    def save_to_disk(self, dataset_path: str, fs=None):
        """
        Saves a dataset to a dataset directory, or in a filesystem using either :class:`datasets.filesystem.S3FileSystem` or any implementation of ``fsspec.spec.AbstractFileSystem``.

        Args:
            dataset_path (``str``): path (e.g. ``dataset/train``) or remote uri (e.g. ``s3://my-bucket/dataset/train``) of the dataset directory where the dataset will be saved to
            fs (Optional[:class:`datasets.filesystem.S3FileSystem`,``fsspec.spec.AbstractFileSystem``],  `optional`, defaults ``None``): instance of :class:`datasets.filesystem.S3FileSystem` or ``fsspec.spec.AbstractFileSystem`` used to download the files from remote filesystem.
        """
        assert (
            not self.list_indexes()
        ), "please remove all the indexes using `dataset.drop_index` before saving a dataset"
        self = pickle.loads(pickle.dumps(self))
  ```
2021-02-19T14:29:03Z
https://github.com/huggingface/datasets/issues/1911#issuecomment-782109584
https://api.github.com/repos/huggingface/datasets/issues/1911
MDEyOklzc3VlQ29tbWVudDc4MjEwOTU4NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/782109584/reactions
2021-02-19T14:35:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/782109584
https://avatars.githubusercontent.com/u/20911334?v=4
https://api.github.com/users/ayubSubhaniya/events{/privacy}
https://api.github.com/users/ayubSubhaniya/followers
https://api.github.com/users/ayubSubhaniya/following{/other_user}
https://api.github.com/users/ayubSubhaniya/gists{/gist_id}
https://github.com/ayubSubhaniya
ayubSubhaniya
MDQ6VXNlcjIwOTExMzM0
https://api.github.com/users/ayubSubhaniya/orgs
https://api.github.com/users/ayubSubhaniya/received_events
https://api.github.com/users/ayubSubhaniya/repos
https://api.github.com/users/ayubSubhaniya/starred{/owner}{/repo}
https://api.github.com/users/ayubSubhaniya/subscriptions
User
https://api.github.com/users/ayubSubhaniya
public
NONE
It's been 24 hours and sadly it's still running. With not a single byte written
2021-02-20T07:04:14Z
https://github.com/huggingface/datasets/issues/1911#issuecomment-782575907
https://api.github.com/repos/huggingface/datasets/issues/1911
MDEyOklzc3VlQ29tbWVudDc4MjU3NTkwNw==
https://api.github.com/repos/huggingface/datasets/issues/comments/782575907/reactions
2021-02-20T07:04:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/782575907
https://avatars.githubusercontent.com/u/20911334?v=4
https://api.github.com/users/ayubSubhaniya/events{/privacy}
https://api.github.com/users/ayubSubhaniya/followers
https://api.github.com/users/ayubSubhaniya/following{/other_user}
https://api.github.com/users/ayubSubhaniya/gists{/gist_id}
https://github.com/ayubSubhaniya
ayubSubhaniya
MDQ6VXNlcjIwOTExMzM0
https://api.github.com/users/ayubSubhaniya/orgs
https://api.github.com/users/ayubSubhaniya/received_events
https://api.github.com/users/ayubSubhaniya/repos
https://api.github.com/users/ayubSubhaniya/starred{/owner}{/repo}
https://api.github.com/users/ayubSubhaniya/subscriptions
User
https://api.github.com/users/ayubSubhaniya
public
NONE
Tried finding the root cause but was unsuccessful.
I am using lazy tokenization with `dataset.set_transform()`, it works like a charm with almost same performance as pre-compute.
2021-02-20T15:32:12Z
https://github.com/huggingface/datasets/issues/1911#issuecomment-782699060
https://api.github.com/repos/huggingface/datasets/issues/1911
MDEyOklzc3VlQ29tbWVudDc4MjY5OTA2MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/782699060/reactions
2021-02-20T15:32:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/782699060
https://avatars.githubusercontent.com/u/20911334?v=4
https://api.github.com/users/ayubSubhaniya/events{/privacy}
https://api.github.com/users/ayubSubhaniya/followers
https://api.github.com/users/ayubSubhaniya/following{/other_user}
https://api.github.com/users/ayubSubhaniya/gists{/gist_id}
https://github.com/ayubSubhaniya
ayubSubhaniya
MDQ6VXNlcjIwOTExMzM0
https://api.github.com/users/ayubSubhaniya/orgs
https://api.github.com/users/ayubSubhaniya/received_events
https://api.github.com/users/ayubSubhaniya/repos
https://api.github.com/users/ayubSubhaniya/starred{/owner}{/repo}
https://api.github.com/users/ayubSubhaniya/subscriptions
User
https://api.github.com/users/ayubSubhaniya
public
MEMBER
Hi ! This very probably comes from the hack you used.

The pickling line was added an a sanity check because save_to_disk uses the same assumptions as pickling for a dataset object. The main assumption is that memory mapped pyarrow tables must be reloadable from the disk. In your case it's not possible since you altered the pyarrow table.
I would suggest you to rebuild a valid Dataset object from your new pyarrow table. To do so you must first save your new table to a file, and then make a new Dataset object from that arrow file.

You can save the raw arrow table (without all the `datasets.Datasets` metadata) by calling `map` with `cache_file_name="path/to/outut.arrow"` and `function=None`. Having `function=None` makes the `map` write your dataset on disk with no data transformation.

Once you have your new arrow file, load it with `datasets.Dataset.from_file` to have a brand new Dataset object :)

In the future we'll have a better support for the fast filtering method from pyarrow so you don't have to do this very unpractical workaround. Since it breaks somes assumptions regarding the core behavior of Dataset objects, this is very discouraged.
2021-02-21T20:24:13Z
https://github.com/huggingface/datasets/issues/1911#issuecomment-782920333
https://api.github.com/repos/huggingface/datasets/issues/1911
MDEyOklzc3VlQ29tbWVudDc4MjkyMDMzMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/782920333/reactions
2021-02-21T20:25:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/782920333
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Thanks, @lhoestq for your response. Will try your solution and let you know.
2021-02-23T07:34:44Z
https://github.com/huggingface/datasets/issues/1911#issuecomment-783975480
https://api.github.com/repos/huggingface/datasets/issues/1911
MDEyOklzc3VlQ29tbWVudDc4Mzk3NTQ4MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/783975480/reactions
2021-02-23T07:34:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/783975480
https://avatars.githubusercontent.com/u/20911334?v=4
https://api.github.com/users/ayubSubhaniya/events{/privacy}
https://api.github.com/users/ayubSubhaniya/followers
https://api.github.com/users/ayubSubhaniya/following{/other_user}
https://api.github.com/users/ayubSubhaniya/gists{/gist_id}
https://github.com/ayubSubhaniya
ayubSubhaniya
MDQ6VXNlcjIwOTExMzM0
https://api.github.com/users/ayubSubhaniya/orgs
https://api.github.com/users/ayubSubhaniya/received_events
https://api.github.com/users/ayubSubhaniya/repos
https://api.github.com/users/ayubSubhaniya/starred{/owner}{/repo}
https://api.github.com/users/ayubSubhaniya/subscriptions
User
https://api.github.com/users/ayubSubhaniya
public
I have a text dataset of size 220M.

For pre-processing, I need to tokenize this and filter rows with the large sequence.

My tokenization took roughly 3hrs. I used map() with batch size 1024 and multi-process with 96 processes.

filter() function was way to slow, so I used a hack to use pyarrow filter table function, which is damm fast. Mentioned [here](https://github.com/huggingface/datasets/issues/1796)

```dataset._data = dataset._data.filter(...)```
It took 1 hr for the filter.

Then i use `save_to_disk()` on processed dataset and it is running forever.

I have been waiting since 8 hrs, it has not written a single byte. 

Infact it has actually read from disk more than 100GB, screenshot below shows the stats using `iotop`. 
Second process is the one.
<img width="1672" alt="Screenshot 2021-02-19 at 6 36 53 PM" src="https://user-images.githubusercontent.com/20911334/108508197-7325d780-72e1-11eb-8369-7c057d137d81.png">


I am not able to figure out, whether this is some issue with dataset library or that it is due to my hack for filter() function.