Support custom fingerprinting with `Dataset.from_generator`
COLLABORATOR
The `fingerprint` parameter serves a slightly different purpose - we use it to inject a new fingerprint after transforming a `Dataset` (computed from the previous fingerprint + transform + transform args), e.g., to be able to compute the cache file for a transform. There is no concept of `fingerprint` before a `Dataset` is fully initialized, but we still need to hash the args (e.g., generator func) of the "dataset creation methods"  (`from_generator`, `from_csv`, etc.) to compute the cache directory (to store the initial version and transformed dataset versions)

I agree it should be easier to bypass the hashing mechanism in this instance, too. However, we should probably first address https://github.com/huggingface/datasets/issues/5080 before solving this (e.g., maybe exposing `hash` in `load_dataset`/`load_dataset_builder`.
2023-08-30T17:33:21Z
https://github.com/huggingface/datasets/issues/6194#issuecomment-1699582921
https://api.github.com/repos/huggingface/datasets/issues/6194
IC_kwDODunzps5lTZPJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1699582921/reactions
2023-08-30T17:33:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1699582921
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Adding +1 here:

If the generator needs to access some external resources or state, then it's not always straightforward to make it pickle-able. So I'd like to be able to override how the default cache key derivation needs to pickle the generator (and of course, I'd accept responsibility for that part of cache consistency).

Appears to be a recurrent roadbump: #6118 #5963 #5819 #5750 #4983 
2023-09-06T07:29:10Z
https://github.com/huggingface/datasets/issues/6194#issuecomment-1707816603
https://api.github.com/repos/huggingface/datasets/issues/6194
IC_kwDODunzps5lyzab
https://api.github.com/repos/huggingface/datasets/issues/comments/1707816603/reactions
2023-09-06T07:29:10Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1707816603
https://avatars.githubusercontent.com/u/356550?v=4
https://api.github.com/users/mlin/events{/privacy}
https://api.github.com/users/mlin/followers
https://api.github.com/users/mlin/following{/other_user}
https://api.github.com/users/mlin/gists{/gist_id}
https://github.com/mlin
mlin
MDQ6VXNlcjM1NjU1MA==
https://api.github.com/users/mlin/orgs
https://api.github.com/users/mlin/received_events
https://api.github.com/users/mlin/repos
https://api.github.com/users/mlin/starred{/owner}{/repo}
https://api.github.com/users/mlin/subscriptions
User
https://api.github.com/users/mlin
public
NONE
Silly hack incoming:

```python
import uuid

class _DatasetGeneratorPickleHack:
    def __init__(self, generator, generator_id=None):
        self.generator = generator
        self.generator_id = (
            generator_id if generator_id is not None else str(uuid.uuid4())
        )

    def __call__(self, *args, **kwargs):
        return self.generator(*kwargs, **kwargs)

    def __reduce__(self):
        return (_DatasetGeneratorPickleHack_raise, (self.generator_id,))


def _DatasetGeneratorPickleHack_raise(*args, **kwargs):
    raise AssertionError("cannot actually unpickle _DatasetGeneratorPickleHack!")
```

Now `Dataset.from_generator(_DatasetGeneratorPickleHack(gen))` works even if `gen` is unpicklable, because Dataset just pickles the shim object that avoids actually traversing `gen`. Then, one can work out how to set `generator_id` meaningfully to allow cache reuse.
2023-09-06T10:28:44Z
https://github.com/huggingface/datasets/issues/6194#issuecomment-1708080653
https://api.github.com/repos/huggingface/datasets/issues/6194
IC_kwDODunzps5lzz4N
https://api.github.com/repos/huggingface/datasets/issues/comments/1708080653/reactions
2023-09-06T10:28:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1708080653
https://avatars.githubusercontent.com/u/356550?v=4
https://api.github.com/users/mlin/events{/privacy}
https://api.github.com/users/mlin/followers
https://api.github.com/users/mlin/following{/other_user}
https://api.github.com/users/mlin/gists{/gist_id}
https://github.com/mlin
mlin
MDQ6VXNlcjM1NjU1MA==
https://api.github.com/users/mlin/orgs
https://api.github.com/users/mlin/received_events
https://api.github.com/users/mlin/repos
https://api.github.com/users/mlin/starred{/owner}{/repo}
https://api.github.com/users/mlin/subscriptions
User
https://api.github.com/users/mlin
public
CONTRIBUTOR
I'd like some way to do this too.  I find that sometimes the hash doesn't cover enough, and that the dataset is not regenerated even when underlying data has changed, and by supplying a custom fingerprint I could do a better job of controlling when my dataset is regenerated.
2023-09-30T16:56:51Z
https://github.com/huggingface/datasets/issues/6194#issuecomment-1741813408
https://api.github.com/repos/huggingface/datasets/issues/6194
IC_kwDODunzps5n0fag
https://api.github.com/repos/huggingface/datasets/issues/comments/1741813408/reactions
2023-09-30T16:56:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1741813408
https://avatars.githubusercontent.com/u/53510?v=4
https://api.github.com/users/hartmans/events{/privacy}
https://api.github.com/users/hartmans/followers
https://api.github.com/users/hartmans/following{/other_user}
https://api.github.com/users/hartmans/gists{/gist_id}
https://github.com/hartmans
hartmans
MDQ6VXNlcjUzNTEw
https://api.github.com/users/hartmans/orgs
https://api.github.com/users/hartmans/received_events
https://api.github.com/users/hartmans/repos
https://api.github.com/users/hartmans/starred{/owner}{/repo}
https://api.github.com/users/hartmans/subscriptions
User
https://api.github.com/users/hartmans
public
NONE
This is what I did and it works:  

https://github.com/stevemadere/s3-datasets/blob/e475a566a16d3051656a66f8ff4d3baa4c55a66c/src/tokengenerators/text_ds_2_tokens_generator.py#L200

2024-02-29T03:46:53Z
https://github.com/huggingface/datasets/issues/6194#issuecomment-1970345207
https://api.github.com/repos/huggingface/datasets/issues/6194
IC_kwDODunzps51cRT3
https://api.github.com/repos/huggingface/datasets/issues/comments/1970345207/reactions
2024-02-29T03:46:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1970345207
https://avatars.githubusercontent.com/u/4029925?v=4
https://api.github.com/users/stevemadere/events{/privacy}
https://api.github.com/users/stevemadere/followers
https://api.github.com/users/stevemadere/following{/other_user}
https://api.github.com/users/stevemadere/gists{/gist_id}
https://github.com/stevemadere
stevemadere
MDQ6VXNlcjQwMjk5MjU=
https://api.github.com/users/stevemadere/orgs
https://api.github.com/users/stevemadere/received_events
https://api.github.com/users/stevemadere/repos
https://api.github.com/users/stevemadere/starred{/owner}{/repo}
https://api.github.com/users/stevemadere/subscriptions
User
https://api.github.com/users/stevemadere
public
NONE
I ran into the same thing - my actual generator reads from a disk source that might have new data (images) available at some point and it ends up ignoring calling the generator. Thanks for the hack @mlin 👋 
2024-10-22T06:12:00Z
https://github.com/huggingface/datasets/issues/6194#issuecomment-2428342640
https://api.github.com/repos/huggingface/datasets/issues/6194
IC_kwDODunzps6QvZFw
https://api.github.com/repos/huggingface/datasets/issues/comments/2428342640/reactions
2024-10-22T06:12:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2428342640
https://avatars.githubusercontent.com/u/1226567?v=4
https://api.github.com/users/urjitbhatia/events{/privacy}
https://api.github.com/users/urjitbhatia/followers
https://api.github.com/users/urjitbhatia/following{/other_user}
https://api.github.com/users/urjitbhatia/gists{/gist_id}
https://github.com/urjitbhatia
urjitbhatia
MDQ6VXNlcjEyMjY1Njc=
https://api.github.com/users/urjitbhatia/orgs
https://api.github.com/users/urjitbhatia/received_events
https://api.github.com/users/urjitbhatia/repos
https://api.github.com/users/urjitbhatia/starred{/owner}{/repo}
https://api.github.com/users/urjitbhatia/subscriptions
User
https://api.github.com/users/urjitbhatia
public
CONTRIBUTOR
just wanted to pitch my support for an easy control over the generator id. requiring that generators are pickleable just to get a unique id is limiting: plenty of classes (maybe even hf.datasets own) are written with no pickle support in mind. also as mentioned above the state of a generator might extend beyond its pickle.
2024-12-22T01:12:16Z
https://github.com/huggingface/datasets/issues/6194#issuecomment-2558289202
https://api.github.com/repos/huggingface/datasets/issues/6194
IC_kwDODunzps6YfGUy
https://api.github.com/repos/huggingface/datasets/issues/comments/2558289202/reactions
2024-12-22T01:14:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/2558289202
https://avatars.githubusercontent.com/u/12157034?v=4
https://api.github.com/users/vttrifonov/events{/privacy}
https://api.github.com/users/vttrifonov/followers
https://api.github.com/users/vttrifonov/following{/other_user}
https://api.github.com/users/vttrifonov/gists{/gist_id}
https://github.com/vttrifonov
vttrifonov
MDQ6VXNlcjEyMTU3MDM0
https://api.github.com/users/vttrifonov/orgs
https://api.github.com/users/vttrifonov/received_events
https://api.github.com/users/vttrifonov/repos
https://api.github.com/users/vttrifonov/starred{/owner}{/repo}
https://api.github.com/users/vttrifonov/subscriptions
User
https://api.github.com/users/vttrifonov
public
### Feature request

When using `Dataset.from_generator`, the generator is hashed when building the fingerprint. Similar to `.map`, it would be interesting to let the user bypass this hashing by accepting a `fingerprint` argument to `.from_generator`.

### Motivation

Using the `.from_generator` constructor with a non-picklable generator fails. By accepting a `fingerprint` argument to `.from_generator`, the user would have the opportunity to manually fingerprint the dataset and thus bypass the crash.

### Your contribution

If validated, I can try to submit a PR for this.