feat: improve error message on Keys mismatch. closes #4917
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-08-31T14:48:58Z
https://github.com/huggingface/datasets/pull/4919#issuecomment-1233039814
https://api.github.com/repos/huggingface/datasets/issues/4919
IC_kwDODunzps5JfrHG
https://api.github.com/repos/huggingface/datasets/issues/comments/1233039814/reactions
2022-09-05T08:46:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1233039814
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
We are having an unrelated issue that makes several tests fail. We are working on that. Once fixed, you will be able to merge the main branch into this, so that you get the fix and the tests pass...
2022-08-31T17:25:01Z
https://github.com/huggingface/datasets/pull/4919#issuecomment-1233216841
https://api.github.com/repos/huggingface/datasets/issues/4919
IC_kwDODunzps5JgWVJ
https://api.github.com/repos/huggingface/datasets/issues/comments/1233216841/reactions
2022-08-31T17:25:01Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1233216841
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Hi @lhoestq what do you think?

Let me give you a code sample:
```py
>>> import datasets
>>> foo = datasets.Dataset.from_dict({'foo':[0,1], 'bar':[2,3]})
>>> foo.save_to_disk('foo')
# edit foo/dataset_info.json e.g. rename the 'foo' feature to 'baz'
>>> datasets.load_from_disk('foo')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-4863e606b330> in <module>
----> 1 datasets.load_from_disk('foo')

~/code/datasets/src/datasets/load.py in load_from_disk(dataset_path, fs, keep_in_memory)
   1851         raise FileNotFoundError(f"Directory {dataset_path} not found")
   1852     if fs.isfile(Path(dest_dataset_path, config.DATASET_INFO_FILENAME).as_posix()):
-> 1853         return Dataset.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)
   1854     elif fs.isfile(Path(dest_dataset_path, config.DATASETDICT_JSON_FILENAME).as_posix()):
   1855         return DatasetDict.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)

~/code/datasets/src/datasets/arrow_dataset.py in load_from_disk(dataset_path, fs, keep_in_memory)
   1230             info=dataset_info,
   1231             split=split,
-> 1232             fingerprint=state["_fingerprint"],
   1233         )
   1234 

~/code/datasets/src/datasets/arrow_dataset.py in __init__(self, arrow_table, info, split, indices_table, fingerprint)
    687             self.info.features = inferred_features
    688         else:  # make sure the nested columns are in the right order
--> 689             self.info.features = self.info.features.reorder_fields_as(inferred_features)
    690 
    691         # Infer fingerprint if None

~/code/datasets/src/datasets/features/features.py in reorder_fields_as(self, other)
   1771                 return source
   1772 
-> 1773         return Features(recursive_reorder(self, other))
   1774 
   1775     def flatten(self, max_depth=16) -> "Features":

~/code/datasets/src/datasets/features/features.py in recursive_reorder(source, target, stack)
   1760                                f"{source.keys()-target.keys()} are missing from dataset.arrow "
   1761                                f"and {target.keys()-source.keys()} are missing from dataset_info.json"+stack_position)
-> 1762                     raise ValueError(message)
   1763                 return {key: recursive_reorder(source[key], target[key], stack + f".{key}") for key in target}
   1764             elif isinstance(source, list):

ValueError: Keys mismatch: between {'baz': Value(dtype='int64', id=None), 'bar': Value(dtype='int64', id=None)} (dataset_info.json) and {'foo': Value(dtype='int64', id=None), 'bar': Value(dtype='int64', id=None)} (inferred from dataset.arrow).
{'baz'} are missing from dataset.arrow and {'foo'} are missing from dataset_info.json
```