Saving Graph/Structured Data in Datasets
MEMBER
It should probably work out of the box to save structured data. If you want to show an example we can help you.
2021-05-29T14:23:40Z
https://github.com/huggingface/datasets/issues/2426#issuecomment-850841785
https://api.github.com/repos/huggingface/datasets/issues/2426
MDEyOklzc3VlQ29tbWVudDg1MDg0MTc4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/850841785/reactions
2021-05-29T14:23:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/850841785
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
An example of a toy dataset is like:
```json
[
    {
        "name": "mike",
        "friends": [
            "tom",
            "lily"
        ],
        "articles": [
            {
                "title": "aaaaa",
                "reader": [
                    "tom",
                    "lucy"
                ]
            }
        ]
    },
    {
        "name": "tom",
        "friends": [
            "mike",
            "bbb"
        ],
        "articles": [
            {
                "title": "xxxxx",
                "reader": [
                    "tom",
                    "qqqq"
                ]
            }
        ]
    }
]
```
We can use the friendship relation to build a directional graph, and a user node can be represented using the articles written by himself. And the relationship between articles can be built when the article has read by the same user.
This dataset can be used to model the heterogeneous relationship between users and articles, and this graph can be used to build recommendation systems to recommend articles to the user, or potential friends to the user.
2021-05-30T01:38:04Z
https://github.com/huggingface/datasets/issues/2426#issuecomment-850925254
https://api.github.com/repos/huggingface/datasets/issues/2426
MDEyOklzc3VlQ29tbWVudDg1MDkyNTI1NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/850925254/reactions
2021-05-30T01:38:04Z
https://api.github.com/repos/huggingface/datasets/issues/comments/850925254
https://avatars.githubusercontent.com/u/3295342?v=4
https://api.github.com/users/gsh199449/events{/privacy}
https://api.github.com/users/gsh199449/followers
https://api.github.com/users/gsh199449/following{/other_user}
https://api.github.com/users/gsh199449/gists{/gist_id}
https://github.com/gsh199449
gsh199449
MDQ6VXNlcjMyOTUzNDI=
https://api.github.com/users/gsh199449/orgs
https://api.github.com/users/gsh199449/received_events
https://api.github.com/users/gsh199449/repos
https://api.github.com/users/gsh199449/starred{/owner}{/repo}
https://api.github.com/users/gsh199449/subscriptions
User
https://api.github.com/users/gsh199449
public
COLLABORATOR
Hi,

you can do the following to load this data into a `Dataset`:
```python
from datasets import Dataset
examples = [
    {
        "name": "mike",
        "friends": [
            "tom",
            "lily"
        ],
        "articles": [
            {
                "title": "aaaaa",
                "reader": [
                    "tom",
                    "lucy"
                ]
            }
        ]
    },
    {
        "name": "tom",
        "friends": [
            "mike",
            "bbb"
        ],
        "articles": [
            {
                "title": "xxxxx",
                "reader": [
                    "tom",
                    "qqqq"
                ]
            }
        ]
    }
]

keys = examples[0].keys()
values = [ex.values() for ex in examples]
dataset = Dataset.from_dict({k: list(v) for k, v in zip(keys, zip(*values))})
```

Let us know if this works for you.
2021-05-30T11:28:29Z
https://github.com/huggingface/datasets/issues/2426#issuecomment-850984319
https://api.github.com/repos/huggingface/datasets/issues/2426
MDEyOklzc3VlQ29tbWVudDg1MDk4NDMxOQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/850984319/reactions
2021-05-30T11:28:29Z
https://api.github.com/repos/huggingface/datasets/issues/comments/850984319
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Thank you so much, and that works! I also have a question that if the dataset is very large, that cannot be loaded into the memory. How to create the Dataset?
2021-06-01T12:49:27Z
https://github.com/huggingface/datasets/issues/2426#issuecomment-852097396
https://api.github.com/repos/huggingface/datasets/issues/2426
MDEyOklzc3VlQ29tbWVudDg1MjA5NzM5Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/852097396/reactions
2021-06-01T12:49:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/852097396
https://avatars.githubusercontent.com/u/3295342?v=4
https://api.github.com/users/gsh199449/events{/privacy}
https://api.github.com/users/gsh199449/followers
https://api.github.com/users/gsh199449/following{/other_user}
https://api.github.com/users/gsh199449/gists{/gist_id}
https://github.com/gsh199449
gsh199449
MDQ6VXNlcjMyOTUzNDI=
https://api.github.com/users/gsh199449/orgs
https://api.github.com/users/gsh199449/received_events
https://api.github.com/users/gsh199449/repos
https://api.github.com/users/gsh199449/starred{/owner}{/repo}
https://api.github.com/users/gsh199449/subscriptions
User
https://api.github.com/users/gsh199449
public
COLLABORATOR
If your dataset doesn't fit in memory, store it in a local file and load it from there. Check out [this chapter](https://huggingface.co/docs/datasets/master/loading_datasets.html#from-local-files) in the docs for more info.
2021-06-01T19:02:20Z
https://github.com/huggingface/datasets/issues/2426#issuecomment-852371041
https://api.github.com/repos/huggingface/datasets/issues/2426
MDEyOklzc3VlQ29tbWVudDg1MjM3MTA0MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/852371041/reactions
2021-06-01T19:02:20Z
https://api.github.com/repos/huggingface/datasets/issues/comments/852371041
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
NONE
Nice! Thanks for your help.
2021-06-02T01:21:03Z
https://github.com/huggingface/datasets/issues/2426#issuecomment-852637180
https://api.github.com/repos/huggingface/datasets/issues/2426
MDEyOklzc3VlQ29tbWVudDg1MjYzNzE4MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/852637180/reactions
2021-06-02T01:21:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/852637180
https://avatars.githubusercontent.com/u/3295342?v=4
https://api.github.com/users/gsh199449/events{/privacy}
https://api.github.com/users/gsh199449/followers
https://api.github.com/users/gsh199449/following{/other_user}
https://api.github.com/users/gsh199449/gists{/gist_id}
https://github.com/gsh199449
gsh199449
MDQ6VXNlcjMyOTUzNDI=
https://api.github.com/users/gsh199449/orgs
https://api.github.com/users/gsh199449/received_events
https://api.github.com/users/gsh199449/repos
https://api.github.com/users/gsh199449/starred{/owner}{/repo}
https://api.github.com/users/gsh199449/subscriptions
User
https://api.github.com/users/gsh199449
public
Thanks for this amazing library! And my question is I have structured data that is organized with a graph. For example, a dataset with users' friendship relations and user's articles. When I try to save a python dict in the dataset, an error occurred ``did not recognize Python value type when inferring an Arrow data type''.
Although I also know that storing a python dict in pyarrow datasets is not the best practice, but I have no idea about how to save structured data in the Datasets. 

Thank you very much for your help.