What is the Right way to use discofuse dataset??
MEMBER
Thanks for opening https://huggingface.co/datasets/discofuse/discussions/3, let's continue the discussion over there if you don't mind
2023-06-14T12:10:16Z
https://github.com/huggingface/datasets/issues/5951#issuecomment-1591070908
https://api.github.com/repos/huggingface/datasets/issues/5951
IC_kwDODunzps5e1dC8
https://api.github.com/repos/huggingface/datasets/issues/comments/1591070908/reactions
2023-06-14T12:10:16Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1591070908
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
I have posted there also sir, please check
@lhoestq
2023-06-14T13:24:33Z
https://github.com/huggingface/datasets/issues/5951#issuecomment-1591205426
https://api.github.com/repos/huggingface/datasets/issues/5951
IC_kwDODunzps5e194y
https://api.github.com/repos/huggingface/datasets/issues/comments/1591205426/reactions
2023-06-14T13:25:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1591205426
https://avatars.githubusercontent.com/u/125154243?v=4
https://api.github.com/users/akesh1235/events{/privacy}
https://api.github.com/users/akesh1235/followers
https://api.github.com/users/akesh1235/following{/other_user}
https://api.github.com/users/akesh1235/gists{/gist_id}
https://github.com/akesh1235
akesh1235
U_kgDOB3Wzww
https://api.github.com/users/akesh1235/orgs
https://api.github.com/users/akesh1235/received_events
https://api.github.com/users/akesh1235/repos
https://api.github.com/users/akesh1235/starred{/owner}{/repo}
https://api.github.com/users/akesh1235/subscriptions
User
https://api.github.com/users/akesh1235
public
[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)
**Below is the following way, as per my understanding , Is it correct :question: :question:**

The **columns/features from `DiscoFuse dataset`** that will be the **input to the `encoder` and `decoder`** are:

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)

1. **coherent_first_sentence**

2. **coherent_second_sentence**

3. **incoherent_first_sentence**

4. **incoherent_second_sentence**

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)

The **`encoder` will take these four columns as input and encode them into a sequence of hidden states. The `decoder` will then take these hidden states as input and decode them into a new sentence that fuses the two original sentences together.**

The **discourse type, connective_string, has_coref_type_pronoun, and has_coref_type_nominal columns will not be used as input to the encoder or decoder.** These columns are used to provide additional information about the dataset, but they are not necessary for the task of sentence fusion.

Please correct me if I am wrong; otherwise, if this understanding is right, how shall I implement this task practically?