text: Argumentation mining aims at automatically extracting the premises-claim discourse structures in natural language texts. There is a great demand for argumentation corpora for customer reviews. However, due to the controversial nature of the argumentation annotation task, there exist very few large-scale argumentation corpora for customer reviews. In this work, we novelly use the crowdsourcing technique to collect argumentation annotations in Chinese hotel reviews. As the first Chinese argumentation dataset, our corpus includes 4814 argument component annotations and 411 argument relation annotations, and its annotations qualities are comparable to some widely used argumentation corpora in other languages.
  @cite_9: We present a novel approach for rapidly developing a corpus with discourse annotations using crowdsourcing. Although discourse annotations typically require much time and cost owing to their complex nature, we realize discourse annotations in an extremely short time while retaining good quality of the annotations by crowdsourcing two annotation subtasks. In fact, our experiment to create a corpus comprising 30,000 Japanese sentences took less than eight hours to run. Based on this corpus, we also develop a supervised discourse parser and evaluate its performance to verify the usefulness of the acquired corpus.
summary: Crowdsourcing has been used to annotate discourse structures. @cite_9 designed a two-stage crowdsourcing mechanism to annotate two levels of discourse relations in Japanese texts crawled from multiple online genres. In their work, discourse relations include contrast, concession, cause-effect, etc., and these relations closely resemble some relations in argumentation structures (e.g. the attack relation between arguments can be viewed as contrast, and the premise-claim relation is closely related to the cause-effect relation). They did not report the IRA of the crowdsourcing workers, but instead, testified the quality of their discourse corpus by training a discourse parser on their corpus. Results suggested that the quality of their corpus is comparable to the state-of-the-art English discourse corpus, indicating that crowdsourcing can be reliably used for annotating relations between clauses.