
| Agreement      | PR   |        |      |      |      |      |     |
|----------------|------|--------|------|------|------|------|-----|
| Data           | Base | Clique | Node | Pair | Full | TR1  | EM  |
| Train size = 3 |      |        |      |      |      |      |     |
| All            | 83.3 | 4.2    | 3.9  | 2.6  | 2.6  | 2.1  | 3.7 |
| 50F            | 55.2 | 2.7    | 3.5  | 2.9  | 2.9  | 3.5  | 1.0 |
| 50M            | 54.6 | 0.6    | 0.9  | 1.3  | 1.1  | 4.5  | 3.6 |
| 60F            | 66.9 | 2.9    | 2.6  | 0.8  | 0.6  | 1.5  | 1.5 |
| 60M            | 67.3 | 3.4    | 2.3  | 1.8  | 2.2  | -0.1 | 3.4 |
| 70F            | 73.5 | 1.7    | 1.2  | 1.4  | 1.0  | 0.7  | 1.1 |
| 70M            | 76.1 | 6.5    | 5.8  | 3.8  | 4.5  | 3.7  | 6.9 |
| 80F            | 85.6 | 4.5    | 4.1  | 3.7  | 3.5  | 0.2  | 4.4 |
| 80M            | 86.6 | 6.5    | 6.0  | 3.8  | 3.4  | 3.6  | 4.5 |
| 90F            | 94.3 | 2.3    | 2.1  | 0.5  | 1.1  | 1.2  | 1.7 |
| 90M            | 96.1 | 0.3    | 0.6  | -0.1 | 0.0  | 0.6  | 0.4 |
| Train size = 7 |      |        |      |      |      |      |     |
| All            | 87.3 | 1.8    | 2.0  | 1.0  | 0.9  | 0.7  | 2.1 |
| 50F            | 52.5 | 4.2    | 5.6  | 4.8  | 4.8  | 4.2  | 3.4 |
| 50M            | 63.4 | 0.0    | 0.1  | 0.9  | 0.4  | 3.5  | 2.7 |
| 60F            | 76.2 | 1.9    | 1.5  | 0.1  | 0.1  | 0.5  | 1.2 |
| 60M            | 75.0 | 1.3    | 2.6  | 0.9  | 0.3  | -1.5 | 2.4 |
| 70F            | 79.6 | 2.3    | 3.3  | 0.1  | 0.0  | -0.8 | 2.5 |
| 70M            | 82.4 | 3.2    | 3.5  | 2.2  | 2.2  | 1.7  | 3.8 |
| 80F            | 90.3 | 1.5    | 1.4  | 1.1  | 1.2  | 0.6  | 1.6 |
| 80M            | 90.5 | 2.7    | 2.6  | 1.3  | 1.0  | 0.9  | 2.4 |
| 90F            | 96.6 | 0.6    | 0.2  | 0.1  | 0.2  | 0.2  | 0.8 |
| 90M            | 96.5 | 0.3    | 0.7  | 0.2  | 0.2  | -0.2 | 0.5 |

training clearly performs the best, and its gains are specially large for datasets whose base accuracy is in the 60–80% range, and which have big agreement sets. Overall, its F1 is 87.9% in contrast to the base accuracy of 83.7%. Even with a training size of seven, F1 improves from 87.4 to 89.2 (not shown in the figure). In contrast, the staged approach overall performs worse than Base and shows large swings in accuracy across datasets. It is highly sensitive to the ordering of sources, and the hard label-transfer often causes error-propagation to all downstream sources. Collective inference improves accuracy in a few cases but overall provides only a small gain of 0.3% beyond Base.

## 5.2 Comparing Collective Training Objectives

We now compare the various approximations of the agreement term - Clique, Node, Instance Pair, Full (Full BP), and TR1 (OneStep TRW) as described in Section 2.3. Table 2 shows the gains in F1 for all the approaches over the base model.

Observe that Clique and Node agreement are two of the best performing methods. We explore two possible reasons for why they score over other approaches that fuse the influence of multiple cliques. One partial explanation is that 15% of the cliques in our agreement set are noisy. In such a case, fused methods would try hard at maximizing the likelihood of a wrongly fused graph. In contrast, the Clique and Node agreement models decompose over cliques, so they can choose to ignore the terms corresponding to erroneous cliques during optimization. A second reason common to all the losing approaches is the inexact nature of the optimization of their training objectives. To understand which of these is a plausible reason, we remove all noisy cliques using the ground truth and compare Clique and Instance Pair agreement. Accuracy improves by less than 0.6 F1 in both, and Clique continues to score over Instance Pair. This indicates that inexact gradient computation is perhaps a major reason why more complex fused approaches perform worse.

We also see that there is little difference between the Clique and Node agreement models. While one possible reason is the weakness of any first-order dependency in the true model, we find another interesting reason for this behavior. We note that in a general n-gram agreement clique, only a few positions might be erroneous. For