e from instruction tuning than dense models.

The core assertions of the paper are:

1. MoE models underperform compared to dense models of the same computational capacity when directly fine-tuned on individual downstream tasks without instruction tuning.

2. However, when instruction tuning is introduced, MoE models outperform dense models on downstream tasks and on held-out zero-shot and few-shot tasks.

3. The performance of MoE models scales better with respect to the number of tasks, than the number of