linguistic:voice_change, information:filtering, linguistic:paraphrasing, structural:reformatting
Emergent abilities

Performance of bigger models on various tasks, when plotted on a log-log scale, appears as a linear extrapolation of performance achieved by smaller models. However, this linearity may be punctuated by " break(s) " in the scaling law, where the slope of the line changes abruptly, and where larger models acquire "emergent abilities". They arise from the complex interaction of the model's components and are not explicitly programmed or designed.

The most intriguing among emergent abilities is in-context learning from example demonstrations. In-context learning is involved in tasks, such as:

Schaeffer et. al. argue that the emergent abilities are not unpredictably acquired, but predictably acquired according to a smooth scaling law. The authors considered a toy statistical model of an LLM solving multiple-choice questions, and showed that this statistical model, modified to account for other types of tasks, applies to these tasks as well.

Let x {\displaystyle x} be the number of parameter count, and y {\displaystyle y} be the performance of the model.