are simply specialized to specific tasks seen during pretraining.

The authors designed a suite of 11 tasks, each with a default and counterfactual version. The tasks spanned various domains, including arithmetic, programming, basic syntactic reasoning, logical reasoning, spatial reasoning, drawing, and music. The counterfactual tasks were designed to be similar in difficulty to the default tasks but differed in their input-output mappings.

The study found that while LMs performed above-random on counterfa