ctual tasks, their performance significantly degraded compared to the default tasks. This suggests that while LMs may possess some degree of abstract task-solving skills, they often rely on narrow, non-transferable procedures for task-solving.

The authors also introduced a counterfactual comprehension check (CCC) to ensure that the LMs understood the counterfactual conditions specified in the prompts. The CCCs were simpler tasks that still allowed for the discrimination of default from counterfactual condi