  With LLMs shifting their role from statistical modeling of language to
serving as general-purpose AI agents, how should LLM evaluations change?
Arguably, a key ability of an AI agent is to flexibly combine, as needed, the
basic skills it has learned. The capability to combine skills plays an
important role in (human) pedagogy and also in a paper on emergence phenomena
(Arora & Goyal, 2023).
  This work introduces Skill-Mix, a new evaluation to measure ability to
combine skills. Using a list of $N$ skills the evaluator repeatedly picks
random subsets of $k$ skills and asks the LLM to produce text combining that
subset of skills. Since the number of subsets grows like $N^k$, for even modest
$k$ this evaluation will, with high probability, require the LLM to produce
text significantly different from any text in the training set. The paper
develops a methodology for (a) designing and administering such an evaluation,
and (b) automatic grading (plus spot-checking by humans) of the results using
GPT-4 as well as the open LLaMA-2 70B model.
  Administering a version of to popular chatbots gave results that, while
generally in line with prior expectations, contained surprises. Sizeable
differences exist among model capabilities that are not captured by their
ranking on popular LLM leaderboards ("cramming for the leaderboard").
Furthermore, simple probability calculations indicate that GPT-4's reasonable
performance on $k=5$ is suggestive of going beyond "stochastic parrot" behavior
(Bender et al., 2021), i.e., it combines skills in ways that it had not seen
during training.
  We sketch how the methodology can lead to a Skill-Mix based eco-system of
open evaluations for AI capabilities of future models.
