Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: A Systematic Characterization of Sampling Algorithms for Open-ended
  Language Generation   This work studies the widely adopted ancestral sampling algorithms for
auto-regressive language models, which is not widely studied in the literature.
We use the quality-diversity (Q-D) trade-off to investigate three popular
sampling algorithms (top-k, nucleus and tempered sampling). We focus on the
task of open-ended language generation. We first show that the existing
sampling algorithms have similar performance. After carefully inspecting the
transformations defined by different sampling algorithms, we identify three key
properties that are shared among them: entropy reduction, order preservation,
and slope preservation. To validate the importance of the identified
properties, we design two sets of new sampling algorithms: one set in which
each algorithm satisfies all three properties, and one set in which each
algorithm violates at least one of the properties. We compare their performance
with existing sampling algorithms, and find that violating the identified
properties could lead to drastic performance degradation, as measured by the
Q-D trade-off. On the other hand, we find that the set of sampling algorithms
that satisfies these properties performs on par with the existing sampling
algorithms. Our data and code are available at
https://github.com/moinnadeem/characterizing-sampling-algorithms

28
The categories of this paper are: alg-geom
29
The categories of this paper are: funct-an
26
The categories of this paper are: bayes-an
33
The categories of this paper are: plasm-ph
31
The categories of this paper are: ao-sci
32
The categories of this paper are: atom-ph
17
The categories of this paper are: q-fin
24
The categories of this paper are: dg-ga
3
The categories of this paper are: physics
9
The categories of this paper are: nlin
25
The categories of this paper are: patt-sol
4
The categories of this paper are: cond-mat
8
The categories of this paper are: hep-ex
22
The categories of this paper are: chao-dyn
10
The categories of this paper are: q-bio
6
The categories of this paper are: astro-ph