Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: More Than Words: Collocation Tokenization for Latent Dirichlet
  Allocation Models   Traditionally, Latent Dirichlet Allocation (LDA) ingests words in a
collection of documents to discover their latent topics using word-document
co-occurrences. However, it is unclear how to achieve the best results for
languages without marked word boundaries such as Chinese and Thai. Here, we
explore the use of Pearson's chi-squared test, t-statistics, and Word Pair
Encoding (WPE) to produce tokens as input to the LDA model. The Chi-squared, t,
and WPE tokenizers are trained on Wikipedia text to look for words that should
be grouped together, such as compound nouns, proper nouns, and complex event
verbs. We propose a new metric for measuring the clustering quality in settings
where the vocabularies of the models differ. Based on this metric and other
established metrics, we show that topics trained with merged tokens result in
topic keys that are clearer, more coherent, and more effective at
distinguishing topics than those unmerged models.

11
The categories of this paper are: hep-lat
30
The categories of this paper are: q-alg
10
The categories of this paper are: q-bio
24
The categories of this paper are: dg-ga
15
The categories of this paper are: nucl-ex
5
The categories of this paper are: gr-qc
2
The categories of this paper are: math
22
The categories of this paper are: chao-dyn
36
The categories of this paper are: cmp-lg
12
The categories of this paper are: nucl-th
23
The categories of this paper are: solv-int
4
The categories of this paper are: cond-mat
37
The categories of this paper are: supr-con
28
The categories of this paper are: alg-geom
34
The categories of this paper are: chem-ph
14
The categories of this paper are: math-ph