  Word embeddings have been widely used in biomedical Natural Language
Processing (NLP) applications as they provide vector representations of words
capturing the semantic properties of words and the linguistic relationship
between words. Many biomedical applications use different textual resources
(e.g., Wikipedia and biomedical articles) to train word embeddings and apply
these word embeddings to downstream biomedical applications. However, there has
been little work on evaluating the word embeddings trained from these
resources.In this study, we provide an empirical evaluation of word embeddings
trained from four different resources, namely clinical notes, biomedical
publications, Wikipedia, and news. We performed the evaluation qualitatively
and quantitatively. For the qualitative evaluation, we manually inspected five
most similar medical words to a given set of target medical words, and then
analyzed word embeddings through the visualization of those word embeddings.
For the quantitative evaluation, we conducted both intrinsic and extrinsic
evaluation. Based on the evaluation results, we can draw the following
conclusions. First, the word embeddings trained on clinical notes and
biomedical publications can capture the semantics of medical terms better, and
find more relevant similar medical terms, and are closer to human experts'
judgments, compared to these trained on Wikipedia and news. Second, there does
not exist a consistent global ranking of word embedding quality for downstream
biomedical NLP applications. However, adding word embeddings as extra features
will improve results on most downstream tasks. Finally, the word embeddings
trained on biomedical domain corpora do not necessarily have better performance
than those trained on other general domain corpora for any downstream
biomedical NLP tasks.
