text: In this paper, we propose a joint algorithm for the word segmentation on Chinese social media. Previous work mainly focus on word segmentation for plain Chinese text, in order to develop a Chinese social media processing tool, we need to take the main features of social media into account, whose grammatical structure is not rigorous, and the tendency of using colloquial and Internet terms makes the existing Chinese-processing tools inefficient to obtain good performance on social media. In our approach, we combine CRF and MMSEG algorithm and extend features of traditional CRF algorithm to train the model for word segmentation, We use Internet lexicon in order to improve the performance of our model on Chinese social media. Our experimental result on Sina Weibo shows that our approach outperforms the state-of-the-art model.
  @cite_0: During the last decade,especially since the First International Chinese Word Segmentation Bakeoff was held in July 2003,the study in automatic Chinese word segmentation has been greatly improved.Those improvements could be summarized as following:(1) on the computation sense Chinese words in real text have been well-defined by "segmentation guidelines + lexicon + segmented corpus";(2) practical results show that performance of statistic segmentation systems outperforms that of handcrafted rule-based systems;(3) the evaluation in terms of Bakeoff data shows that the accuracy drop caused by outof-vocabulary(OOV) words is at least five times greater than that of segmentation ambiguities;(4) the better performance of OOV recognition the higher accuracy of the segmentation system in whole,and the accuracy of statistic segmentation systems with character-based tagging approach outperforms any other word-based system.
summary: Currently research on natural language process for social media is still at the starting stage, in the 2006 SIGHAN Chinese word segmentation competition, approaches based on sequence annotation has been widely used. Microsoft used conditional random field and the size of feature window equals one. @cite_0 It turns out that features have been simplified, but the performance was still very good. DaLian Science and Technology University built two models, one is based on the use of the word CRF model, the other is based on MMSM model. During our experiments, we used CRF algorithm as well, and it plays the key role when we build the model. Our work mainly depends on the improvement of conditional random field.