
-
Previous Article
A mathematical analysis for the forecast research on tourism carrying capacity to promote the effective and sustainable development of tourism
- DCDS-S Home
- This Issue
-
Next Article
Collaborative filtering recommendation algorithm towards intelligent community
Uyghur morphological analysis using joint conditional random fields: Based on small scaled corpus
1. | Xinjiang Technical Institute of Physical and Chemistry, Chinese Academy of Sciences, Urumqi 830011, China |
2. | University of Chinese Academy of Sciences, Beijing 100049, China |
3. | Institute of Mathematics and Information of Hotan Teachers College, Hotan 848000, China |
As a fundamental research in the field of natural language processing, the Uyghur morphological analysis is used mainly to determine the part of speech (POS) and segmental morphemes (stem and affix) of a word in a given sentence, as well as to automatically annotate the grammatical function of the morphemes based on the context. It is necessary to provide various information for other tasks of natural language processing including syntactic analysis, machine translation, automatic summarization, and semantic analysis, etc. In order to increase the morphological analysis efficiency, this paper puts forward a hybrid approach to create a statistical model for Uyghur morphological tagging through a small-scale corpus. Experimental results show that this plan can obtain an overall accuracy of 92.58 % with a limited training corpus.
References:
[1] |
B. Aisha and M. Sun, A statistical method for Uyghur tokenization, in International Conference on Natural Language Processing and Knowledge Engineering, (2009), 1-5.
doi: 10.1109/NLPKE.2009.5313764. |
[2] |
Uyghur Language, Available from: https://en.wikipedia.org/wiki/Uyghur_language. Google Scholar |
[3] |
S. Dandapat, S. Sarkar and A. Basu, Automatic part-of-speech tagging for bengali: An approach for morphologically rich languages in a poor resource scenario, in ACL 2007, Proceedings of the Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic, 2007. Google Scholar |
[4] |
T. Ibrahim and B. Yuan, A survey on minority language information processing research and application in xinjiang, Journal of Chinese Information Processing, 6 (2011), 149-156. Google Scholar |
[5] |
T. Klymchuk, Regularizing algorithm for mixed matrix pencils, Applied Mathematics and Nonlinear Sciences, 2 (2017), 123-130. Google Scholar |
[6] |
O. Kohonen, S. Virpioja, L. Leppanen and K. Lagus, Semi-supervised extensions to morfessor baseline, Proceedings of the Morpho Challenge 2010 Workshop, 2010. Google Scholar |
[7] |
T. Kudo, K. Yamamoto and Y. Matsumoto, Applying conditional random fields to japanese morphological analysis, in Conference on Empirical Methods in Natural Language Processing, EMNLP 2004, A Meeting of Sigdat, A Special Interest Group of the Acl, Held in Conjunction with ACL 2004, 25-26 July 2004, Barcelona, Spain, 6 (2004), 230-237. Google Scholar |
[8] |
Lafferty, D. John, McCallum, Andrew, Pereira and C. N. Fernando, Conditional random fields: Probabilistic models for segmenting and labeling sequence data, 2001. Google Scholar |
[9] |
T. Litip, The possibility of handling phonetic harmony by computer in Uyghur, Journal of the Central University for Nationalities, 5 (2004), 108-113. Google Scholar |
[10] |
A. Mairehaba, W.-B. Jiang, Z.-Y. Wang, Y. Tuergen and Q. LIU,
Directed graph model of Uyghur morphological analysis, Journal of Software, 12 (2012), 3115-3129.
doi: 10.3724/SP.J.1001.2012.04205. |
[11] |
A. Mijit, N. Graham, M. Masato, M. Shinsuke, K. Tatsuya and H. Askar,
Uyghur Morpheme-based Language Models and ASR, Ipsj Sig Notes, (2010), 581-584.
doi: 10.1109/ICOSP.2010.5656065. |
[12] |
M. Orhun, A. C. eyd Tantug and A. Esref, Rule Based Analysis of the Uyghur Nouns, International Journal on Asian Language Processing, 1 (2009), 33-44. Google Scholar |
[13] |
L. Tohti, Modern Uyghur Reference Grammar, China Social Science Press, Beijing, 2012. Google Scholar |
[14] |
E. Tursun, D. Ganguly, T. Osman, Y. Yating, G. Abdukerim, Z. Junlin and L. Qun,
A semisupervised Tag-Transition-Based markovian model for Uyghur morphology analysis, ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16 (2016), 8-23.
doi: 10.1145/2968410. |
[15] |
A. Wumaier, T. Yibulayin, Z. Kadeer and S. Tian, Conditional random fields combined fsm stemming method for uyghur, in IEEE International Conference on Computer Science and Information Technology, (2009), 295-299.
doi: 10.1109/ICCSIT.2009.5234727. |
[16] |
H. Xue, Y. Yang, T. Osman, X. Li and R. Zhang, Uyghur word segmentation using a combination of rules and statistics, Advances in information Sciences and Service Sciences(AISS), 3 (2011), 105-113. Google Scholar |
[17] |
H. Zhang, Q. Cai, W. Jiang, Y. Lv and Q. Liu, Joint voice harmony restoration and morphological segmentation for morphology analysis, Journal of Chinese Information Processing, 6 (2014), 9-17. Google Scholar |
[18] |
L. Zhu, Y. Pan and J. Wang,
Affine transformation based ontology sparse vector learning algorithm, Applied Mathematics and Nonlinear Sciences, 2 (2017), 111-122.
doi: 10.21042/AMNS.2017.1.00009. |
show all references
References:
[1] |
B. Aisha and M. Sun, A statistical method for Uyghur tokenization, in International Conference on Natural Language Processing and Knowledge Engineering, (2009), 1-5.
doi: 10.1109/NLPKE.2009.5313764. |
[2] |
Uyghur Language, Available from: https://en.wikipedia.org/wiki/Uyghur_language. Google Scholar |
[3] |
S. Dandapat, S. Sarkar and A. Basu, Automatic part-of-speech tagging for bengali: An approach for morphologically rich languages in a poor resource scenario, in ACL 2007, Proceedings of the Meeting of the Association for Computational Linguistics, June 23-30, 2007, Prague, Czech Republic, 2007. Google Scholar |
[4] |
T. Ibrahim and B. Yuan, A survey on minority language information processing research and application in xinjiang, Journal of Chinese Information Processing, 6 (2011), 149-156. Google Scholar |
[5] |
T. Klymchuk, Regularizing algorithm for mixed matrix pencils, Applied Mathematics and Nonlinear Sciences, 2 (2017), 123-130. Google Scholar |
[6] |
O. Kohonen, S. Virpioja, L. Leppanen and K. Lagus, Semi-supervised extensions to morfessor baseline, Proceedings of the Morpho Challenge 2010 Workshop, 2010. Google Scholar |
[7] |
T. Kudo, K. Yamamoto and Y. Matsumoto, Applying conditional random fields to japanese morphological analysis, in Conference on Empirical Methods in Natural Language Processing, EMNLP 2004, A Meeting of Sigdat, A Special Interest Group of the Acl, Held in Conjunction with ACL 2004, 25-26 July 2004, Barcelona, Spain, 6 (2004), 230-237. Google Scholar |
[8] |
Lafferty, D. John, McCallum, Andrew, Pereira and C. N. Fernando, Conditional random fields: Probabilistic models for segmenting and labeling sequence data, 2001. Google Scholar |
[9] |
T. Litip, The possibility of handling phonetic harmony by computer in Uyghur, Journal of the Central University for Nationalities, 5 (2004), 108-113. Google Scholar |
[10] |
A. Mairehaba, W.-B. Jiang, Z.-Y. Wang, Y. Tuergen and Q. LIU,
Directed graph model of Uyghur morphological analysis, Journal of Software, 12 (2012), 3115-3129.
doi: 10.3724/SP.J.1001.2012.04205. |
[11] |
A. Mijit, N. Graham, M. Masato, M. Shinsuke, K. Tatsuya and H. Askar,
Uyghur Morpheme-based Language Models and ASR, Ipsj Sig Notes, (2010), 581-584.
doi: 10.1109/ICOSP.2010.5656065. |
[12] |
M. Orhun, A. C. eyd Tantug and A. Esref, Rule Based Analysis of the Uyghur Nouns, International Journal on Asian Language Processing, 1 (2009), 33-44. Google Scholar |
[13] |
L. Tohti, Modern Uyghur Reference Grammar, China Social Science Press, Beijing, 2012. Google Scholar |
[14] |
E. Tursun, D. Ganguly, T. Osman, Y. Yating, G. Abdukerim, Z. Junlin and L. Qun,
A semisupervised Tag-Transition-Based markovian model for Uyghur morphology analysis, ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16 (2016), 8-23.
doi: 10.1145/2968410. |
[15] |
A. Wumaier, T. Yibulayin, Z. Kadeer and S. Tian, Conditional random fields combined fsm stemming method for uyghur, in IEEE International Conference on Computer Science and Information Technology, (2009), 295-299.
doi: 10.1109/ICCSIT.2009.5234727. |
[16] |
H. Xue, Y. Yang, T. Osman, X. Li and R. Zhang, Uyghur word segmentation using a combination of rules and statistics, Advances in information Sciences and Service Sciences(AISS), 3 (2011), 105-113. Google Scholar |
[17] |
H. Zhang, Q. Cai, W. Jiang, Y. Lv and Q. Liu, Joint voice harmony restoration and morphological segmentation for morphology analysis, Journal of Chinese Information Processing, 6 (2014), 9-17. Google Scholar |
[18] |
L. Zhu, Y. Pan and J. Wang,
Affine transformation based ontology sparse vector learning algorithm, Applied Mathematics and Nonlinear Sciences, 2 (2017), 111-122.
doi: 10.21042/AMNS.2017.1.00009. |




Features | Description |
Unary context features of the word | |
Binary context features of the word | |
n characters selected from the beginning of the word | |
n characters selected from the end of the word | |
POS tag transition feature |
Features | Description |
Unary context features of the word | |
Binary context features of the word | |
n characters selected from the beginning of the word | |
n characters selected from the end of the word | |
POS tag transition feature |
Features | Description |
Unary context features of the morpheme | |
Binary context features of the morpheme | |
Morphological tag transition feature |
Features | Description |
Unary context features of the morpheme | |
Binary context features of the morpheme | |
Morphological tag transition feature |
Number of sentences | Number of words (including punctuation marks) | Number of Uyghur words | |
Training set | 1000 | 12433 | 10391 |
Development set | 200 | 2564 | 2151 |
Test set | 200 | 2492 | 2075 |
Number of sentences | Number of words (including punctuation marks) | Number of Uyghur words | |
Training set | 1000 | 12433 | 10391 |
Development set | 200 | 2564 | 2151 |
Test set | 200 | 2492 | 2075 |
Method | Accuracy (%) | |||
Stemming | Morpheme segmentation | POS | Overall | |
Tag sequence Markov model | 90.18 | 83.25 | 86.17 | 75.13 |
Joint CRF model | 91.98 | 85.79 | 92.7 | 77.95 |
Tag sequence Markov model, |
92.65 | 88.47 | 88.12 | 79.65 |
Joint CRF model, |
92.85 | 89.76 | 92.6 | 80.73 |
Method | Accuracy (%) | |||
Stemming | Morpheme segmentation | POS | Overall | |
Tag sequence Markov model | 90.18 | 83.25 | 86.17 | 75.13 |
Joint CRF model | 91.98 | 85.79 | 92.7 | 77.95 |
Tag sequence Markov model, |
92.65 | 88.47 | 88.12 | 79.65 |
Joint CRF model, |
92.85 | 89.76 | 92.6 | 80.73 |
Method(Joint CRF model, |
Accuracy (%) | |||
Stemming | Morpheme segmentation | POS | Overall | |
Joint CRF model, When filtering rules are not used |
92.85 | 89.76 | 92.6 | 80.73 |
Joint CRF model, When filtering rules are used |
97.4 | 94.58 | 96.35 | 92.58 |
Tag sequence transition model, When filtering rules are used |
94.35 | 93.22 | 94.78 | 91.81 |
Method(Joint CRF model, |
Accuracy (%) | |||
Stemming | Morpheme segmentation | POS | Overall | |
Joint CRF model, When filtering rules are not used |
92.85 | 89.76 | 92.6 | 80.73 |
Joint CRF model, When filtering rules are used |
97.4 | 94.58 | 96.35 | 92.58 |
Tag sequence transition model, When filtering rules are used |
94.35 | 93.22 | 94.78 | 91.81 |
[1] |
Seung-Yeal Ha, Shi Jin. Local sensitivity analysis for the Cucker-Smale model with random inputs. Kinetic & Related Models, 2018, 11 (4) : 859-889. doi: 10.3934/krm.2018034 |
[2] |
Lekbir Afraites, Abdelghafour Atlas, Fahd Karami, Driss Meskine. Some class of parabolic systems applied to image processing. Discrete & Continuous Dynamical Systems - B, 2016, 21 (6) : 1671-1687. doi: 10.3934/dcdsb.2016017 |
[3] |
Davi Obata. Symmetries of vector fields: The diffeomorphism centralizer. Discrete & Continuous Dynamical Systems, 2021 doi: 10.3934/dcds.2021063 |
[4] |
Jean-François Biasse. Improvements in the computation of ideal class groups of imaginary quadratic number fields. Advances in Mathematics of Communications, 2010, 4 (2) : 141-154. doi: 10.3934/amc.2010.4.141 |
[5] |
Lei Lei, Wenli Ren, Cuiling Fan. The differential spectrum of a class of power functions over finite fields. Advances in Mathematics of Communications, 2021, 15 (3) : 525-537. doi: 10.3934/amc.2020080 |
[6] |
Wei Xi Li, Chao Jiang Xu. Subellipticity of some complex vector fields related to the Witten Laplacian. Communications on Pure & Applied Analysis, , () : -. doi: 10.3934/cpaa.2021047 |
[7] |
Scott Schmieding, Rodrigo Treviño. Random substitution tilings and deviation phenomena. Discrete & Continuous Dynamical Systems, 2021, 41 (8) : 3869-3902. doi: 10.3934/dcds.2021020 |
[8] |
Reza Lotfi, Yahia Zare Mehrjerdi, Mir Saman Pishvaee, Ahmad Sadeghieh, Gerhard-Wilhelm Weber. A robust optimization model for sustainable and resilient closed-loop supply chain network design considering conditional value at risk. Numerical Algebra, Control & Optimization, 2021, 11 (2) : 221-253. doi: 10.3934/naco.2020023 |
[9] |
Mehmet Duran Toksari, Emel Kizilkaya Aydogan, Berrin Atalay, Saziye Sari. Some scheduling problems with sum of logarithm processing times based learning effect and exponential past sequence dependent delivery times. Journal of Industrial & Management Optimization, 2021 doi: 10.3934/jimo.2021044 |
[10] |
Nishant Sinha. Internal state recovery of Espresso stream cipher using conditional sampling resistance and TMDTO attack. Advances in Mathematics of Communications, 2021, 15 (3) : 539-556. doi: 10.3934/amc.2020081 |
[11] |
Guillaume Bal, Wenjia Jing. Homogenization and corrector theory for linear transport in random media. Discrete & Continuous Dynamical Systems, 2010, 28 (4) : 1311-1343. doi: 10.3934/dcds.2010.28.1311 |
[12] |
Jan Prüss, Laurent Pujo-Menjouet, G.F. Webb, Rico Zacher. Analysis of a model for the dynamics of prions. Discrete & Continuous Dynamical Systems - B, 2006, 6 (1) : 225-235. doi: 10.3934/dcdsb.2006.6.225 |
[13] |
Pablo D. Carrasco, Túlio Vales. A symmetric Random Walk defined by the time-one map of a geodesic flow. Discrete & Continuous Dynamical Systems, 2021, 41 (6) : 2891-2905. doi: 10.3934/dcds.2020390 |
[14] |
Yuta Tanoue. Improved Hoeffding inequality for dependent bounded or sub-Gaussian random variables. Probability, Uncertainty and Quantitative Risk, 2021, 6 (1) : 53-60. doi: 10.3934/puqr.2021003 |
[15] |
Florian Dorsch, Hermann Schulz-Baldes. Random Möbius dynamics on the unit disc and perturbation theory for Lyapunov exponents. Discrete & Continuous Dynamical Systems - B, 2021 doi: 10.3934/dcdsb.2021076 |
[16] |
Sohana Jahan. Discriminant analysis of regularized multidimensional scaling. Numerical Algebra, Control & Optimization, 2021, 11 (2) : 255-267. doi: 10.3934/naco.2020024 |
[17] |
Fumihiko Nakamura. Asymptotic behavior of non-expanding piecewise linear maps in the presence of random noise. Discrete & Continuous Dynamical Systems - B, 2018, 23 (6) : 2457-2473. doi: 10.3934/dcdsb.2018055 |
[18] |
Lingyu Li, Zhang Chen. Asymptotic behavior of non-autonomous random Ginzburg-Landau equation driven by colored noise. Discrete & Continuous Dynamical Systems - B, 2021, 26 (6) : 3303-3333. doi: 10.3934/dcdsb.2020233 |
[19] |
Xuping Zhang. Pullback random attractors for fractional stochastic $ p $-Laplacian equation with delay and multiplicative noise. Discrete & Continuous Dynamical Systems - B, 2021 doi: 10.3934/dcdsb.2021107 |
[20] |
Anhui Gu. Weak pullback mean random attractors for non-autonomous $ p $-Laplacian equations. Discrete & Continuous Dynamical Systems - B, 2021, 26 (7) : 3863-3878. doi: 10.3934/dcdsb.2020266 |
2019 Impact Factor: 1.233
Tools
Metrics
Other articles
by authors
[Back to Top]