doi: 10.4304/jsw.9.7.1818-1826
Challenges of Diacritical Marker or Hudhaa Character in Tokenization of Oromo Text
- School of Computer Science and Technology, Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, China
Abstract
The problem of tokenization in natural language processing is to find a way to get every token in a text. For languages like Oromo, for which, much effort has not been done yet regarding language processing, the task of tokenization by no means cannot be overlooked. This paper reports on Oromo tokenizer that we designed and tested by accommodating the challenge of diacritical marker-Hudhaa which is a sign to represent in-word glottal sound in Oromo language. In this work, we have studied the effect of using acute accent for diacritical mark rather than using other confusing marks like right-quote to write Hudhaa. Accuracy is a prime factor in evaluating any Natural Language Processing (NLP) system. So we measured the accuracy of our system on 1.2MB (9686 sentences having 164932 words) of Oromo text data and an accuracy of 99.94% was achieved by this algorithm.
Keywords
- Diacritical
- Marker;
- Glottal;
- Hudhaa;
- Oro
How to Cite
Abraham Tesso Nedjo, Degen Huang, and Xiaoxia Liu, "Challenges of Diacritical Marker or Hudhaa Character in Tokenization of Oromo Text," Journal of Software, vol. 9, no. 7, pp. 1818-1826, 2014. https://doi.org/ 10.4304/jsw.9.7.1818-1826
Copyright & License
Copyright © 2014 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).