Volume 9 Number 7 (Jul. 2014)
Home > Archive > 2014 > Volume 9 Number 7 (Jul. 2014) >
JSW 2014 Vol.9(7): 1818-1826 ISSN: 1796-217X
doi: 10.4304/jsw.9.7.1818-1826

Challenges of Diacritical Marker or Hudhaa Character in Tokenization of Oromo Text

Abraham Tesso Nedjo, Degen Huang, Xiaoxia Liu
School of Computer Science and Technology, Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, China

Abstract—The problem of tokenization in natural language processing is to find a way to get every token in a text. For languages like Oromo, for which, much effort has not been done yet regarding language processing, the task of tokenization by no means cannot be overlooked. This paper reports on Oromo tokenizer that we designed and tested by accommodating the challenge of diacritical marker-Hudhaa which is a sign to represent in-word glottal sound in Oromo language. In this work, we have studied the effect of using acute accent for diacritical mark rather than using other confusing marks like right-quote to write Hudhaa. Accuracy is a prime factor in evaluating any Natural Language Processing (NLP) system. So we measured the accuracy of our system on 1.2MB (9686 sentences having 164932 words) of Oromo text data and an accuracy of 99.94% was achieved by this algorithm.

Index Terms—Diacritical Marker; Glottal; Hudhaa; Oromo; Tokenization

[PDF]

Cite: Abraham Tesso Nedjo, Degen Huang, Xiaoxia Liu, "Challenges of Diacritical Marker or Hudhaa Character in Tokenization of Oromo Text," Journal of Software vol. 9, no. 7, pp. 1818-1826, 2014.

General Information

ISSN: 1796-217X (Online)
Frequency: Monthly
Editor-in-Chief: Prof. Antanas Verikas
Executive Editor: Ms. Yoyo Y. Zhou
Abstracting/ Indexing: DBLP, EBSCO, ProQuest, INSPEC, ULRICH's Periodicals Directory, WorldCat, etc
E-mail: jsw@iap.org
  • Aug 21, 2019 News!

    Papers published in JSW Vol 14, No 1- Vol 14 No 8 have been indexed by DBLP     [Click]

  • Jun 25, 2019 News!

    Vol.13, No.9 has been indexed by EI (Inspec).   [Click]

  • Aug 01, 2018 News!

    [CFP] 2019 the annual meeting of JSW Editorial Board, ICCSM 2019, will be held in Barcelona, Spain, July 14-16, 2019.   [Click]

  • Jul 10, 2019 News!

    Vol 14, No.8 has been published with online version 4 original aritcles from 2 countries are published in this issue.    [Click]

  • Jul 22, 2019 News!

    Welcome Prof Ferhat Khendek from Canada to join the Editorial board of JSW    [Click]