Volume 6 Number 3 (Mar. 2011)
Home > Archive > 2011 > Volume 6 Number 3 (Mar. 2011) >
JSW 2011 Vol.6(3): 413-420 ISSN: 1796-217X
doi: 10.4304/jsw.6.3.413-420

An Improvement to TF-IDF: Term Distribution based Term Weight Algorithm

Tian Xia1, Yanmei Chai2
1Shanghai Second Polytechnic University, Shanghai, China
2Central University of Finance and Economics, Beijing, China


Abstract—In the process of document formalization, term weight algorithm plays an important role. It greatly interferes the precision and recall results of the natural language processing (NLP) systems. Currently, TF-IDF term weight algorithm is widely applied into language models to build NLP Systems. Since term frequency is not the only discriminator which is necessary to be considered in term weighting and make each weight suitable to indicate the term’s importance, we are motivated to investigate other statistical characteristics of terms and found an important discriminator: term distribution. Furthermore, we found that, in a single document, a term with higher frequency and close to hypodispersion distribution usually contains much semantic information and should be given higher weight. One the other hand, in a document collection, the term with higher frequency and hypo-dispersion distribution usually contains less information. Based on this hypothesis, by leveraging the Pearson Chisquare Test Statistic, a Term Distribution based Local Term Weight Algorithm and Global Term Weight Algorithm are put forward respectively in this paper. Also, the experiment results at the end of this paper approve the reliability and efficiency of the algorithms.

Index Terms—TF, IDF, Term Weight, Natural Language Processing

[PDF]

Cite: Tian Xia, Yanmei Chai, "An Improvement to TF-IDF: Term Distribution based Term Weight Algorithm," Journal of Software vol. 6, no. 3, pp. 413-420, 2011.

General Information

ISSN: 1796-217X (Online)
Frequency:  Bimonthly 
Editor-in-Chief: Prof. Antanas Verikas
Executive Editor: Ms. Yoyo Y. Zhou
Abstracting/ Indexing: DBLP, EBSCO, CNKIGoogle Scholar, ProQuest, INSPEC(IET), ULRICH's Periodicals Directory, WorldCat, etc
E-mail: jsw@iap.org
  • Apr 26, 2021 News!

    Vol 14, No 4- Vol 14, No 12 has been indexed by IET-(Inspec)     [Click]

  • Nov 18, 2021 News!

    Papers published in JSW Vol 16, No 1- Vol 16, No 6 have been indexed by DBLP   [Click]

  • Dec 24, 2021 News!

     Vol 15, No 1- Vol 15, No 6 has been indexed by IET-(Inspec)   [Click]

  • Nov 18, 2021 News!

    [CFP] 2022 the annual meeting of JSW Editorial Board, ICCSM 2022, will be held in Rome, Italy, July 21-23, 2022   [Click]

  • Jul 26, 2022 News!

     Vol 17, No 5 has been published with online version     [Click]