Volume 7 Number 11 (Nov. 2012)
Home > Archive > 2012 > Volume 7 Number 11 (Nov. 2012) >
JSW 2012 Vol.7(11): 2622-2629 ISSN: 1796-217X
doi: 10.4304//jsw.7.11.2622-2629

A Length-variable Feature Code Based Fuzzy Duplicates Elimination Approach for Large Scale Chinese WebPages

Hongzhi Guo1, 2, Qingcai Chen1, 2, Cong Xin1, Xiaolong Wang1, 2
1Harbin Institute of Technology Shenzhen Graduate School, Shenzhen, P.R. China
2Dept. of Computer Science and Technology, Harbin Institute of Technology, Harbin, P.R. China

Abstract—Most of the existing Chinese webpage duplicate elimination approaches do not focus on noisy and fuzzy duplicates elimination. In this paper, we propose an efficient and noise-tolerant Chinese webpage duplicate elimination approach based on Length-variable Feature Code. First, an Independent Extraction Unit is defined to eliminate the impact of short paragraphs on feature code extraction. Then the concept of repeatability is introduced by using the longest common substring to enhance the noise tolerant capability. Experimental results on 10 million webpage dataset show that the proposed approach can efficiently deal with duplicates from massive WebPages with the duplicate elimination precision of 99.03%.

Index Terms—duplicates, near-duplicates, Independent Extracting Unit, Length-variable Feature Code, repeatability


Cite: Hongzhi Guo, Qingcai Chen, Cong Xin, Xiaolong Wang "A Length-variable Feature Code Based Fuzzy Duplicates Elimination Approach for Large Scale Chinese WebPages," Journal of Software vol. 7, no. 11, pp. 页码, 2012.

General Information

ISSN: 1796-217X (Online)
Frequency: Monthly (2006-2019); Bimonthly (Since 2020)
Editor-in-Chief: Prof. Antanas Verikas
Executive Editor: Ms. Yoyo Y. Zhou
Abstracting/ Indexing: DBLP, EBSCO, Google Scholar, ProQuest, INSPEC, ULRICH's Periodicals Directory, WorldCat, etc
E-mail: jsw@iap.org
  • Dec 06, 2019 News!

    Vol 14, No 1- Vol 14, No 4 has been indexed by EI (Inspec)   [Click]

  • Jun 22, 2020 News!

    Papers published in JSW Vol 14, No 1- Vol 15 No 4 have been indexed by DBLP     [Click]

  • Jun 22, 2020 News!

    The papers published in Vol 15, No 5 have all received dois from Crossref    [Click]

  • Aug 01, 2018 News!

    [CFP] 2020 the annual meeting of JSW Editorial Board, ICCSM 2020, will be held in Rome, Italy, July 17-19, 2020   [Click]

  • Jun 22, 2020 News!

    Vol 15, No 5 has been published with online version     [Click]