Volume 7 Number 11 (Nov. 2012)
Home > Archive > 2012 > Volume 7 Number 11 (Nov. 2012) >
JSW 2012 Vol.7(11): 2424-2436 ISSN: 1796-217X
doi: 10.4304/jsw.7.11.2424-2436

Efficient and Effective Filtering of Duplication Detection in Large Database Applications

Ji Zhang
Department of Mathematics and Computing University of Southern Queensland Toowoomba, QLD 4350, Australia

Abstract—In this paper, a robust filtering technique, called PC-Filter (PC stands for partition comparison), is proposed for effective and efficient duplicate record detection in large databases. PC-Filter distinguishes itself from all of existing methods by using record partitions in duplicate detection. PC-Filter operates in three steps. It first sorts the whole database and splits the sorted database into a number of record partitions. The Partition Comparison Graph (PCG) is then generated by performing fast partition pruning. Finally, duplicate records are effectively detected through internal and external partition comparison based on PCG. Four closure properties, used as heuristics, have been devised to achieve a remarkable efficiency of the filter based on triangle inequity of record similarity. The partition size is well specified such that the time complexity of PC-Filter can be optimized. By equipping existing detection methods with PC-Filter, we are able to well solve the major problems that the existing methods suffer.

[PDF]

Cite: Ji Zhang, "Efficient and Effective Filtering of Duplication Detection in Large Database Applications," Journal of Software vol. 7, no. 11, pp. 2424-2436, 2012.

General Information

ISSN: 1796-217X (Online)
Frequency: Monthly
Editor-in-Chief: Prof. Antanas Verikas
Executive Editor: Ms. Yoyo Y. Zhou
Abstracting/ Indexing: DBLP, EBSCO, ProQuest, INSPEC, ULRICH's Periodicals Directory, WorldCat, CNKI,etc
E-mail: jsw@iap.org
  • Jun 25, 2019 News!

    Papers published in JSW Vol. 14, No. 1- Vol. 14 No. 6 have been indexed by DBLP.    [Click]

  • Jun 25, 2019 News!

    Vol.13, No.9 has been indexed by EI (Inspec).   [Click]

  • Aug 01, 2018 News!

    [CFP] 2019 the annual meeting of JSW Editorial Board, ICCSM 2019, will be held in Barcelona, Spain, July 14-16, 2019.   [Click]

  • Jul 10, 2019 News!

    Vol 14, No.8 has been published with online version 4 original aritcles from 2 countries are published in this issue.    [Click]

  • Jul 22, 2019 News!

    Welcome Prof Ferhat Khendek from Canada to join the Editorial board of JSW    [Click]