Volume 9 Number 4 (Apr. 2014)
Home > Archive > 2014 > Volume 9 Number 4 (Apr. 2014) >
JSW 2014 Vol.9(4): 1028-1035 ISSN: 1796-217X
doi: 10.4304/jsw.9.4.1028-1035

The Large-scale Dynamic Data Rapid Reduction Algorithm Based on Map-Reduce

Jing-ling Yuan1, Jing Xie1, 2, Yan Yuan1, 2, Lin Li1, 2
1Computer Science and Technology school, Wuhan University of Technology, Wuhan, China
2School of Urban Design, Wuhan University, Wuhan, China


Abstract—With the advent of the era of “Big Data”, the application of the large-scale data is becoming popular. Efficiently using and analyzing the data has become an interesting research topic. Traditional knowledge reduction algorithms read small data samples once into a computer main memory for reduction, but it is not suitable for large-scale data. This paper takes large-scale sensor monitoring dynamic data as the research object and puts forward an incremental reduction algorithm based on Map-Reduce. Using a Hash fast partitioning strategy this algorithm divides the dynamic data set into multiple subdatasets to compute, which has greatly reduced the calculation time and space complexity of each node. Finally,experiments are conducted on the data from UCI Machine Learning Repository using Hadoop platform to prove that the algorithm is efficient and suitable for large-scale dynamic data. Compared to the traditional algorithms, the highest speedup of the parallel algorithm can be increased up to 1.55 times.

Index Terms—Large-scale dynamic data, increment knowledge reduction, Hash algorithm, Map-Reduce

[PDF]

Cite: Jing-ling Yuan, Jing Xie, Yan Yuan, Lin Li, "The Large-scale Dynamic Data Rapid Reduction Algorithm Based on Map-Reduce," Journal of Software vol. 9, no. 4, pp. 1028-1035, 2014.

General Information

ISSN: 1796-217X (Online)
Frequency: Monthly
Editor-in-Chief: Prof. Antanas Verikas
Executive Editor: Ms. Yoyo Y. Zhou
Abstracting/ Indexing: DBLP, EBSCO, ProQuest, INSPEC, ULRICH's Periodicals Directory, WorldCat, etc
E-mail: jsw@iap.org
  • Nov 18, 2019 News!

    Papers published in JSW Vol 14, No 1- Vol 14 No 10 have been indexed by DBLP     [Click]

  • Jun 25, 2019 News!

    Vol.13, No.9 has been indexed by EI (Inspec).   [Click]

  • Aug 01, 2018 News!

    [CFP] 2020 the annual meeting of JSW Editorial Board, ICCSM 2020, will be held in Rome, Italy, July 17-19, 2020   [Click]

  • Jul 10, 2019 News!

    Vol 14, No.8 has been published with online version 4 original aritcles from 2 countries are published in this issue.    [Click]

  • Nov 18, 2019 News!

    Vol 14, No 11 has been published with online version 4 original aritcles from 4 countries are published in this issue     [Click]