doi: 10.4304/jsw.9.4.1028-1035
The Large-scale Dynamic Data Rapid Reduction Algorithm Based on Map-Reduce
- 1 Computer Science and Technology school, Wuhan University of Technology, Wuhan, China
- 2 School of Urban Design, Wuhan University, Wuhan, China
Abstract
With the advent of the era of “Big Data”, the application of the large-scale data is becoming popular. Efficiently using and analyzing the data has become an interesting research topic. Traditional knowledge reduction algorithms read small data samples once into a computer main memory for reduction, but it is not suitable for large-scale data. This paper takes large-scale sensor monitoring dynamic data as the research object and puts forward an incremental reduction algorithm based on Map-Reduce. Using a Hash fast partitioning strategy this algorithm divides the dynamic data set into multiple subdatasets to compute, which has greatly reduced the calculation time and space complexity of each node. Finally，experiments are conducted on the data from UCI Machine Learning Repository using Hadoop platform to prove that the algorithm is efficient and suitable for large-scale dynamic data. Compared to the traditional algorithms, the highest speedup of the parallel algorithm can be increased up to 1.55 times.
Keywords
- Large-scale dynamic data
- increment know
How to Cite
Jing-ling Yuan, Jing Xie, Yan Yuan, and Lin Li, "The Large-scale Dynamic Data Rapid Reduction Algorithm Based on Map-Reduce," Journal of Software, vol. 9, no. 4, pp. 1028-1035, 2014. https://doi.org/ 10.4304/jsw.9.4.1028-1035
Copyright & License
Copyright © 2014 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).