Partition Selection for Large-Scale Data Management Using KNN Join Processing
Joint Authors
Hu, Yue
Peng, Ge
Wang, Zehua
Cui, Yanrong
Qin, Hang
Source
Mathematical Problems in Engineering
Issue
Vol. 2020, Issue 2020 (31 Dec. 2020), pp.1-14, 14 p.
Publisher
Hindawi Publishing Corporation
Publication Date
2020-09-08
Country of Publication
Egypt
No. of Pages
14
Main Subjects
Abstract EN
For the data processing with increasing avalanche under large datasets, the k nearest neighbors (KNN) algorithm is a particularly expensive operation for both classification and regression predictive problems.
To predict the values of new data points, it can calculate the feature similarity between each object in the test dataset and each object in the training dataset.
However, due to expensive computational cost, the single computer is out of work to deal with large-scale dataset.
In this paper, we propose an adaptive vKNN algorithm, which adopts on the Voronoi diagram under the MapReduce parallel framework and makes full use of the advantages of parallel computing in processing large-scale data.
In the process of partition selection, we design a new predictive strategy for sample point to find the optimal relevant partition.
Then, we can effectively collect irrelevant data, reduce KNN join computation, and improve the operation efficiency.
Finally, we use a large number of 54-dimensional datasets to conduct a large number of experiments on the cluster.
The experimental results show that our proposed method is effective and scalable with ensuring accuracy.
American Psychological Association (APA)
Hu, Yue& Peng, Ge& Wang, Zehua& Cui, Yanrong& Qin, Hang. 2020. Partition Selection for Large-Scale Data Management Using KNN Join Processing. Mathematical Problems in Engineering،Vol. 2020, no. 2020, pp.1-14.
https://search.emarefa.net/detail/BIM-1200758
Modern Language Association (MLA)
Hu, Yue…[et al.]. Partition Selection for Large-Scale Data Management Using KNN Join Processing. Mathematical Problems in Engineering No. 2020 (2020), pp.1-14.
https://search.emarefa.net/detail/BIM-1200758
American Medical Association (AMA)
Hu, Yue& Peng, Ge& Wang, Zehua& Cui, Yanrong& Qin, Hang. Partition Selection for Large-Scale Data Management Using KNN Join Processing. Mathematical Problems in Engineering. 2020. Vol. 2020, no. 2020, pp.1-14.
https://search.emarefa.net/detail/BIM-1200758
Data Type
Journal Articles
Language
English
Notes
Includes bibliographical references
Record ID
BIM-1200758