![](/images/graphics-bg.png)
A Log-Based Anomaly Detection Method with Efficient Neighbor Searching and Automatic K Neighbor Selection
Joint Authors
Wang, Bingming
Ying, Shi
Yang, Zhe
Source
Issue
Vol. 2020, Issue 2020 (31 Dec. 2020), pp.1-17, 17 p.
Publisher
Hindawi Publishing Corporation
Publication Date
2020-06-02
Country of Publication
Egypt
No. of Pages
17
Main Subjects
Abstract EN
Using the k-nearest neighbor (kNN) algorithm in the supervised learning method to detect anomalies can get more accurate results.
However, when using kNN algorithm to detect anomaly, it is inefficient at finding k neighbors from large-scale log data; at the same time, log data are imbalanced in quantity, so it is a challenge to select proper k neighbors for different data distributions.
In this paper, we propose a log-based anomaly detection method with efficient selection of neighbors and automatic selection of k neighbors.
First, we propose a neighbor search method based on minhash and MVP-tree.
The minhash algorithm is used to group similar logs into the same bucket, and MVP-tree model is built for samples in each bucket.
In this way, we can reduce the effort of distance calculation and the number of neighbor samples that need to be compared, so as to improve the efficiency of finding neighbors.
In the process of selecting k neighbors, we propose an automatic method based on the Silhouette Coefficient, which can select proper k neighbors to improve the accuracy of anomaly detection.
Our method is verified on six different types of log data to prove its universality and feasibility.
American Psychological Association (APA)
Wang, Bingming& Ying, Shi& Yang, Zhe. 2020. A Log-Based Anomaly Detection Method with Efficient Neighbor Searching and Automatic K Neighbor Selection. Scientific Programming،Vol. 2020, no. 2020, pp.1-17.
https://search.emarefa.net/detail/BIM-1209021
Modern Language Association (MLA)
Wang, Bingming…[et al.]. A Log-Based Anomaly Detection Method with Efficient Neighbor Searching and Automatic K Neighbor Selection. Scientific Programming No. 2020 (2020), pp.1-17.
https://search.emarefa.net/detail/BIM-1209021
American Medical Association (AMA)
Wang, Bingming& Ying, Shi& Yang, Zhe. A Log-Based Anomaly Detection Method with Efficient Neighbor Searching and Automatic K Neighbor Selection. Scientific Programming. 2020. Vol. 2020, no. 2020, pp.1-17.
https://search.emarefa.net/detail/BIM-1209021
Data Type
Journal Articles
Language
English
Notes
Includes bibliographical references
Record ID
BIM-1209021