Leveraging Log Instructions in Log-based Anomaly Detection
Date Issued
2022-07-10
Author(s)
Bogatinovski, Jasmin
Madjarov, Gjorgji
Nedelkoski, Sasho
Cardoso, Jorge
Kao, Odej
Abstract
Artificial Intelligence for IT Operations (AIOps)
describes the process of maintaining and operating large IT
systems using diverse AI-enabled methods and tools for, e.g.,
anomaly detection and root cause analysis, to support the remediation, optimization, and automatic initiation of self-stabilizing
IT activities. The core step of any AIOps workflow is anomaly
detection, typically performed on high-volume heterogeneous
data such as log messages (logs), metrics (e.g., CPU utilization),
and distributed traces. In this paper, we propose a method for
reliable and practical anomaly detection from system logs. It
overcomes the common disadvantage of related works, i.e., the
need for a large amount of manually labeled training data, by
building an anomaly detection model with log instructions from
the source code of 1000+ GitHub projects. The instructions from
diverse systems contain rich and heterogenous information about
many different normal and abnormal IT events and serve as a
foundation for anomaly detection. The proposed method, named
ADLILog, combines the log instructions and the data from the
system of interest (target system) to learn a deep neural network
model through a two-phase learning procedure. The experimental
results show that ADLILog outperforms the related approaches
by up to 60% on the F1 score while satisfying core non-functional
requirements for industrial deployments such as unsupervised
design, efficient model updates, and small model sizes.
describes the process of maintaining and operating large IT
systems using diverse AI-enabled methods and tools for, e.g.,
anomaly detection and root cause analysis, to support the remediation, optimization, and automatic initiation of self-stabilizing
IT activities. The core step of any AIOps workflow is anomaly
detection, typically performed on high-volume heterogeneous
data such as log messages (logs), metrics (e.g., CPU utilization),
and distributed traces. In this paper, we propose a method for
reliable and practical anomaly detection from system logs. It
overcomes the common disadvantage of related works, i.e., the
need for a large amount of manually labeled training data, by
building an anomaly detection model with log instructions from
the source code of 1000+ GitHub projects. The instructions from
diverse systems contain rich and heterogenous information about
many different normal and abnormal IT events and serve as a
foundation for anomaly detection. The proposed method, named
ADLILog, combines the log instructions and the data from the
system of interest (target system) to learn a deep neural network
model through a two-phase learning procedure. The experimental
results show that ADLILog outperforms the related approaches
by up to 60% on the F1 score while satisfying core non-functional
requirements for industrial deployments such as unsupervised
design, efficient model updates, and small model sizes.
Subjects
File(s)![Thumbnail Image]()
Loading...
Name
2207.03206.pdf
Size
452.94 KB
Format
Adobe PDF
Checksum
(MD5):8a6d4c12d810af80c71419f91c263228