Umeå University's logo

The emerging 5G networks promises more throughput, faster, and more reliable services, but as the network complexity and dynamics increases, it becomes more difficult to troubleshoot the systems. Vendors are spending a lot of time and effort on early anomaly detection in their development cycle and majority of the time is spent on manually analyzing system logs. While main research in anomaly detection uses performance metrics, anomaly detection using functional behaviour is still lacking in depth analysis. In this paper we show how a boosted ensemble of Long Short Term Memory classifiers can detect anomalies in the 5G Radio Access Network system logs. Acquiring system logs from a live 5G network is difficult due to confidentiality issues, live network disturbance, and problems to repeat scenarios. Therefore, we perform our evaluation on logs from a 5G test bed that simulate realistic traffic in a city. Our ensemble learns the functional behaviour of an application by training on logs from normal execution time. It can then detect deviations from normal behaviour and also be retrained on false positive cases found during validation. Anomaly detection in RAN shows that our ensemble called BoostLog, outperforms a single LSTM classifier and further testing on HDFS logs confirms that BoostLog also can be used in other domains. Instead of using domain experts to manually analyse system logs, BoostLog can be used by less experienced trouble shooters to automatically detect anomalies faster and more reliable.

The fifth generation (5G) RAN is a complex system consisting of virtualized and distributed parts with many concurrent threads interacting. Several services are time critical, and it is important for RAN vendors to understand where and why abnormal delays occur. Research within the latency anomaly detection area mainly focuses on utilizing performance metrics to analyze end-to-end latency or to detect delays at certain interfaces. One challenge is that many delays might occur due to application overload, scheduling, software bugs, or conflicts within the applications and in these cases, it is important to get a detailed view of the entire system. To improve the ability to analyze large complex systems, such as RAN, we present LogLatency, a novel combination learner approach that utilizes smart instrumented system logs. LogLatency employs statistical, probabilistic, and clustering methods to automatically learn the functional behavior of RAN and identify where latency anomalies occur. In our evaluation, which used an advanced 5G testbed, we conclude that LogLatency is superior to existing techniques and provides the fine granular view needed to improve 5G RAN in the future.

Developers of complex system like 5G Radio Access Networks (RAN) need algorithms that can automatically locate the root causes of software bugs. Existing methods mainly use supervised learning to track down root causes and only a few of these provide enough information to identify the function in which a software bug occurs. Supervised learning methods work well when scenarios can be repeated, and the normal behavior is somewhat similar. In RAN, where thousands of different configurations are used, software is updated frequently, and each node has its own traffic intensity, using unsupervised learning that does not require any pre-training can be more suitable. The few existing methods that use unsupervised learning to locate the root cause of software bugs can only detect delays or software hangs, and are not able to identify the many types of bugs that occur in RAN. We propose a multi-step method that uses unsupervised learning to analyze kernel and user space traces in system logs. The methods can guide developers by suggesting top-k candidate functions that are likely to contain a software bug. Our methods, MultiSpace and CallGraph were evaluated using an advanced 5G testbed in which many different software bugs that are common in RAN, were injected. The results shows that MultiSpace and CallGraph, can detect a wider range of software bugs than previous methods and only adds an average CPU load of 1.3% on the testbed. An important aspect is also that our methods scale well with large amount of data produced by real time systems, like RAN, and can analyze the data much faster.

Most existing large distributed systems have poor observability and cannot use the full potential of machine learning-based behavior analysis. The system logs, which contain the primary source of information, are unstructured and lack the context needed to track procedures and learn the system’s behavior. This work presents a new trace guideline that enables a component-and procedure-based split of the system logs for the future 5G Radio Access Network (RAN). As the system can be broken into smaller pieces, models can more accurately learn the system’s behavior and use the context to improve anomaly detection and observability. The evaluation result is astonishing; where previously state-of-the-art methods struggle to learn the behavior, a fast, dictionary-based algorithm can detect all anomalies and keep false positives close to zero. Troubleshooters can also more quickly identify anomalies and gain useful insights into the component interaction in RAN.

To meet the low latency requirements of 5G Radio Access Networks (RAN), it is essential to learn where performance bottlenecks occur. As parts are distributed and virtualized, it becomes troublesome to identify where unwanted delays occur. Today, vendors spend huge manual effort analyzing key performance indicators (KPIs) and system logs to detect these bottlenecks. The 5G architecture allows a flexible scaling of microservices to handle the variation in traffic. But knowing how, when, and where to scale is difficult without a detailed latency analysis. In this article, we propose a novel method that combines procedural learning with latency analysis of system log events. The method, which we call LogGenie, learns the latency pattern of the system at different load scenarios and automatically identifies the parts with the most significant increase in latency. Our evaluation in an advanced 5G testbed shows that LogGenie can provide a more detailed analysis than previous research has achieved and help troubleshooters locate bottlenecks faster. Finally, through experiments, we show how a latency prediction model can dynamically fine-tune the behavior where bottlenecks occur. This lowers resource utilization, makes the architecture more flexible, and allows the system to fulfill its latency requirements.