Umeå University's logo

Analyzing the data generated by complex information systems to identify undesirable behaviors in event log traces---so-called conformance checking---is a key challenge. With the rise of deep learning and, more specifically, generative AI applications, one promising line of research is the auto-generation of (symbolic) temporal reasoning queries that can then be applied in a semi-automatic manner. Recent work has demonstrated that utilizing fine-tuned open-source large language models (LLMs) for this purpose is promising and, in some aspects, superior to other approaches to automated conformance checking.

This thesis further expands this research direction by integrating additional state-of-the-art LLMs, such as GPT-4o and Llama, and supporting the provision of process-specific natural language context to evaluate their effectiveness. While previous framework is designed to work directly with event log schemata, this study explores whether incorporating human-readable text descriptions as supplementary input improves performance for non-fine-tuned models. A naïve baseline is introduced to validate that all models outperform random predictions, ensuring the robustness of the evaluation.

The results show that fine-tuning significantly enhances performance, with the xSemAD model achieving consistently higher F1-scores across most constraint types compared to state-of-the-art LLMs. However, text descriptions did not yield the expected performance improvements, highlighting the complexity of aligning contextual information with process semantics. Additionally, for some inherently ambiguous constraints, such as Choice and Alternate Succession model performance was only marginally better than the naïve baseline. These findings emphasize the importance of task-specific adaptation and the need for advanced methods to address complex constraints.

By demonstrating the potential of fine-tuned LLMs for semantic anomaly detection, this thesis contributes to advancing automated conformance checking and lays the groundwork for future research. Proposed directions include improving textual context generation, exploring alternative ground truth sources, and developing specialized techniques for handling complex constraints.