Umeå University's logo

Industrial applications of Artificial Intelligence (AI) can be hindered by the issues of explainability and trust from end users. Human-computer interaction and eXplainable AI (XAI) concerns become imperative in such scenarios. However, the prior evidence of applying more general principles and techniques in specialized industrial scenarios is often limited. In this case study, we focus on designing interactive interfaces of XAI solutions for operators in the pulp and paper industry. The explanation techniques supported and compared include counterfactual and feature importance explanations. We applied the user centered design methodology, including the analysis of requirements elicited from operators during site visits and interactive interface prototype evaluation eventually conducted on site with five operators. Our results indicate that the operators preferred the combination of counterfactual and feature importance explanations. The study also provides lessons learned for researchers and practitioners.

This paper critically evaluates the attempts to align Artificial Intelligence (AI) systems, especially Large Language Models (LLMs), with human values and intentions through Reinforcement Learning from Feedback methods, involving either human feedback (RLHF) or AI feedback (RLAIF). Specifically, we show the shortcomings of the broadly pursued alignment goals of honesty, harmlessness, and helpfulness. Through a multidisciplinary sociotechnical critique, we examine both the theoretical underpinnings and practical implementations of RLHF techniques, revealing significant limitations in their approach to capturing the complexities of human ethics, and contributing to AI safety. We highlight tensions inherent in the goals of RLHF, as captured in the HHH principle (helpful, harmless and honest). In addition, we discuss ethically-relevant issues that tend to be neglected in discussions about alignment and RLHF, among which the trade-offs between user-friendliness and deception, flexibility and interpretability, and system safety. We offer an alternative vision for AI safety and ethics which positions RLHF approaches within a broader context of comprehensive design across institutions, processes and technological systems, and suggest the establishment of AI safety as a sociotechnical discipline that is open to the normative and political dimensions of artificial intelligence.

Machine Learning Operations (MLOps) involves software development practices for Machine Learning (ML), including data management, preprocessing, model training, deployment, and monitoring. While MLOps have received significant interest, much less work has been published addressing MLOps in industrial production settings lately, particularly if solutions are not cloud-based. This article addresses this shortcoming based on our and our partner’s real industrial experience in various projects. While there is a broad range of challenges for MLOps in cyber-physical production systems (CPPS), we focus on those related to data, models, and operations as we assume these will directly benefit the reader and provide solutions such as lightweight integration, integration of domain knowledge, periodic calibration, and interactive interfaces. In this way, we want to support practitioners in setting up industrial MLOps environments in CPPS. Further, we discuss explainability as an additional part of MLOps, which should be explored in more detail in the future.

Machine learning (ML) has become popular in the industrial sector as it helps to improve operations, increase efficiency, and reduce costs. However, deploying and managing ML models in production environments can be complex. This is where Machine Learning Operations (MLOps) comes in. MLOps aims to facilitate this deployment and management process. One of the MLOps challenges is understanding how ML models reason, which is key to trust and acceptance. Here, explainable AI (XAI) can help. Better error identification and improved model accuracy are only two resulting advantages. An often neglected fact is that deployed models are bypassed when model performance or explanations do not meet user expectations. In this paper, we provide a novel reference architecture to address the challenge of integrating explanations and feedback capabilities into MLOps. Our architecture is implemented in a series of industrial use cases in the project EXPLAIN. The proposed MLOps software architecture has several advantages. It provides an efficient way to manage ML models in production environments. Further, it allows for integrating explanations into the development and deployment processes.

Understanding when and why to apply any given eXplainable Artificial Intelligence (XAI) technique is not a straightforward task. There is no single approach that is best suited for a given context. This paper aims to address the challenge of selecting the most appropriate explainer given the context in which an explanation is required. For AI explainability to be effective, explanations and how they are presented needs to be oriented towards the stakeholder receiving the explanation. If—in general—no single explanation technique surpasses the rest, then reasoning over the available methods is required in order to select one that is context-appropriate. Due to the transparency they afford, we propose employing argumentation techniques to reach an agreement over the most suitable explainers from a given set of possible explainers.

In this paper, we propose a modular reasoning system consisting of a given mental model of the relevant stakeholder, a reasoner component that solves the argumentation problem generated by a multi-explainer component, and an AI model that is to be explained suitably to the stakeholder of interest. By formalizing supporting premises—and inferences—we can map stakeholder characteristics to those of explanation techniques. This allows us to reason over the techniques and prioritise the best one for the given context, while also offering transparency into the selection decision.

This paper presents an empirical study on the explainability of transformer models analyzing time series data, a largely unexplored area in the field of AI explainability. The study is part of an ongoing EU-funded project which applies a human-centered approach to developing explainable AI solutions for the process industry. Here, we investigate the choice of explainer mechanisms and human factor needs when developing eXplainable Artificial Intelligence (XAI) for operators of two industrial contexts: copper mining and paper manufacturing. On-site evaluations were conducted in these settings involving control room operators to test the prototype developed in the project. The results indicate that the method of feature importance alone was not sufficient to provide explanations that are tailored to individuals and situations, as required by users. Overall, our empirical data supports insights from previous research on human centered XAI and demonstrates the value of involving end users in the design process of effective XAI solutions. We also provide design implications which address human factor needs for such solutions in industrial settings.

This paper explores the integration of mixed-initiative systems in human-AI teams to improve coordination and communication in Search and Rescue (SAR) scenarios, leveraging dynamic control sharing to enhance operational effectiveness.

This article surveys the Variable Autonomy (VA) robotics literature that considers two contributory elements to Trustworthy AI: transparency and explainability. These elements should play a crucial role when designing and adopting robotic systems, especially in VA where poor or untimely adjustments of the system's level of autonomy can lead to errors, control conflicts, user frustration, and ultimate disuse of the system. Despite this need, transparency and explainability is, to the best of our knowledge, mostly overlooked in VA robotics literature or is not considered explicitly. In this article, we aim to present and examine the most recent contributions to the VA literature concerning transparency and explainability. In addition, we propose a way of thinking about VA by breaking these two concepts down based on: the mission of the human-robot team; who the stakeholder is; what needs to be made transparent or explained; why they need it; and how it can be achieved. Last, we provide insights and propose ways to move VA research forward. Our goal with this article is to raise awareness and inter-community discussions among the Trustworthy AI and the VA robotics communities.

Several high profile incidents that involve Artificial Intelligence (AI) have captured public attention and increased demand for regulation. Low public trust and attitudes towards AI reinforce the need for concrete policy around its development and use. However, current guidelines and standards rolled out by institutions globally are considered by many as high-level and open to interpretation, making them difficult to put into practice. This paper presents ongoing research in the field of Responsible AI and explores numerous methods of operationalising AI ethics. If AI is to be effectively regulated, it must not be considered as a technology alone—AI is embedded in the fabric of our societies and should thus be treated as a socio-technical system, requiring multi-stakeholder involvement and employment of continuous value-based methods of assessment. When putting guidelines and standards into practice, context is of critical importance. The methods and frameworks presented in this paper emphasise this need and pave the way towards operational AI ethics.

This dissertation presents AWKWARD: a hybrid architecture for the development of socially aware agents in Multi-Agent Systems (MAS). AWKWARD bridges Artificial Intelligence (AI) methods for their individual and combined strengths; Behaviour Oriented Design (BOD) is used to develop reactive planning agents, the OperA framework is used to modeland validate agent behaviour as per social norms, and Reinforcement Learning (RL) is used to optimise plan structures that induce desirable social outcomes. In concert, OperA and BOD help AWKWARD agents achieve real-time adjustment of reactive plans in response to social obligations. As systems scale, however, reactive plans become challenging to optimise by hand. In this work, AWKWARD’s extensibility is demonstrated to tackle this problem. The RL module is presented as an additional AI method that enables the automatic restructuring of reactive plans. A sample implementation of AWKWARD is developed in DOTA2—a game where success is heavily dependent on social interactions. The results gathered demonstrate the social outcome achieved from plan adjustments in real time, in both experiments of norm enforcement using OperA, and experiments of norm reinforcement using the extended RL module. Each sub-component, including the reactive planner itself, is a decision-making entity that dictates agent behaviour at various stages of the system’s life cycle. The level of decision-making control of each module is adjusted at various stages, making AWKWARD a system with Variable Autonomy (VA). The concept of VA isdiscussed as one that can aid in maintaining human control over a system. However, as with any VA system, challenges of transparency and transfer of control, amongst others, present themselves. Suggestions for tackling these challenges are presented as next steps for the maturation of AWKWARD as a platform, and the DOTA2 implementation as a research and educational platform.