Umeå University's logo

This paper examines the emerging field of Hybrid Human-Artificial Intelligence (HHAI) through a scoping literature review of contributions to the HHAI conferences from 2022 to 2025, with the goal of identifying its core characteristics. HHAI focuses on the synergistic collaboration between human and machine intelligence, aiming to augment rather than replace human capabilities. To understand the fundamentals of HHAI as a developing research field, our research team - comprising both computer science and social science experts - conducted a scoping analysis of conference papers using a defined set of attributes, including submission type, author affiliations and disciplinary backgrounds, research topics and keywords, methodologies, and theoretical frameworks. The results of our scoping review suggest an emerging multidisciplinary character of HHAI as a research field. We found diversity in the disciplinary composition of the HHAI community, as well as in the key research topics and methodological approaches in HHAI conference literature. We see both qualitative and quantitative approaches across papers, though still with a predominant computer science presence. We conclude with suggestions for fostering more interdisciplinary engagement.

Dominant approaches to explainability in AI emphasise post hoc technical transparency, overlooking the socio-technical contexts in which systems are developed, deployed, and experienced. This paper argues that beginning AI adoption processes with Question Zero (Q0), “Should we adopt an AI system in the first place?”, reframes explainability as an essential requirement across the entire AI lifecycle rather than a narrow compliance task. Q0 challenges entrenched techno-solutionist assumptions that position AI as the default or best option, encouraging early integration of considerations of explainability in system design. By foregrounding this, Q0 shifts explainability towards purpose aligned, stakeholder aware forms that move beyond generic model centred outputs. Further, the paper reflects on how explainability can also strengthen Q0 by providing tools to assess the proportionality of AI adoption, clarify problem framing, and make visible the alternatives excluded during design. Taken together, these dual perspectives, Q0 for explainability and explainability for Q0, offer multidimensional opportunities for enhancing explainability. The paper illustrates this argument through reflection on the need, and value of, applying QO in the context of the European Commission’s Apply AI Strategy.

This paper critically evaluates the attempts to align Artificial Intelligence (AI) systems, especially Large Language Models (LLMs), with human values and intentions through Reinforcement Learning from Feedback methods, involving either human feedback (RLHF) or AI feedback (RLAIF). Specifically, we show the shortcomings of the broadly pursued alignment goals of honesty, harmlessness, and helpfulness. Through a multidisciplinary sociotechnical critique, we examine both the theoretical underpinnings and practical implementations of RLHF techniques, revealing significant limitations in their approach to capturing the complexities of human ethics, and contributing to AI safety. We highlight tensions inherent in the goals of RLHF, as captured in the HHH principle (helpful, harmless and honest). In addition, we discuss ethically-relevant issues that tend to be neglected in discussions about alignment and RLHF, among which the trade-offs between user-friendliness and deception, flexibility and interpretability, and system safety. We offer an alternative vision for AI safety and ethics which positions RLHF approaches within a broader context of comprehensive design across institutions, processes and technological systems, and suggest the establishment of AI safety as a sociotechnical discipline that is open to the normative and political dimensions of artificial intelligence.

Does vision-and-language (VL) training change the linguistic representations of language models in meaningful ways? Most results in the literature have shown inconsistent or marginal differences, both behaviorally and representationally. In this work, we start from the hypothesis that the domain in which VL training could have a significant effect is lexical-conceptual knowledge, in particular its taxonomic organization. Through comparing minimal pairs of text-only LMs and their VL-trained counterparts, we first show that the VL models often outperform their text-only counterparts on a text-only question-answering task that requires taxonomic understanding of concepts mentioned in the questions. Using an array of targeted behavioral and representational analyses, we show that the LMs and VLMs do not differ significantly in terms of their taxonomic knowledge itself, but they differ in how they represent questions that contain concepts in a taxonomic relation vs. a non-taxonomic relation. This implies that the taxonomic knowledge itself does not change substantially through additional VL training, but VL training does improve the deployment of this knowledge in the context of a specific task, even when the presentation of the task is purely linguistic.

The field of hybrid human-artificial intelligence (HHAI), although primarily driven by developments in AI, also requires fundamentally new approaches and solutions. Multidisciplinary in nature, it calls for collaboration across various research domains, such as AI, HCI, the cognitive and social sciences, philosophy and ethics, and complex systems, to name but a few.

This book presents the proceedings of HHAI 2024, the 3rd International Conference on Hybrid Human-Artificial Intelligence, held from 10-14 June 2024 in Malmö, Sweden. The focus of HHAI 2024 was on artificially-intelligent systems that cooperate synergistically, proactively and purposefully with humans, amplifying rather than replacing human intelligence. A total of 62 submissions were received for the main track of the conference, of which 31 were accepted for presentation after a thorough double blind review process. These comprised 9 full papers, 5 blue sky papers, and 17 working papers, making the final acceptance rate for full papers 29%. Acceptance rate across all tracks of the main program was 50%. This book contains all submissions accepted for the main track, as well as the proposals for the Doctoral Consortium and extended abstracts from the Posters and Demos track. Topics covered include human-AI interaction and collaboration; learning, reasoning and planning with humans and machines in the loop; fair, ethical, responsible, and trustworthy AI; societal awareness of AI; and the role of design and compositionality of AI systems in interpretable/collaborative AI, among others.

Providing a current overview of research and development, the book will be of interest to all those working in the field and facilitate the ongoing exchange and development of ideas across a range of disciplines.

We humans learn language and how to interact with the world through our different senses, grounding our language in what we can see, touch, hear, and smell. We call these streams of information different modalities, and our efficient processing and synthesis of the interactions between different modalities is a cornerstone of our intelligence. Therefore, it is important to study how we can build multimodal language models, where machine learning models learn from more than just text. This is particularly important in the era of large language models (LLMs), where their general capabilities are unclear and unreliable. This thesis investigates learning and reasoning in multimodal language models, and their capabilities to compositionally generalise in visual question answering tasks. Compositional generalisation is the process in which we produce and understand novel sentences, by systematically combining words and sentences to uncover the meaning in language, and has proven a challenge for neural networks. Previously, the literature has focused on compositional generalisation in text-only language models. One of the main contributions of this work is the extensive investigation of text-image language models. The experiments in this thesis compare three neural network-based models, and one neuro-symbolic method, and operationalise language grounding as the ability to reason with relevant functions over object affordances.

In order to better understand the capabilities of multimodal models, this thesis introduces CLEVR-Math as a synthetic benchmark of visual mathematical reasoning. The CLEVR-Math dataset involve tasks such as adding and removing objects from 3D scenes based on textual instructions, such as \emph{Remove all blue cubes. How many objects are left?}, and is given as a curriculum of tasks of increasing complexity. The evaluation set of CLEVR-Math includes extensive testing of different functional and object attribute generalisations. We open up the internal representations of these models using a technique called probing, where linear classifiers are trained to recover concepts such as colours or named entities from the internal embeddings of input data. The results show that while models are fairly good at generalisation with attributes (i.e.~solving tasks involving never before seen objects), it is a big challenge to generalise over functions and to learn abstractions such as categories. The results also show that complexity in the training data is a driver of generalisation, where an extended curriculum improves the general performance across tasks and generalisation tests. Furthermore, it is shown that training from scratch versus transfer learning has significant effects on compositional generalisation in models.

The results identify several aspects of how current methods can be improved in the future, and highlight general challenges in multimodal language models. A thorough investigation of compositional generalisation suggests that the pre-training of models allow models access to inductive biases that can be useful to solve new tasks. Contrastingly, models trained from scratch show much lower overall performance on the synthetic tasks at hand, but show lower relative generalisation gaps. In the conclusions and outlook, we discuss the implications of these results as well as future research directions.

This paper explores the integration of mixed-initiative systems in human-AI teams to improve coordination and communication in Search and Rescue (SAR) scenarios, leveraging dynamic control sharing to enhance operational effectiveness.

Fairness is central to the ethical and responsible development and use of AI systems, with a large number of frameworks and formal notions of algorithmic fairness being available. However, many of the fairness solutions proposed revolve around technical considerations and not the needs of and consequences for the most impacted communities. We therefore want to take the focus away from definitions and allow for the inclusion of societal and relational aspects to represent how the effects of AI systems impact and are experienced by individuals and social groups. In this paper, we do this by means of proposing the ACROCPoLis framework to represent allocation processes with a modeling emphasis on fairness aspects. The framework provides a shared vocabulary in which the factors relevant to fairness assessments for different situations and procedures are made explicit, as well as their interrelationships. This enables us to compare analogous situations, to highlight the differences in dissimilar situations, and to capture differing interpretations of the same situation by different stakeholders.

We introduce CLEVR-Math, a multi-modal math word problems dataset consisting of simple math word problems involving addition/subtraction, represented partly by a textual description and partly by an image illustrating the scenario. The text describes actions performed on the scene that is depicted in the image. Since the question posed may not be about the scene in the image, but about the state of the scene before or after the actions are applied, the solver envision or imagine the state changes due to these actions. Solving these word problems requires a combination of language, visual and mathematical reasoning. We apply state-of-the-art neural and neuro-symbolic models for visual question answering on CLEVR-Math and empirically evaluate their performances. Our results show how neither method generalise to chains of operations. We discuss the limitations of the two in addressing the task of multi-modal word problem solving.