Umeå University's logo

This paper explores the co-design of an Interactive Digital Narrative (IDN) aimed at supporting the development of executive function (EF) in young children through culturally grounded interactive storytelling. Adopting a user-centered design approach, the research followed a five-phase iterative process involving nine participants from various disciplines, including social sciences, education, and Artificial Intelligence (AI), as well as parents. Three co-design workshops were conducted: (1) to integrate Swedish culture and values into the IDN, (2) to embed EF-stimulating interactions based on parent-child play, and (3) to conceptualise an AI-enhanced support tool for parent-child storytelling and cognitive development. Thematic analysis was employed to extract design insights from the audio recording of the workshops. These insights informed the development of a high-fidelity prototype featuring culturally resonant story elements that can support the development of EF in young children. Additionally, by positioning parents as companions rather than instructors and by embedding EF-supportive design into a joyful storytelling context, this work provides a framework for designing IDNs that are both pedagogically effective and playful for young children. This paper presents initial results of co-design of this IDN.

Human-robot interactions increasingly require adaptive instruction delivery, yet robots struggle to calibrate instruction detail levels without explicit user input. We present a system that automatically modulates instruction granularity using real-time affect detection through multi-modal fusion of thermal imaging, facial expressions, and contextual information. Our transformer-based architecture integrates these signals to enable decisions about instruction delivery based on detected user states. In a between-subjects study (N=40), participants completed assembly tasks under either manual adjustment or automatic adaptation conditions. Results showed significantly fewer manual adjustments in the adaptive condition (0.7 vs 2.0 per session), with comparable user satisfaction across conditions. This work shows the effectiveness of affect-driven adaptive instruction in human-robot interaction, contributing to more responsive robotic interfaces while providing guidelines for balancing automation with user control.

As children face global challenges, creating environments that nurture hope and empower them to shape a fair, transparent future is essential. Conversational AI systems (CAIs) offer opportunities for cognitive and emotional growth, with trust built through transparent, responsive interactions. This workshop offers participants a hands-on opportunity to analyze child-CAI interactions, bringing their own use cases alongside pre-recorded examples from five countries provided by organizers. In collaboration with a diverse group of stakeholders, the focus will be on identifying the human factors that influence trust in child-AI interactions, aiming to advance guidelines for building transparent, trustworthy conversational AI systems.

As AI systems increasingly engage in social interactions, comprehending human social dynamics is crucial. Affect recognition enables systems to respond appropriately to emotional nuances in social situations. However, existing multimodal approaches lack accounting for the social appropriateness of detected emotions within their contexts. This paper presents a novel methodology leveraging sentence embeddings to distinguish socially appropriate and inappropriate interactions for more context-aware AI systems. Our approach measures the semantic distance between facial expression descriptions and predefined reference points. We evaluate our method using a benchmark dataset and a real-world robot deployment in a library, combining GPT-4(V) for expression descriptions and ada-2 for sentence embeddings to detect socially inappropriate interactions. Our results underscore the importance of considering contextual factors for effective social interaction understanding through context-aware affect recognition, contributing to the development of socially intelligent AI capable of interpreting and responding to human affect appropriately.

Mental disorders including neurodevelopmental diffculties are frequent, creating a substantial disparity between the demand for mental health care and the available resources. The potential of therapeutic technologies to address this treatment gap is immense, offering scalable solutions to enhance access. Yet, the intricate nature of mental disorders, woven with diverse risk factors, poses challenges to a comprehensive understanding of their mechanisms and assessment of this potential. Current assessments of mental disorders heavily rely on the expertise of trained clinicians, making it imperative to explore innovative avenues such as "digital phenotyping" to capture nuanced behaviors. However, integrating technology into healthcare encounters obstacles exacerbated by the divergent cultures of medical professionals and engineers. While technical feasibility is a priority for engineers, it often needs to match the acceptability standards set by healthcare professionals. Navigating the complexity of the healthcare ecosystem compounds the challenge of identifying precise needs. Furthermore, the time-intensive nature of clinical research methods hinders the swift evaluation of effcacy. To surmount these hurdles, we advocate for the incorporation of user-centered design methodologies and participatory research in the development of therapeutic technologies. This chapter delves into the multifaceted challenges of designing technologies, such as robots, for therapeutic programs focused on individuals with neurodevelopmental disorders. By proposing solutions that prioritize participatory co-design environments, we aim to empower individuals from diverse backgrounds to collaboratively support those undergoing therapy with technology, ensuring its effcacy and benefts.

Background: Body image plays a crucial role in both physical and mental health, influencing self-esteem, eating behaviors, and psychological well-being. Young adults are particularly vulnerable to body dissatisfaction, defined as negative thoughts or feelings about one’s appearance. The benefits of positive body image, characterized by body appreciation and acceptance, are widely recognized, but few digital interventions are designed to support it for young adults.

Objective: We designed a conversational artificial intelligence (AI) agent integrating biomedical information on eating disorders and the principles of cognitive behavioral therapy to enable open-domain conversations on body image. The study explores young adults’ strategies to maintain a positive body image without the agent, the characteristics of conversations with the agent, and the advantages and drawbacks of having a conversation for body image concerns.

Methods: A qualitative study consisting of in-depth interviews with young adults was conducted among 15 young adults (aged 20-30 years) who used the AI agent in their homes for a 1-week period. Data comprise preinterviews exploring young adults’ maintenance of body image without the AI agent, text-based conversations with an AI agent (n=933 messages), and postinterviews on the perceived impact of conversations on body image awareness. Interview transcripts were analyzed through thematic analysis. Content analysis was applied to analyze the conversations with the AI agent.

Results: Young adults’ body image awareness was connected to self-acceptance, confidence, and valuing body functionality. Participants used several strategies to maintain body image without the AI agent, ranging from social support networks to exercise and positive self-talk. The conversations with the AI agent were categorized into (1) body image awareness, (2) body image–related eating and behavioral regulation, (3) body-focused mindfulness, and (4) social conversation with the agent. Three themes of perceived advantages and drawbacks regarding the conversations with the agent were identified as (1) facilitating body image awareness and self-reflection, (2) availability of conversational support, and (3) discontinuities in user engagement.

Conclusions: Young adults’ body image awareness is closely linked to self-acceptance and self-appreciation. In this context, the AI agent was perceived as an available, accessible, and nonjudgmental conversational support in raising body image awareness through self-reflection and self-compassion. Challenges remain in sustaining long-term user engagement, which address the need for multidimensional personalization of the agent.

Accurate recognition of human emotions is a crucial challenge in affective computing and human-robot interaction (HRI). Emotional states play a vital role in shaping behaviors, decisions, and social interactions. However, emotional expressions can be influenced by contextual factors, leading to misinterpretations if context is not considered. Multimodal fusion, combining modalities like facial expressions, speech, and physiological signals, has shown promise in improving affect recognition. This paper proposes a transformer-based multimodal fusion approach that leverages facial thermal data, facial action units, and textual context information for context-aware emotion recognition. We explore modality-specific encoders to learn tailored representations, which are then fused and processed by a shared transformer encoder to capture temporal dependencies and interactions. The proposed method is evaluated on a dataset collected from participants engaged in a tangible tabletop Pacman game designed to induce various affective states. Our results demonstrate improvements from incorporating contextual information and multimodal fusion, achieving 89% F1 score with our full model compared to 65% for action units alone and 30% for thermal data alone.

With the growing use of digital platforms for professional chat support, waiting time has become a major source of frustration, especially in digital mental health services for children, where many leave the queue before receiving help. This paper explores how an AI-powered interactive feature can improve children’s queue experience by identifying key design and functional requirements and evaluating its impact on engagement, boredom, and perceived waiting time. Through a Design Thinking process, two iterations were conducted: a low-fidelity prototype co-designed with children and experts, and a high-fidelity prototype featuring an LLM-based storytelling agent. The findings underscore the importance of safety, transparency, emotional expression, and personalization in child-AI interactions, showing that storytelling agents can reduce boredom and enhance engagement while children wait.

Storytelling is an effective evidence-based practice as an accepted intervention by therapists for the therapy of children with autism spectrum disorder (ASD). Digital storytelling, particularly using smart tangibles, offers a structured, interactive and engaging environment for children with ASD allowing for repetition, offering feedback with visual supports, and giving the child more authority over the learning experience.

This study presents a co-designed approach to digital storytelling activities with smart tangibles for autism therapy, aimed at enhancing multiple social and behavioral skills. Through co-design sessions with therapists, activity flows and scenarios were developed to target specific skill improvements. These include free play exploration, positive stimulus introduction, fostering cooperation to address disturbances, and incorporating magical objects to facilitate peer turn-taking. Additionally, real-life connections were emphasized to promote emotional regulation and multicultural understanding while further activities are designed to overcome routine issues, build tolerance to change, and enhance cognitive structuring.

Feasibility was demonstrated through integration into therapy sessions of five children, where therapists independently utilized the system, fostering immersive and interactive storytelling experiences. Overall, the co-designed activities offer insights into enhancing therapy interventions for children with ASD beyond specific contexts, contributing to the broader design of autism therapy activities.

We present here some lessons learned from observations and applications of child-robot interaction research in diverse ecosystems such as schools, therapy centers, and hospitals, where the interaction was facilitated in real-world circumstances rather than lab settings. Specifically, we use observational results from our reflections on multiple child-robot interaction practices in the wild conducted over a 9-year research period. Using these exploratory studies, we outline some general design considerations and adaptation guidelines for improving the design and implementation of robotic systems in healthcare and education that might lead to more practical, feasible, and ethically sustainable results.