As Vision-Language Models are increasingly deployed in safety-critical applications, the trustworthiness of their explanations becomes crucial. Explainable AI (XAI) methods for Vision-Language Models often suffer from semantic hallucination, where attribution maps highlight prominent image regions even when prompted with incorrect text descriptions (e.g., highlighting a dog when prompted “cat”). Although this problem is widespread, a formal mathematical analysis of XAI methods and CLIP embeddings is largely missing in the literature. We demonstrate that this phenomenon is not specific to a single architecture but is a fundamental consequence of Linear Semantic Leakage in high-dimensional embedding spaces. We propose a unified theoretical framework, Linear Semantic Attribution (LSA), which generalizes across discriminative methods. We introduce OSP, a geometric intervention that utilizes the residual property of OMP to disentangle unique semantic signals from shared concepts. We prove theoretically and demonstrate empirically that OSP minimizes hallucination by orthogonalizing the query vector against distractor concepts, rendering the attribution model blind to shared features while preserving fidelity for correct prompts. Our code is available at: https://github.com/emirhanbilgic/Orthogonal-Semantic-Projection
IR Lens: A Tool for Interpreting Cross-Encoder Models
Mihai
Branga-Peicu, Mathias
Vast, Basile
Van Cooten, and
4 more authors
In Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval, Jun 2026
Transformer-based ranking models, such as MonoBERT, are central to Information Retrieval; yet their inner workings remain largely opaque. This hinders not only our understanding of the systems implementing them, but also our ability to improve them. To alleviate this limitation, we introduce IR Lens, a new interpretability tool tailored to cross-encoders based on two key components: 1) Neuron Integrated Gradients to expose the contributions of model parts at multiple levels, and 2) targeted ablations to support hypothesis tracking. With its interactive graphical interface, IR Lens enables IR practitioners to explore, analyze, and manipulate neuron-level mechanisms in cross-encoders, facilitating a deeper understanding of neural ranking models. By extending the reach of existing interpretability methods, we believe IR Lens has the potential to support the improvement of cross-encoders.
Designing Movement Generation Models in Collaboration With Voguing And Dancehall Dancers
Léo
Chédin, Jules
Françoise, Baptiste
Caramiaux, and
1 more author
In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, Jun 2026
Recent advances in Artificial Intelligence have enabled powerful generative models, yet few are tailored to dancers’ practices. We present a long-term collaboration with a Voguing and Dancehall collective to design movement generation models trained on their repertoire. Our initial study with the dancers revealed that, despite limited physical realism, the generated movements inspired them. Iterative development led to Korai, an interactive tool for monitoring training, visualizing motion data, and prompting generation, which improved output quality. A subsequent structured observation study compared three model variants with high, medium, and low fidelity to the original dataset’s style. Results show that dancers favored either highly faithful or highly unfaithful outputs, rejecting medium fidelity as neither authentic to their style nor creatively stimulating. Our findings highlight how direct collaboration with dancers not only informs model design but also deepens understanding of AI’s role in supporting creative movement practices.
Designing Accessible Interfaces to Enhance Collective Musical Engagement for and with Children with Autism
Théo
Jourdan, Alejandro
Vanzandt-Escobar, and Baptiste
Caramiaux
We introduce PASTA (Perceptual Assessment System for explanaTion of Artificial Intelligence), a novel human-centric framework for evaluating eXplainable AI (XAI) techniques in computer vision. Our first contribution is the creation of the PASTA-dataset, the first large-scale benchmark that spans a diverse set of models and both saliency-based and concept-based explanation methods. This dataset enables robust, comparative analysis of XAI techniques based on human judgment. Our second contribution is an automated, data-driven benchmark that predicts human preferences using the PASTA-dataset. This scoring called PASTA-score method offers scalable, reliable, and consistent evaluation aligned with human perception. Additionally, our benchmark allows for comparisons between explanations across different modalities, an aspect previously unaddressed. We then propose to apply our scoring method to probe the interpretability of existing models and to build more human interpretable XAI methods.
Sensemaking in User-Driven Algorithm Auditing: A Case Study on Gender Bias in an Image Captioning Model
Behnoosh
Mohammadzadeh, Jules
Françoise, Michèle
Gouiffès, and
1 more author
In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (Best Paper 🏆), Apr 2026
Non-experts increasingly engage in user-driven algorithm auditing, interacting directly with AI systems to probe, document, and reflect on biased behavior. Yet, auditing remains challenging due to model opacity and limited support for navigating and interpreting outputs. This paper explores the design and evaluation of interfaces grounded in the sensemaking framework to support non-experts in auditing gender bias in image captioning. In a between-subjects study, 60 participants audited an image captioning model using one of three interface conditions: a Baseline interface, a Masking Tool for image manipulation, or a Filtering Tool for organizing captions. Our findings show that interface design shaped what participants noticed, how they interpreted model behavior, and supported their hypotheses. The Image Masking Tool enabled fine-grained testing of visual cues and context, while the Text Filtering Tool revealed broader asymmetries in gendered language. We argue that incorporating sensemaking into auditing practices can advance accountability and transparency in machine learning systems.
Artists on a Decade of AI Evolution: An Interview Study of Affordances, Culture, and Artistic Practice with Machine Learning
Téo
Sanchez, Mariya
Dzhimova, Stacy
Hsueh, and
3 more authors
In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, Apr 2026
In the mid-2010s, media artists began developing practices using machine learning (ML) as an artistic medium. Since 2022, the rise of large generative models, the mainstreaming of AI as consumer products, and intensifying ethical disputes have reconfigured the conditions of their artistic practice. This paper aims to understand how artists working with ML over the past decade respond to these shifts, shedding light on how practices, tools, and culture co-evolve. We address this question through thematic analysis of semi-structured interviews with 30 artists active before 2020. Our findings show how artists experience narrowing aesthetics and reduced malleability of post-2020 ML systems, have diverging views on where to locate moral responsibility with large AI models, and face shifting cultural reception that challenges the legibility of their work. We map how artists envision their practice going forward and discuss those orientations with respect to HCI conversations on design and creativity.
Exploring people’s testing strategies in ML-based image classification
Téo
Sanchez, Fani-Marina
Kalamara, Simone
Stumpf, and
1 more author
In Proceedings of the 1st International Conference on Human-Computer Interaction in the Alps, Feb 2026
Human testers—often end-users themselves—can judge system errors in context and reveal failures that automated methods miss. Inspired by software testing practices, we conducted an exploratory study in which 15 participants tested a satellite image classifier using an interactive tool that allowed them to collect data and create test cases. While participants shared a common understanding of the model’s behavior and generally adopted a failure-driven approach, we observed significant variability in their testing behaviors, including the number of test cases created, the timing of seeking feedback, the distribution of effort across classes, and the types of failures identified. Although links between specific strategies and outcomes remain unclear, our findings provide a first step toward understanding human testing of ML models and inform future research on human-driven AI auditing.
2025
Generative AI and Creative Work: Narratives, Values, and Impacts
Baptiste
Caramiaux, Kate
Crawford, Q. Vera
Liao, and
2 more authors
Generative AI has gained a significant foothold in the creative and artistic sectors. In this context, the concept of creative work is influenced by discourses originating from technological stakeholders and mainstream media. The framing of narratives surrounding creativity and artistic production not only reflects a particular vision of culture but also actively contributes to shaping it. In this article, we review online media outlets and analyze the dominant narratives around AI’s impact on creative work that they convey. We found that the discourse promotes creativity freed from its material realisation through human labor. The separation of the idea from its material conditions is achieved by automation, which is the driving force behind productive efficiency assessed as the reduction of time taken to produce. And the withdrawal of the skills typically required in the execution of the creative process is seen as a means for democratising creativity. This discourse tends to correspond to the dominant techno-positivist vision and to assert power over the creative economy and culture.
Generative AI in Documentary Photography: Exploring Opportunities and Challenges for Visual Storytelling
Lenny
Martinez, Baptiste
Caramiaux, and Sarah
Fdili Alaoui
In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, Apr 2025
Generative AI is increasingly used to create images from text, but its role in documentary photography remains under-explored. This paper investigates how generative AI can be integrated into documentary practice while maintaining ethical standards. Through interviews with six documentary photographers, we explored their views on AI’s potential to support community-driven storytelling. While AI presents opportunities for creative expression and community involvement, concerns about trust, authenticity, and decontextualization of images persist. Photographers expressed doubts about AI’s ability to accurately represent lived experiences, fearing it could compromise narrative integrity. Our findings suggest that AI tools should be designed to enhance collaboration and transparency in storytelling, complementing rather than replacing traditional documentary methods. This study contributes to the ongoing discourse on AI in photography, advocating for the development of tools that preserve the ethical foundations of documentary storytelling while empowering communities.
2024
Prototyping with Uncertainties: Data, Algorithms, and Research through Design
Elisa
Giaccardi, Dave
Murray-Rust, Johan
Redström, and
1 more author
ACM Transactions on Computer-Human Interaction, Dec 2024
Seen both as a resource and an obstacle to clarity, uncertainty is a concept that permeates many areas of design. As the concept gains prominence in Human-Computer Interaction (HCI), this special issue specifically explores the interplay between uncertainty and prototyping in Research through Design (RtD). We first outline three histories of uncertainty in design, in relation to its philosophical significance, its role in statistical and algorithmic processes, and its importance in prototyping. The convergence of these aspects is crucial as design evolves toward more agentive and entangled systems, introducing challenges such as Design as a Probabilistic Outcome. We then investigate the design spaces for engaging with “being uncertain” that emerge from the papers: from nuancing the relationship between designers and quantitative data to blurring the line between humans, fungi, and algorithms. Finally, we illuminate some preliminary threads for how RtD can navigate and engage with these shifting technological and design landscapes thoughtfully.
Studying Collaborative Interactive Machine Teaching in Image Classification
Behnoosh
Mohammadzadeh, Jules
Françoise, Michèle
Gouiffès, and
1 more author
In Proceedings of the 29th International Conference on Intelligent User Interfaces, Apr 2024
While human-centered approaches to machine learning explore various human roles within the interaction loop, the notion of Interactive Machine Teaching (IMT) emerged with a focus on leveraging the teaching skills of humans as a teacher to build machine learning systems. However, most systems and studies are devoted to single users. In this article, we study collaborative interactive machine teaching in the context of image classification to analyze how people can structure the teaching process collectively and to understand their experience. Our contributions are threefold. First, we developed a web application called TeachTOK that enables groups of users to curate data and train a model together incrementally. Second, we conducted a study in which ten participants were divided into three teams that competed to build an image classifier in nine days. Qualitative results of participants’ discussions in focus groups reveal the emergence of collaboration patterns in the machine teaching task, how collaboration helps revise teaching strategies and participants’ reflections on their interaction with the TeachTOK application. From these findings we provide implications for the design of more interactive, collaborative and participatory machine learning-based systems.
Comparing Teaching Strategies of a Machine Learning-based Prosthetic Arm
Vaynee
Sungeelee, Nathanaël
Jarrassé, Téo
Sanchez, and
1 more author
In Proceedings of the 29th International Conference on Intelligent User Interfaces, Apr 2024
Pattern-recognition-based arm prostheses rely on recognizing muscle activation to trigger movements. The effectiveness of this approach depends not only on the performance of the machine learner but also on the user’s understanding of its recognition capabilities, allowing them to adapt and work around recognition failures. We investigate how different model training strategies to select gesture classes and record respective muscle contractions impact model accuracy and user comprehension. We report on a lab experiment where participants performed hand gestures to train a classifier under three conditions: (1) the system cues gesture classes randomly (control), (2) the user selects gesture classes (teacher-led), (3) the system queries gesture classes based on their separability (learner-led). After training, we compare the models’ accuracy and test participants’ predictive understanding of the prosthesis’ behavior. We found that teacher-led and learner-led strategies yield faster and greater performance increases, respectively. Combining two evaluation methods, we found that participants developed a more accurate mental model when the system queried the least separable gesture class (learner-led). Our results conclude that, in the context of machine learning-based myoelectric prosthesis control, guiding the user to focus on class separability during training can improve recognition performances and support users’ mental models about the system’s behavior. We discuss our results in light of several research fields : myoelectric prosthesis control, motor learning, human-robot interaction, and interactive machine teaching.
Interactive curriculum learning increases and homogenizes motor smoothness
Vaynee
Sungeelee, Antoine
Loriette, Olivier
Sigaud, and
1 more author
One of the challenges of technology-assisted motor learning is how to adapt practice to facilitate learning. Random practice has been shown to promote long-term learning. However, it does not adapt to the learner’s specific learning requirements. Previous attempts to adapt learning considered the skill level of learners from past training sessions. This study investigates the effects of personalizing practice in real time, through a curriculum learning approach, where a curriculum of tasks is built by considering consecutive performance differences for each task. 12 participants were allocated to each of three training conditions in an experiment which required performing a steering task to drive a cursor in an arc channel. The curriculum learning approach was compared to two other conditions: random practice and another adaptive practice, which does not consider the learning evolution. The curriculum learning practice outperformed the random practice in effectively increasing movement smoothness at post-test and outperformed both the random practice and the adaptive practice on transfer tests. The adaptation of practice through the curriculum learning approach also made learners’ skills more uniform. Based on these findings, we anticipate that future research will explore the use of curriculum learning in interactive training tools to support motor skill learning, such as rehabilitation.
Trust in AI-assisted Decision Making: Perspectives from Those Behind the System and Those for Whom the Decision is Made
Oleksandra
Vereschak, Fatemeh
Alizadeh, Gilles
Bailly, and
1 more author
In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, May 2024
Trust between humans and AI in the context of decision-making has acquired an important role in public policy, research and industry. In this context, Human-AI Trust has often been tackled from the lens of cognitive science and psychology, but lacks insights from the stakeholders involved. In this paper, we conducted semi-structured interviews with 7 AI practitioners and 7 decision subjects from various decision domains. We found that 1) interviewees identified the prerequisites for the existence of trust and distinguish trust from trustworthiness, reliance, and compliance; 2) trust in AI-integrated systems is strongly influenced by other human actors, more than the system’s features; 3) the role of Human-AI trust factors is stakeholder-dependent. These results provide clues for the design of Human-AI interactions in which trust plays a major role, as well as outline new research directions in Human-AI Trust.
This paper discusses AI in the context of cultural heritage. First, I contextualize what we call AI, particularly with respect to infrastructure. With this representation in mind, my first objective is to outline the opportunities that AI can bring to these sec- tors, as identified in a series of reports and white papers edited by European institu- tions. In these reports, we have, however, barely grasped the need for stakeholders in these sectors to have their say on how they see this technology and how it should be integrated into their practice and organizations. My second objective is thus to highlight the fact that AI is not just a source of opportunities, as this would obscu- re the sociocultural and sociotechnical implications of integrating AI into existing practices.
Culture and Politics of Machine Learning in NIME: A Preliminary Qualitative Inquiry
Théo
Jourdan and Baptiste
Caramiaux
In New Interfaces for Musical Expression (NIME), May 2023
For several years, the various practices around ML techniques have been increasingly present and diversified. However, the literature associated with these techniques rarely reveals the cultural and political sides of these practices. In order to explore how practitioners in the NIME community engage with ML techniques, we conducted interviews with seven researchers in the NIME community and analysed them through a thematic analysis. Firstly, we propose findings at the level of the individual, resisting technological determinism and redefining sense making in interactive ML. Secondly, we propose findings at the level of the community, revealing mitigated adoption with respect to ML. This paper aims to provide the community with some reflections on the use of ML in order to initiate a discussion about cultural, political and ethical issues surrounding these techniques as their use grows within the community.
Machine Learning for Musical Expression: A Systematic Literature Review
Théo
Jourdan and Baptiste
Caramiaux
In New Interfaces for Musical Expression (NIME), May 2023
For several decades NIME community has always been appropriating machine learning (ML) to apply for various tasks such as gesture-sound mapping or sound synthesis for digital musical instruments. Recently, the use of ML methods seems to have increased and the objectives have diversified. Despite its increasing use, few contributions have studied what constitutes the culture of learning technologies for this specific practice. This paper presents an analysis of 69 contributions selected from a systematic review of the NIME conference over the last 10 years. This paper aims at analysing the practices involving ML in terms of the techniques and the task used and the ways to interact this technology. It thus contributes to a deeper understanding of the specific goals and motivation in using ML for musical expression. This study allows us to propose new perspectives in the practice of these techniques.
Describing movement learning using metric learning
Antoine
Loriette, Wanyu
Liu, Frédéric
Bevilacqua, and
1 more author
Analysing movement learning can rely on human evaluation, e.g. annotating video recordings, or on computing means in applying metrics on behavioural data. However, it remains challenging to relate human perception of movement similarity to computational measures that aim at modelling such similarity. In this paper, we propose a metric learning method bridging the gap between human ratings of movement similarity in a motor learning task and computational metric evaluation on the same task. It applies metric learning on a Dynamic Time Warping algorithm to derive an optimal set of movement features that best explain human ratings. We evaluated this method on an existing movement dataset, which comprises videos of participants practising a complex gesture sequence toward a target template, as well as the collected data that describes the movements. We show that it is possible to establish a linear relationship between human ratings and our learned computational metric. This learned metric can be used to describe the most salient temporal moments implicitly used by annotators, as well as movement parameters that correlate with motor improvements in the dataset. We conclude with possibilities to generalise this method for designing computational tools dedicated to movement annotation and evaluation of skill learning.
Effect of sonification types in upper-limb movement: a quantitative and qualitative study in hemiparetic and healthy participants
Iseline
Peyre, Agnès
Roby-Brami, Maël
Segalen, and
5 more authors
Journal of NeuroEngineering and Rehabilitation, Oct 2023
Movement sonification, the use of real-time auditory feedback linked to movement parameters, have been proposed to support rehabilitation. Nevertheless, if promising results have been reported, the effect of the type of sound used has not been studied systematically. The aim of this study was to investigate in a single session the effect of different types of sonification both quantitatively and qualitatively on patients with acquired brain lesions and healthy participants.
Interaction Knowledge: Understanding the ‘Mechanics’ of Digital Tools
Miguel A.
Renom, Baptiste
Caramiaux, and Michel
Beaudouin-Lafon
In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Apr 2023
User interfaces typically feature tools to act on objects and rely on the ability of users to discover or learn how to interact with them. Previous work in HCI has used the Theory of Affordances to explain how users understand the possibilities for action in digital environments. A complementary theory from cognitive neuroscience, Technical Reasoning, posits that users accumulate abstract knowledge of object properties and technical principles known as mechanical knowledge, essential in tool use. Drawing from this theory, we introduce interaction knowledge as the “mechanical” knowledge of digital environments. We provide evidence of its relevance by reporting on an experiment where participants performed tasks in a digital environment with ambiguous possibilities for interaction. We analyze how interaction knowledge was transferred across two digital domains, text editing and graphical editing, and conclude that interaction knowledge models an essential type of knowledge for interacting in the digital world.
Probing Respiratory Care With Generative Deep Learning
Hugo
Scurto, Thomas
Similowski, Samuel
Bianchini, and
1 more author
This paper combines design, machine learning and social computing to explore generative deep learning as both tool and probe for respiratory care. We first present GANspire, a deep learning tool that generates fine-grained breathing waveforms, which we crafted in collaboration with one respiratory physician, attending to joint materialities of human breathing data and deep generative models. We then relate a probe, produced with breathing waveforms generated with GANspire, and led with a group of ten respiratory care experts, responding to its material attributes. Qualitative annotations showed that respiratory care experts interpreted both realistic and ambiguous attributes of breathing waveforms generated with GANspire, according to subjective aspects of physiology, activity and emotion. Semi-structured interviews also revealed experts’ broader perceptions, expectations and ethical concerns on AI technology, based on their clinical practice of respiratory care, and reflexive analysis of GANspire. These findings suggest design implications for technological aids in respiratory care, and show how ambiguity of deep generative models can be leveraged as a resource for qualitative inquiry, enabling socio-material research with generative deep learning. Our paper contributes to the CSCW community by broadening how generative deep learning may be approached not only as a tool to design human-computer interactions, but also as a probe to provoke open conversations with communities of practice about their current and speculative uses of AI technology.
2022
"Explorers of Unknown Planets": Practices and Politics of Artificial Intelligence in Visual Arts
Baptiste
Caramiaux and Sarah
Fdili Alaoui
Proceedings of the ACM on Human-Computer Interaction, Nov 2022
Alongside recent advances in artificial intelligence (AI), a new art practice has emerged in recent years that borrows and transforms these advances in the production of artworks. The actors of this emergent practice are coming from contemporary art, media and digital arts. These artists have developed an original practice of AI within their creative field. In this article, we propose a qualitative study to explore the nature of this practice. We interviewed five internationally renowned artists about how AI is integrated into their work. Through a thematic analysis of the interviews, we first find that their practice relies on crafting algorithms and data as materials. We uncover how they explicitly use this material unpredictability rather than avoid it. Secondly, we highlight the politics of their practice that consist of resisting the culture of AI research, as well as its inherent power dynamics. We also highlight how their relationship with the technology is imbued with ethics and how they rethink their role with respect to the technology. In this paper, we aim to provide the CSCW community with a way to expand the framework in which AI can be understood not only as a tool but also as cultural and political design material.
Gestural Sound Toolkit: Reflections on an Interactive Design Project
Baptiste
Caramiaux, Alessandro
Altavilla, Jules
Françoise, and
1 more author
In International Conference on New Interfaces for Musical Expression, Nov 2022
The Technical Reasoning hypothesis in cognitive neuroscience posits that humans engage in physical tool use by reasoning about mechanical interactions among objects. By modeling the use of objects as tools based on their abstract properties, this theory explains how tools can be re-purposed beyond their assigned function. This paper assesses the relevance of Technical Reasoning to digital tool use. We conducted an experiment with 16 participants that forced them to re-purpose commands to complete a text layout task. We analyzed self-reported scores of creative personality and experience with text editing, and found a significant association between re-purposing performance and creativity, but not with experience. Our results suggest that while most participants engaged in Technical Reasoning to re-purpose digital tools, some experienced “functional fixedness.” This work contributes Technical Reasoning as a theoretical model for the design of digital tools.
Deep Learning Uncertainty in Machine Teaching
Téo
Sanchez, Baptiste
Caramiaux, Pierre
Thiel, and
1 more author
In Proceedings of the 27th International Conference on Intelligent User Interfaces (Best Paper 🏆), Mar 2022
Machine Learning models can output confident but incorrect predictions. To address this problem, ML researchers use various techniques to reliably estimate ML uncertainty, usually performed on controlled benchmarks once the model has been trained. We explore how the two types of uncertainty—aleatoric and epistemic—can help non-expert users understand the strengths and weaknesses of a classifier in an interactive setting. We are interested in users’ perception of the difference between aleatoric and epistemic uncertainty and their use to teach and understand the classifier. We conducted an experiment where non-experts train a classifier to recognize card images, and are tested on their ability to predict classifier outcomes. Participants who used either larger or more varied training sets significantly improved their understanding of uncertainty, both epistemic or aleatoric. However, participants who relied on the uncertainty measure to guide their choice of training data did not significantly improve classifier training, nor were they better able to guess the classifier outcome. We identified three specific situations where participants successfully identified the difference between aleatoric and epistemic uncertainty: placing a card in the exact same position as a training card; placing different cards next to each other; and placing a non-card, such as their hand, next to or on top of a card. We discuss our methodology for estimating uncertainty for Interactive Machine Learning systems and question the need for two-level uncertainty in Machine Teaching.
2021
Marcelle: Composing Interactive Machine Learning Workflows and Interfaces
Jules
Françoise, Baptiste
Caramiaux, and Téo
Sanchez
In The 34th Annual ACM Symposium on User Interface Software and Technology, Mar 2021
Software tools for generating digital sound often present users with high-dimensional, parametric interfaces, that may not facilitate exploration of diverse sound designs. In this article, we propose to investigate artificial agents using deep reinforcement learning to explore parameter spaces in partnership with users for sound design. We describe a series of user-centred studies to probe the creative benefits of these agents and adapting their design to exploration. Preliminary studies observing users’ exploration strategies with parametric interfaces and testing different agent exploration behaviours led to the design of a fully-functioning prototype, called Co-Explorer, that we evaluated in a workshop with professional sound designers. We found that the Co-Explorer enables a novel creative workflow centred on human–machine partnership, which has been positively received by practitioners. We also highlight varied user exploration behaviours throughout partnering with our system. Finally, we frame design guidelines for enabling such co-exploration workflow in creative digital applications.
Prototyping Machine Learning Through Diffractive Art Practice
Hugo
Scurto, Baptiste
Caramiaux, and Frédéric
Bevilacqua
In Designing Interactive Systems Conference 2021, Jan 2021
The spread of AI-embedded systems involved in human decision making makes studying human trust in these systems critical. However, empirically investigating trust is challenging. One reason is the lack of standard protocols to design trust experiments. In this paper, we present a survey of existing methods to empirically investigate trust in AI-assisted decision making and analyse the corpus along the constitutive elements of an experimental protocol. We find that the definition of trust is not commonly integrated in experimental protocols, which can lead to findings that are overclaimed or are hard to interpret and compare across studies. Drawing from empirical practices in social and cognitive studies on human-human trust, we provide practical guidelines to improve the methodology of studying Human-AI trust in decision-making contexts. In addition, we bring forward research opportunities of two types: one focusing on further investigation regarding trust methodologies and the other on factors that impact Human-AI trust.
2020
Machine Learning Approaches for Motor Learning: A Short Review
Baptiste
Caramiaux, Jules
Françoise, Wanyu
Liu, and
2 more authors
Machine learning approaches have seen considerable applications in human movement modeling, but remain limited for motor learning. Motor learning requires accounting for motor variability, and poses new challenges as the algorithms need to be able to differentiate between new movements and variation of known ones. In this short review, we outline existing machine learning models for motor learning and their adaptation capabilities. We identify and describe three types of adaptation: Parameter adaptation in probabilistic models, Transfer and meta-learning in deep neural networks, and Planning adaptation by reinforcement learning. To conclude, we discuss challenges for applying these models in the domain of motor learning support systems.
Making Mappings: Examining the Design Process
Travis J
West, Baptiste
Caramiaux, and Marcelo M
Wanderley
In New Interfaces for Musical Expression (NIME’20), Jun 2020
We conducted a study which examines mappings from a relatively unexplored perspective: how they are made. Twelve skilled NIME users designed a mapping from a T-Stick to a subtractive synthesizer, and were interviewed about their approach to mapping design. We present a thematic analysis of the interviews, with reference to data recordings captured while the designers worked. Our results suggest that the mapping design process is an iterative process that alternates between two working modes: diffuse exploration and directed experimentation.
2019
AI in the media and creative industries
Baptiste
Caramiaux, Fabien
Lotte, Joost
Geurts, and
16 more authors
Our goal is to understand how dancers learn complex dance phrases. We ran three workshops where dancers learned dance fragments from videos. In workshop 1, we analyzed how dancers structure their learning strategies by decomposing movements. In workshop 2, we introduced MoveOn, a technology probe that lets dancers decompose video into short, repeatable clips to support their learning. This served as an effective analysis tool for identifying the changes in focus and understanding their decomposition and recomposition processes. In workshop 3, we compared the teacher’s and dancers’ decomposition strategies, and how dancers learn on their own compared to teacher-created decompositions. We found that they all ungroup and regroup dance fragments, but with different foci of attention, which suggests that teacher-imposed decomposition is more effective for introductory dance students, whereas personal decomposition is more suitable for expert dancers. We discuss the implications for designing technology to support analysis, learning and teaching of dance through movement decomposition.
2018
Dissociable effects of practice variability on learning motor and timing skills
Baptiste
Caramiaux, Frédéric
Bevilacqua, Marcelo M.
Wanderley, and
1 more author
Machine learning is the capacity of a computational system to learn structure from data in order to make predictions on new data. This chapter draws on music, machine learning, and human-computer interaction to elucidate an understanding of machine learning algorithms as creative tools for music and the sonic arts. It motivates a new understanding of learning algorithms as human-computer interfaces: like other interfaces, learning algorithms can be characterized by the ways their affordances intersect with goals of human users. The chapter also argues that the nature of interaction between users and algorithms impacts the usability and usefulness of those algorithms in profound ways. This human-centred view of machine learning motivates a concluding discussion of what it means to employ machine learning as a creative tool.
Embracing First-Person Perspectives in Soma-Based Design
Kristina
Höök, Baptiste
Caramiaux, Cumhur
Erkut, and
19 more authors
A set of prominent designers embarked on a research journey to explore aesthetics in movement-based design. Here we unpack one of the design sensitivities unique to our practice: a strong first person perspective—where the movements, somatics and aesthetic sensibilities of the designer, design researcher and user are at the forefront. We present an annotated portfolio of design exemplars and a brief introduction to some of the design methods and theory we use, together substantiating and explaining the first-person perspective. At the same time, we show how this felt dimension, despite its subjective nature, is what provides rigor and structure to our design research. Our aim is to assist researchers in soma-based design and designers wanting to consider the multiple facets when designing for the aesthetics of movement. The applications span a large field of designs, including slow introspective, contemplative interactions, arts, dance, health applications, games, work applications and many others.
Perceiving Agent Collaborative Sonic Exploration In Interactive Reinforcement Learning
Hugo
Scurto, Frédéric
Bevilacqua, and Baptiste
Caramiaux
In Proceedings of the 15th Sound and Music Computing Conference (SMC 2018), Jul 2018
We present the first implementation of a new framework for sound and music computing, which allows humans to explore musical environments by communicating feedback to an artificial agent. It is based on an interactive reinforcement learning workflow, which enables agents to incrementally learn how to act on an environment by balancing exploitation of human feedback knowledge and exploration of new musical content. In a controlled experiment , participants successfully interacted with these agents to reach a sonic goal in two cases of different complexities. Subjective evaluations suggest that the exploration path taken by agents, rather than the fact of reaching a goal, may be critical to how agents are perceived as collaborative. We discuss such quantitative and qualitative results and identify future research directions toward deploying our "co-exploration" approach in real-world contexts.
2015
Adaptive Gesture Recognition with Variation Estimation for Interactive Systems
Baptiste
Caramiaux, Nicola
Montecchio, Atau
Tanaka, and
1 more author
ACM Transactions on Interactive Intelligent Systems, Jan 2015
This article presents a gesture recognition/adaptation system for human–computer interaction applications that goes beyond activity classification and that, as a complement to gesture labeling, characterizes the movement execution. We describe a template-based recognition method that simultaneously aligns the input gesture to the templates using a Sequential Monte Carlo inference technique. Contrary to standard template-based methods based on dynamic programming, such as Dynamic Time Warping, the algorithm has an adaptation process that tracks gesture variation in real time. The method continuously updates, during execution of the gesture, the estimated parameters and recognition results, which offers key advantages for continuous human–machine interaction. The technique is evaluated in several different ways: Recognition and early recognition are evaluated on 2D onscreen pen gestures; adaptation is assessed on synthetic data; and both early recognition and adaptation are evaluated in a user study involving 3D free-space gestures. The method is robust to noise, and successfully adapts to parameter variation. Moreover, it performs recognition as well as or better than nonadapting offline template-based methods.
Form follows sound: designing interactions from sonic memories
Baptiste
Caramiaux, Alessandro
Altavilla, Scott G.
Pobiner, and
1 more author
In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Jan 2015
Designing Accessible Interfaces to Enhance Collective Musical Engagement for and with Children with AutismJun 2026
Gestural Sound Toolkit: Reflections on an Interactive Design ProjectIn International Conference on New Interfaces for Musical Expression, Nov 2022
What Becomes of" Work" in AI" Artwork"?In CHI’22 Conference-Workshop on Outsourcing Artificial Intelligence: Responding to the Reassertion of the Human Element into Automation, Nov 2022
Marcelle: Composing Interactive Machine Learning Workflows and InterfacesIn The 34th Annual ACM Symposium on User Interface Software and Technology, Mar 2021
Exploring the role of artifacts in collective dance re-stagingProceedings of the ACM on Human-Computer Interaction, Mar 2021
How do people train a machine? Strategies and (Mis) UnderstandingsProceedings of the ACM on Human-Computer Interaction, Mar 2021
Prototyping Machine Learning Through Diffractive Art PracticeIn Designing Interactive Systems Conference 2021, Jan 2021
Learning metrics on spectrotemporal modulations reveals the perception of musical instrument timbreNature Human Behaviour, Jan 2021
Dissociable effects of practice variability on learning motor and timing skillsPloS one, Nov 2018
Form follows sound: designing interactions from sonic memoriesIn Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Jan 2015
