Publications | Baptiste Caramiaux

2025

Generative AI and Creative Work: Narratives, Values, and Impacts

Baptiste Caramiaux, Kate Crawford, Q. Vera Liao, and 2 more authors

Feb 2025

arXiv:2502.03940 [cs]

Abs DOI PDF

Generative AI has gained a significant foothold in the creative and artistic sectors. In this context, the concept of creative work is influenced by discourses originating from technological stakeholders and mainstream media. The framing of narratives surrounding creativity and artistic production not only reflects a particular vision of culture but also actively contributes to shaping it. In this article, we review online media outlets and analyze the dominant narratives around AI’s impact on creative work that they convey. We found that the discourse promotes creativity freed from its material realisation through human labor. The separation of the idea from its material conditions is achieved by automation, which is the driving force behind productive efficiency assessed as the reduction of time taken to produce. And the withdrawal of the skills typically required in the execution of the creative process is seen as a means for democratising creativity. This discourse tends to correspond to the dominant techno-positivist vision and to assert power over the creative economy and culture.
Benchmarking XAI Explanations with Human-Aligned Evaluations

Rémi Kazmierczak, Steve Azzolin, Eloïse Berthier, and 9 more authors

Aug 2025

arXiv:2411.02470 [cs]

Abs DOI PDF

We introduce PASTA (Perceptual Assessment System for explanaTion of Artificial Intelligence), a novel human-centric framework for evaluating eXplainable AI (XAI) techniques in computer vision. Our first contribution is the creation of the PASTA-dataset, the first large-scale benchmark that spans a diverse set of models and both saliency-based and concept-based explanation methods. This dataset enables robust, comparative analysis of XAI techniques based on human judgment. Our second contribution is an automated, data-driven benchmark that predicts human preferences using the PASTA-dataset. This scoring called PASTA-score method offers scalable, reliable, and consistent evaluation aligned with human perception. Additionally, our benchmark allows for comparisons between explanations across different modalities, an aspect previously unaddressed. We then propose to apply our scoring method to probe the interpretability of existing models and to build more human interpretable XAI methods.
Computational Hermeneutics: Evaluating Generative AI as a Cultural Technology

Cody Kommers, Ruth Ahnert, Maria Antoniak, and 35 more authors

Aug 2025

Abs DOI PDF

\textlessdiv\textgreater \textlessdiv\textgreater \textlessdiv\textgreater \textlessp\textgreaterGenerative AI (GenAI) systems are increasingly recognized as cultural technologies, yet current evaluation frameworks often treat cul
Generative AI in Documentary Photography: Exploring Opportunities and Challenges for Visual Storytelling

Lenny Martinez, Baptiste Caramiaux, and Sarah Fdili Alaoui

In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, Apr 2025

Abs DOI PDF

Generative AI is increasingly used to create images from text, but its role in documentary photography remains under-explored. This paper investigates how generative AI can be integrated into documentary practice while maintaining ethical standards. Through interviews with six documentary photographers, we explored their views on AI’s potential to support community-driven storytelling. While AI presents opportunities for creative expression and community involvement, concerns about trust, authenticity, and decontextualization of images persist. Photographers expressed doubts about AI’s ability to accurately represent lived experiences, fearing it could compromise narrative integrity. Our findings suggest that AI tools should be designed to enhance collaboration and transparency in storytelling, complementing rather than replacing traditional documentary methods. This study contributes to the ongoing discourse on AI in photography, advocating for the development of tools that preserve the ethical foundations of documentary storytelling while empowering communities.

2024

Prototyping with Uncertainties: Data, Algorithms, and Research through Design

Elisa Giaccardi, Dave Murray-Rust, Johan Redström, and 1 more author

ACM Transactions on Computer-Human Interaction, Dec 2024

Publisher: Association for Computing Machinery (ACM)

Abs DOI PDF

Seen both as a resource and an obstacle to clarity, uncertainty is a concept that permeates many areas of design. As the concept gains prominence in Human-Computer Interaction (HCI), this special issue specifically explores the interplay between uncertainty and prototyping in Research through Design (RtD). We first outline three histories of uncertainty in design, in relation to its philosophical significance, its role in statistical and algorithmic processes, and its importance in prototyping. The convergence of these aspects is crucial as design evolves toward more agentive and entangled systems, introducing challenges such as Design as a Probabilistic Outcome. We then investigate the design spaces for engaging with “being uncertain” that emerge from the papers: from nuancing the relationship between designers and quantitative data to blurring the line between humans, fungi, and algorithms. Finally, we illuminate some preliminary threads for how RtD can navigate and engage with these shifting technological and design landscapes thoughtfully.
Studying Collaborative Interactive Machine Teaching in Image Classification

Behnoosh Mohammadzadeh, Jules Françoise, Michèle Gouiffès, and 1 more author

In Proceedings of the 29th International Conference on Intelligent User Interfaces, Apr 2024

Abs DOI PDF

While human-centered approaches to machine learning explore various human roles within the interaction loop, the notion of Interactive Machine Teaching (IMT) emerged with a focus on leveraging the teaching skills of humans as a teacher to build machine learning systems. However, most systems and studies are devoted to single users. In this article, we study collaborative interactive machine teaching in the context of image classification to analyze how people can structure the teaching process collectively and to understand their experience. Our contributions are threefold. First, we developed a web application called TeachTOK that enables groups of users to curate data and train a model together incrementally. Second, we conducted a study in which ten participants were divided into three teams that competed to build an image classifier in nine days. Qualitative results of participants’ discussions in focus groups reveal the emergence of collaboration patterns in the machine teaching task, how collaboration helps revise teaching strategies and participants’ reflections on their interaction with the TeachTOK application. From these findings we provide implications for the design of more interactive, collaborative and participatory machine learning-based systems.
Comparing Teaching Strategies of a Machine Learning-based Prosthetic Arm

Vaynee Sungeelee, Nathanaël Jarrassé, Téo Sanchez, and 1 more author

In Proceedings of the 29th International Conference on Intelligent User Interfaces, Apr 2024

Abs DOI PDF

Pattern-recognition-based arm prostheses rely on recognizing muscle activation to trigger movements. The effectiveness of this approach depends not only on the performance of the machine learner but also on the user’s understanding of its recognition capabilities, allowing them to adapt and work around recognition failures. We investigate how different model training strategies to select gesture classes and record respective muscle contractions impact model accuracy and user comprehension. We report on a lab experiment where participants performed hand gestures to train a classifier under three conditions: (1) the system cues gesture classes randomly (control), (2) the user selects gesture classes (teacher-led), (3) the system queries gesture classes based on their separability (learner-led). After training, we compare the models’ accuracy and test participants’ predictive understanding of the prosthesis’ behavior. We found that teacher-led and learner-led strategies yield faster and greater performance increases, respectively. Combining two evaluation methods, we found that participants developed a more accurate mental model when the system queried the least separable gesture class (learner-led). Our results conclude that, in the context of machine learning-based myoelectric prosthesis control, guiding the user to focus on class separability during training can improve recognition performances and support users’ mental models about the system’s behavior. We discuss our results in light of several research fields : myoelectric prosthesis control, motor learning, human-robot interaction, and interactive machine teaching.
Interactive curriculum learning increases and homogenizes motor smoothness

Vaynee Sungeelee, Antoine Loriette, Olivier Sigaud, and 1 more author

Scientific Reports, Feb 2024

Publisher: Nature Publishing Group

Abs DOI PDF

One of the challenges of technology-assisted motor learning is how to adapt practice to facilitate learning. Random practice has been shown to promote long-term learning. However, it does not adapt to the learner’s specific learning requirements. Previous attempts to adapt learning considered the skill level of learners from past training sessions. This study investigates the effects of personalizing practice in real time, through a curriculum learning approach, where a curriculum of tasks is built by considering consecutive performance differences for each task. 12 participants were allocated to each of three training conditions in an experiment which required performing a steering task to drive a cursor in an arc channel. The curriculum learning approach was compared to two other conditions: random practice and another adaptive practice, which does not consider the learning evolution. The curriculum learning practice outperformed the random practice in effectively increasing movement smoothness at post-test and outperformed both the random practice and the adaptive practice on transfer tests. The adaptation of practice through the curriculum learning approach also made learners’ skills more uniform. Based on these findings, we anticipate that future research will explore the use of curriculum learning in interactive training tools to support motor skill learning, such as rehabilitation.
Trust in AI-assisted Decision Making: Perspectives from Those Behind the System and Those for Whom the Decision is Made

Oleksandra Vereschak, Fatemeh Alizadeh, Gilles Bailly, and 1 more author

In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, May 2024

Abs DOI PDF

Trust between humans and AI in the context of decision-making has acquired an important role in public policy, research and industry. In this context, Human-AI Trust has often been tackled from the lens of cognitive science and psychology, but lacks insights from the stakeholders involved. In this paper, we conducted semi-structured interviews with 7 AI practitioners and 7 decision subjects from various decision domains. We found that 1) interviewees identified the prerequisites for the existence of trust and distinguish trust from trustworthiness, reliance, and compliance; 2) trust in AI-integrated systems is strongly influenced by other human actors, more than the system’s features; 3) the role of Human-AI trust factors is stakeholder-dependent. These results provide clues for the design of Human-AI interactions in which trust plays a major role, as well as outline new research directions in Human-AI Trust.

2023

AI with Museums and Cultural Heritage

Baptiste Caramiaux

In AI in Museums, Dec 2023

Abs DOI PDF

This paper discusses AI in the context of cultural heritage. First, I contextualize what we call AI, particularly with respect to infrastructure. With this representation in mind, my first objective is to outline the opportunities that AI can bring to these sec- tors, as identified in a series of reports and white papers edited by European institu- tions. In these reports, we have, however, barely grasped the need for stakeholders in these sectors to have their say on how they see this technology and how it should be integrated into their practice and organizations. My second objective is thus to highlight the fact that AI is not just a source of opportunities, as this would obscu- re the sociocultural and sociotechnical implications of integrating AI into existing practices.
Culture and Politics of Machine Learning in NIME: A Preliminary Qualitative Inquiry

Théo Jourdan and Baptiste Caramiaux

In New Interfaces for Musical Expression (NIME), May 2023

Abs PDF

For several years, the various practices around ML techniques have been increasingly present and diversified. However, the literature associated with these techniques rarely reveals the cultural and political sides of these practices. In order to explore how practitioners in the NIME community engage with ML techniques, we conducted interviews with seven researchers in the NIME community and analysed them through a thematic analysis. Firstly, we propose findings at the level of the individual, resisting technological determinism and redefining sense making in interactive ML. Secondly, we propose findings at the level of the community, revealing mitigated adoption with respect to ML. This paper aims to provide the community with some reflections on the use of ML in order to initiate a discussion about cultural, political and ethical issues surrounding these techniques as their use grows within the community.
Machine Learning for Musical Expression: A Systematic Literature Review

Théo Jourdan and Baptiste Caramiaux

In New Interfaces for Musical Expression (NIME), May 2023

Abs PDF

For several decades NIME community has always been appropriating machine learning (ML) to apply for various tasks such as gesture-sound mapping or sound synthesis for digital musical instruments. Recently, the use of ML methods seems to have increased and the objectives have diversified. Despite its increasing use, few contributions have studied what constitutes the culture of learning technologies for this specific practice. This paper presents an analysis of 69 contributions selected from a systematic review of the NIME conference over the last 10 years. This paper aims at analysing the practices involving ML in terms of the techniques and the task used and the ways to interact this technology. It thus contributes to a deeper understanding of the specific goals and motivation in using ML for musical expression. This study allows us to propose new perspectives in the practice of these techniques.
Describing movement learning using metric learning

Antoine Loriette, Wanyu Liu, Frédéric Bevilacqua, and 1 more author

PLOS ONE, Feb 2023

Publisher: Public Library of Science

Abs DOI PDF

Analysing movement learning can rely on human evaluation, e.g. annotating video recordings, or on computing means in applying metrics on behavioural data. However, it remains challenging to relate human perception of movement similarity to computational measures that aim at modelling such similarity. In this paper, we propose a metric learning method bridging the gap between human ratings of movement similarity in a motor learning task and computational metric evaluation on the same task. It applies metric learning on a Dynamic Time Warping algorithm to derive an optimal set of movement features that best explain human ratings. We evaluated this method on an existing movement dataset, which comprises videos of participants practising a complex gesture sequence toward a target template, as well as the collected data that describes the movements. We show that it is possible to establish a linear relationship between human ratings and our learned computational metric. This learned metric can be used to describe the most salient temporal moments implicitly used by annotators, as well as movement parameters that correlate with motor improvements in the dataset. We conclude with possibilities to generalise this method for designing computational tools dedicated to movement annotation and evaluation of skill learning.
Effect of sonification types in upper-limb movement: a quantitative and qualitative study in hemiparetic and healthy participants

Iseline Peyre, Agnès Roby-Brami, Maël Segalen, and 5 more authors

Journal of NeuroEngineering and Rehabilitation, Oct 2023

Abs DOI PDF

Movement sonification, the use of real-time auditory feedback linked to movement parameters, have been proposed to support rehabilitation. Nevertheless, if promising results have been reported, the effect of the type of sound used has not been studied systematically. The aim of this study was to investigate in a single session the effect of different types of sonification both quantitatively and qualitatively on patients with acquired brain lesions and healthy participants.
Interaction Knowledge: Understanding the ‘Mechanics’ of Digital Tools

Miguel A. Renom, Baptiste Caramiaux, and Michel Beaudouin-Lafon

In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, Apr 2023

Abs DOI PDF

User interfaces typically feature tools to act on objects and rely on the ability of users to discover or learn how to interact with them. Previous work in HCI has used the Theory of Affordances to explain how users understand the possibilities for action in digital environments. A complementary theory from cognitive neuroscience, Technical Reasoning, posits that users accumulate abstract knowledge of object properties and technical principles known as mechanical knowledge, essential in tool use. Drawing from this theory, we introduce interaction knowledge as the “mechanical” knowledge of digital environments. We provide evidence of its relevance by reporting on an experiment where participants performed tasks in a digital environment with ambiguous possibilities for interaction. We analyze how interaction knowledge was transferred across two digital domains, text editing and graphical editing, and conclude that interaction knowledge models an essential type of knowledge for interacting in the digital world.
Probing Respiratory Care With Generative Deep Learning

Hugo Scurto, Thomas Similowski, Samuel Bianchini, and 1 more author

Proc. ACM Hum.-Comput. Interact., Oct 2023

Place: New York, NY, USA Publisher: Association for Computing Machinery

Abs DOI PDF

This paper combines design, machine learning and social computing to explore generative deep learning as both tool and probe for respiratory care. We first present GANspire, a deep learning tool that generates fine-grained breathing waveforms, which we crafted in collaboration with one respiratory physician, attending to joint materialities of human breathing data and deep generative models. We then relate a probe, produced with breathing waveforms generated with GANspire, and led with a group of ten respiratory care experts, responding to its material attributes. Qualitative annotations showed that respiratory care experts interpreted both realistic and ambiguous attributes of breathing waveforms generated with GANspire, according to subjective aspects of physiology, activity and emotion. Semi-structured interviews also revealed experts’ broader perceptions, expectations and ethical concerns on AI technology, based on their clinical practice of respiratory care, and reflexive analysis of GANspire. These findings suggest design implications for technological aids in respiratory care, and show how ambiguity of deep generative models can be leveraged as a resource for qualitative inquiry, enabling socio-material research with generative deep learning. Our paper contributes to the CSCW community by broadening how generative deep learning may be approached not only as a tool to design human-computer interactions, but also as a probe to provoke open conversations with communities of practice about their current and speculative uses of AI technology.

2022

"Explorers of Unknown Planets": Practices and Politics of Artificial Intelligence in Visual Arts

Baptiste Caramiaux and Sarah Fdili Alaoui

Proceedings of the ACM on Human-Computer Interaction, Nov 2022

Abs DOI PDF

Alongside recent advances in artificial intelligence (AI), a new art practice has emerged in recent years that borrows and transforms these advances in the production of artworks. The actors of this emergent practice are coming from contemporary art, media and digital arts. These artists have developed an original practice of AI within their creative field. In this article, we propose a qualitative study to explore the nature of this practice. We interviewed five internationally renowned artists about how AI is integrated into their work. Through a thematic analysis of the interviews, we first find that their practice relies on crafting algorithms and data as materials. We uncover how they explicitly use this material unpredictability rather than avoid it. Secondly, we highlight the politics of their practice that consist of resisting the culture of AI research, as well as its inherent power dynamics. We also highlight how their relationship with the technology is imbued with ethics and how they rethink their role with respect to the technology. In this paper, we aim to provide the CSCW community with a way to expand the framework in which AI can be understood not only as a tool but also as cultural and political design material.
Gestural Sound Toolkit: Reflections on an Interactive Design Project

Baptiste Caramiaux, Alessandro Altavilla, Jules Françoise, and 1 more author

In International Conference on New Interfaces for Musical Expression, Nov 2022

PDF
What Becomes of" Work" in AI" Artwork"?

Baptiste Caramiaux, Sarah Fdili Alaoui, and Stacy Hsueh

In CHI’22 Conference-Workshop on Outsourcing Artificial Intelligence: Responding to the Reassertion of the Human Element into Automation, Nov 2022

PDF
Exploring Technical Reasoning in Digital Tool Use

Miguel A. Renom, Baptiste Caramiaux, and Michel Beaudouin-Lafon

In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, Apr 2022

Abs DOI PDF

The Technical Reasoning hypothesis in cognitive neuroscience posits that humans engage in physical tool use by reasoning about mechanical interactions among objects. By modeling the use of objects as tools based on their abstract properties, this theory explains how tools can be re-purposed beyond their assigned function. This paper assesses the relevance of Technical Reasoning to digital tool use. We conducted an experiment with 16 participants that forced them to re-purpose commands to complete a text layout task. We analyzed self-reported scores of creative personality and experience with text editing, and found a significant association between re-purposing performance and creativity, but not with experience. Our results suggest that while most participants engaged in Technical Reasoning to re-purpose digital tools, some experienced “functional fixedness.” This work contributes Technical Reasoning as a theoretical model for the design of digital tools.
Deep Learning Uncertainty in Machine Teaching

Téo Sanchez, Baptiste Caramiaux, Pierre Thiel, and 1 more author

In Proceedings of the 27th International Conference on Intelligent User Interfaces, Mar 2022

Abs DOI PDF

Machine Learning models can output confident but incorrect predictions. To address this problem, ML researchers use various techniques to reliably estimate ML uncertainty, usually performed on controlled benchmarks once the model has been trained. We explore how the two types of uncertainty—aleatoric and epistemic—can help non-expert users understand the strengths and weaknesses of a classifier in an interactive setting. We are interested in users’ perception of the difference between aleatoric and epistemic uncertainty and their use to teach and understand the classifier. We conducted an experiment where non-experts train a classifier to recognize card images, and are tested on their ability to predict classifier outcomes. Participants who used either larger or more varied training sets significantly improved their understanding of uncertainty, both epistemic or aleatoric. However, participants who relied on the uncertainty measure to guide their choice of training data did not significantly improve classifier training, nor were they better able to guess the classifier outcome. We identified three specific situations where participants successfully identified the difference between aleatoric and epistemic uncertainty: placing a card in the exact same position as a training card; placing different cards next to each other; and placing a non-card, such as their hand, next to or on top of a card. We discuss our methodology for estimating uncertainty for Interactive Machine Learning systems and question the need for two-level uncertainty in Machine Teaching.

2021

Marcelle: Composing Interactive Machine Learning Workflows and Interfaces

Jules Françoise, Baptiste Caramiaux, and Téo Sanchez

In The 34th Annual ACM Symposium on User Interface Software and Technology, Mar 2021

PDF
Artificial Intelligence in Music and Performance: A Subjective Art-Research Inquiry

Baptiste Caramiaux and Marco Donnarumma

In Handbook of Artificial Intelligence for Music, Mar 2021

DOI PDF
Exploring the role of artifacts in collective dance re-staging

Jean-Philippe Rivière, Sarah Fdili Alaoui, Baptiste Caramiaux, and 1 more author

Proceedings of the ACM on Human-Computer Interaction, Mar 2021

Publisher: ACM New York, NY, USA

PDF
How do people train a machine? Strategies and (Mis) Understandings

Téo Sanchez, Baptiste Caramiaux, Jules Françoise, and 2 more authors

Proceedings of the ACM on Human-Computer Interaction, Mar 2021

Publisher: ACM New York, NY, USA

PDF
Designing Deep Reinforcement Learning for Human Parameter Exploration

Hugo Scurto, Bavo Van Kerrebroeck, Baptiste Caramiaux, and 1 more author

ACM Transactions on Computer-Human Interaction, Jan 2021

Abs DOI PDF

Software tools for generating digital sound often present users with high-dimensional, parametric interfaces, that may not facilitate exploration of diverse sound designs. In this article, we propose to investigate artificial agents using deep reinforcement learning to explore parameter spaces in partnership with users for sound design. We describe a series of user-centred studies to probe the creative benefits of these agents and adapting their design to exploration. Preliminary studies observing users’ exploration strategies with parametric interfaces and testing different agent exploration behaviours led to the design of a fully-functioning prototype, called Co-Explorer, that we evaluated in a workshop with professional sound designers. We found that the Co-Explorer enables a novel creative workflow centred on human–machine partnership, which has been positively received by practitioners. We also highlight varied user exploration behaviours throughout partnering with our system. Finally, we frame design guidelines for enabling such co-exploration workflow in creative digital applications.
Prototyping Machine Learning Through Diffractive Art Practice

Hugo Scurto, Baptiste Caramiaux, and Frédéric Bevilacqua

In Designing Interactive Systems Conference 2021, Jan 2021

PDF
Learning metrics on spectrotemporal modulations reveals the perception of musical instrument timbre

Etienne Thoret, Baptiste Caramiaux, Philippe Depalle, and 1 more author

Nature Human Behaviour, Jan 2021

Publisher: Nature Publishing Group

PDF
How to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies

Oleksandra Vereschak, Gilles Bailly, and Baptiste Caramiaux

Proceedings of the ACM on Human-Computer Interaction, Oct 2021

Abs DOI PDF

The spread of AI-embedded systems involved in human decision making makes studying human trust in these systems critical. However, empirically investigating trust is challenging. One reason is the lack of standard protocols to design trust experiments. In this paper, we present a survey of existing methods to empirically investigate trust in AI-assisted decision making and analyse the corpus along the constitutive elements of an experimental protocol. We find that the definition of trust is not commonly integrated in experimental protocols, which can lead to findings that are overclaimed or are hard to interpret and compare across studies. Drawing from empirical practices in social and cognitive studies on human-human trust, we provide practical guidelines to improve the methodology of studying Human-AI trust in decision-making contexts. In addition, we bring forward research opportunities of two types: one focusing on further investigation regarding trust methodologies and the other on factors that impact Human-AI trust.

2020

Machine Learning Approaches for Motor Learning: A Short Review

Baptiste Caramiaux, Jules Françoise, Wanyu Liu, and 2 more authors

Frontiers in Computer Science, May 2020

Publisher: Frontiers

Abs DOI PDF

Machine learning approaches have seen considerable applications in human movement modeling, but remain limited for motor learning. Motor learning requires accounting for motor variability, and poses new challenges as the algorithms need to be able to differentiate between new movements and variation of known ones. In this short review, we outline existing machine learning models for motor learning and their adaptation capabilities. We identify and describe three types of adaptation: Parameter adaptation in probabilistic models, Transfer and meta-learning in deep neural networks, and Planning adaptation by reinforcement learning. To conclude, we discuss challenges for applying these models in the domain of motor learning support systems.
Making Mappings: Examining the Design Process

Travis J West, Baptiste Caramiaux, and Marcelo M Wanderley

In New Interfaces for Musical Expression (NIME’20), Jun 2020

Abs PDF

We conducted a study which examines mappings from a relatively unexplored perspective: how they are made. Twelve skilled NIME users designed a mapping from a T-Stick to a subtractive synthesizer, and were interviewed about their approach to mapping design. We present a thematic analysis of the interviews, with reference to data recordings captured while the designers worked. Our results suggest that the mapping design process is an iterative process that alternates between two working modes: diffuse exploration and directed experimentation.

2019

AI in the media and creative industries

Baptiste Caramiaux, Fabien Lotte, Joost Geurts, and 16 more authors

Apr 2019

PDF
Capturing Movement Decomposition to Support Learning and Teaching in Contemporary Dance

Jean-Philippe Rivière, Sarah Fdili Alaoui, Baptiste Caramiaux, and 1 more author

Proc. ACM Hum.-Comput. Interact., Nov 2019

Abs DOI PDF

Our goal is to understand how dancers learn complex dance phrases. We ran three workshops where dancers learned dance fragments from videos. In workshop 1, we analyzed how dancers structure their learning strategies by decomposing movements. In workshop 2, we introduced MoveOn, a technology probe that lets dancers decompose video into short, repeatable clips to support their learning. This served as an effective analysis tool for identifying the changes in focus and understanding their decomposition and recomposition processes. In workshop 3, we compared the teacher’s and dancers’ decomposition strategies, and how dancers learn on their own compared to teacher-created decompositions. We found that they all ungroup and regroup dance fragments, but with different foci of attention, which suggests that teacher-imposed decomposition is more effective for introductory dance students, whereas personal decomposition is more suitable for expert dancers. We discuss the implications for designing technology to support analysis, learning and teaching of dance through movement decomposition.

2018

Dissociable effects of practice variability on learning motor and timing skills

Baptiste Caramiaux, Frédéric Bevilacqua, Marcelo M. Wanderley, and 1 more author

PloS one, Nov 2018

Publisher: Public Library of Science San Francisco, CA USA

PDF
The Machine Learning Algorithm as Creative Musical Tool

Rebecca A. Fiebrink and Baptiste Caramiaux

In The Oxford Handbook of Algorithmic Music, Feb 2018

Abs DOI PDF

Machine learning is the capacity of a computational system to learn structure from data in order to make predictions on new data. This chapter draws on music, machine learning, and human-computer interaction to elucidate an understanding of machine learning algorithms as creative tools for music and the sonic arts. It motivates a new understanding of learning algorithms as human-computer interfaces: like other interfaces, learning algorithms can be characterized by the ways their affordances intersect with goals of human users. The chapter also argues that the nature of interaction between users and algorithms impacts the usability and usefulness of those algorithms in profound ways. This human-centred view of machine learning motivates a concluding discussion of what it means to employ machine learning as a creative tool.
Embracing First-Person Perspectives in Soma-Based Design

Kristina Höök, Baptiste Caramiaux, Cumhur Erkut, and 19 more authors

Informatics, Mar 2018

Publisher: Multidisciplinary Digital Publishing Institute

Abs DOI PDF

A set of prominent designers embarked on a research journey to explore aesthetics in movement-based design. Here we unpack one of the design sensitivities unique to our practice: a strong first person perspective—where the movements, somatics and aesthetic sensibilities of the designer, design researcher and user are at the forefront. We present an annotated portfolio of design exemplars and a brief introduction to some of the design methods and theory we use, together substantiating and explaining the first-person perspective. At the same time, we show how this felt dimension, despite its subjective nature, is what provides rigor and structure to our design research. Our aim is to assist researchers in soma-based design and designers wanting to consider the multiple facets when designing for the aesthetics of movement. The applications span a large field of designs, including slow introspective, contemplative interactions, arts, dance, health applications, games, work applications and many others.
Perceiving Agent Collaborative Sonic Exploration In Interactive Reinforcement Learning

Hugo Scurto, Frédéric Bevilacqua, and Baptiste Caramiaux

In Proceedings of the 15th Sound and Music Computing Conference (SMC 2018), Jul 2018

Abs PDF

We present the first implementation of a new framework for sound and music computing, which allows humans to explore musical environments by communicating feedback to an artificial agent. It is based on an interactive reinforcement learning workflow, which enables agents to incrementally learn how to act on an environment by balancing exploitation of human feedback knowledge and exploration of new musical content. In a controlled experiment , participants successfully interacted with these agents to reach a sonic goal in two cases of different complexities. Subjective evaluations suggest that the exploration path taken by agents, rather than the fact of reaching a goal, may be critical to how agents are perceived as collaborative. We discuss such quantitative and qualitative results and identify future research directions toward deploying our "co-exploration" approach in real-world contexts.

2015

Adaptive Gesture Recognition with Variation Estimation for Interactive Systems

Baptiste Caramiaux, Nicola Montecchio, Atau Tanaka, and 1 more author

ACM Transactions on Interactive Intelligent Systems, Jan 2015

Abs DOI PDF

This article presents a gesture recognition/adaptation system for human–computer interaction applications that goes beyond activity classification and that, as a complement to gesture labeling, characterizes the movement execution. We describe a template-based recognition method that simultaneously aligns the input gesture to the templates using a Sequential Monte Carlo inference technique. Contrary to standard template-based methods based on dynamic programming, such as Dynamic Time Warping, the algorithm has an adaptation process that tracks gesture variation in real time. The method continuously updates, during execution of the gesture, the estimated parameters and recognition results, which offers key advantages for continuous human–machine interaction. The technique is evaluated in several different ways: Recognition and early recognition are evaluated on 2D onscreen pen gestures; adaptation is assessed on synthetic data; and both early recognition and adaptation are evaluated in a user study involving 3D free-space gestures. The method is robust to noise, and successfully adapts to parameter variation. Moreover, it performs recognition as well as or better than nonadapting offline template-based methods.
Form follows sound: designing interactions from sonic memories

Baptiste Caramiaux, Alessandro Altavilla, Scott G. Pobiner, and 1 more author

In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Jan 2015

PDF

2014

Mapping through listening

Baptiste Caramiaux, Jules Françoise, Norbert Schnell, and 1 more author

Computer Music Journal, Jan 2014

Publisher: MIT Press

PDF