Marcelle is an open-source framework designed to make it easy to build custom, interactive interfaces for Machine Learning (ML) pipelines. With Marcelle, developers and researchers can rapidly prototype and deploy ML workflows that can be integrated into any web application.
Quick description
Originally created as a tool for teaching Interactive Machine Learning (IML) and Human–AI Interaction (HAI), Marcelle has evolved into a core framework supporting our research at the intersection of Human–Computer Interaction (HCI) and Artificial Intelligence (AI).
For example, using Marcelle, one can easily build an interactive web interface for developing a teachable image recognition system that learns directly from webcam input. Such interfaces allow users to collect data, train models in real time, visualize results, and experiment with AI behaviors within the browser.
Marcelle also allows to connect easily with external computation backends. It can interface with Python-based scripts (that use PyTorch, Tensorflow or Scikitlearn) through Ray or FeathersJS backend. This enables an integration with existing data processing pipelines, training scripts, or real-time inference services. In this case Marcelle can be used to design interactions and interfaces with these services.
While human-centered approaches to machine learning explore various human roles within the interaction loop, the notion of Interactive Machine Teaching (IMT) emerged with a focus on leveraging the teaching skills of humans as a teacher to build machine learning systems. However, most systems and studies are devoted to single users. In this article, we study collaborative interactive machine teaching in the context of image classification to analyze how people can structure the teaching process collectively and to understand their experience. Our contributions are threefold. First, we developed a web application called TeachTOK that enables groups of users to curate data and train a model together incrementally. Second, we conducted a study in which ten participants were divided into three teams that competed to build an image classifier in nine days. Qualitative results of participants’ discussions in focus groups reveal the emergence of collaboration patterns in the machine teaching task, how collaboration helps revise teaching strategies and participants’ reflections on their interaction with the TeachTOK application. From these findings we provide implications for the design of more interactive, collaborative and participatory machine learning-based systems.
2022
Deep Learning Uncertainty in Machine Teaching
Téo
Sanchez, Baptiste
Caramiaux, Pierre
Thiel, and
1 more author
In Proceedings of the 27th International Conference on Intelligent User Interfaces, Mar 2022
Machine Learning models can output confident but incorrect predictions. To address this problem, ML researchers use various techniques to reliably estimate ML uncertainty, usually performed on controlled benchmarks once the model has been trained. We explore how the two types of uncertainty—aleatoric and epistemic—can help non-expert users understand the strengths and weaknesses of a classifier in an interactive setting. We are interested in users’ perception of the difference between aleatoric and epistemic uncertainty and their use to teach and understand the classifier. We conducted an experiment where non-experts train a classifier to recognize card images, and are tested on their ability to predict classifier outcomes. Participants who used either larger or more varied training sets significantly improved their understanding of uncertainty, both epistemic or aleatoric. However, participants who relied on the uncertainty measure to guide their choice of training data did not significantly improve classifier training, nor were they better able to guess the classifier outcome. We identified three specific situations where participants successfully identified the difference between aleatoric and epistemic uncertainty: placing a card in the exact same position as a training card; placing different cards next to each other; and placing a non-card, such as their hand, next to or on top of a card. We discuss our methodology for estimating uncertainty for Interactive Machine Learning systems and question the need for two-level uncertainty in Machine Teaching.
2021
Marcelle: Composing Interactive Machine Learning Workflows and Interfaces
Jules
Françoise, Baptiste
Caramiaux, and Téo
Sanchez
In The 34th Annual ACM Symposium on User Interface Software and Technology, Mar 2021
Marcelle: Composing Interactive Machine Learning Workflows and InterfacesIn The 34th Annual ACM Symposium on User Interface Software and Technology, Mar 2021
How do people train a machine? Strategies and (Mis) UnderstandingsProceedings of the ACM on Human-Computer Interaction, Mar 2021Publisher: ACM New York, NY, USA