Ph.D Thesis 2024-2027 – Lifelong learning for mixed initiative musical co-creativity

Context

Current Human-Ai co-creative systems, also known as Mixed Initiative Co-Creative systems (MICC), have opened new possibilities for musicking [1] by facilitating novel creative processes and modes of interactions [2]. However, these systems lack the ability for long-term adaptation between user and machine. Current ways of adapting are either unilateral — the user progressively learning how to operate or play with the Ai system [3] — or self engineered — a musician-engineer modifying their own system over time [4]. The goal of this PhD project is to address this limitation by developing and evaluation Ai methods designed for lifelong learning in the scope of MICC musicking.

This PhD topic bases itself on the converging strength of three communities: Music Information Retrieval (MIR), Lifelong Learning and Personalisation in Long-Term Human Robot Interaction (LEAP-HRI) and Computational Creativity (CC).In the recent years, a lot of focus in music generation has been in the creation of autonomous neural network based systems [5], while other research focuses more on interactive systems and co-creativity for computer assisted composition [6], co-improvisation [7], or the integration of an artificial agent in an orchestra [8].In Human-Robot Interaction, the use of lifelong learning techniques [9] has been of interest to foster long-term, adaptable relationships between humans and Ai. Different strategies can be used, such as reinforcement learning [10] or curiosity-driven learning [11]. The artificial agents can also be complemented with methods for personalisation [12], allowing the Ai to tailor its behaviour and interactions to meet the specific needs and preferences of individual human users.The evaluation of creative systems is a complex issue but one of the cornerstone of the CC community which proposes different framework such as Rhodes’ Four Ps (Person, Process, Product, Press) [13], or Lin et al. [14] design space to explore and evaluate the different dynamics of co-creation.

Work Plan

The primary objective is to apply and extend state-of-the-art lifelong learning techniques for ML-based music generation models [14]. This application to music is novel and as such will pioneer the development of data representations and evaluation metrics that can successfully navigate the stability–plasticity dilemma (i.e., balancing how much new input should interfere with existing knowledge) within the context of musical creation. Two different strategies will be implemented and tested: supervised methods based on user feedback using reinforcement learning [10], and an unsupervised methods based on curiosity-driven learning [11]. The project will then focus on agent personalisation [6,12] allowing the Ai system to adapt based on user-specific characteristics and preferences. Areas of interest will include engagement, inclusivity, performance, and system maintenance. This research will start by focusing on single users and then be extended to allow a single agent to adapt to the needs and styles of multiple musicians simultaneously (for example, in a band setting). Finally, with sustainability in mind, this project will also investigate methods for adapting the system while using minimal computational resources and data.

The development of the models implemented in this project will be carried out in constant interaction with expert musicians, fully integrating these collaborations into the iterative process of designing the models and architectures. During these collaborations, the experimental sessions will be combined with filmed interviews and listening sessions, using ethnographical methods, in order to gather a wealth of feedback from the musicians and validate and refine the technological choices. In particular, musicians will be asked to play with a systems over several months to evaluate the evolution of the partnership they develop with the Ai. Deliverables will be publications in the MIR and ML communities, and open-source code releases.

Candidates profile

Candidates should have a Master’s degree in Computer Science, with experience in either machine learning, human-computer interaction, or computer music. A strong interest for music and a musical practice is highly recommended. We particularly welcome applications from women and under-represented groups in music and computer science research.

Work environment

The Algomus team is a friendly team of 10+ scientists and students. The team meets while sharing and talking on music and science. The Algomus team is used to publish at major conferences and journals in the field. The PhD student will be guided in writing and submitting papers to these conferences, whether on his/her own work or on collaborative work inside or outside the team. The PhD student will regularly participate at conferences and other events in the field, and will be encouraged to collaborate with other scientists and artists. Each PhD student in the Algomus team also undertakes an international research stay of 2-3 months during the course of his or her thesis.

References

  • [1] Small, C. (1998). Musicking: The meanings of performing and listening. Wesleyan University Press.
  • [2] Jordanous (2017). Co-creativity and perceptions of computational agents in co-creativity. International Conference on Computational Creativity.
  • [3] Nika et al. (2017). DYCI2 agents: merging the ‘free’, ‘reactive’ and ‘scenario-based’ music generation paradigms. International Computer Music Conference.
  • [4] Lewis (2021). Co-creation: Early steps and future prospects. Artisticiel/Cyber-Improvisations.
  • [5] Herremans et al. (2017). A functional taxonomy of music generation systems. ACM Computing Surveys, 50(5).
  • [6] Sturm, B. L., et al. (2016). Music transcription modelling and composition using deep learning. arXiv:1604.08723.
  • [7] Déguernel et al. (2018). Probabilistic factor oracles for multidimensional machine improvisation. Computer Music Journal, 43:2.
  • [8] Parmentier et al. (2021). A modular tool for automatic Soundpainting query recognition and music generation in Max/MSP. Sound and Music Computing.
  • [9] Parisi et al. (2019). Continual lifelong learning with neural networks: A review. Neural networks, 113.
  • [10] Scurto et al. (2021). Designing deep reinforcement learning for human parameter exploration. ACM Transactions on Computer-Human Interaction, 28(1).
  • [11] Colas et al. (2020). Language as a cognitive tool to imagine goals in curiosity driven exploration. NeurIPS.
  • [12] Irfan et al. (2022). Personalised socially assistive robot for cardiac rehabilitation: Critical reflections on long-term interactions in the real world. User Modeling and User-Adapted Interaction.
  • [13] Jordanous (2016). Four PPPPerspectives on computational creativity in theory and in practice. Connection Science, 28(2).
  • [14] Lin et al. (2023). Beyond prompts: Exploring the design space of Mixed-Initiative Co-Creativity Systems. International Conference on Computational Creativity.