Intro to Machine Learning for Interactive Arts

class: center, middle
.title[Creative Coding and Software Design 3]
<br/><br/>
.subtitle[Intro to Machine Learning for Interactive Arts]
<br/><br/><br/><br/><br/><br/>
.date[Oct 2024] 
<br/><br/><br/>
.note[Created with [Liminal](https://github.com/jonathanlilly/liminal) using [Remark.js](http://remarkjs.com/) + [Markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) +  [KaTeX](https://katex.org)]

???

Author: Grigore Burloiu, UNATC
    
---
name: toc
class: left
# ★ Table of Contents ★      
      
1. [Semester overview](#semester-overview)
2. [Machine Learning](#machine-learning)
3. [What we cover](#what-we-cover)
4. [Interactive Machine Learning](#interactive-machine-learning)
5. [Links](#links)

---
layout: true  .toc[[★](#toc)]
        
---
name: semester-overview  
class: left
# Semester overview

- 2h lab: new concepts in practice
- 1h lecture: theory, discussion

[Syllabus](https://itpma.notion.site/itpma/Creative-Coding-and-Software-Design-3-c18bb7e2da834cabb027a681abefec2c)

[Classroom](https://classroom.google.com/)

---
name: machine-learning
class: left
# Machine Learning

*learn* structure / functionality **from data**

AI?

terminology :(

---
class: center

---
## ML modern history

since the 2010s: AI ~ ML ~ (deep) artificial neural networks

- historically not the case!
- 2012: [deep NNs win ImageNet competition](https://en.wikipedia.org/wiki/AlexNet)
- 2013: [word2vec maps word associations](https://en.wikipedia.org/wiki/Word2vec)
- 2015-17: [superhuman performance in the game of Go](https://en.wikipedia.org/wiki/AlphaGo)

"we should have been using neural networks all along!"

WRONG! -[Tomas Mikolov](https://cs.nyu.edu/~welleck/episode25.html)

---

## ML for art

2009: [Wekinator](http://www.wekinator.org/)

- (from [fablab](https://fablab.ruc.dk/hand-gesture-recognition-using-handpose-osc-and-wekinator/))

---

## DL for art

2015: [DeepDream](https://www.tensorflow.org/tutorials/generative/deepdream)

[ML art](https://www.libreai.com/a-short-overview-on-ai-art/) is:
- a subset of *generative* art
- sometimes *interactive* (in training and/or execution)

[ethical questions](https://www.youtube.com/watch?v=NMYTALX9kEw)

---

## ML / DL myths

circa 2018

| **myth** (*you don't need*) |   | **truth** (*you can*) |
|-------------------|-|-------|
| expensive computers |   | use machines in the cloud for free |
| math and coding |   | do a lot with user-friendly tools |
| lots of data |   | start from pre-trained models |
| lots of time |   | do inference in (almost) real time |

- (adapted from [course.fast.ai](https://course.fast.ai/#Is-this-course-for-me?))

---

## Getting into ML: two approaches

classic: bottom-up
- [A. Ng's MOOC](https://www.coursera.org/learn/machine-learning), classic textbooks
- (can lead to [burnout](https://www.reddit.com/r/MachineLearning/comments/73n9pm/d_confession_as_an_ai_researcher_seeking_advice/))

maker: top-down
- [R. Fiebrink’s MOOC](https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists-v/) / [mimic](https://mimicproject.com/guides/kadenze)
- [fast.ai](https://course.fast.ai/) / [fastbook](https://github.com/fastai/fastbook)
- [ml4a](https://ml4a.github.io/classes/itp-F18/), [ml4web](https://github.com/yining1023/machine-learning-for-the-web), [dl4music](http://www.jordipons.me/apps/teaching-materials/) …

---
name: what-we-cover
# What we cover

| **interactive ML** |   | **Deep Learning** |
|-------------------|-|-------|
| small models (few layers) |   | big models (many layers) |
| fast training (local CPU) |   | slow training (GPU / cloud) |
| analysis, action |   | analysis, generation**  |

DL can also be part of an interactive system:
- almost* real-time inference 
- *transfer learning*

- *2022 update: faster-than-realtime inference [is](https://github.com/acids-ircam/nn_tilde) [common](https://github.com/ggerganov/whisper.cpp)

**GenAI, potential vs dangers - [Paul Trillo at ArsE'24](https://youtu.be/kKM0F33CCkc?t=4853)

---
name: interactive-machine-learning
# Interactive Machine Learning

human-in-the-loop (training and inference)

[Wekinator](https://twitter.com/search?q=wekinator), [InteractML](https://interactml.com/) (Rebecca Fiebrink, since 2008!)

[Wolf3D](https://twitter.com/stoj_io/status/840222647489318914) sound to action

[Poetry in Motion](https://rednoise.org/rita/gallery/PoetryInMotion/): movement to text generation

---

## Why use iML?

mapping / control design
- by example / modelling     (vs fixed rules)

dimensions
- 1 : 1                 1 : many                 many : 1                many : many

workflow
- experimental, iterative

creating something truly *new*?

iML + GenAI: learning interaction
- [DeepMind Genie](https://sites.google.com/view/genie-2024/): Generative Interactive Environments

- <iframe width="190" height="80" src="https://www.youtube.com/embed/b3REaKYEW3k" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

- [MLST interview](https://www.youtube.com/watch?v=kbt0ZFoI2Hc)

---
name: links       
class: left
# Links

[Wekinator](http://www.wekinator.org/) 
- [FluCoMa](https://flucoma.org/)
- [ml.star](https://www.benjamindaysmith.com/#/ml-machine-learning-toolkit-in-max/)
- [MuBu for Max](https://forum.ircam.fr/projects/detail/mubu/)
- [IIL tools](https://iil.is/outputs#open-source)

- [fastai](https://docs.fast.ai/) / [PyTorch](https://pytorch.org/)
- [ml5](https://ml5js.org/) / [Tensorflow.js](https://teachablemachine.withgoogle.com/)
- [Runway](https://runwayml.com/)

[Google](https://github.com/dvschultz/ml-art-colabs) [Colab](https://ljvmiranda921.github.io/notebook/2021/08/11/vqgan-list/)

See more [resources](../resources#machine-learning).

---
## Follow

Twitter: [Andreas Refsgaard](https://twitter.com/AndreasRef), [Max Woolf](https://twitter.com/minimaxir), [vadim epstein](https://twitter.com/eps696), [Adverb](https://twitter.com/advadnoun), [Emily Short](https://twitter.com/emshort), [Chris Donahue](https://twitter.com/chrisdonahuey), [AK](https://twitter.com/ak92501), [Janelle Shane](https://twitter.com/JanelleCShane), [Rebecca Fiebrink](https://twitter.com/RebeccaFiebrink), [Parag K. Mital](https://twitter.com/pkmital), [Jesse Engel](https://twitter.com/jesseengel), [dadabots](https://twitter.com/dadabots), [Kyle McDonald](https://twitter.com/kcimc), [Memo Akten](https://twitter.com/memotv)...

Lectures/MOOCs: [Rebecca Fiebrink](https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists/info), [Gene Kogan](https://ml4a.net/) [+](https://www.youtube.com/playlist?list=PLaN6Cxwpu9UKR2mPc39bZEJoyAoCwRw_q), [Yining Shi](https://github.com/yining1023/machine-learning-for-the-web), [Artificial Images](https://www.youtube.com/channel/UCaZuPdmZ380SFUMKHVsv_AA), [Daniel Shiffman](https://www.youtube.com/c/TheCodingTrain)