| layout | title |
|---|---|
default |
Exploring Human-in-the-loop Program Synthesis with Live Coding |
Mark Santolucito, Barnard College
Live Coding is a budding performative practice whereby artists write code that generates media (audio, visuals, etc.) live on stage. In live coding, the process of writing code is itself the final product, rather than a fixed implementation of a tool, as typically seen in software development. We propose that this unique performance style, with a heavy emphasis on the evolution of code, is an ideal space to explore program synthesis. In this paper, we present a synthesis-aided live coding interface for drum sequencer patterns. Our live coding environment allows performers to live coding in Javascript, uses a programming-by-example model for synthesis. The synthesis toolchain uses SyGuS and a set of templates specific to live coding music. We describe the tool and its implementation, and highlight key areas of future exploration.
Live Coding is a performative practice of coding still in its nascent stages of definition. Live coding, by its nature, is an evolving practice that eludes a formal definition. Generally, a live coding performance consists of code that continually generates some media (often audio). The performer changes the code throughout the performance, thereby shaping the media that is being produced. A key component to live coding allowing viewers to watch the evolution of the code itself.
During the evolution of code throughout a live coding performance, there is no single correct state of the code that the performer must achieve. The process of a writing code for a live coding performance is exploratory. As a result, the implicit specification of the intended code is constantly shifting. This constantly changing specification make live coding an ideal testbed for program synthesis.
Live coding is closely related in many ways to Live Programming. In both cases, there is a focus on live re-evaluation of code. Live programming is largely framed as an style of IDE and a software development environment, whereas live coding specifically focuses on the performative practice of the evolution of code. One possible interpretation of these two terms is that live coding is a performance practice that takes place inside a live programming environment.
There has been great progress in the space of Live Programming with the work of Hempel et. al, UIST '19 on the Sketch-n-Sketch tool that looks at output directed programming. In this work, the authors explore how direct manipulation of the intended output can be turned back into program transformations automatically. The authors of this work mention the possible integration of synthesis (as opposed to hand-coded transformations) as a future direction. Here we explore how general purpose synthesis tools might be integrated into a direct manipulation style live programming editor.
This setting is particularly valuable, as it allows us to explore more of the human-in-the-loop model of program synthesis. Live coding is, by its nature, an iterative process where specifications are continuously refined. By augmenting live coding environments with synthesis, we can gain insight into how a synthesis driven feedback loop of development might be most effectively designed.
To explore the intersection of live coding and program synthesis, we built a live coding environment, called the "Shiny Happy WebAudio MIDI-fied Live Coding Drum Machine", or the SHWAMLCDM for short. A live demo is available online.
Following a classic live progrmaming interface model, we have a code editor to left, and the output to the right. As the code to the left is updated, so too is the graphical display on the right. Similarly, as the output is updated, so too is the code updated to align with the output. One of our key design goals is that the code and output are always in alignment.
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/k5I2fkEr_qE?controls=0" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" style=" margin: 0 auto; display: block; " allowfullscreen></iframe>As shown in the above video, our live coding environment uses a synthesis driven program-repair model to constantly updated the code to match the provided pattern. On the interface level, this is a programming-by-demonstration interaction - the user demonstrates the intended beat with the GUI, and the repair engine generates code to match the data in the GUI. The implementation of this transforms this problem into a programming-by-example (PBE) problem. As programming-by-example problem, we then use CVC4, a SMT solver with support for SyGuS, to run SyGuS queries. We map the grammatical elements of SyGuS to generic Javascript code, and put this result inside Javascript templates before displaying the code to the user.
More specifically, to obtain a PBE problem, we start by treating each beat (e.g. Tom1, Kick, Snare, etc) as a separate problem. Each beat itself is an array of length 16 with values representing the note to be played (0 for no beat, 1 for a quiet beat, 2 for a loud beat) at the time step corresponding to the index of the array. As is typical in step sequencers, time is quantized into 16 steps. To transform this array into a PBE problem, we view the array as a function mapping time to notes.
As an example, the pattern below would generate the following PBE constraints.
(constraint (== f(0) 1))
(constraint (== f(1) 0))
(constraint (== f(2) 1))
(constraint (== f(3) 0))
(constraint (== f(4) 1))
Notice that we truncate the constraints at the position of the last non-zero beat. We do this so that synthesis has some opportunity to generalize the given pattern. In the above example, CVC4 will output the following function in SyGuS format.
(define-fun f ((i Int)) Int (- 1 (mod i 2)))
Transformed from SyGuS into Javascript via a simple syntactic mapping, we then obtain the following term:
1 - (i % 2)
To map this term over the array of beats, we use a code snippet template as follows:
b.rhythm1 = new Array(16).fill(0).map((val,i) => { return 1 - (i % 2); });
This is a good solution as it not only satisfies the given examples, it also reasonably generalizes across the rest of the beat, hypothesizing that the pattern the user would like to complete is as follows:
However, this is of course not the pattern the user has provided yet. With our current .map code alone, the code and the GUI pattern are no longer in alignment.
One of the key design goals of this system is that the code is always in a state that would generate the GUI data.
To remedy this situation, we overwrite the second half of the array which is all zeros.
This allows us to keep our general solution, while still ensuring the code does not overstep in its predictions.
b.rhythm1 = new Array(16).fill(0).map((val,i) => { return 1 - (i % 2); });
b.rhythm1.splice(5,11,...Array(11).fill(0));
If the user likes this proposed code, the user may comment out the second line so that the GUI pattern adopts the generalized code solution.
There are situations where synthesis is not able to find new code that fits the updated pattern in the alloted time (we set a timeout of 500 ms). In this case, the synthesis engine keeps the old synthesis result and adds a direct indexing operation to adjust the code according (e.g. b.rhythm1[7]=2).
Another interesting edge case in the synthesis process is when a synthesis request is still running, and the GUI is updated by the user. To remedy this, we tag each synthesis request with the state of the GUI when it begins. If we receive a synthesis result with a tag that does not match the current state of the GUI, we ignore it and wait for the new synthesis query to finish.
This prototype of a synthesis-enabled live coding environment demonstrates the potential for further exploration in this space.
One main question is how we can more effectively leverage synthesis.
Currently, our synthesis is limited to LIA problems and only replaces subexpressions in .map statements.
CVC4's SyGuS support is powerful enough to support much more complex synthesis, but the key challenge is how to integrate this into a real language.
One possible direction is to connect the synthesis requests of each individual beat pattern together. At the moment, there is no way for, as an example, a result for the Kick pattern to be reused in the Snare pattern. Allowing for this would make the synthesis requests more complex, but potentially allow us to generate more compact code. At times, this may even speed up synthesis when a previous pattern can be effectively reused for a new pattern.
The speed of synthesis is also a major concern - especially in a live coding environment where reactivity is important to the performance. Since we run each pattern as a separate synthesis query, the total synthesis time over all six patterns with a 500ms timeout can be 3 seconds when run serially. We are able to overcome this to some extent by running each synthesis request in parallel, but there is a clear near for faster synthesis tools.
There are additionally many interaction design questions in the space of displaying proposed synthesis solutions. The work of Lerner, CHI 2020 proposes projection boxes as a way of demonstrating code behavior to users. This might be adopted for the synthesis context to good effect.
The most pressing matter in this work is doing a user study with live coding performers.
In our initial conversations, one point that was brought up is the need for a music specific DSL.
In pursuing a DSL, we also believe it is important that the synthesis itself remains able to synthesis general purpose Javascript code, allowing us to take full advantage of existing synthesis tools.