Universal audio synthesizer control
Using deep learning to understand and control musical audio synthesizers
FlowSynth - Deep synthesizer control
Sound synthesizers are pervasive in music and they now even entirely define new music genres. However, their complexity and sets of parameters renders them difficult to master. We created an innovative generative probabilistic model that learns an invertible mapping between a continuous auditory latent space of a synthesizer audio capabilities and the space of its parameters. We approach this task using variational auto-encoders and normalizing flows Using this new learning model, we can learn the principal macro-controls of a synthesizer, allowing to travel across its organized manifold of sounds, performing parameter inference from audio to control the synthesizer with our voice, and even address semantic dimension learning where we find how the controls fit to given semantic concepts, all within a single model.
Check out our demo on YouTube
Deep learning models have provided extremely successful methods in most application fields, by enabling unprecedented accuracy in various tasks, including audio generation. However, the consistently overlooked downside of deep models is their massive complexity and tremendous computation cost. In the context of music creation and composition, model reduction becomes eminently important to provide these systems to users in real-time settings and on dedicated lightweight embedded hardware, which are particularly pervasive in the audio generation domain. Hence, in order to design a stand alone and real time instrument, we first need to craft an extremely lightweight model in terms of computation and memory footprint. To make this task even more easier, we relied on the Nvidia Jetson Nano which is a nanocomputer containing 128-core GPUs (graphical unit processors) and 4 CPUs. The compression problem is the core of the PhD topic of Ninon Devis and a full description can be found here.
Targets of our instrument
We designed our instrument so that it follows several aspects that we found crucial:
Musical: the generative model we choose is particularly interesting as it produces sounds that are impossible to synthesize without using samples.
Controllable: the interface was relevantly chosen, being handy and easy to manipulate.
Real-time: the hardware behaves as traditional instrument and is as reactive.
Stand alone: it is playable without any computer.