Possible to use VST model programmatically?

[olaviinha]

Anybody have any ideas if it's possible to use the new VST models programmatically for synthesizing audio on Python / Colab, and if so, how to do it?

[jesseengel]

The SavedModels are specifically saved to work with the VST C++ code, so they it would be a little tricky, but you should be able to load them as typical saved models and call them in a for loop.

You would need to call Feature Extraction, Predict Controls, and Synthesize each in the inner loop or their own loops and carry the state manually. Code showing what's going on inside the saved models: https://github.com/magenta/ddsp/blob/main/ddsp/training/inference.py#L191

Extract Features Model: https://github.com/magenta/ddsp-vst/tree/main/models/pitch-detection