-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AudioWorklet support #28
Comments
Hey, no the AudioWorklet API is not yet supported unfortunately, as few other nodes. Hopefully it will be in some future but I can't give any roadmap nor due date. I will had some notes on that in the README |
I'd be interested in looking into this. The biggest hurdle is probably to get the AudioWorkerProcessor to run in the web-audio-api-rs audio rendering thread. I think this could be possible by spawning a new napi::Env in the rendering thread, call Env::run_script on that and implement AudioWorkletProcessor in the rust side that then calls the JS AudioWorkletProcessor defined in Env::run_script. I'm very new to rust and node.js internals, so this may not be the way to go. Could somebody more familiar with these comment if this architecture is worth pursuing? |
Hi @jampekka that would be very welcome. Your suggestion makes sense and I have looked into this before, but I was unable to get an env set up on the render thread. It is obviously not allowed to use the Env of the main thread, but nowhere in the n-api docs I could find a way to construct a new one. I will investigate further and report back in this issue. If you do the same, that would be great |
I have made a bit of research and found this https://docs.rs/napi/latest/napi/attr.module_init.html (examples are a bit weird) which seems to correspond to
Maybe this can help? |
Hey @b-ma, I read those docs but I think that is not related to the actual executor. But more for the module loading (require ...) and running a one-time initialization method on the module. But I may be mistaken. I'm simultaneously looking into https://nodejs.org/api/vm.html which may be interesting provided it does not share the event loop with the main thread
Another option is to spawn a worker https://nodejs.org/api/worker_threads.html
Which would then serve as an extra thread alongside the render thread. We then use message passing to invoke |
The more I dig into this, the more confused I get how to do it. My original idea to create a new Env using napi probably doesn't work: I can't find a way to create a new Env, and my impression from the Node-API docs is that this shouldn't be done. @orottier: https://nodejs.org/api/vm.html does look promising. I guess to do this we should be able to access the created vm.Script object on the rust side and try to somehow run code in it in the rendering thread? Yet another possibility would be to create a totally new node.js instance by embedding it in the web-audio-api-rs rendering thread. Embedding a new instance seems quite straightforward (from C++ at least): https://github.com/nodejs/node/blob/main/doc/api/embedding.md |
The So on the rust side this could be just as simple as calling the registered processor in the render thread. This would be along the lines of implementing e.g. const vm = require('node:vm');
const script_code = `
function process(inputs, outputs) {
// Just to show we can write the value
outputs[0][0] = inputs[0][0];
return true;
};
register(process);
`;
var processor = null;
function register(proc) {
processor = proc;
}
const processor_context = {register};
vm.createContext(processor_context);
const script = new vm.Script(script_code);
script.runInContext(processor_context);
inputs = [new Float32Array([1, 1, 1])];
outputs = [new Float32Array([0, 0, 0])];
console.log(outputs[0]); // Float32Array(3) [ 0, 0, 0 ]
var ret = processor(inputs, outputs);
console.log(ret); // true
console.log(outputs[0]); // Float32Array(3) [ 1, 0, 0 ] |
Looks nice indeed (I need to dig into this vm stuff, looks really interesting), But I'd just like to add that at some point we will need to cast from rust I'm a bit afraid that we might have to deal directly wether with the V8 api actually, cf. https://v8docs.nodesource.com/node-4.8/d5/dda/classv8_1_1_isolate.html, or as you said, with embedding a new node instance in the render thread |
napi-rs seems to have Here's a simple napi-rs POC that defines a callback in a new VM that is called in Rust to do #![deny(clippy::all)]
use napi_derive::napi;
use napi::bindgen_prelude::*;
#[napi]
pub fn process_callback<T: Fn(Float32Array, &mut Float32Array) -> Result<()>>(callback: T) -> Float32Array {
let input = Float32Array::new([1.0, 2.0, 3.0].to_vec());
let mut output = Float32Array::new([0.0, 0.0, 0.0].to_vec());
let _ = callback(input, &mut output);
return output;
} const {processCallback} = require('./index')
const vm = require('node:vm');
// This defines our process function in a different VM
const script_code = `
function process(input, output) {
// Just to show we can write the value
output[0] = input[0]*1337;
return true;
};
register(process);
`;
// The processor in the VM is stored here
var processor = null;
function register(proc) {
processor = proc;
}
// Run the VM, which populates the processor
const processor_context = {register};
vm.createContext(processor_context);
const script = new vm.Script(script_code);
script.runInContext(processor_context);
// Run the processor in rust side
output = processCallback(processor);
console.log(output); // Float32Array(3) [ 1337, 0, 0 ] |
That's the big question. Can we detach a pointer to the VM and then work with it independently from the main context/thread. Or is a VM just an isolation mechanism and cannot be run independently from the main Env. I mean, to call
This could be interesting because a drawback of using I'm wondering if this requires us to include V8 as a dependency on the Rust side, or if we can 'reuse' the fact that we are already running in a NodeJs context. The binary size will increase greatly if we can't, and it does not really feel right to have V8 as a dependency of a NodeJs lib.. - In any case, I would like to get started with something so I will probably experiment with the Worker setup in the coming weeks. But let's continue with other explorations @b-ma @jampekka, that is very helpful! |
Hey @b-ma I remember you had another comment here, did that not work out? I have another terrible idea:
So that looks like the Worker setup (with the same drawbacks regarding real-time safety) but this has the benefits that I can write more Rust and less NodeJs, so it's easier for me to tinker with. I will start with it and keep you posted! |
No, after more checks this was a dead end, everything was launched in same thread... For info, also had a look at https://nodejs.org/api/worker_threads.html (which would be kind of ideal as
Hehe quite brute force indeed... I was thinking about launching a new node process with another napi entry point, which looks quite similar finally... So... Let's see where it goes! |
Alright. I have my terrible experiment running at orottier/web-audio-api-rs#503 |
Just FYI my current Worker experiment is stuck because
Anyway, I will try more things, maybe just run the Regarding the question if Workers are the right choice, I suddenly remembered this from hoch: |
calling
.audioWorklet
on an initializedAudioContext
returnsundefined
. Is working with web audio audio worklets supported?The text was updated successfully, but these errors were encountered: