Work in progress for AudioWorkletNode via Worker thread #124

orottier · 2024-05-03T10:06:36Z

Many many todo, but it works now for single-channel source nodes! examples/audio-worklet.mjs
Relates to #28

Todo

env.run_script seems like it has no access to the global scope. It does not know registerProcessor and AudioWorkletProcessor. So maybe we should just call `eval`.

index.d.ts

b-ma · 2024-05-03T10:26:50Z

Cool! I will have a look soon

generator/rs/lib.tmpl.rs

b-ma · 2024-05-03T17:24:53Z

Just made small change to remove the hardcoded path and have it work on my machine, this is really nice! Great work!

TODO pass to the render call

…-worker

Cargo.toml

src/audio_worklet_node.rs

b-ma · 2024-05-04T09:48:22Z

[WARNING] make sure to check #124 (comment) to not reintroduce the index.js file and changes in index.d.ts

b-ma · 2024-05-04T10:51:29Z

I can advance on

conform to the spec (needs context.addModule, only pass the name in the AudioWorkletNode ctor)

next, can be done on JS side rather simply I think

orottier · 2024-05-04T11:15:12Z

Thanks! Interesting stuff happening regarding the index.js. I will take care

can be done on JS side rather simply I think

Okay that would be helpful!

b-ma · 2024-05-12T09:35:55Z

For remaining avoid allocations in process call and safety issues, I think the following algorithm would do the trick

Check inputs layout [n][m] compared to js_inputs (only m can change at runtime, as number of inputs is defined at construction)
If layout is not the same we need to rebuild js_inputs (allocations only happen at this step)
- create new js_inputs Array
- for each input
  - create new js_input Array
  - shallow copy old js_input (i.e. existing Float32Array channels) into new js_input
  - adjust missing or extra channels
  - freeze js_input Array
- freeze js_inputs Array
- re-assign js_inputs to processor[kWorkletInputs]
Repeat 1. and eventually 2. with outputs and js_ouputs
Copy inputs buffers into js_inputs using copy_from_slice
Execute js_process
Copy js_ouputs buffers into outputs using copy_from_slice

b-ma · 2024-05-12T09:58:35Z

Other things I see that could be nice to have I think:

ensure we run run_audio_worklet_global_scope and setImmediate only once per render quantum, but I don't really see how we can achieve that without some hint from the graph... If you have any idea, that's welcome
rely on the fact that parameterDescriptors are ordered to avoid transferring parameter keys, the NapiAudioWorlketProcessor::param_values could just be Vec<&'static [f32]>

Then I start to be out of ideas too :)

In any case, here are the latest wpt results:

  RESULTS:
  - # pass: 260
  - # fail: 42
  - # type error issues: 0
> wpt duration: 1:34.799 (m:ss.mmm)

Almost 80% of the tests (332)!

orottier · 2024-05-13T09:46:23Z

Hey, I skimmed your commits and they look good. Nice to see progress on allocation free JS because it may be the garbage collector playing us parts in realtime safety.
Indeed very nice results on the WPT.

I have been benchmarking a bit and it's surprising to see the click actually happen inside the run_audio_worklet_global_scope call, not outside. This is nice because apparently we are not fighting the event loop but something inside our call into the worklets (still it may be a garbage collect)..

In any case, if I change

-    while let Ok(msg) =
-        process_call_receiver(worklet_id).recv_timeout(std::time::Duration::from_micros(100))
-    {
+    while let Ok(msg) = process_call_receiver(worklet_id).try_recv() {

The load goes up from 11% CPU to 14% on my machine (because we removed some idling) but all the clicks/pop/underruns also seem to dissappear. Can you confirm?

b-ma · 2024-05-13T12:39:08Z

I just implemented the algorithm I described upper, would be nice to have your review when you find some time (no hurry :), should indeed help minimizing garbage collection I think

The load goes up from 11% CPU to 14% on my machine (because we removed some idling) but all the clicks/pop/underruns also seem to dissappear. Can you confirm?

Actually, this is hard to tell because I do very rarely have clicks on my side now, but if it's better on your side then let's go that way! (maybe we should just make sure it doesn't create issue with the process exit, but has you had ideas to refactor anyway...)

orottier

The code looks good to me, I have left some nitpicks.

However, the garbage collector stats before this change are equal to afterwards:

node --trace-gc examples/audio-worklet.mjs
gives e.g.
[46986:0x130028000] 5658 ms: Scavenge 5.9 (9.0) -> 4.3 (9.0) MB, 0.17 / 0.00 ms (average mu = 1.000, current mu = 1.000) task;
Meaning it reclaimed from 5.9 to 4.3 MB. More info: https://nodejs.org/en/learn/diagnostics/memory/using-gc-traces

After the change:
[47391:0x128028000] 5354 ms: Scavenge 5.9 (8.7) -> 4.3 (8.7) MB, 0.12 / 0.00 ms (average mu = 1.000, current mu = 1.000) task;

Not sure what to make from this..

orottier · 2024-05-13T19:02:45Z

src/audio_worklet_node.rs

-
-            for (output_number, output) in outputs.iter().enumerate() {
-                let mut channels = js_outputs.get_element::<JsObject>(output_number as u32)?;
+            if !check_same_io_layout(&js_outputs, outputs) {


I was in doubt if it was possible for output channel counts to change at runtime, but it is indeed possible:
https://webaudio.github.io/web-audio-api/#configuring-channels-with-audioworkletnodeoptions (note at item 4)

Yup indeed, I think there might be an issue on the rust side on this point, the audioworkletnode-channel-count.https.html wpt test is a good example of what is expected. The test is currently crashing for some reason I didn't really understand, but it seems the rs output length keeps to be 1 even when input is 3

orottier · 2024-05-13T19:05:28Z

src/audio_worklet_node.rs

+                            let src = js_channel_buffer.as_ptr();
+                            let dst = channel.as_ptr() as *mut f32;
+
+                            unsafe {


We can just safely use copy_from_slice, no?

That was my first try but I did manage to find a way to iter_mut() the &'static [&'static [&'static [f32]]] output...

orottier · 2024-05-13T19:07:27Z

src/audio_worklet_node.rs

-                    channels.delete_element(i)?;
+                for (channel_number, channel) in input.iter().enumerate() {
+                    let js_channel = js_input.get_element::<JsTypedArray>(channel_number as u32)?;
+                    let mut js_channel_value = js_channel.into_value()?;


Would be good to check at some point if we don't allocate with some of these calls

Normally except in rebuild_io_layout there should be no allocation, but clearly needs to be confirmed

I'm just thinking we could also have a pool of preallocated Float32Array to mitigate further this possible issue

orottier · 2024-05-13T19:17:51Z

The GC pauses appear to be of the range 0.08 - 0.17 ms and seem not related to the excessive call durations I'm measuring.. So what is stalling my audio thread!?

b-ma · 2024-05-14T06:56:17Z

However, the garbage collector stats before this change are equal to afterwards:

Hum, no clear idea neither... did you make it run for a long time ? I'm no expert but the few times I had to check such thing, the GC was called every 4-5 sec, so as our example is 8 sec we could just see the result of some initialisation stuff?

The GC pauses appear to be of the range 0.08 - 0.17 ms and seem not related to the excessive call durations I'm measuring.. So what is stalling my audio thread!?

Yup weird, no idea neither... note that I didn't remove the recv_timeout in my last changes, might be somewhere to look at?

orottier · 2024-05-14T19:18:53Z

Okay after loads of logging I have come to the surprising conclusion that the call .recv_timeout(std::time::Duration::from_micros(100)) is indeed the culprit. It takes up to 8 milliseconds sometimes which is pretty insane to think about! Reading the issue list of crossbeam seems to indicate that they are doing some weird stuff with spinning and backoffs and possibly that does not work nice on ARM? In any case this is not suitable for our latency-sensitive ping pong of the audio thread and the worker.

So as I already mentioned changing the call to just try_recv solves my underrun issues. I did a bit of further profiling because I was worried the worker thread would max out a single CPU core but I don't see it happening, so possible the event loop does a bit of idling. Which is probably fine for now. The load per core does not exceed 15% for audio-worklet.mjs

I definitely want to look into this further, possible using some primitives from the parking_lot crate to perform the ping pong with CondVars maybe. But I guess that is not a must have before merging.

orottier · 2024-05-15T06:34:00Z

Regarding

TODO: In the case of OfflineAudioContext, it is probably desired to block here

We could maybe add a method await_all_processors_registered in js/AudioWorklet.js to ensure the number of create-processor is equal to the number of processor-registered and call it before the OfflineAudioContext startRendering?

b-ma · 2024-05-15T06:36:43Z

Cool, nice that you found the issue! I think we could start to consider merging this, and put all remaining ideas / improvements to a new issue.

Is there anything you think we should really do before?

orottier · 2024-05-15T06:39:39Z

Yeah sounds good! Perhaps only my final comment

b-ma · 2024-05-15T06:41:42Z

Yup sure, I will have a look

…fore actual startRendering

b-ma · 2024-05-15T11:02:47Z

Follow up in #127

b-ma · 2024-05-15T11:03:08Z

\o/ great work!

jampekka · 2024-05-15T13:00:23Z

Awesome work! 🔥

orottier · 2024-05-15T15:13:47Z

Quite the milestone indeed :) Congrats to us. And also thanks @jampekka for chiming in on the exploratory phase. An extra pair of eyes is always very motivating.

orottier · 2024-05-17T07:02:41Z

Please let me know when you have released this on npm. I think it is worth the announcement on the web audio slack. Also I would like to get some tips and tricks from Paul and Hoch in the WG afterwards

b-ma · 2024-05-17T08:12:13Z

Done [email protected] !

Also I would like to get some tips and tricks from Paul and Hoch in the WG afterwards

Yup, I will try to attend too next times

orottier added 10 commits April 24, 2024 14:05

Copied ScriptProcessor rs and js to AudioWorklet, no changes yet

686efe7

Constructor and binding for AudioWorkletNode - no functionality

cb96017

Napi(Offline)AudioContext is now a struct instead of tuple

47032c5

Spawn a worker, run some things in worker - does not work

992ef28

env.run_script seems like it has no access to the global scope. It does not know registerProcessor and AudioWorkletProcessor. So maybe we should just call `eval`.

New plan: spawn a worker per AudioWorkletNode - should be simpler

87f076d

Get output buffer out of the AudioWorklet to Rust

0e06db7

Got the WhiteNoiseProcessor working! only 1-channel source nodes for now

7e85530

Restore the butchered examples/script-processor.mjs

01039e6

Cleanup of js/AudioWorkletNode.js

a2f02f7

Proper file ext for example, validated that processorOptions are passed

e8e9fd5

orottier requested a review from b-ma May 3, 2024 10:06

orottier commented May 3, 2024

View reviewed changes

index.d.ts Outdated Show resolved Hide resolved

orottier commented May 3, 2024

View reviewed changes

generator/rs/lib.tmpl.rs Outdated Show resolved Hide resolved

remove hardcoded path

9f74daa

orottier added 5 commits May 3, 2024 20:27

worklet: pass input, bind this to the processor, coerce proper return

d66078b

worklet: read parameterDescriptors from JS and pass to Rust

5a80d3a

TODO pass to the render call

Merge remote-tracking branch 'origin/main' into feature/audio-worklet…

1f6ea1e

…-worker

Move worklet rs code from lib template to audio_worklet_node.rs

aa6677b

Worklet: make parameters work on the rust side, todo expose via JS

43fefd7

orottier commented May 4, 2024

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

orottier commented May 4, 2024

View reviewed changes

src/audio_worklet_node.rs Outdated Show resolved Hide resolved

refactor: use lower-level napi macros

3845591

b-ma added 2 commits May 4, 2024 12:13

feat: expose worklet audio params to js

689f0ff

cleaning

e1ce543

Rename worklet SendItem to ProcessorArguments

533f004

b-ma added 2 commits May 12, 2024 09:42

simplify update of Float32Array data for parameters

1011c60

cleaning

d528460

store processors in dedicated object

931d059

b-ma added 2 commits May 13, 2024 12:58

reuse Float32Array in JS side

6916f1a

clippy

e423654

orottier commented May 13, 2024

View reviewed changes

Worker command poll: do not use recv_timeout due to realtime issues

7497e3e

b-ma marked this pull request as ready for review May 15, 2024 06:36

ensure all AudioWorkletProcessor have finished their instanciation be…

cf19f89

…fore actual startRendering

b-ma mentioned this pull request May 15, 2024

AudioWorklet improvements #127

Open

11 tasks

b-ma merged commit 3a0a5ba into main May 15, 2024
5 checks passed

b-ma mentioned this pull request May 15, 2024

AudioWorklet support #28

Closed

b-ma deleted the feature/audio-worklet-worker branch May 15, 2024 11:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Work in progress for AudioWorkletNode via Worker thread #124

Work in progress for AudioWorkletNode via Worker thread #124

orottier commented May 3, 2024 •

edited by b-ma

Loading

b-ma commented May 3, 2024

b-ma commented May 3, 2024

b-ma commented May 4, 2024

b-ma commented May 4, 2024

orottier commented May 4, 2024

b-ma commented May 12, 2024 •

edited

Loading

b-ma commented May 12, 2024 •

edited

Loading

orottier commented May 13, 2024

b-ma commented May 13, 2024 •

edited

Loading

orottier left a comment •

edited

Loading

orottier May 13, 2024

b-ma May 14, 2024

orottier May 13, 2024

b-ma May 14, 2024

orottier May 13, 2024

b-ma May 14, 2024

orottier commented May 13, 2024

b-ma commented May 14, 2024 •

edited

Loading

orottier commented May 14, 2024 •

edited

Loading

orottier commented May 15, 2024

b-ma commented May 15, 2024 •

edited

Loading

orottier commented May 15, 2024

b-ma commented May 15, 2024

b-ma commented May 15, 2024

b-ma commented May 15, 2024

jampekka commented May 15, 2024

orottier commented May 15, 2024

orottier commented May 17, 2024

b-ma commented May 17, 2024

Work in progress for AudioWorkletNode via Worker thread #124

Work in progress for AudioWorkletNode via Worker thread #124

Conversation

orottier commented May 3, 2024 • edited by b-ma Loading

b-ma commented May 3, 2024

b-ma commented May 3, 2024

b-ma commented May 4, 2024

b-ma commented May 4, 2024

orottier commented May 4, 2024

b-ma commented May 12, 2024 • edited Loading

b-ma commented May 12, 2024 • edited Loading

orottier commented May 13, 2024

b-ma commented May 13, 2024 • edited Loading

orottier left a comment • edited Loading

Choose a reason for hiding this comment

orottier May 13, 2024

Choose a reason for hiding this comment

b-ma May 14, 2024

Choose a reason for hiding this comment

orottier May 13, 2024

Choose a reason for hiding this comment

b-ma May 14, 2024

Choose a reason for hiding this comment

orottier May 13, 2024

Choose a reason for hiding this comment

b-ma May 14, 2024

Choose a reason for hiding this comment

orottier commented May 13, 2024

b-ma commented May 14, 2024 • edited Loading

orottier commented May 14, 2024 • edited Loading

orottier commented May 15, 2024

b-ma commented May 15, 2024 • edited Loading

orottier commented May 15, 2024

b-ma commented May 15, 2024

b-ma commented May 15, 2024

b-ma commented May 15, 2024

jampekka commented May 15, 2024

orottier commented May 15, 2024

orottier commented May 17, 2024

b-ma commented May 17, 2024

orottier commented May 3, 2024 •

edited by b-ma

Loading

b-ma commented May 12, 2024 •

edited

Loading

b-ma commented May 12, 2024 •

edited

Loading

b-ma commented May 13, 2024 •

edited

Loading

orottier left a comment •

edited

Loading

b-ma commented May 14, 2024 •

edited

Loading

orottier commented May 14, 2024 •

edited

Loading

b-ma commented May 15, 2024 •

edited

Loading