Comlink C#

[chris-kruining]

Hi @amaitland,

I saw you starred my project thanks!

I hope you don't mind me asking for a bit of help here.

I have a few issues with trying to implement the sending / receiving of messages.
currently this is my implementation

using System;
using CefSharp;
using Comlink.Core;

namespace Comlink.CefSharp
{
    public class CefSharpWindow : IWindow
    {
        private readonly IWebBrowser _window;
        public CefSharpWindow(IWebBrowser window)
        {
            _window = window;
            _window.JavascriptMessageReceived += (s, a) =>
            {
                // TODO(Chris Kruining) Properly handle the message
                Message?.Invoke(a.ConvertMessageTo<String>());
            };
        }

        public void Start()
        {
            throw new NotImplementedException();
        }

        public event Action<String> Message;
        public void PostMessage(String message, params ITransferable[]? transferables)
        {
            // TODO(Chris Kruining) Send message to _window
        }
    }
}

As you can see, I have 2 TODO's in here, and for both I'm unsure on how to solve them. Because if I cast the message to a String, do I get the unparsed message sent by the page? And how would I go about sending a string to the page.

And can I support transferable object as the postMessage specs say in any way?

Thanks in advance :D

[amaitland]

Because if I cast the message to a String, do I get the unparsed message sent by the page?

The ConvertMessageTo is used to create a POCO.

By string are you meaning getting the data as JSON? Calling JSON.stringify on the object that you pass to PostMessage is probably the simplest option if you are after JSON.

And how would I go about sending a string to the page.

Can you be more specific? Send it to where exactly? Call a javascript function? Fire a javascript event?

And can I support transferable object as the postMessage specs say in any way?

In the context of your project what exactly are you calling a transferable? How exactly are you picturing this to work?

I'm never actually used Comlink, only read about it so assume I know nothing 😄

[chris-kruining]

The ConvertMessageTo is used to create a POCO.
comlink does indeed serialize the messages into JSON strings, I forgot for a sec this is not spec-based, just convention :P. Does this mean my ConvertMessageTo<String> is the right call here, because the listeners of the Message event do expect a JSON string.

Can you be more specific? Send it to where exactly? Call a javascript function? Fire a javascript event?
I suppose firing a JS event would be the most simple solution.

In the context of your project what exactly are you calling a transferable? How exactly are you picturing this to work?
MDN postMessage docs => I'm referring to the transfer argument.

Again, thanks in advance for your time and effort!!!

[amaitland]

Simply casting the Message property to a string should be sufficient if the message is already JSON. http://cefsharp.github.io/api/85.3.x/html/P_CefSharp_JavascriptMessageReceivedEventArgs_Message.htm

I suppose firing a JS event would be the most simple solution.

Calling a function would likely be the simplest solution, if you wish to setup an event then you'd need some JavaScript like https://developer.mozilla.org/en-US/docs/Web/API/EventTarget#Example

Fundamentally you'd use ExecuteScriptAsync or EvaluateScriptAsync to call your function or dispatch your event.
https://github.com/cefsharp/CefSharp/wiki/General-Usage#1-how-do-you-call-a-javascript-method-from-net

MDN postMessage docs => I'm referring to the transfer argument

I'm aware of what transferable means in the context of webworker PostMessage. If commlink c# had documentation what would it define a transferable as?

• What types are supported?
• What are some typical use cases?

Is a sequence of Transferable objects that are transferred with the message. The ownership of these objects is given to the destination side and they are no longer usable on the sending side.

Say you pass an arraybuffer to commlink, what type does that map to in .Net? Are you creating a custom type? You are crossing a process boundary, so the data will need to be sent between processes.

[chris-kruining]

Simply casting the Message property to a string should be sufficient if the message is already JSON. http://cefsharp.github.io/api/85.3.x/html/P_CefSharp_JavascriptMessageReceivedEventArgs_Message.htm

Ok, thanks!

Calling a function would likely be the simplest solution, if you wish to setup an event then you'd need some JavaScript like https://developer.mozilla.org/en-US/docs/Web/API/EventTarget#Example

Fundamentally you'd use ExecuteScriptAsync or EvaluateScriptAsync to call your function or dispatch your event.
https://github.com/cefsharp/CefSharp/wiki/General-Usage#1-how-do-you-call-a-javascript-method-from-net

Ok, I'll look into this, I was hoping I could achieve this without having to inject JS, but no biggie at all. Do you racon overriding the window.parent would be a good idea? Because that would be the postMessage target in my case, but I don't know if that would do more harm than good.

I'm aware of what transferable means in the context of webworker PostMessage. If commlink c# had documentation what would it define a transferable as?

Exactly the same as the postMessage docs, Comlink doesn't do anything with or to them, just passes them through. Comlink is "just" a serialization/deserialization abstraction layer for the message parameter of postMessage.

Say you pass an arraybuffer to commlink, what type does that map to in .Net? Are you creating a custom type? You are crossing a process boundary, so the data will need to be sent between processes.

Honestly no clue, I'd say whatever type is defined in C#, as an arraybuffer (ar any transferable for that matter) is just a block of memory. so the whole point of them is to "only" transfer onwership and don't do any serialization. But I wonder if this would even be feasible for cefsharp to implement (would it even be possible for cefsharp, or would this need to be escalated to cef?).

A quick somewhat off-topic side-thought, Would you recon that implementing flatbuffers be possible assuming the above mentioned transferables work?

[amaitland]

Exactly the same as the postMessage docs, Comlink doesn't do anything with or to them, just passes them through. Comlink is "just" a serialization/deserialization abstraction layer for the message parameter of postMessage.

Whilst the method might be called the same as that's the convention the CefSharp implementation sends an IPC message that crosses a process boundary, and two different runtimes (and three different programming languages). The JS implementation of postMessage is moving memory around within the V8 engine (may even just be moving pointers around at the end of the day).

Honestly no clue, I'd say whatever type is defined in C#, as an arraybuffer (ar any transferable for that matter) is just a block of memory. so the whole point of them is to "only" transfer onwership and don't do any serialization. But I wonder if this would even be feasible for cefsharp to implement (would it even be possible for cefsharp, or would this need to be escalated to cef?).

Perhaps you can use CefV8Value::CreateArrayBuffer in conjunction with a shared section of memory. Not something I've tried. You can ask on https://magpcss.org/ceforum/index.php and see if anyone else has attempted something similar in the context of CEF.

If we look at other types like MesagePort, how does that behave you if you were to transfer it?

A quick somewhat off-topic side-thought, Would you recon that implementing flatbuffers be possible assuming the above mentioned transferables work?

Not sure.

[chris-kruining]

Whilst the method might be called the same as that's the convention the CefSharp implementation sends an IPC message that crosses a process boundary, and two different runtimes (and three different programming languages). The JS implementation of postMessage is moving memory around within the V8 engine (may even just be moving pointers around at the end of the day).

I had my suspicions this would be the case, but non the less the same question. Do you think it is feasible to add this to cef&cefsharp. Or would this add to many edgecases and/or security risks?

Perhaps you can use CefV8Value::CreateArrayBuffer in conjunction with a shared section of memory. Not something I've tried. You can ask on https://magpcss.org/ceforum/index.php and see if anyone else has attempted something similar in the context of CEF.

I'll look into this, thanks!

If we look at other types like MesagePort, how does that behave you if you were to transfer it?

in terms of JS between a page and a worker, it would be as if the MessagePort is created locally in the worker. But I must admit I never explicitly done this, so I'm afraid I can't give a proper answer. Maybe Surma from Google is willing to chime in here (assuming he knows the answer).

I hope I'm not bothering you with these strange use-cases, thanks for your input thus far :D

[amaitland]

. Do you think it is feasible to add this to cef&cefsharp.

Maybe for ArrayBuffer, though that's really a big maybe at this point in time. CEF only supports ArrayBuffer at the moment (and that support is a little limited).

The other transferable object types it's not clear what the use case would actually be. It's not clear how the MessagePort, ImageBitmap and OffscreenCanvas would be useful if transferred into .Net (if such a thing was actually possible). I could see that perhaps you might wish to proxy calls to them from .Net.

I hope I'm not bothering you with these strange use-cases

It's an interesting idea 😄

[chris-kruining]

The other transferable object types it's not clear what the use case would actually be. It's not clear how the MessagePort, ImageBitmap and OffscreenCanvas would be useful if transferred into .Net (if such a thing was actually possible). I could see that perhaps you might wish to proxy calls to them from .Net.

well, maybe we could take a less literal aproach here;
MessagePort -> set's up parallel postMessage channels.
ImageBitmap -> move an image from side A to B or vice versa
OffscreenCanvas -> basically a glorified image stream.

I mean, it wouldn't as fast as just moving around a pointer, but transferring binary data seems at the least possible to me. Or am I thinking way to naïve here?

[amaitland]

Yes you can copy the data. Do you have a real world application for transferable at the moment? I think attempting to use it in real life would give you a better sense of how what the actually implementation should look/behave like.

[chris-kruining]

Sorry for the late reaction.

I don't have an example, that's what makes this whole thing hard for me as well.

Or at least, Comlink internally sets up MessagePorts, but I have no clue if that would be something we need to cover on the C# side.

[amaitland]

Moving conversation to new discussions section, maybe a little bit of noise.

[ricksanche2]

I'm new here(in general), maybe this is related(sorry if not), but what is the most efficient way to transmit data from C# to JavaScript? In my use case, I use c# to capture system audio(WASAPI loopback) and calculate FastFourierTransform data then send to browser js(60 fps) to draw visualizer. I know there is ExecuteScriptAsync(data) and JavaScript Binding, and they are all IPC underneath, but in my testing both use huge amount of CPU(~10% on 4 core i7-4900mq) just sending a size 2048 array of float 60 times a second(no fft involved, sending a prepared array). Is it because they not designed to stream data and have other construct that adds overhead? Is there a more appropriate way to send the data? Or maybe I should just use webrtc to send the captured audio stream to browser? Because the only reason to do it in C# is it can capture system audio.

[ricksanche2]

build the string in advance get 3~5 % CPU usage. Before(prepared array) it wasn't locked above 10% either, like 7 ~ 11 fluctuating.

[amaitland]

What is the baseline cpu usage for the page?

[ricksanche2]

0.5 ~ 2 %. Mostly around 1%.

[ricksanche2]

I remember seeing somewhere an array on the c# side could be attached to the browser window object or did I remember wrong? Could sharing a space in memory between c# and browser be accomplished？

[amaitland]

I remember seeing somewhere an array on the c# side could be attached to the browser window object or did I remember wrong?

It is possible, not implemented yet