Support for numpy in Javascript through another serilization like MessagePack

[VigneshVSV]

HTTP can support MessagePack headers for content type. Also for other lower TCP based protocols, MessagePack serialization can be very useful.

Numpy arrays are serialized as fast as python's pickle by MessagePack format with msgspec.msgpack . However, to unpack it on the javascript side, a suitable deserialization is necessary. MessagePack is also safer than pickle.

The expected server side code is as follows:

class MyThing(Thing)
    
    _numpy_event = Event() 
    
    @action()
    def numpy_gen(self) -> memoryview:
        return memoryview(self._large_numpy_array)
       
    def event_pub(self) -> None:
         while True:
             self._numpy_event.push(memoryview(self._large_numpy_array)
             time.sleep(1)

if __name__ == '___main__':
     MyThing(instance_name='my-thing', serializer='msgpack').run_with_http_server()

However, a javascript library which can read this memoryview data is not known.

Serializer	Spectrum 1e3	Spectrum 1e4 Double Entries	Spectrum 1e5 Entries	Spectrum 1e6 Entries
JSONSerializer	0.05121	0.61983	6.29486	65.279
PickleSerializer	0.01586	0.01927	0.26054	8.31536
MsgpackSerializer	0.01237	0.09472	1.13373	15.2071

Rough steps:

• research existing serializers in Javascript
• Implement a C/C++ based numpy to javascript parser
• Show code examples to be used in node-wot. In node-wot, you can always access the binary buffer and pass it to a custom deserializer. Better may be to register a content-type with their existing content type management.

This is a javascript related issue, not python. This can also be released as a separate package or add-on in npm.