Output to typed array?

[viziionary]

I'm interested in using GPU.js as a shader. Problem is, it seems to me, the shading is fairly fast, I mean, 8.2M values in a typical full screen rgba array buffer (1080 x 1920 x 4 channels = 8,294,400) get handled using a fairly high number of unoptimized operations in ~12ms, (congratz on the performance! I havent even tried using your input method yet but your documentation doesnt mention outputting as a typed array, only that this inputs the array more efficiently) but then the typed array I input to the kernel is now an untyped array in the output, and to render it I have to convert to a typed array and that is slowwww, like so:

renderer.render(scene, camera, rt); // render scene with three.js ~1ms :)
renderer.readRenderTargetPixels(rt, 0, 0, smallWidth, smallHeight, frameBuffer); // get scene as buffer ~ 1ms :)
outputBufferRaw = pixelateMatrix(frameBuffer, [width], [height], [scale], [size], [purity]); // shade the buffer ~12ms :)))
outputBuffer = new Uint8ClampedArray(outputBufferRaw); // convert the now untyped array output from kernel ~100ms :(((
output = new ImageData(outputBuffer, width, height); // create ImageData from typed array ~1ms :)
context.putImageData(output, 0, 0); // render result ~1ms :)

Maybe I'm missing something? Is there already a way to get the output as a typed array?

[viziionary]

By the way, this beautiful pixelart shader is what I'm doing with GPU.js, and I dont recall if you remember before, but I created an issue here once where I mentioned a neural network project I planned to integrate gpu.js with, and my plan, aside from using gpu.js for shaders, is to make a procedurally generated universe game with that artstyle shown above where each creature is not only evolved uniquely to fit the conditions on the planet like this, but each creature also will operate with its own neural network that I designed from the ground up with GPU.js powering it :) This is all very much still in development, but as I promised before, I will share the project once it's at a respectable stage.

[robertleeplummerjr]

I just started working on a method that allows you to add an image as an argument into the kernel, like so:

const gpu = new GPU();
const kernel = gpu.createKernel(function(image) {
  // .., do stuff with image
}, { output: [x, y], graphical: true });

const image = new Image();
image.src = 'some-source.png';
image.onload = () => {
  kernel(image);
};

Does this help at all? As far as typed array, can you put a jsfiddle together or similar that shows the performance issue?

[robertleeplummerjr]

I wanted to mention, brain.js v2 will use gpu.js.
Some of the features it has:

• FeedForward neural networks
• Time step neural networks
• Recurrent neural networks
• Convolution neural networks
• Any combination of the above
• 100% Complete network composition, you can even use your own layers!
• Super easy api that even a child can learn
• 100% GPU driven

The work on image inputs with gpu.js will allow us to use images directly in the network in brain.js as well. I'd love to get more collaborators, if you are interested, as it isn't 100% done yet (probably about 90% or so done), but friendly competition is nearly as good.

[robertleeplummerjr]

Also, I am so excited about your plan, this is exactly the type of innovation we esteem.

[viziionary]

Well lets say you wanted to use gpu.js as a shader (instead of having to learn to write shader code) and you were rendering from three.js. You want to take the output from the three.js context (the source render) and apply a shader (let's say darken then whole image by 50%, a simple operation) and render the new image to a canvas, in real time, at 60FPS. How would you do it?

I read about the graphical output option but it seemed to me like it might not be designed for real time output, but rather as an easy way to see results from the kernel. It sends the result to a canvas that I can then access, but that does add a step where I have to pull the image data from the canvas within gpu.js and put it on my render target canvas. Maybe it would be better if you let the user set their own target canvas for the output?

[robertleeplummerjr]

You want to take the output from the three.js context

I think you can just use the textures from three.js. This will likely need a bit of research. One area that you'd need to utilize is sharing of contexts and canvas.

const gpu = new GPU({ canvas, webGl });

[robertleeplummerjr]

Next you'd just read the values out of webgl as a texture perhaps: https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/readPixels

[viziionary]

@robertleeplummerjr Ok your solution was perfect.

https://stackoverflow.com/a/50617342/7765114

The results I'm getting from that optimization are 30x faster, bringing the shading and rendering time down from ~30ms to ~1ms. That's 8.2944 million multi-operation tasks carried out (for a 1080x1920 render, x4 rgba values per pixel) in just ~1ms by your library with WebGL. That's insanity. Can that even be right? Is my math wrong on that? Maybe you're making an optimization to run only 1080x1920 (2.073 million) tasks and handle the 4 rgba values as a loop within each task? Still comes out to well over 8.2 million operations (by my count ~30 math operations per pixel in my shader, so 62 million individual math operations at least?) in ~1ms if my bench-marking is not lying to me. I see nvidia's self driving AI is performing 24 trillion operations/s, so I guess the 62 billion ops/s that comes out to is within the realm of possibility for my 1060.

And there are still further optimizations to be made to send that buffer to the GPU faster with the use method I read about in your documentation, plus probably fancy math optimizations I could make, I just dont have any motivation to make the optimization because it's already so fast.

[viziionary]

By the way you should mention the context sharing capability in the section of your readme where you already mention graphical output, it's buried pretty deep in the documentation and I had no idea about it even after looking for something like that.

[robertleeplummerjr]

Lets open this and add the context stuff in a readme. Very glad to hear your results!

[robertleeplummerjr]

Would you happen to have an example we could use in documentation you could share here?

[viziionary]

Yes, well not in working order atm but Ill prepare one. Also I am interested in your AI project. I'll take a look. And on that note, for large scale GPU powered neural networks, typically more GPUs are used than what's standard, I'm wondering: It's doubtful that the web browser would be capable of leveraging 3 or more GPUs properly, but I'm wondering if, when you extend GPU.js to work properly in node.js, you could make it so that any number of GPUs on a mobo would be utilized fully? I'm interested in the possibility of getting something like this to try some wild neural networking for advanced object recognition training or whatever other crazy neural networking that would make feasible. Anyway, that's a separate issue. I'll provide that example tomorrow when I have time :)