Add support for headless OpenGL

[nsthorat]

From @nsthorat on October 19, 2017 15:0

deeplearn.js can be used in a headless nodejs environment with this tool:
https://github.com/stackgl/headless-gl

Floating point textures are not supported in headless-gl, but we have a switch that can store floats as 4 unsigned bytes using fixed precision floats. This already may work.

The relevant flag is WEBGL_FLOAT_TEXTURE_ENABLED, which is controlled by this function:
https://github.com/PAIR-code/deeplearnjs/blob/master/src/environment.ts#L99

We will have to detect when we are in the nodejs environment and make sure this returns false (it may already return false by running this test).

headless-gl will return a new WebGLRenderingContext, this can be passed directly to the constructor of NDArrayMathGPU to be used in this context.

We don't want to add headless-gl as a real dependencies, so for an example script, we should have people manually run npm install headless-gl. If we can show a simple example of this in our repository, that would be phenomenal.

We should then be able to support headless environments!

Copied from original issue: tensorflow/tfjs-core#234

[nsthorat]

From @arthurjdam on October 20, 2017 2:8

Side-note; referencing #158, I'm working on (fully GPU accelerated) headless support via OpenCL (not WebCL, as I'd like to remove the need for a (v)fb). It's not exactly what you're mentioning here but it should serve the same purpose eventually.

Making some (rather slow, but steady) progress in here: arthurjdam/deeplearnjs/feature-opencl. It's -very- messy right now. Will get better as I move along :-)

[nsthorat]

Yup, similar, though I think headless OpenGL we may be able do without many changes. OpenCL, in theory, should be much faster, in the headless OpenGL we're 1) using GPGPU shaders which is going to be slower than CL and 2) we have to use 4 byte integer quantization so we lose a little precision and we're slower than working directly with 32 bit floats.

[nsthorat]

How is this going Arthur?

[nsthorat]

From @arthurjdam on November 1, 2017 1:45

Progress is not particularly fast, but it's going! Matmul and unary ops are working, though it's taking some time to figure out how to piece it together so it's relatively aligned with WebGL's interface. I'm structuring some things out, but any insights on how to nicely perform initialization and batching would be super-helpful!

Using OpenCL 1.2 (since that just happens to be what my Macbook supports), performance tests look pretty solid, and allow for delegation to multiple cpu's/gpu's, as long as I don't try to make anything return doubles (so it's 32-bit floats for everyone) I'm getting quite good behavior from everything.

[nsthorat]

From @arthurjdam on November 8, 2017 2:37

Out of sheer laziness, I've be playing with adding node-webgl (webgl-type bindings around opengl) which seems to be relatively straight-forward. I've been running into an error that I could use a hand on if anyone's more knowledgeable: https://github.com/arthurjdam/deeplearnjs/tree/feature-headless-gl

The current build will throw an error when binding vertexAttrib buffers, but I'm not yet sure why. Commenting out bindVertexBufferToProgramAttribute for clipSpacePos (gpgpu_util.ts:185) will make it -at least not throw errors- but obviously breaks the engine. Any ideas?

[nsthorat]

From @dsmilkov on November 9, 2017 2:56

Unfortunately, I'm not a WebGL/OpenGL expert, but maybe this page talking about the differences between WebGL and OpenGL might help.

Also might be good to ask the node-webgl author @mikeseven for ideas.

[nsthorat]

From @arthurjdam on November 9, 2017 3:55

Yeah I think I've processed that doc on the web/open-gl differences. Fingers crossed @mikeseven can shed some light on the situation!

[nsthorat]

From @mikeseven on November 9, 2017 6:58

Sure, what is the question exactly?

--mike

From: Arthur
Sent: Wednesday, November 8, 19:55
Subject: Re: [PAIR-code/deeplearnjs] Add support for headless OpenGL (#234)
To: PAIR-code/deeplearnjs
Cc: Bourges-sevenier, Mikael, Mention

Yeah I think I've processed that doc on the web/open-gl differences. Fingers crossed @mikesevenhttps://github.com/mikeseven can shed some light on the situation!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/PAIR-code/deeplearnjs/issues/234#issuecomment-343040334, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAxYLL3lnISn-ESKgjlXr1Sz8uNKjGh2ks5s0ne0gaJpZM4P_XuP.

[nsthorat]

From @arthurjdam on November 9, 2017 19:18

@mikeseven I sent an issue in the node-webgl repo with some explanation mikeseven/node-webgl#59

[nsthorat]

From @mikeseven on November 9, 2017 20:28

I believe the error is due to changes in nodejs internals. I need to upgrade my packages to the new nodejs/v8 native interfaces.

--mike

---

From: Arthur [email protected]
Sent: Thursday, November 9, 2017 11:18:38 AM
To: PAIR-code/deeplearnjs
Cc: Bourges-sevenier, Mikael; Mention
Subject: Re: [PAIR-code/deeplearnjs] Add support for headless OpenGL (#234)

@mikesevenhttps://github.com/mikeseven I sent an issue in the node-webgl repo with some explanation mikeseven/node-webgl#59mikeseven/node-webgl#59

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/PAIR-code/deeplearnjs/issues/234#issuecomment-343261844, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAxYLKWr07-nOZm36ZuCmx4T-3aGtxU6ks5s01ANgaJpZM4P_XuP.

[nsthorat]

From @arthurjdam on November 10, 2017 3:24

Thanks for looking into it @mikeseven! Let me know if I can do anything to help
By the by; not sure if this helps, but, the demos (i.e. the test/lesson0*.js's) work just fine, so not sure what the discrepancy is. Tested under a few different Node 8 versions, though nothing lower.

[nsthorat]

From @mikeseven on November 10, 2017 3:45

The first lessons are very simple use of GL API. This shouldn't have changed but the other samples especially using vertex buffers and async call backs have changed because v8 has changed significantly.

At the same time, nodejs has introduced n-api recently and it's better than the previous native API.

Since n-api is the future, I need to rewrite these parts for node-webgl and node-webcl and test if performance and async call backs are still behaving the same.

I'm not sure when I will have time as I'm pretty full until end of November but when I have some free time, I'll start ;-)

--mike

---

From: Arthur [email protected]
Sent: Thursday, November 9, 2017 7:24:58 PM
To: PAIR-code/deeplearnjs
Cc: Bourges-sevenier, Mikael; Mention
Subject: Re: [PAIR-code/deeplearnjs] Add support for headless OpenGL (#234)

Thanks for looking into it @mikesevenhttps://github.com/mikeseven! Let me know if I can do anything to help
By the by; not sure if this helps, but, the demos (i.e. the test/lesson0*.js's) work just fine, so not sure what the discrepancy is. Tested under a few different Node 8 versions, though nothing lower.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/PAIR-code/deeplearnjs/issues/234#issuecomment-343364394, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAxYLJaMfWHG3X6m-K2a8S0kb_y8tEeMks5s08IKgaJpZM4P_XuP.

[nsthorat]

From @erikwilson on November 24, 2017 23:38

I found that if I can use NodeJs with DeepLearn if I added a shim for location:

global.window = {
  location: {
    search: ""
  }
}

and used NDArrayMathCPU instead of NDArrayMathGPU for doing math. In fact for small graphs using the CPU can be 10x faster than using the GPU, is there a general matrix size where using the GPU becomes more efficient?

I played with using headless-gl and node-webgl modules a little but they seemed to be lacking enough WebGL extensions not to be worthwhile.

Also the location search dependency should probably be removed...

[nsthorat]

From @erikwilson on November 25, 2017 15:36

Sorry I was using version 0.3.8 of deeplearn, it looks like the shim isn't needed for 0.3.10 and later.

[nsthorat]

Adding Dan Foody's work: https://github.com/dfoody/headless-gl

Would be good to have a more "official" version, so keeping this open.

[mikeseven]

Ah yes this is using chrome webgl. I may need to update to the latest specs. Node-webgl is a native implementation using desktop opengl so there are some differences but it's faster as long as the new specs don't force me to translate data... gotta check.

…

--mike

________________________________ From: Nikhil Thorat <[email protected]> Sent: Monday, April 2, 2018 5:53:18 PM To: tensorflow/tfjs Cc: Bourges-sevenier, Mikael; Mention Subject: Re: [tensorflow/tfjs] Add support for headless OpenGL (#78) Adding Dan Foody's work: https://github.com/dfoody/headless-gl Would be good to have a more "official" version, so keeping this open. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#78 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAxYLCR3ogu43CAci9MY_3YUjjevLD14ks5tksf-gaJpZM4TESfd>.

[nkreeger]

duplicate of #811

[DavidGOrtega]

Hi,

as I said in #811 this is something that I'm actually approaching.
I'm borrowing from headless-gl and and webgl-raub.

My issues right now is that some shaders does not compile for me in openGL 2.0 and I have to change them, however the code are inside the ts files. Maybe would be a good idea to put them apart.

I have also some random thoughts.
On Windows Chrome/Chromium tfjs is actually running under DirectX through Angle and you can even specify the pure OpenGL backend VS DirectX.
However the performance is not what i expected. CPU bindings are much faster than Angle DirectX or OpenGL. When I fallback to SwiftShader the performance is like web CPU, in between 10 and 100 times slower than CPU bindings.

So, I guess that the bottleneck is right now in tfjs performance of js code which is also unfortunately slower than TVM WebGL

Why don't you speak with the SwiftShader guys? They are also googlers.

[nkreeger]

I just open-sourced the binding code - we don't have an NPM package yet: https://github.com/google/node-gles

How are you binding to ANGLE? The performance should match what Chrome does on Windows.

[DavidGOrtega]

@nkreeger super cool!

I'm reviewing the code and forking it...

How are you binding to ANGLE? The performance should match what Chrome does on Windows.

I'm not doing that yet. I said that I set Chrome on Windows as a baseline to see what to expect since Chrome uses Angle by default. I'm working to bind OpenGL.

Surprises me that you can bind to Angle on Mac!

[danbri]

Did anything ever come of these efforts?

[danbri]

Just found https://github.com/tensorflow/tfjs/tree/master/tfjs-backend-nodegl/src - seems like progress but quiet lately?

[d11wtq]

👋 I'm also very keen to know if there's a way to use a WebGL backend for nodejs. I'm on a M1 (arm64) MacBook Pro running MacOS Monterey and as far as I can tell it's impossible to get GPU acceleration on this machine, which is a shame as all new Macs are M1 now. I'd like to use Nodejs rather than running everything in the browser. Is there any way to get this going? I've been googling for a couple of days and trying various things but I think most solutions are for the Python framework, not NodeJS (e.g. Apple's Metal PluggableDevice). This WebGL-on-node idea seems like the most promising path to making this work.