Proposal for Integrating React-XR with Reagraph for VR/AR Support

[thobhanifreddy]

I'd like to propose an exciting idea: integrating the react-xr library with Reagraph to introduce Virtual Reality (VR) and Augmented Reality (AR) capabilities.

Given the rising popularity of VR/AR, especially with devices like Meta Quest and Apple's Vision Pro, it's an opportune time to explore this integration. Since Reagraph already supports 3D graphs through @react-three/fiber, this addition seems not only feasible but also a natural progression in enhancing our user experience with immersive technologies.

[amcdnl]

This is something I'd love to add! I have a MQ3 and plan to get a AVP once it comes out. If you want to take a stab, I'd happily work with you to get it integrated.

[thobhanifreddy]

Hi @amcdnl, I've made some progress on integrating the react-xr library into Reagraph. I've pushed these initial changes to the react-xr branch on my fork, which you can find here: thobhanifreddy@c22c54e

So far, I've successfully enabled the rendering of Nodes in VR mode. However, I've hit a bit of a roadblock with the CameraControls, primarily due to my limited experience with three.js.

In the meantime, I've been exploring various debugging methods. The most straightforward approach I've found is to simplify complex renderings to a basic blue box and then using webxr emulator.

<div className={css.canvas}>
          <XRButton mode="VR" onError={error => console.log(error)} />
          <Canvas
            legacy
            linear
            flat
            gl={gl}
            camera={camera}
            onPointerMissed={onCanvasClick}
            vr
          >
            <XR>
              <Provider
                createStore={() =>
                  createStore({
                    selections,
                    actives,
                    theme,
                    collapsedNodeIds
                  })
                }
              >
                <Controllers />
                <Hands />
                <CameraControls
                  mode={cameraMode}
                  ref={controlsRef}
                  animated={animated}
                  disabled={disabled}
                >
                  <mesh position={[0, 0, -500]}>
                    <boxGeometry args={[100, 100, 100]} />
                    <meshBasicMaterial color="blue" />
                  </mesh>
                </CameraControls>
              </Provider>
            </XR>
          </Canvas>
        </div>

To test it in MQ3 I am running ngrok tunnel.

I would greatly appreciate any insights or suggestions you might have, especially regarding the CameraControls issue.

Looking forward to collaborating further on XR :)

[amcdnl]

You are probably going to need to use a different Camera Controls for the XR setup. This might be a good place to look for the right camera conrols - https://github.com/pmndrs/drei

[amcdnl]

We are probably going to have to add a flag for XR so we can inject more things like the hands/etc like isXR and then that can be set by the app consuming the graph.

[amcdnl]

What is that Chrome Dev Tools WebXR plugin? Is that activated when you open a XR site or something?

[thobhanifreddy]

@amcdnl

• Thanks I will look into camera control
• Yes I agree
• I am using meta quest emulator: https://github.com/meta-quest/immersive-web-emulator | https://chromewebstore.google.com/detail/immersive-web-emulator/cgffilbpcibhmcfbgggfhfolhkfbhmik