MuJoCo AR

mujoco_ar is a plugin for MuJoCo that lets you control frames using your iOS/Android device's AR data.

Recent Updates

• Android support thanks to @Lr-2002's contribution.
• Trigger vibrations on your iOS device for haptic feedback (release log).

MuJoCo Demos

Position Control
MuJoCo Fruits Pick and Place		MuJoCo PushT
Position and Rotation Control
MuJoCo Study Desk		MuJoCo Blocks Stacking

Real Demo

Examples of MuJoCo AR linked to the end-effectors of multiple manipulators can be found in this fork of Mink.

Installation

You can install mujoco_ar package using pip:

pip install mujoco_ar

You can download the app from the App Store for iOS devices or here for Android devices.

Usage

Quick MuJoCo Setup

This setup allows you to directly control a MuJoCo frame (body, geom, or site), with the frame's position and orientation matching the ARKit data received from the connected iOS device.

from mujoco_ar import MujocoARConnector

# Initialize the connector with your desired parameters
mj_ar = MujocoARConnector(mujoco_model=my_model,mujoco_data=my_data)

# Link a MuJoCo frame (link_body(), link_geom() or link_site())
mj_ar.link_body(name="eef_target")

# Start the connector
mj_ar.start()

Full MuJoCo Setup

In addition to what the quick setup allows you to do, this setup allows you to automate the applying of a translation, rotation or scaling of the recieved pose. Additionally, you can pass functions to button_fn and toggle_fn to be triggered when the button or toggle are activated

from mujoco_ar import MujocoARConnector

# Initialize the connector with your desired parameters
mj_ar = MujocoARConnector(
    mujoco_model=my_model, 
    mujoco_data=my_data, 
    port=8888,                                           # Optional, defaults to 8888 if not provided
    debug=False                                          # Optional, defaults to False if not provided
)

# Link a MuJoCo frame (link_body(), link_geom() or link_site())
mj_ar.link_body(
    name="eef_target",
    scale=1.0,                                           # Optional, defaults to 1.0 if not provided
    position_origin=np.array([0.0, 0.0, 0.0]),           # Optional, defaults to [0, 0, 0] if not provided
    rotation_origin=np.identity(3),                      # Optional, defaults to I(3) if not provided
    toggle_fn=my_toggle_function,                        # Optional, calls nothing if not provided
    button_fn=my_button_function,                        # Optional, calls nothing if not provided
    disable_pos=False,                                   # Optional, defaults to False if not provided
    disable_rot=False                                    # Optional, defaults to False if not provided
)

# Start the connector
mj_ar.start()

Flexible Setup (works without MuJoCo):

You can retrieve the ARKit data including the position, rotation, button, and toggle states directly from a connected iOS device, making it flexible for usage in various applications beyond physics simulations. Try running mjpython demos/flexible_setup.py.

from mujoco_ar import MujocoARConnector

# Initialize the connector with your desired parameters
connector = MujocoARConnector()

# Start the connector
connector.start()

# Retrieve the latest AR data (after connecting the iOS device, see the guide below)
data = connector.get_latest_data()  # Returns {"position": (3, 1), "rotation": (3, 3), "button": bool, "toggle": bool}

Additional Functions

connector.vibrate(sharpness=0.8, intensity=0.4, duration=0.01) # Trigger a vibration on the connected device
connector.pause_updates()  # Temporarily stops receiving updates from the connected device.
connector.resume_updates() # Resumes receiving updates from the connected device.
connector.reset_position() # Resets the current position as the origin (0,0,0).

FAQ

How can I reduce latency?

• If you're experiencing latency, try connecting your PC to your device's hotspot. This should significantly reduce latency if you're far from a router since the communication happens locally via WebSockets.

Can I use it for a non-MuJoCo application?

• Yes, check the Flexible Setup out where you can retrive the pure ARKit position and rotation and use it as you wish. You wouldn't need to pass in the MuJoCo model and data in such a case.

How to contribute?

Report any bugs you encounter to the issue tracker. Also, feel free to suggest any features. Those could be regarding the app (iOS swift code) or the python package.

Acknowledgements

• Thanks to @Lr-2002 for creating an Android app that works with the package.
• Thanks to @kevinzakka for the mink and mjctrl libraries, which are used in the provided demos.

Author

Omar Rayyan ([email protected])