Dynamics-Guided Diffusion Model for Robot Manipulator Design

Paper | Website | Video

Xiaomeng Xu, Huy Ha, Shuran Song

Dependencies

Required packages can be installed by:

pip install -r requirements.txt

Data Preparation

Download object dataset

2D objects

Download 2D object icons from Icons50 dataset.

3D objects

Download 3D object meshes from MuJoCo scanned object dataset.

Generate simulation data

Replace OBJECT_DIR in sim/sim_2d.py and sim/sim_3d.py with the directory to object dataset.

Install v-hacd.

2D

bash sim/run_sim_2d.sh

3D

bash sim/run_sim_3d.sh

Note for data generation: Sometimes the sampled objects or manipulators may have weird shapes and thus lead to qhull error when doing convex decomposition. And ray is used to parallelize cpu-based data generation, which sometimes may lead to timeout issues. Therefore, it is expected if you see some error message and the data for some object-manipulator pairs is not generated, but it should be fine as long as you see most data is being generated.

Training

Download pretrained model checkpoints

Train Dynamics Model

2D

bash dynamics/train_dynamics_2d.sh

3D

bash dynamics/train_dynamics_3d.sh

Train Diffusion Model

2D

bash generator/train_diffusion_2d.sh

3D

bash generator/train_diffusion_3d.sh

Inference

Generate Task-Specific Manipulators

2D

bash generator/guided_sample_2d.sh

3D

bash generator/guided_sample_3d.sh

Citation

If you find DGDM useful for your work, please cite:

@misc{xu2024dynamicsguided,
	title={Dynamics-Guided Diffusion Model for Robot Manipulator Design}, 
	author={Xiaomeng Xu and Huy Ha and Shuran Song},
	year={2024},
	eprint={2402.15038},
	archivePrefix={arXiv},
	primaryClass={cs.RO}
}

Contact

If you have any questions, please feel free to contact Xiaomeng Xu ([email protected])