This GitHub repository is part of the Master Thesis Robot Perception and Manipulation in Life Science Laboratories that was created in a cooperation between DTU Electro and Novo Nordisk.
The Thesis explores the opportunities for systems that use computer vision and robotic manipulation to fulfill tasks in life science laboratories.
Please refer to the specific folders of the project for installation details for the specific usage. In general it is assumed you are on a Linux system. The code is tested for Ubuntu 20.04 LTS with the NVIDIA Driver Version: 550.90.07 and CUDA Version: 12.4.
git clone https://github.com/hmhauter/lab-perception-and-maipulationTo further advance laboratory automation, a bridge is required to connect the unordered human workspace with the structured world of automation equipment. This work focuses on 96-well plates, commonly used in research labs, and presents a robotic system to transfer these plates, for example, from a laboratory workbench to a liquid handler. The developed system reduces the time and effort needed by researchers by automating the pick-and-place tasks of 96-well plates.
This study explores the integration of computer vision and robot manipulation in research laboratories using accessible hardware. The robot observes its environment with a camera, detecting plates using a You Only Look Once (YOLO) model trained on mainly synthetic data. The depth and segmented images are combined to accurately estimate the plate's pose with point cloud registration for precise grasping by the parallel gripper. Optical character recognition (OCR) is employed to determine the correct plate orientation by identifying the characters
Results demonstrate a robust detection model capable of adapting to varying laboratory conditions. Additionally, a pick-and-place routine specifically tailored to 96-well plates is developed with an accuracy of 92% for placing the plate in ±2 mm. The work brings novelty to the state of the art by the high-accuracy OCR on embossed, engraved and printed characters on transparent and translucent plates, achieving a 97% success rate in correct plate rotation determination with the a simple ring light setup. Additionally, the work serves as a first investigation for future improvement in optimizing camera positions for character visibility using active vision techniques.
The project delivers an easy extendable setup based on the Robot Operating System (ROS) that is tested under real-world conditions, successfully replicating researcher tasks with an overall success rate of 87%. This proof-of-concept highlights the potential of combining computer vision and robotic manipulation in laboratory settings, facilitating future enhancements such as integrating a mobile base for automating routine tasks.
An overview of the compnents and subsystems is summarized in the following Figure:
An explanation of the system can be found in the following video:
0001-4146.mp4
This project is licensed under the terms of the MIT license.
This project was developed in collaboration with Novo Nordisk, a global healthcare company that is dedicated to improving the lives of people with diabetes and other chronic diseases. Novo Nordisk provided support and resources for this project, including expert guidance and access to equipment and facilities.
For any inquiries or feedback, feel free to contact us:
- LinkedIn: Helena Hauter
- Website: Novo Nordisk