Level with examples demonstrating streaming depth, color (IR and external RGB) and pointcloud in Unreal Engine 4.
UnrealEngine4 wrapper shows how to use RealSense features via Blueprints (UE4 version 4.19+). At the moment it provides next functionality:
- enumerate devices/sensors/options/profiles
- load and save configuration presets
- tune sensor options
- access live sensor streams via textures
- record and playback sensor streams
- process raw data in shader
- align depth/color streams
- generate and render point cloud mesh
Download and install RealSense SDK 2.0
Download wrapper and generate project files for RealSenseUE.uproject
Copy C:\Program Files (x86)\Intel RealSense SDK 2.0\bin\x64\realsense2.dll to either:
- The engine bin folder C:\Program Files\Epic Games\UE_4.19\Engine\Binaries\Win64\ OR
- The plugin build folder RealSense-UE4-Wrapper\Plugins\RealSense\Binaries\Win64\
In case RealSense SDK was installed in custom folder:
- edit file RealSense-UE4-Wrapper\Plugins\RealSense\Source\RealSense\RealSense.Build.cs
- update line
string RealSenseDirectory = "REALSENSE_SDK_PATH_HERE"
Build RealSenseUE.sln using DevelopmentEditor configuration
Start and play RealSenseUE.uproject to see some live data
Check out prefab BP_RealSenseAPI, it shows functionality exposed to Blueprints and a few examples:
- ExEnumerateHardware - get connected devices/sensors and print some info
- ExModifyOption - modify specific sensor option
- ExPresetLoadSave - load and save device configuration preset
- ExStreamConfiguration - enable/disable streams, set video mode
- ExRecordPlayback - record and playback demo
Check out prefab BP_RealSenseInspector, it simplifies stream configuration and management, also binds live data to scene objects. BP_RealSenseAPI should be placed to scene as well if new one is being created.
Hardware inspector allows to check connected hardware and tune sensor options via editor. This feature is for debugging only and changes are not saved automatically. It's possible to load and save configuration presets using corresponding buttons in hardware inspector dropdown or via blueprint functions.
Preset files can be downloaded here.
Picture: hardware inspector
DeviceSerial allows to select specific device by serial. If empty default device will be used.
PipelineMode switches between CaptureOnly/Record/Playback:
- CaptureOnly - fetch and render live data
- Record - fetch and render live data but also record it to specified file
- Playback - load previously recorded file and loop it content
EnablePolling switches poll/wait update mode. Poll mode is checking for data with specified rate, wait mode is blocking until data is available.
Picture: mode inspector
Each stream has it's own dropdown with enable/disable flag and video mode selector. These options should be configured before starting RealSenseInspector. Restarting is required to change video mode. Enable/disable flag can be used to pause/resume already running stream.
Depth can provide two textures: raw and colorized. Colorized depth is soft-processed based on Equalize/MinMax/Colormap settings. Avoid using colorized depth on high resolutions/rates.
In case running both depth and infrared they should be configured to same video mode.
Picture: depth stream inspector
Point cloud support is experimental and has a lot of flaws. Current implementation uses third-party plugin for dynamic mesh generation and should be optimized.
- PclScale - allows to scale the whole cloud
- PclVoxelSize - single point/voxel size in mm
- PclDensity - percentage of rendered points
- PclRenderRate - how often the PCL mesh is updated (0.5 means two times per second)
Picture: point cloud inspector
RealSenseContext is global object initialized at plugin load. It wraps native RealSense library and provides device management functions.
Picture: get context blueprint
After getting the context devices can be enumerated. Each device contains a few sensors, each sensor has some options. Sensor profile contains video mode and frame rate.
Picture: enumerate devices blueprint
Picture: enumerate sensors blueprint
Picture: enumerate options blueprint
Sensor options can be tuned one by one or using preset load/save functions.
Picture: get and set options value
Picture: save and load preset blueprint
RealSenseInspector provides stream configuration and management functions. It updates textures and PCL mesh.
Picture: find Inspector blueprint
Properties described in StreamConfiguration section also can be modified from Blueprints.
Picture: modify stream mode blueprint
After configuration RealSenseInspector should be started with any PipelineMode.
Picture: set mode blueprint
Time to get texture objects and PCL mesh.
Picture: get stream texture blueprint
Picture: transform PCL mesh blueprint
It's also possible to access wrapper from native code, check RealSenseTestActor.cpp for example:
#include "RealSensePlugin.h"
#include "RealSenseContext.h"
#include "RealSenseDevice.h"
#include "RealSenseNative.h"
ARealSenseTestActor::ARealSenseTestActor()
{
auto Context = IRealSensePlugin::Get().GetContext();
Context->QueryDevices();
for (auto Device : Context->Devices)
{
UE_LOG(LogRealSenseDemo, Display, TEXT("Wrapper device %s"), *(Device->Name));
}
rs2::context_ref RsContext(Context->GetHandle());
auto DeviceList = RsContext.query_devices();
for (auto Device : DeviceList)
{
FString DevName(ANSI_TO_TCHAR(Device.get_info(RS2_CAMERA_INFO_NAME)));
UE_LOG(LogRealSenseDemo, Display, TEXT("Native device %s"), *DevName);
}
}- Frame drop/timeout on high resolutions might be laggy (depends on the hosts hardware)
- High CPU usage (RealSenseSDK issue)
- ColorizedDepth is CPU bound
- PointCloud (PCL) has sometimes texture corruptions at thhe egdes (RealSenseSDK issue)
- PointCloud (PCL) update/rendering performance is somtime slow
- rebuilding project files can sometimes fail after switching Unreal Engine versions. Cleaning the solution or Deleting the intermediate directory should fix this.
This project is licensed under the Apache License, Version 2.0.