Fisheye sensor

[annaornatskaya]

Description

Added a new sensor - fisheye camera.

Fisheye image example:

Fisheye camera works better with the following BP_Sky settings:

BP_SKY -> PostProcces -> PostProccesVolume:

1. Film -> Toe = 0
2. Exposure -> MinEV100 = 6
3. Exposure -> MaxEV100 = 6
4. ImageEffects -> Vignette intensity = 0
5. RenderingFeatures -> AmbientOcclusion -> Intencity = 0
6. Screen space reflection -> Intencity=0
7. Light Propagation Volume -> Intencity = 0

As an example fisheye sensor is added to the sensors list in manual_control.py and can be used like other camera-sensors.
['sensor.camera.fisheye', cc.Raw, 'Camera Fisheye', {}]

Fisheye sensor attributes:

• x_size, y_size - image size
• max_angle - maximum angle
• d1 - d4 - distortion coefficients
• c_x, c_y - optical center
• f_x, f_y - focal lengths

For ex. (attributes used for images above):
bp.set_attribute('x_size', str(1280))

bp.set_attribute('y_size', str(720))

bp.set_attribute('max_angle', str(210))

bp.set_attribute('d_1', str(0.08309221636708493))

bp.set_attribute('d_2', str(0.01112126630599195))

bp.set_attribute('d_3', str(-0.008587261043925865))

bp.set_attribute('d_4', str(0.0008542188930970716))

bp.set_attribute('f_x', str(320))

bp.set_attribute('f_y', str(320))

bp.set_attribute('c_x', str(640))

bp.set_attribute('c_y', str(480))

In UE4_patch_fisheye-sensor.zip you can find a patch for UE4 which is obligatory for the fisheye sensor.
UE4_patch_fisheye-sensor.zip

Where has this been tested?

• Platform(s): Ubuntu 18.04
• Python version(s): 3.6
• Unreal Engine version(s): 4.24

---

This change is 

[bernatx]

Thanks Anna for this PR. These days there is no much activity on the team due holidays, but we will take a look as soon we are back. Regards.

[guntherkrehl]

Hi there,

Really impressive results! I love it and would love to see it get into the next release :)
Is it possible to also add the semantic segmentation and depth camera as fisheye returns as well?

Best,
Gunther

[cassfalg]

That sounds interesting. I did something similar by manually creating cube map images (adding 6 90° fov sensors per intended fisheye camera) and generating the fisheye images separately. That is potentially a bit more flexible for creating machine learning datasets since I can extract images and create various different fisheye lens models out of them as needed. Still, this seems to be a more elegant and integrated solution.

Can you please elaborate a bit what you're doing here? Especially what fisheye camera model you're using? What parameters would I need to choose to get a perfect equidistant fisheye lens with 210 degrees field of view for example?

Having glanced at the code, it seems like you created a shader that does the necessary computations to directly generate the fisheye image on the GPU?

I am confused by this code part in CubemapTexturePropertiesFisheye (from the UE4 patch):

+    for (int j = 0; j < 10; j++ )
+    {
+        float th2 = theta * theta;
+        float th4 = th2 * th2;
+        float th6 = th4 * th2;
+        float th8 = th6 * th2;
+        theta = r / (1.0f + d1*th2 + d2*th4 + d3*th6 + d4*th8);
+    }

The for loop does not seem to be necessary, as it calculates the same thing 10 times?

I have seem similar even degreed polynomials in OpenCV, but they don't correspond to what I would expect from the inverse lens mapping function in a fisheye lens. If we take this diagram of terms:
camera_coordinates_definitions.pdf
Then for equidistant projection the radius r is proportional to theta, with the focal length as scaling factor. Other lens mapping functions that I've seen in automotive industry use polynomials as lens mapping functions, with a degree of 4-6. How would that be mapped to this?

Have you thought about caching the displacement map / the direction vector for use in subsequent images? To me it looks like you calculate the direction vector for each pixel in each image that is generated. Since the direction only depends on the camera intrinsics they do not change from image to image. That would save the trigonometric calculations.

As to the vignetting effects in the first image, I got rid of those by changing some render settings in ./CarlaUE4/Config/DefaultEngine.ini:

[...]
[/Script/Engine.RendererSettings]
[...]
r.BlackBorders=0
r.DepthOfFieldQuality=0 # fisheye cameras don't really have that
r.DisableDistortion=1
r.MotionBlurQuality=0
r.SceneColorFringeQuality=0 # might be chromatic aberrations
r.Tonemapper.Quality=0 # might be vignette

[germanros1987]

@annaornatskaya we have found certain issues with your PR, could we discuss possible changes/ideas?

[annaornatskaya]

Hi there,

Really impressive results! I love it and would love to see it get into the next release :)
Is it possible to also add the semantic segmentation and depth camera as fisheye returns as well?

Best,
Gunther

Hello Gunther,

It would be great to have segmentation and depth fisheyes. 
For segmentation and depth 2D cameras post process materials are used, so it should be find another way to get the segmentation and depth information for the fisheye camera (based on ue4 scene capture cube). If you have any ideas I’d appreciate you sharing them.

Now for the segmentation I use GTMaterial as post process in BP_Sky -> PostProcessing. It doesn’t work like the real segmentation(objects are still segmented but colours don't match with segmentation camera colours) but it still can help in some cases.

[annaornatskaya]

Can you please elaborate a bit what you're doing here? Especially what fisheye camera model you're using?

Kannala-Brandt camera model.

What parameters would I need to choose to get a perfect equidistant fisheye lens with 210 degrees field of view for example?

d_1, d_2, d_3, d_4 = 0
max_angle = 210

Having glanced at the code, it seems like you created a shader that does the necessary computations to directly generate the fisheye image on the GPU?

That's true.

I am confused by this code part in CubemapTexturePropertiesFisheye (from the UE4 patch):

+    for (int j = 0; j < 10; j++ )
+    {
+        float th2 = theta * theta;
+        float th4 = th2 * th2;
+        float th6 = th4 * th2;
+        float th8 = th6 * th2;
+        theta = r / (1.0f + d1*th2 + d2*th4 + d3*th6 + d4*th8);
+    }

The for loop does not seem to be necessary, as it calculates the same thing 10 times?

No, that’s not the same calculates. Every time theta becomes more accurate.

I have seem similar even degreed polynomials in OpenCV, but they don't correspond to what I would expect from the inverse lens mapping function in a fisheye lens. If we take this diagram of terms:
camera_coordinates_definitions.pdf
Then for equidistant projection the radius r is proportional to theta, with the focal length as scaling factor. Other lens mapping functions that I've seen in automotive industry use polynomials as lens mapping functions, with a degree of 4-6. How would that be mapped to this?

Have you thought about caching the displacement map / the direction vector for use in subsequent images? To me it looks like you calculate the direction vector for each pixel in each image that is generated. Since the direction only depends on the camera intrinsics they do not change from image to image. That would save the trigonometric calculations.

That’s a good idea. Now I’m looking for a better way to do it. If you have any ideas I’d appreciate you sharing them.

As to the vignetting effects in the first image, I got rid of those by changing some render settings in ./CarlaUE4/Config/DefaultEngine.ini:

[...]
[/Script/Engine.RendererSettings]
[...]
r.BlackBorders=0
r.DepthOfFieldQuality=0 # fisheye cameras don't really have that
r.DisableDistortion=1
r.MotionBlurQuality=0
r.SceneColorFringeQuality=0 # might be chromatic aberrations
r.Tonemapper.Quality=0 # might be vignette

As I understand, it will change the settings for a whole project. I use different BP_Sky settings for different cases.

[annaornatskaya]

@annaornatskaya we have found certain issues with your PR, could we discuss possible changes/ideas?

yes, @dmitry0000 and I are open to any ideas and suggestions.

[cassfalg]

Can you please elaborate a bit what you're doing here? Especially what fisheye camera model you're using?

Kannala-Brandt camera model.

I've only glanced at it to date, but found one paper about it that is an interesting read:
https://www.researchgate.net/publication/6899685_A_Generic_Camera_Model_and_Calibration_Method_for_Conventional_Wide-Angle_and_Fish-Eye_Lenses
That is what you were using? Since there are so many different ways to handle this I found it important to provide those details.

Kinda sad that they glossed over the "without loss of generality we will use an odd polynomial" part for equation 6 -- from a Math standpoint that is not obvious. I am guessing that it yields superior accuracy with fewer parameters, but the mathematician in me really would like the proof for that...

I am confused by this code part in CubemapTexturePropertiesFisheye (from the UE4 patch):

+    for (int j = 0; j < 10; j++ )
+    {
+        float th2 = theta * theta;
+        float th4 = th2 * th2;
+        float th6 = th4 * th2;
+        float th8 = th6 * th2;
+        theta = r / (1.0f + d1*th2 + d2*th4 + d3*th6 + d4*th8);
+    }

The for loop does not seem to be necessary, as it calculates the same thing 10 times?

No, that’s not the same calculates. Every time theta becomes more accurate.

Oh, silly me, really... I am guessing that you're calculating the inverse here? Since you need the inverse lens mapping function for the image to world mapping and got supplied the world to image lens mapping function?

You could also think about calculating the inverse lens mapping function somewhere once -- or have the user supply it. In my application I calculated a numerical approximation of the inverse via a polyfit algorythm. An odd polynomial would probably also work for that (educated guess, the math probably works out, but have not really looked into it). It might also be worth it to handle equidistant projection as a special case, since in many use cases it might be sufficient, and for equidistant projection since it's linear you can easily calculate the (accurate) inverse. In my case the whole lens mapping function was a template parameter for C++ code, a functor or lambda or so that got called with theta = lensMappingFunction(uDistance, vDistance) basically. I did offset and aspect ratio correction before calling it.

Have you thought about caching the displacement map / the direction vector for use in subsequent images? To me it looks like you calculate the direction vector for each pixel in each image that is generated. Since the direction only depends on the camera intrinsics they do not change from image to image. That would save the trigonometric calculations.

That’s a good idea. Now I’m looking for a better way to do it. If you have any ideas I’d appreciate you sharing them.

In my case I had the 6 cube mapping images, so I created a displacement map as a 3 channel 32 bit integer "image" (OpenCV dispMap = new Mat_<Vec3i>(fisheyeRes, CV_32SC3)). Each "pixel" in this displacement map image tells me where in the cube map images to look for the source pixel of the fisheye pixel. One channel has the source image (1-6), the other two have the u and v pixel coordinates into that cube map image. Not sure if that works for your method, I did not dig that deep. You could alternatively save the vector for each fisheye pixel, or the polar coordinates (theta, phi) or so.

As to the vignetting effects in the first image, I got rid of those by changing some render settings in ./CarlaUE4/Config/DefaultEngine.ini:

[...]
[/Script/Engine.RendererSettings]
[...]
r.BlackBorders=0
r.DepthOfFieldQuality=0 # fisheye cameras don't really have that
r.DisableDistortion=1
r.MotionBlurQuality=0
r.SceneColorFringeQuality=0 # might be chromatic aberrations
r.Tonemapper.Quality=0 # might be vignette

As I understand, it will change the settings for a whole project. I use different BP_Sky settings for different cases.

Yes. Not an issue for my case.

About depth images: Please be aware of issues #2494 and #2287. I'm not sure if that is still the case or not.

[germanros1987]

@annaornatskaya we have found certain issues with your PR, could we discuss possible changes/ideas?

yes, @dmitry0000 and I are open to any ideas and suggestions.

The most effective approach would be to have a discussion in our Discord network. Please, could you sign up and send me a private message? PM: germanros

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[cassfalg]

Just curious if there has been any progress with this PR? Hence the bump.

[jimmyw404]

I'm also very interested in this pull request being merged!!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jimmyw404]

I am still very interested in this!

[mistoFENG]

Hi @annaornatskaya ,
I used your fisheye camera and found that when I add two more, the FPS will be very low, do you know what is going on, thank you！My GPU is 1080Ti

[mittkongg]

Hi @annaornatskaya , I used your fisheye camera and found that when I add two more, the FPS will be very low, do you know what is going on, thank you！My GPU is 1080Ti

可以用

[crabiner]

I'm also interested in this PR being merged!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[konu-droid]

Will this change be merged into the main branch ? Interested in having fisheye camera for automated parking application testing

[rreddy78]

When I try to apply the patch I get an error:

UnrealEngine_4.26$ git apply --summary UE4_patch_fisheye-sensor.patch
error: corrupt patch at line 99

Any idea why ?

Same is the case even with 4.24:

UnrealEngine_4.24$ git status
On branch 4.24
nothing to commit, working tree clean
sxv1kor@BMH7160115:~/Downloads/GitHub/UnrealEngine_4.24$ git apply --summary UE4_patch_fisheye-sensor.patch
error: corrupt patch at line 99

Also last change in the patch file seems to do nothing:

@@ -951,7 +953,7 @@ void FScene::UpdateSceneCaptureContents(USceneCaptureComponentCube* CaptureCompo
 				ENQUEUE_RENDER_COMMAND(CaptureCommand)(
 					[SceneRenderer, TextureRenderTarget, EventName, TargetFace](FRHICommandListImmediate& RHICmdList)
 				{
-					UpdateSceneCaptureContent_RenderThread(RHICmdList, SceneRenderer, TextureRenderTarget, TextureRenderTarget, EventName, FResolveParams(FResolveRect(), TargetFace), false, FGenerateMipsParams());
+					UpdateSceneCaptureContent_RenderThread(RHICmdList, SceneRenderer, TextureRenderTarget, TextureRenderTarget, EventName, FResolveParams(FResolveRect(), TargetFace), false, FGenerateMipsParams()); 

Hello @annaornatskaya:
I could apply the patch changes to Unreal 4.24 and 4.26, but on 4.26 I get build errors even though the files have not changed:
#5678 (comment)
Could you please post the patch for 4.26 if possible. Thank you..

EDIT:
Solved it. Use RHICmdList.GetBoundVertexShader() instead GetVertexShader()
RHICmdList.GetBoundPixelShader() instead of GetPixelShader()

• GraphicsPSOInit.BoundShaderState.VertexShaderRHI = VertexShader.GetVertexShader();
• GraphicsPSOInit.BoundShaderState.PixelShaderRHI = PixelShader.GetPixelShader();

[rreddy78]

Hi @annaornatskaya

Do you know what this build error that I am getting could be caused due to ?

Assertion failed: AlignedOffset == TypeLayout.Size [File:/home/sxv1kor/Downloads/GitHub/UnrealEngine_4.26/Engine/Source/Runtime/Core/Private/Serialization/MemoryImage.cpp] [Line: 224]
  [FCubemapTexturePropertiesVSFisheye] Calculated Size: 280, Real Size: 288
Took 0.942226s to run UE4Editor, ExitCode=133

EIDT:

Might be due to the lack of LAYOUT_FIELD macros:
https://forums.unrealengine.com/t/assertion-failed-alignedoffset-typelayout-size/243865

[rreddy78]

Hi @annaornatskaya

I have built the code with Unreal 4.26 and testing with latest CARLA. I have defined the fisheye camera in the carla-ros-bridge as follows:

                {
                    "type": "sensor.camera.fisheye",
                    "id": "fisheye",
                    "spawn_point": {"x": 2.0, "y": 0.06, "z": 2.0, "roll": 0.0, "pitch": 0.0, "yaw": 0.0},
                    "x_size": 1280,
                    "y_size": 720,
                    "max_angle": 210,
                    "d_1": 0.08309221636708493,
                    "d_2": 0.01112126630599195,
                    "d_3": -0.008587261043925865,
                    "d_4": 0.0008542188930970716,
                    "f_x": 320,
                    "f_y": 320,
                    "c_x": 640,
                    "c_y": 480
                },

But I get the following warnings I the sensor does not receive images:

[INFO] [1675075694.702107, 6.685779]: Object (type='sensor.camera.fisheye', id='fisheye') spawned successfully as 50.

[WARN] [1675075694.780804, 6.685779]: Created Unsupported Camera Actor(id=50, type=sensor.camera.fisheye, attributes={'d_4': '0.0008542188930970716', 'd_3': '-0.008587261043925865', 'd_2': '0.01112126630599195', 'c_y': '480', 'd_1': '0.08309221636708493', 'c_x': '640', 'f_y': '320', 'f_x': '320', 'y_size': '720', 'max_angle': '210', 'x_size': '1280', 'role_name': 'fisheye'})

[INFO] [1675075694.820235, 6.685779]: Created ActorControl(id=10003)
[WARN] [1675075695.916660, 6.735779]: Camera(50): Expected Frame 251 not received
[WARN] [1675075696.968695, 6.785779]: Camera(50): Expected Frame 252 not received
[WARN] [1675075698.030259, 6.835779]: Camera(50): Expected Frame 253 not received
[WARN] [1675075699.090293, 6.885779]: Camera(50): Expected Frame 254 not received
[WARN] [1675075700.135342, 6.935779]: Camera(50): Expected Frame 255 not received

[Qdriving]

When I try to apply the patch I get an error:

UnrealEngine_4.26$ git apply --summary UE4_patch_fisheye-sensor.patch
error: corrupt patch at line 99

Any idea why ?

Same is the case even with 4.24:

UnrealEngine_4.24$ git status
On branch 4.24
nothing to commit, working tree clean
sxv1kor@BMH7160115:~/Downloads/GitHub/UnrealEngine_4.24$ git apply --summary UE4_patch_fisheye-sensor.patch
error: corrupt patch at line 99

Also last change in the patch file seems to do nothing:

@@ -951,7 +953,7 @@ void FScene::UpdateSceneCaptureContents(USceneCaptureComponentCube* CaptureCompo
 				ENQUEUE_RENDER_COMMAND(CaptureCommand)(
 					[SceneRenderer, TextureRenderTarget, EventName, TargetFace](FRHICommandListImmediate& RHICmdList)
 				{
-					UpdateSceneCaptureContent_RenderThread(RHICmdList, SceneRenderer, TextureRenderTarget, TextureRenderTarget, EventName, FResolveParams(FResolveRect(), TargetFace), false, FGenerateMipsParams());
+					UpdateSceneCaptureContent_RenderThread(RHICmdList, SceneRenderer, TextureRenderTarget, TextureRenderTarget, EventName, FResolveParams(FResolveRect(), TargetFace), false, FGenerateMipsParams()); 

Hello @annaornatskaya: I could apply the patch changes to Unreal 4.24 and 4.26, but on 4.26 I get build errors even though the files have not changed: #5678 (comment) Could you please post the patch for 4.26 if possible. Thank you..

EDIT: Solved it. Use RHICmdList.GetBoundVertexShader() instead GetVertexShader() RHICmdList.GetBoundPixelShader() instead of GetPixelShader()

• GraphicsPSOInit.BoundShaderState.VertexShaderRHI = VertexShader.GetVertexShader();
• GraphicsPSOInit.BoundShaderState.PixelShaderRHI = PixelShader.GetPixelShader();

Hello rreddy78,
I am follow your step, but still get error: corrupt patch at line 99 in UE4.26. Is there any ohter modify needed?

Thanks for your help.

[rreddy78]

I am follow your step, but still get error: corrupt patch at line 99 in UE4.26. Is there any ohter modify needed?

I didn't use the patch as it is. It does not work. I manually took out the changes from the patch file and made the changes in corresponding files.

[Qdriving]

I am follow your step, but still get error: corrupt patch at line 99 in UE4.26. Is there any ohter modify needed?

I didn't use the patch as it is. It does not work. I manually took out the changes from the patch file and made the changes in corresponding files.

OK, I know.

Thanks.

[dongkeyan]

Hi @annaornatskaya

I have built the code with Unreal 4.26 and testing with latest CARLA. I have defined the fisheye camera in the carla-ros-bridge as follows:

                {
                    "type": "sensor.camera.fisheye",
                    "id": "fisheye",
                    "spawn_point": {"x": 2.0, "y": 0.06, "z": 2.0, "roll": 0.0, "pitch": 0.0, "yaw": 0.0},
                    "x_size": 1280,
                    "y_size": 720,
                    "max_angle": 210,
                    "d_1": 0.08309221636708493,
                    "d_2": 0.01112126630599195,
                    "d_3": -0.008587261043925865,
                    "d_4": 0.0008542188930970716,
                    "f_x": 320,
                    "f_y": 320,
                    "c_x": 640,
                    "c_y": 480
                },

But I get the following warnings I the sensor does not receive images:

[INFO] [1675075694.702107, 6.685779]: Object (type='sensor.camera.fisheye', id='fisheye') spawned successfully as 50.

[WARN] [1675075694.780804, 6.685779]: Created Unsupported Camera Actor(id=50, type=sensor.camera.fisheye, attributes={'d_4': '0.0008542188930970716', 'd_3': '-0.008587261043925865', 'd_2': '0.01112126630599195', 'c_y': '480', 'd_1': '0.08309221636708493', 'c_x': '640', 'f_y': '320', 'f_x': '320', 'y_size': '720', 'max_angle': '210', 'x_size': '1280', 'role_name': 'fisheye'})

[INFO] [1675075694.820235, 6.685779]: Created ActorControl(id=10003) [WARN] [1675075695.916660, 6.735779]: Camera(50): Expected Frame 251 not received [WARN] [1675075696.968695, 6.785779]: Camera(50): Expected Frame 252 not received [WARN] [1675075698.030259, 6.835779]: Camera(50): Expected Frame 253 not received [WARN] [1675075699.090293, 6.885779]: Camera(50): Expected Frame 254 not received [WARN] [1675075700.135342, 6.935779]: Camera(50): Expected Frame 255 not received

@rreddy78
Hi, I have the same problem. I can't get image raw data by "sensor.listen" function, have you solved this problem?

[rreddy78]

@rreddy78 Hi, I have the same problem. I can't get image raw data by "sensor.listen" function, have you solved this problem?
Yes. You have to add a FisheyeCamera type in camera.py and adapt accordingly. Please see the representative changes below

git diff carla_ros_bridge/src/carla_ros_bridge/actor_factory.py
diff --git a/carla_ros_bridge/src/carla_ros_bridge/actor_factory.py b/carla_ros_bridge/src/carla_ros_bridge/actor_factory.py
index a696355..997d795 100755
--- a/carla_ros_bridge/src/carla_ros_bridge/actor_factory.py
+++ b/carla_ros_bridge/src/carla_ros_bridge/actor_factory.py
@@ -23,7 +23,7 @@ import carla_common.transforms as trans
 from carla_ros_bridge.actor import Actor
 from carla_ros_bridge.actor_control import ActorControl
 from carla_ros_bridge.actor_list_sensor import ActorListSensor
-from carla_ros_bridge.camera import Camera, RgbCamera, DepthCamera, SemanticSegmentationCamera, DVSCamera
+from carla_ros_bridge.camera import Camera, RgbCamera, DepthCamera, SemanticSegmentationCamera, DVSCamera, FisheyeCamera
 from carla_ros_bridge.collision_sensor import CollisionSensor
 from carla_ros_bridge.ego_vehicle import EgoVehicle
 from carla_ros_bridge.gnss import Gnss
@@ -381,6 +381,9 @@ class ActorFactory(object):
                 elif carla_actor.type_id.startswith("sensor.camera.dvs"):
                     actor = DVSCamera(uid, name, parent, spawn_pose, self.node,
                                       carla_actor, self.sync_mode)
+                elif carla_actor.type_id.startswith("sensor.camera.fisheye"):
+                    actor = FisheyeCamera(uid, name, parent, spawn_pose, self.node,
+                                      carla_actor, self.sync_mode)
                 else:
                     actor = Camera(uid, name, parent, spawn_pose, self.node,
                                    carla_actor, self.sync_mode)
sxv1kor@BANO-C-0012R:~/carla-ros-bridge/catkin_ws/src/ros-bridge$ git diff carla_ros_bridge/src/carla_ros_bridge/camera.py
diff --git a/carla_ros_bridge/src/carla_ros_bridge/camera.py b/carla_ros_bridge/src/carla_ros_bridge/camera.py
index c9404e2..0e852ee 100644
--- a/carla_ros_bridge/src/carla_ros_bridge/camera.py
+++ b/carla_ros_bridge/src/carla_ros_bridge/camera.py
@@ -91,8 +91,12 @@ class Camera(Sensor):
         camera_info = CameraInfo()
         # store info without header
         camera_info.header = self.get_msg_header()
-        camera_info.width = int(self.carla_actor.attributes['image_size_x'])
-        camera_info.height = int(self.carla_actor.attributes['image_size_y'])
+        if self.__class__.__name__ == "FisheyeCamera":
+            camera_info.width = int(self.carla_actor.attributes['x_size'])
+            camera_info.height = int(self.carla_actor.attributes['y_size'])
+        else:
+            camera_info.width = int(self.carla_actor.attributes['image_size_x'])
+            camera_info.height = int(self.carla_actor.attributes['image_size_y'])

[dongkeyan]

@rreddy78 Hi, I have the same problem. I can't get image raw data by "sensor.listen" function, have you solved this problem?
Yes. You have to add a FisheyeCamera type in camera.py and adapt accordingly. Please see the representative changes below

git diff carla_ros_bridge/src/carla_ros_bridge/actor_factory.py
diff --git a/carla_ros_bridge/src/carla_ros_bridge/actor_factory.py b/carla_ros_bridge/src/carla_ros_bridge/actor_factory.py
index a696355..997d795 100755
--- a/carla_ros_bridge/src/carla_ros_bridge/actor_factory.py
+++ b/carla_ros_bridge/src/carla_ros_bridge/actor_factory.py
@@ -23,7 +23,7 @@ import carla_common.transforms as trans
 from carla_ros_bridge.actor import Actor
 from carla_ros_bridge.actor_control import ActorControl
 from carla_ros_bridge.actor_list_sensor import ActorListSensor
-from carla_ros_bridge.camera import Camera, RgbCamera, DepthCamera, SemanticSegmentationCamera, DVSCamera
+from carla_ros_bridge.camera import Camera, RgbCamera, DepthCamera, SemanticSegmentationCamera, DVSCamera, FisheyeCamera
 from carla_ros_bridge.collision_sensor import CollisionSensor
 from carla_ros_bridge.ego_vehicle import EgoVehicle
 from carla_ros_bridge.gnss import Gnss
@@ -381,6 +381,9 @@ class ActorFactory(object):
                 elif carla_actor.type_id.startswith("sensor.camera.dvs"):
                     actor = DVSCamera(uid, name, parent, spawn_pose, self.node,
                                       carla_actor, self.sync_mode)
+                elif carla_actor.type_id.startswith("sensor.camera.fisheye"):
+                    actor = FisheyeCamera(uid, name, parent, spawn_pose, self.node,
+                                      carla_actor, self.sync_mode)
                 else:
                     actor = Camera(uid, name, parent, spawn_pose, self.node,
                                    carla_actor, self.sync_mode)
sxv1kor@BANO-C-0012R:~/carla-ros-bridge/catkin_ws/src/ros-bridge$ git diff carla_ros_bridge/src/carla_ros_bridge/camera.py
diff --git a/carla_ros_bridge/src/carla_ros_bridge/camera.py b/carla_ros_bridge/src/carla_ros_bridge/camera.py
index c9404e2..0e852ee 100644
--- a/carla_ros_bridge/src/carla_ros_bridge/camera.py
+++ b/carla_ros_bridge/src/carla_ros_bridge/camera.py
@@ -91,8 +91,12 @@ class Camera(Sensor):
         camera_info = CameraInfo()
         # store info without header
         camera_info.header = self.get_msg_header()
-        camera_info.width = int(self.carla_actor.attributes['image_size_x'])
-        camera_info.height = int(self.carla_actor.attributes['image_size_y'])
+        if self.__class__.__name__ == "FisheyeCamera":
+            camera_info.width = int(self.carla_actor.attributes['x_size'])
+            camera_info.height = int(self.carla_actor.attributes['y_size'])
+        else:
+            camera_info.width = int(self.carla_actor.attributes['image_size_x'])
+            camera_info.height = int(self.carla_actor.attributes['image_size_y'])

Thanks for your response.
I haven't used ROS to test. I just use Carla/PythonAPI/examples/manual_control.py and this problem occurs. Do you know how to solve it?

[dongkeyan]

I find the capture render target of Fisheyesensor is 3D, example:

I config fisheye parameter: xsize=1280,ysize=720, but the UE4 CaptureRenderTarget pitcture and parameter is :

The image raw size isn't different with my configuration, so is that the reason I can't get image raw data from sensor.listen?

[lucalazzaroni]

Is there any possibility that this feature will be added to new releases of Carla?

[AHAHA124]

I am follow your step, but still get error: corrupt patch at line 99 in UE4.26. Is there any ohter modify needed?

I didn't use the patch as it is. It does not work. I manually took out the changes from the patch file and made the changes in corresponding files.

OK, I know.

Thanks.

@Qdriving
Hello, Have you solved the problem? Would you mind sharing your method？

[Hassan-BABAOUSMAIL]

Kannala-Brandt camera model.

@annaornatskaya Please, can you give me the reference of the distortion model which you have used?
especially the reference for this part:

•    theta = r / (1.0f + d1*th2 + d2*th4 + d3*th6 + d4*th8);
  

[dongkeyan]

I have a problem when using fisheye sensor model. I deploy one fisheye on ego car ,the resolution is 3840x2160. The FPS of carla is 14, but I use RGB sensor model, the fps is 28. So how to solve the low fps problem?

My computer GPU:RTX 4090, CPU: AMD 7950

[Qdriving]

I am follow your step, but still get error: corrupt patch at line 99 in UE4.26. Is there any ohter modify needed?

I didn't use the patch as it is. It does not work. I manually took out the changes from the patch file and made the changes in corresponding files.

OK, I know.
Thanks.

@Qdriving Hello, Have you solved the problem? Would you mind sharing your method？

Hello, you need just merge the modify from the patch manually.

[balckgxj]

Hi, I met the same problem, do you have a solution? Thank you for sharing!

I have a problem when using fisheye sensor model. I deploy one fisheye on ego car ,the resolution is 3840x2160. The FPS of carla is 14, but I use RGB sensor model, the fps is 28. So how to solve the low fps problem?

My computer GPU:RTX 4090, CPU: AMD 7950

[AGREATAPE]

非常感谢您分享您的代码。 但是，当我按照您的指示进行操作时，我遇到了以下问题。 我怎样才能解决这个问题？

Traceback (most recent call last): File "manual_control.py", line 1188, in <module> main() File "manual_control.py", line 1180, in main game_loop(args) File "manual_control.py", line 1096, in game_loop world = World(client.get_world(), hud, args) File "manual_control.py", line 186, in __init__ self.restart() File "manual_control.py", line 252, in restart self.camera_manager = CameraManager(self.player, self.hud, self._gamma) File "manual_control.py", line 982, in __init__ bp = bp_library.find(item[0]) IndexError: blueprint 'sensor.camera.fisheye' not found WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 87 (sensor.other.collision) WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 89 (sensor.other.gnss) WARNING: sensor object went out of the scope but the sensor is still alive in the simulation: Actor 90 (sensor.other.imu)

构建 Carla 后，我修改了 manual_control.py 并执行了所有命令，如 make import、LibCarla PythonAPI、Launch 并收到了此消息。 任何建议都会帮助我！

Are you sure you've changed the carla source code (not just ue4), because it looks like carla didn't find the fisheye model.
Please modify the carla source code according to the following content:
master...annaornatskaya:carla:fisheye-sensor

[ShyLvy]

• Unreal Engine version(s): 4.24

Hello, thank you for providing the fisheye camera patch project. Currently, I have modified the file according to https://github.com/carla-simulator/carla/issues/5678 under Unreal Engine version(s): 4.26. ubuntu20.04
Currently, I use UE4 can see the FisheyeCamera sensor, but switching the sensor to the fisheye camera when starting manual_control.py in pythonAPI will cause the program to crash, and the terminal prompts
Segmentation fault (core dumped)

After preliminary judgment, it is caused by the operation of self.sensor.listen(lambda image: CameraManager._parse_image(weak_self, image)). How can I fix this problem?

[ShyLvy]

• Unreal Engine version(s): 4.24

Hello, thank you for providing the fisheye camera patch project. Currently, I have modified the file according to https://github.com/carla-simulator/carla/issues/5678 under Unreal Engine version(s): 4.26. ubuntu20.04 Currently, I use UE4 can see the FisheyeCamera sensor, but switching the sensor to the fisheye camera when starting manual_control.py in pythonAPI will cause the program to crash, and the terminal prompts Segmentation fault (core dumped)

After preliminary judgment, it is caused by the operation of self.sensor.listen(lambda image: CameraManager._parse_image(weak_self, image)). How can I fix this problem?

我已经解决了pythonAPI的问题,在carla根目录下启动终端,执行
make clean
make PythonAPI
make launch
pythonAPI就可以正常使用了,目前正在尝试carla ros bridge

[StoneZ7]

Hi, I met the same problem, do you have a solution? Thank you for sharing!

I have a problem when using fisheye sensor model. I deploy one fisheye on ego car ,the resolution is 3840x2160. The FPS of carla is 14, but I use RGB sensor model, the fps is 28. So how to solve the low fps problem?
My computer GPU:RTX 4090, CPU: AMD 7950

do you have some solution? Thank you for sharing!

[InigoAgirre]

Hello, I manually modified the UE files for the patch but I get multiple errors when I build.

I would appreciate any idea, or if it is possible to share the files with the changes in case I made any error.
Thanks!

[kkssgg114514]

Hello, I manually modified the UE files for the patch but I get multiple errors when I build.

I would appreciate any idea, or if it is possible to share the files with the changes in case I made any error. Thanks!

You don't seem to be using UE4.24. Try to remove the inline function "Serialize", and you seem to need to replace the declaration of the private part of the newly added class with the macro LAYOUT_FIELD, which does what Serialize and declare. That's what UE4.26 does.

[jchhuang]

Description

Added a new sensor - fisheye camera.

Fisheye image example:

Fisheye camera works better with the following BP_Sky settings:

BP_SKY -> PostProcces -> PostProccesVolume:

1. Film -> Toe = 0
2. Exposure -> MinEV100 = 6
3. Exposure -> MaxEV100 = 6
4. ImageEffects -> Vignette intensity = 0
5. RenderingFeatures -> AmbientOcclusion -> Intencity = 0
6. Screen space reflection -> Intencity=0
7. Light Propagation Volume -> Intencity = 0

As an example fisheye sensor is added to the sensors list in manual_control.py and can be used like other camera-sensors. ['sensor.camera.fisheye', cc.Raw, 'Camera Fisheye', {}]

Fisheye sensor attributes:

• x_size, y_size - image size
• max_angle - maximum angle
• d1 - d4 - distortion coefficients
• c_x, c_y - optical center
• f_x, f_y - focal lengths

For ex. (attributes used for images above): bp.set_attribute('x_size', str(1280))

bp.set_attribute('y_size', str(720))

bp.set_attribute('max_angle', str(210))

bp.set_attribute('d_1', str(0.08309221636708493))

bp.set_attribute('d_2', str(0.01112126630599195))

bp.set_attribute('d_3', str(-0.008587261043925865))

bp.set_attribute('d_4', str(0.0008542188930970716))

bp.set_attribute('f_x', str(320))

bp.set_attribute('f_y', str(320))

bp.set_attribute('c_x', str(640))

bp.set_attribute('c_y', str(480))

In UE4_patch_fisheye-sensor.zip you can find a patch for UE4 which is obligatory for the fisheye sensor. UE4_patch_fisheye-sensor.zip

Where has this been tested?

• Platform(s): Ubuntu 18.04
• Python version(s): 3.6
• Unreal Engine version(s): 4.24

This change is 

what's the meaning of "max angle", does it equal to FOV?

[zdreamer1980]

add a new patch file,in order to match UE4.26.(git clone --depth 1 -b carla https://github.com/CarlaUnreal/UnrealEngine.git ~/UnrealEngine_4.26) Convenient for subsequent reproduction
0001-fisheye-pathc.patch

[IAEjeremy]

Dear all and @annaornatskaya ,
I succeed to implement the Fisheye distortion on Carla 0.9.15 and unreal 2.26 following the files modified by Junchuan Zhang here: https://github.com/GimpelZhang/carla/tree/fisheye

However, like @Hassan-BABAOUSMAIL , I am confused by the formula :
theta = r / (1.0f + d1*th2 + d2*th4 + d3*th6 + d4*th8);

I did some research and for example in Matlab and OpenCV websites which are also using and presenting KB method, the formula is:
theta = r * (1.0f + d1*th2 + d2*th4 + d3*th6 + d4*th8);
With some slights differences such as:
theta = arctan(r); and not theta = r; as provided in the patch.

• Matlab: https://www.mathworks.com/help/vision/ref/cameraintrinsicskb.html
• OpenCV: https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html

Furthermore, could someone explain me why the black borders seems not centered, it seems always "bigger"/"stronger" on the top. Why is it so ? Is there a way to remove this black border ?
Also, when the distortion coefficients are set to 0, we still have the black borders. It is like having a "normal" RGB camera frame with black borders. Is this normal ?

Thank you for your time and feedback in advance,

Br,

Jeremy