Extracted Images are Completely Black

[joachim-rtr]

Bug report

• AirSim Version/#commit: branch ue5.1.1, e6cf02b
• UE/Unity version: 5.1.1
• autopilot version: N/A
• OS Version: Windows10

What's the issue you encountered?

Clean build of Airsim and Blocks. The Blocks environment is started in ComputerVision-Mode and Colosseum/PythonClient/computer_vision/cv_mode.py is executed using Python 3.7.4. The camera moves in the Unreal Engine and images are created in the specified folder. However, all images are completely black. This is the case for all arisim.ImageTypes.

Settings

{
"SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/main/docs/settings.md",
"SettingsVersion": 1.2,
"SimMode": "ComputerVision",
"ViewMode": "Fpv" ,
"CameraDefaults": {
"CaptureSettings": [
{
"ImageType": 0,
"Width": 720,
"Height": 567,
"FOV_Degrees": 42
},
{
"ImageType": 2,
"Width": 720,
"Height": 567,
"FOV_Degrees": 42
},
{
"ImageType": 5,
"Width": 720,
"Height": 567,
"FOV_Degrees": 42
}
]
}
}

How can the issue be reproduced?

See description above:

1. New installation and build of AirSim
2. Run Blocks in UE5.1.1
3. Run the default script "Colosseum/PythonClient/computer_vision/cv_mode.py" using Python 3.7.4
4. Images created in the specified folder are completely black

Include full error message in text form

N/A

Any help is appreciated! :)

[ak6560]

I am also facing the same issue on Blocks and my own costume environments. When I hit record, I notice a frequent jitter in the drone motion and it disappears if the recording stops. I can capture images with no issues in the prebuild TrapCamera environment though. @joachim-rtr did you manage to solve it?

[git-akispa]

Also facing the same issue. Has anyone found a fix?

[git-akispa]

Ok, I found a fix, it is not perfect but it works.

The problem with the photos is the encoding of the alpha channel as can be seen here

So what I did is run this command on a Linux machine: for i in ls *.png; do convert $i -channel alpha -negate result+$i; done

And what it does is that it removes the alpha channel completely, and you have usable images!

It would be great if the script for the is fixed! thanks everyone

[martingbl]

I cannot verify if I am confronted with the same issue but in Unreal itself the whole environment is lit properly but when trying to access images by recording through airsim textures of buildings are not lit properly.

When I disable lumen I receive the exact same result in the editor as in the airsim screenshot outputs. Therefore I assume lumen might not be compatible with airsim/colosseum

[Rong-Tao]

I encountered both problems. In my recorded images I get black images and in my preview, I get improperly lumened images

[git-akispa]

I encountered both problems. In my recorded images I get black images and in my preview, I get improperly lumened images

What exactly is improperly lit, the feed in the game or do you "un-black" the images and then?

[Rong-Tao]

Hi, I have been struggling for days on this if you can help me, It would be massive.
First of all. I can confirm that the 'un-black' works. When I hit the record button and the images would be black and by removing the A channel works.
However, As you can see here, there are black stains and improperly lumined pixels in the subwindow but the original FPV is fine.

This image is obtained using 'unblack' and it is clear that the lumin is somehow broken.

Also, I usually take images using the python API, which does not go through the RGBA process. And the code is attached at the end.
The output from responses[0] is like all b'ff ff ff' all white, and I am getting completely white images. I don't know if it is the issue of lighting or eye adaptation, but I turned it off and set the lighting to be 12 min/max using a post processing shader.
https://drive.google.com/file/d/1ylfs-xVgpOdL7241PkhVtCmKqGcKbMYG/view?usp=sharing
My project is here, it is basically an archviz.
UE Version: 5.2.1
sdk: 10.0.22621

def image_save(client, base_folder = 'img'):
    # Pause the simulation
    
    # Request various types of images
    responses = client.simGetImages([
        airsim.ImageRequest("front_left", airsim.ImageType.Scene, False, False),
        airsim.ImageRequest("front_left", airsim.ImageType.DepthPerspective, True),
        airsim.ImageRequest("front_left", airsim.ImageType.Segmentation, False, False),
        airsim.ImageRequest("front_left", airsim.ImageType.SurfaceNormals, False, False),
        airsim.ImageRequest("front_right", airsim.ImageType.Scene, False, False),
        airsim.ImageRequest("front_right", airsim.ImageType.DepthPerspective, True),
        airsim.ImageRequest("front_right", airsim.ImageType.Segmentation, True, False),
        airsim.ImageRequest("front_right", airsim.ImageType.SurfaceNormals, True, False)
        ])

    # Specify the base folder to save images
    
    folders = ['left_raw', 'left_depth_perspective', 'left_seg', 'left_surface_normals',
            'right_raw', 'right_depth_perspective', 'right_seg', 'right_surface_normals']

    # Ensure folders exist
    for folder in folders:
        path = os.path.join(base_folder, folder)
        if not os.path.exists(path):
            os.makedirs(path)
            
    timestamp = str(responses[0].time_stamp)
    # Process and save each response
    for i, response in enumerate(responses):
        
        camera = 'left' if i < 4 else 'right'

        if response.image_type == airsim.ImageType.Scene:
            folder = f"{camera}_raw"
        elif response.image_type == airsim.ImageType.DepthPerspective:
            folder = f"{camera}_depth_perspective"
        elif response.image_type == airsim.ImageType.Segmentation:
            folder = f"{camera}_seg"
        elif response.image_type == airsim.ImageType.SurfaceNormals:
            folder = f"{camera}_surface_normals"

        filename = os.path.join(base_folder, folder, f"{timestamp}")
        npy_filename = f"{filename}.npy"
        png_filename = f"{filename}.png"

        # Case 1: Pixels as floating-point values
        if response.pixels_as_float:
            print("Type %d, size %d" % (response.image_type, len(response.image_data_float)))
            img = np.array(response.image_data_float).reshape(response.height, response.width, -1)

        # Case 2: Pixels as uint8 values
        else:
            print("Type %d, size %d" % (response.image_type, len(response.image_data_uint8)))
            img = np.frombuffer(response.image_data_uint8, dtype=np.uint8).reshape(response.height, response.width, -1)

        
        img = np.flipud(img)
        img = np.fliplr(img)

        # Save the image using Matplotlib without showing it
        default_dpi = plt.rcParams['figure.dpi']
        plt.figure(figsize=(img.shape[1] / default_dpi, img.shape[0] / default_dpi),
                   dpi=default_dpi)
        if img.shape[2] == 1:
            plt.imshow(img, cmap='gray', aspect='auto')
        else:
            img = img[:, :, ::-1]
            plt.imshow(img, aspect='auto')

        np.save(npy_filename, img)
        plt.axis('off')
        plt.savefig(png_filename, bbox_inches='tight', pad_inches=0)
        plt.close()

[Rong-Tao]

I encountered both problems. In my recorded images I get black images and in my preview, I get improperly lumened images

What exactly is improperly lit, the feed in the game or do you "un-black" the images and then?

Have you encountered any issue I described? What settings are you using? Can you have a try of the Archvis project?

[git-akispa]

I encountered both problems. In my recorded images I get black images and in my preview, I get improperly lumened images

What exactly is improperly lit, the feed in the game or do you "un-black" the images and then?

Have you encountered any issue I described? What settings are you using? Can you have a try of the Archvis project?

Sadly, I only had issues with lighting only when I used very complicated envs. When I run my project on my pc I also get those stains but most of them disappear when I try it on my work's pc.

Those, I have also not been able to remove them : (

[Rong-Tao]

I encountered both problems. In my recorded images I get black images and in my preview, I get improperly lumened images

What exactly is improperly lit, the feed in the game or do you "un-black" the images and then?

Have you encountered any issue I described? What settings are you using? Can you have a try of the Archvis project?

Sadly, I only had issues with lighting only when I used very complicated envs. When I run my project on my pc I also get those stains but most of them disappear when I try it on my work's pc.

Those, I have also not been able to remove them : (

The current method used in Colosseum is to first capture images using UTextureRenderTarget2D and USceneCaptureComponent2D on This Function, followed by rendering with render target.

Then, I speculate that Colosseum employs UE4's rendering method, leading to the major difference between UE4 and UE5: the inability to render global illumination. The suspected solution is to find out how UE5 captures screenshots and probably what is the UE5 equivalent render target. Currently, render target cannot be searched in the official UE5 documentation but only in UE4.27. My understanding is that it has been updated, but the code still runs due to backward compatibility.

May I know what setups are you using and I will give it a shot?

[zhangshaos]

I have met this problem too. My solution is using airsim.write_png to replace arisim.write_file.

resps = self.client.simGetImages([
            ImageRequest('0', ImageType.Scene, False, False),
        ])
for resp in resps:
    if resp.image_type == ImageType.Scene:
        img1d = np.fromstring(resp.image_data_uint8, dtype=np.uint8)
        img_rgb = img1d.reshape((resp.height, resp.width, 3))
        airsim.write_png(os.path.normpath(f'{self.setting.target_dir}/{i}.png'), img_rgb)
        # airsim.write_file(f'{self.setting.target_dir}/{i}_scene.png', resp.image_data_uint8)

[FrankFeng-23]

Hey guys, I just found a workaround by using UE5's built-in command "HighResShot". To automate screenshots, I made a slight modification to the BP_ComputerVisionPawn blueprint so that pressing the K key within the game scene executes "HighResShot 2560x1440". Then, in Python, I used the pyautogui library to simulate the action of pressing K on my keyboard. Now it gives very high-quality taint-free scene images.

[ysh11o-2020]

I think that just solved the problem. The source code of the AirSim plug-in needs to be modified. Solution and code are below:
Plugins\AirSim\Source\AirBlueprintLib.cpp in function CompressImageArray
r777 add "MutableSrcData[Index].A = 255;"
for (int32 Index = 0; Index < width * height; Index++) { uint8 TempRed = MutableSrcData[Index].R; MutableSrcData[Index].R = MutableSrcData[Index].B; MutableSrcData[Index].B = TempRed; MutableSrcData[Index].A = 255; //add this line }

[ysh11o-2020]

[reza-shahriari]

I had the same problem.
just reading images with cv2 and rewriting them solved my problem!

folder = "path/to/images/folder"
imgs = glob.glob(folder+'/*.png')
imgs.extend(glob.glob(folder+'/*.jpg')
for img in imgs:
  im = cv2.imread(img)
  cv2.imwrite(img,im)

[June-Smith]

I have met this problem too. My solution is using to replace .airsim.write_png``arisim.write_file

resps = self.client.simGetImages([
            ImageRequest('0', ImageType.Scene, False, False),
        ])
for resp in resps:
    if resp.image_type == ImageType.Scene:
        img1d = np.fromstring(resp.image_data_uint8, dtype=np.uint8)
        img_rgb = img1d.reshape((resp.height, resp.width, 3))
        airsim.write_png(os.path.normpath(f'{self.setting.target_dir}/{i}.png'), img_rgb)
        # airsim.write_file(f'{self.setting.target_dir}/{i}_scene.png', resp.image_data_uint8)

I tried to use the solution you provided, but the following error occurred. I would like to ask what is the version of airsim in your python environment? Is it the same as the version of the airsim plugin you are using?
——————————————————————————————————————————————
Traceback (most recent call last):
File "C:\Users\test\PycharmProjects\airsim_test\test2.py", line 13, in
responses = client.simGetImages(
File "C:\Users\test.conda\envs\airsim\lib\site-packages\airsim\client.py", line 309, in simGetImages
responses_raw = self.client.call('simGetImages', requests, vehicle_name, external)
File "C:\Users\test.conda\envs\airsim\lib\site-packages\msgpackrpc\session.py", line 41, in call
return self.send_request(method, args).get()
File "C:\Users\test.conda\envs\airsim\lib\site-packages\msgpackrpc\future.py", line 45, in get
raise error.RPCError(self._error)
msgpackrpc.error.RPCError: rpclib: function 'simGetImages' (called with 3 arg(s)) threw an exception. The exception contained this information: bad cast.