Rendering zero copy HDR AVFrames decoded via Vulkan (NVidia GPU) results in green frames

[streamingdv]

Hi,

currently I'm working on a Java wrapper for libplacebo to render my real time stream. So far everythin works great and I'm currently testing it on my Windows development machine. My application supports DXVA, D3D11, NVIDEO_CUDA and VULKAN decoding on Windows. As far as I have understood the source code of libplacebo, AVFrames decoded via Vulkan, VAAPI are supporting zero copy rendering via libplacebo (when VkWin32SurfaceCreateInfoKHR is used on Windows for example). However, I encountered a strange problem during testing and I'm not sure if it's a driver issue, ffmpeg issue, or libplacebo issue. I have following results on my Windows machine:

H264 decoding

SOFTWARE decoding + libplacebo rendering = ✓
NVIDEO_CUDA decoding + libplacebo rendering = ✓
DXVA decoding + libplacebo rendering = ✓
D3D11 decoding + libplacebo rendering = ✓
VULKAN decoding + libplacebo rendering = ✓
VULKAN decoding (zero copy)+ libplacebo rendering = ✓

H265 decoding

SOFTWARE decoding + libplacebo rendering = ✓
NVIDEO_CUDA decoding + libplacebo rendering = ✓
DXVA decoding + libplacebo rendering = ✓
D3D11 decoding + libplacebo rendering = ✓
VULKAN decoding + libplacebo rendering = ✓
VULKAN decoding (zero copy)+ libplacebo rendering = ✓

H265 + HDR10 decoding

SOFTWARE decoding + libplacebo rendering = ✓
NVIDEO_CUDA decoding + libplacebo rendering = ✓
DXVA decoding + libplacebo rendering = ✓
D3D11 decoding + libplacebo rendering = ✓
VULKAN decoding + libplacebo rendering = ✓
VULKAN decoding (zero copy)+ libplacebo rendering = ✗ (green frames)

The only difference I do when when using zero copy is this

    if (this.m_VideoDecoderCtx.hw_device_ctx() == null || !hwDownload) {
        decodedFrame = avFrame; // use the frame as is for Software frames an zero copy
    } else {
        final AVFrame hwAvFrame = av_frame_alloc();
        if (av_hwframe_transfer_data(hwAvFrame, avFrame, 0) < 0) {
            av_frame_unref(hwAvFrame);
            decodedFrame = avFrame;
        } else {
            av_frame_unref(avFrame);
            decodedFrame = hwAvFrame;
        }
    }

    return decodedFrame;

When I use vulkan decoding without the zero copy via av_hwframe_transfer_data and pass this frame to libplacebo everything works fine. But with zero copy I get only greenish frames and via the log I get following warnings, errors

Masking blit_src from wrapped texture because the corresponding format 'rx10' does not support PL_FMT_CAP_BLITTABLE

I'm not sure if this issue is related but it sounds similar. Here is an example frame to visualize what I mean with "green frame"

normal hdr via VULKAN decoding + libplacebo rendering

This is how I get the AVFrame and this is how I pass it to libplacebo

    AVFrame avFrame = videoDecoder.decodeFrame(isKeyFrame, frameByteBuffer.limit(), hwDownload); // hwDownload is false for Vulkan + HDR
    boolean result = this.placeboManager.plRenderAvFrame(avFrame.address(), this.placeboVulkan, this.placebo_swapchain,     this.placebo_renderer);

And in my JNI code I do follwowing

extern "C"
JNIEXPORT jboolean JNICALL Java_com_grill_placebo_PlaceboManager_plRenderAvFrame
  (JNIEnv *env, jobject obj, jlong avframe, jlong placebo_vulkan, jlong swapchain, jlong renderer) {
  AVFrame *frame = reinterpret_cast<AVFrame*>(avframe);
  pl_vulkan vulkan = reinterpret_cast<pl_vulkan>(placebo_vulkan);
  pl_swapchain placebo_swapchain = reinterpret_cast<pl_swapchain>(swapchain);
  pl_renderer placebo_renderer = reinterpret_cast<pl_renderer>(renderer);

  pl_tex placebo_tex[4] = {nullptr, nullptr, nullptr, nullptr};
  struct pl_swapchain_frame sw_frame = {0};
  struct pl_frame placebo_frame = {0};
  struct pl_frame target_frame = {0};

  struct pl_avframe_params avparams = {
      .frame = frame,
      .tex = placebo_tex,
      .map_dovi = false,
  };
  bool mapped = pl_map_avframe_ex(vulkan->gpu, &placebo_frame, &avparams);

  if (!mapped) {
      LogCallbackFunction(nullptr, PL_LOG_ERR, "Failed to map AVFrame to Placebo frame!");
      return false;
  }
  // set colorspace hint
  struct pl_color_space hint = placebo_frame.color;
  pl_swapchain_colorspace_hint(placebo_swapchain, &hint);

  pl_rect2df crop;
  bool ret = false;
  pl_render_params render_params = pl_render_fast_params; // pl_render_high_quality_params, pl_render_default_params

  if (!pl_swapchain_start_frame(placebo_swapchain, &sw_frame)) {
      LogCallbackFunction(nullptr, PL_LOG_ERR, "Failed to start Placebo frame!");
      goto cleanup;
  }

  pl_frame_from_swapchain(&target_frame, &sw_frame);

  crop = placebo_frame.crop;
  if (!pl_render_image(placebo_renderer, &placebo_frame, &target_frame, &render_params)) {
      LogCallbackFunction(nullptr, PL_LOG_ERR, "Failed to render Placebo frame!");
      goto cleanup;
  }
  if (!pl_swapchain_submit_frame(placebo_swapchain)) {
      LogCallbackFunction(nullptr, PL_LOG_ERR, "Failed to submit Placebo frame!");
      goto cleanup;
  }
  pl_swapchain_swap_buffers(placebo_swapchain);
  ret = true;

cleanup:
  pl_unmap_avframe(vulkan->gpu, &placebo_frame);

  return ret;
}

Here is a full example of my JNI wrapper code.
Is this something which is known? If not I can provide more information.

I have a reproducable java rendering project which uses javacv for the ffmepg decoding part and my libplacebo java wrapper for the rendering part. If needed I can upload this example to my GitHub account.

[nowrep]

Try this #239

[streamingdv]

@nowrep thanks I will try it out later this week, was in contact with jbaiter and he told me about this. Was not aware that there is an open pull request for that. I really hope that this fixes the issue.

[streamingdv]

@haasn just for information the fix in the pull request

"Only adjust bit_shift for DRM P010 frames"

Mentioned above, fixed the problem for me.

[gitoss]

I'm reportg this issue occurs with AMD, too - because this bug ticket has been closed, I've opened a new one now only refering to nVidia. #272