Skip to content

Latest commit

 

History

History
263 lines (235 loc) · 8.4 KB

2024-05-19.org

File metadata and controls

263 lines (235 loc) · 8.4 KB
Raw

2024/05/23 Progress Report

1 Summary

1.1 Summary

1.1.1 Task List

  1. Browse ~smartcam~ module source code.
  2. Compile ~smartcam~ code.
  3. Modify ~smartcam~ code to use other kernels.
  4. Write image compression $→$ storage prototype pipeline.

1.1.2 @@latex:@@

1.2 Recap (1 / 2)

Xilinx’s Smart Camera application allows for a

MIPI capture $→$ PL processing $→$ RTSP/DP output pipeline. https://xilinx.github.io/kria-apps-docs/kv260/2022.1/build/html/_images/sc_image_landing.jpg

1.3 Recap (2 / 2)

This is done via a GStreamer-Video4Linux2 driver. 2024-05-23-13-44-56_915x560.png

1.4 Objectives

1.4.1 Runtime

  • What is v4l2’s role in acquiring the image from DMA?
  • How does GStreamer use v4l2?
  • How does an acceleration kernel run using v4l2/GStreamer?

1.4.2 Building

  • What does the final software depend on?
  • What does the developer need to have ready in order to build the application?
    • Dependencies?

2 Hardware-Software Architecture

2.1 MIPI-CSI2 Capture (Hardware)

2.1.1 Col left

  • MIPI source outputs to S_AXI_HP0_FPD.
  • NV12 format.

2.1.2 @@latex:@@

NV12Planar 8-bit YUV 4:2:0
YUV420Semi-planar 8-bit YUV 4:2:0

2.1.3 @@latex:@@

https://xilinx.github.io/kria-apps-docs/kv260/2022.1/build/html/_images/hw_cap_pp1.png

2.2 MIPI-CSI2 Capture (Software) (1 / 2)

2.2.1 Col left

  • Source accessible via v4l2

    → /dev/media*

    → /dev/video*

    …

    → v4l2src

    → mediasrc

2.2.2 Col right

https://xilinx.github.io/kria-apps-docs/kv260/2022.1/build/html/_images/video_capture1.png

2.3 MIPI-CSI2 Capture (Software) (2 / 2)

https://xilinx.github.io/kria-apps-docs/kv260/2022.1/build/html/_images/software-overall-data-flow1.png

2.4 Acceleration

Acceleration on the new stack is based on HLS kernels. For vision, there is VVAS. The Smart Camera application uses VVAS with GStreamer.

2.4.1 VVAS

Vitis Video Analytics SDK is Xilinx’s software stack for building video processing applications.

2.4.2 Plugin Structure

Use GStreamer plugins for creating computer vision pipelines.

3 GStreamer

3.1 GStreamer

GStreamer is a pipeline-based multimedia framework linking various media processing systems to create workflows.

gst.png

3.2 GStreamer Basics

3.2.1 Elements

Objects derived from GstElement. Examples:

  • source element: provides data
  • filter element: acts on incoming data

3.2.2 Pads

Objects derived from GstPad. They are the “ports” that intermediate data between elements.

  • Source pad: data flows outward
  • Sink pad: data flows inward

3.2.3 Buffers

  • GstMiniObject, GstBuffer, GstEvent…

3.3 GStreamer Plugins

  • Enables plug-and-play functionality with GStreamer.
  • Elements must be wrapped into a plugin before being used.

3.3.1 What is a plugin?

Essentially, it is a block of loadable code (a shared object/dynamically linked library).

3.3.2 What is the structure of a plugin?

https://gitlab.freedesktop.org/gstreamer/gst-template.git

  • The above serves as good reference.
  • The compiled code generates a libgstplugin.so, which can be loaded by gst-launch-1.0 by simply specifying the path in GST_PLUGIN_PATH.

3.4 VVAS

  • Collection, as well as an API for creating acceleration kernels for usage with GStreamer.

https://xilinx.github.io/VVAS/1.0/build/html/_images/core-API-functions.png

3.5 VVAS Usage in GStreamer (1 / 2)

API usage:

  1. Plugin Initialization (xlnx_kernel_init)
  2. Kernel Start (xlnx_kernel_start)
  3. Waiting for the kernel to finish (vvas_kernel_done)
  4. Denitilizating the plugin (xlnx_kernel_deinit)

3.6 VVAS Usage in GStreamer (2 / 2)

3.6.1 col1

~vvas_xfilter~:

3.6.1.1 JSON config

Contains information regarding:

  • xclbin location
  • vvas-library-repo
  • element-mode (passthrough, inplace, transform)

3.6.1.2 xclbin

Used to program the FPGA.

3.6.2 col2

https://xilinx.github.io/VVAS/main/build/html/_images/xfilter_plugin1.png

4 Source Code Analysis

4.1 Smart Camera — Folder Structure

  • ~smartcam~: Contains the base smartcam application. It is built and used with Docker.

4.1.1 col1

smartcam_repo.png

4.1.2 smartcam/src

ivas_airender: aadasda
ivas_xpp_pipeline: aadasda\ main: Contains the GStreamer pipelines that run everything.

4.2 VVAS Usage in smartcam

~vvas_xfilter~: Plugin to act directly on data with the specified kernel.

xfilter.png

4.3 VVAS Example — Edge Filter with filter2d (1 / 3)

4.3.1 Makefile

filter2d_makefile.png

4.4 VVAS Example — Edge Filter with filter2d (2 / 3)

4.4.1 PS Connectivity

filter2d_connection.png

4.5 VVAS Example — Edge Filter with filter2d (3 / 3)

4.5.1 Launching GStreamer with filter2d kernel

gst-launch-1.0 v4l2src \
  device=/dev/video0 io-mode=mmap !\
  "video/x-raw, width=640, height=480" !\
  videoconvert !\
  "video/x-raw, width=640, height=480,
   format=YUY2, framerate=30/1" !\
  vvas_xfilter kernels-config=/tmp/kernel_xfilter2d_pl.json\
  dynamic-config='{ "filter_preset" : "edge" }' !\
  perf ! kmssink plane-id=39 fullscreen-overlay=true -v

4.6 VVAS Example — Results of the Edge Filter

edges.jpg

5 Conclusion

5.1 Conclusion

  • Got smartcamera to work on Docker inside the KV260
    • It also worked with a modified kernel
  • Next: Write image compression $→$ storage prototype pipeline?
  • I’m also working on making the build process for these packages simple.

5.2 Appendix

5.2.1 AIBox-ReID

Xilinx demo for 4k IP cameras as inputs on the KV260. https://xilinx.github.io/kria-apps-docs/kv260/2022.1/build/html/_images/aib_image_landing.jpg

5.3 Appendix

5.3.1 AI Box Distributed ReID

Distributed demo. https://xilinx.github.io/kria-apps-docs/kv260/2022.1/build/html/_images/aibox-dist-landing.png