Custom Op Clarification

[JPPalacios]

Hello,

I am having a hard time following the new custom op implementation. Do the steps discussed in this thread still apply? I am trying to create some custom pre/post-processing operations for letterboxing/nms, respectively.

Thanks

[iksnagreb]

The general idea discussed there should still apply and it is mostly the same functions/methods you have to implement, but it is distributed differently throughout the repository and some names changed: Since the last release there is a two-level approach to introducing new custom ops, so you have to implement at least two classes now. The first level must derive from the new HWCustomOP class and typically contains all the general shape and datatype handling and maybe calling into the Python or RTL simulation. The second level derives from your first level as well as either the HLSBackend or RTLBackend class to specialize how your operator is actually implemented. Here you must implement all the HLS or RTL (depending on your back-end choice, you can also do both, but then you need two separate classes, one for each back-end) code generation methods as well as potentially calling into the C++ simulation (only relevant for the HLS back-end, RTL should probably contain some stub/fallback method). It is important to put your classes into the correct sub-folder, e.g, src/finn/custom_op/fpgadataflow/hls/ for some HLSBackend operator, and suffix your class name by either _hls or _rtl, to make the new SpecializeLayers transformation consider your new operator.

You might also have to introduce the corresponding convert_to_hw_layers (formerly convert_to_hls_layers) transformation and add support for your operator to the SetFolding transformation, if you want to set/adjust the parallelism automatically.

Also remember to register you custom op to the custom_op dictionary at both levels of the hierarchy, you find this in the __init__.py of the respective level/back-end specialization sub-module, see for example here

I think you have already seen and "liked" my [Squeeze] PR #1153: The actual back-end contribution there is rather trivial, but the "boilerplate" to get the operator working and integrate it with the rest of the framework should look similar for any custom op, so you could take some inspiration from there. It should also show you how to set up test-cases for your new operator.

As I am not too familiar with your letterboxing/nms operator/use-case: Is there a standard ONNX operator or some composition of standard ONNX operators, or is this truly custom? If this can be expressed via standard ONNX operations, maybe you could show a graph of the operator pattern you are trying to implement?

[JPPalacios]

Hi @iksnagreb,

Thanks for your detailed response. I will take another look at your PR, thanks for the reminder!

I am currently implementing a YOLO model where the letterboxing resizes and pads an image to preserve its aspect ratio and NMS selects the best overlapping bounding boxes. I am trying to implement something along these lines: reshape and nms (from docs) nodes.

I have letterboxing (HLS) and nms (on PS) working during inference on PYNQ but I am looking to streamline the "build" process instead of manually stitching these together per build. Thanks again for your help!