[Doc] Add docs for how to develop c# api (PaddlePaddle#1399)

* add doc for developing c sharp api for new models * add english doc * add english doc * fix doc
leyond123 · Feb 28, 2023 · 010f12d · 010f12d
1 parent f80d929
commit 010f12d
Show file tree

Hide file tree

Showing 5 changed files with 319 additions and 0 deletions.
diff --git a/docs/README.md b/docs/README.md
@@ -40,6 +40,8 @@
 - [3. How to Use FastDeploy C++ SDK on Android Platform](en/faq/use_cpp_sdk_on_android.md)
 - [4. Tricks of TensorRT](en/faq/tensorrt_tricks.md)
 - [5. How to Develop a New Model](en/faq/develop_a_new_model.md)
+- [6. How to Develop C API for a New Model](en/faq/develop_c_api_for_a_new_model.md)
+- [7. How to Develop C# API for a New Model](en/faq/develop_c_sharp_api_for_a_new_model.md)
 
 ## More FastDeploy Deployment Module
 

diff --git a/docs/README_CN.md b/docs/README_CN.md
@@ -43,6 +43,7 @@
 - [4. TensorRT使用中的一些技巧](cn/faq/tensorrt_tricks.md)
 - [5. 如何增加新的模型](cn/faq/develop_a_new_model.md)
 - [6. 如何给新模型增加C API](cn/faq/develop_c_api_for_a_new_model.md)
+- [7. 如何给新模型增加C# API](cn/faq/develop_c_sharp_api_for_a_new_model.md)
 
 ## 更多FastDeploy部署模块
 

diff --git a/docs/README_EN.md b/docs/README_EN.md
@@ -42,6 +42,7 @@
 - [4. Tricks of TensorRT](en/faq/tensorrt_tricks.md)
 - [5. How to Develop a New Model](en/faq/develop_a_new_model.md)
 - [6. How to Develop C API for a New Model](en/faq/develop_c_api_for_a_new_model.md)
+- [7. How to Develop C# API for a New Model](en/faq/develop_c_sharp_api_for_a_new_model.md)
 
 ## More FastDeploy Deployment Module
 

diff --git a/docs/cn/faq/develop_c_sharp_api_for_a_new_model.md b/docs/cn/faq/develop_c_sharp_api_for_a_new_model.md
@@ -0,0 +1,158 @@
+[English](../../en/faq/develop_c_sharp_api_for_a_new_model.md) | 中文
+
+# FastDeploy给模型新增C# API
+
+## 相关概念
+
+FastDeploy的核心代码库的实现是基于C++开发的，为了提供给使用者C#接口进行调用，需要使用C接口作为沟通的桥梁。通常在给模型新增C# API时，需要先给模型新增C API。然后再开发C# API。C#会调用C接口，C接口再去调用C++接口，已实现具体的功能。关于如何给模型新增C API，请先参考文档[FastDeploy给模型新增C API](./develop_c_api_for_a_new_model.md)。
+
+按照FastDeploy目前的实现结构，新增一个模型的API通常涉及以下三个部分：
+
+- Model
+
+  模型接口，提供给用户进行模型创建和载入、预测的功能。
+
+- Result
+
+  模型推理的结果
+
+- Visualization
+
+  对推理结果进行可视化的功能
+
+由于C#是面向对象编程的高级语言，在设计C# API的过程中，各个部分的接口形式需要尽量和C++对应的接口保持一致，给使用者相同的使用体验。由于C#接口是通过调用底层的C接口来具体实现的，因此需要了解C#如何调用C接口以及两者数据结构之间的关系。具体可以参考[C# Marshal](https://learn.microsoft.com/en-us/dotnet/framework/interop/marshalling-data-with-platform-invoke)
+。
+
+## 实现流程
+
+下面通过给ppseg系列模型提供C# API为示例讲述如何在当前框架下进行C# API的实现。
+
+1. 提供表示分割模型结果的数据结构
+
+打开文件fastdeploy/vision/common/result.h, 里面定义各种不同类别模型预测结果的数据结构，找到SegmentationResult，将数据结构对应用C#进行表示。
+```c++
+struct FASTDEPLOY_DECL SegmentationResult : public BaseResult {
+  std::vector<uint8_t> label_map;
+  std::vector<float> score_map;
+  std::vector<int64_t> shape;
+  bool contain_score_map = false;
+  ResultType type = ResultType::SEGMENTATION;
+}
+```
+在csharp/fastdeploy/vision/result.cs对应的定义一个C#的SegmentationResult结构
+
+```c#
+public class SegmentationResult{
+  public List<byte> label_map;
+  public List<float> score_map;
+  public List<long> shape;
+  public bool contain_score_map;
+  public ResultType type;
+}
+```
+
+该结构是提供给用户在C# API中实际使用的结构，但是为了能够在调用C接口时进行使用，需要使用C#按照C中定义的对应结构进行定义作为内部使用，和C语言中结构进行映射的C#结构定义在csharp/fastdeploy/types_internal_c.cs中。比如这里面再定义一个FD_SegmentationResult，结构如下
+
+```c#
+[StructLayout(LayoutKind.Sequential)]
+public struct FD_SegmentationResult {
+  public FD_OneDimArrayUint8 label_map;
+  public FD_OneDimArrayFloat score_map;
+  public FD_OneDimArrayInt64 shape;
+  [MarshalAs(UnmanagedType.U1)]
+  public bool contain_score_map;
+  public FD_ResultType type;
+}
+```
+
+接下来需要定义两个函数，负责在SegmentationResult和FD_SegmentationResult这两个结构间进行转换。这两个函数需要定义在ConvertResult类中。
+
+```c#
+public class ConvertResult {
+  public static SegmentationResult
+  ConvertCResultToSegmentationResult(FD_SegmentationResult fd_segmentation_result);
+  public static FD_SegmentationResult
+  ConvertSegmentationResultToCResult(SegmentationResult segmentation_result);
+}
+```
+
+2. 提供模型接口的C# API
+
+打开文件fastdeploy/vision/segmentation/ppseg/model.h，里面定义了分割模型的C++接口，即fastdeploy::vision::segmentation::PaddleSegModel类。在C#中定义一个类对应实现这些接口。
+
+通过对照PaddleSegModel类里暴露的方法，提供对应的方法。一般在C#的类声明里分为两部分，第一部分是提供给用户的接口，第二部分是声明要调用的C接口。
+
+```c#
+public class PaddleSegModel {
+
+  public PaddleSegModel(string model_file, string params_file,
+                         string config_file, RuntimeOption custom_option = null,
+                         ModelFormat model_format = ModelFormat.PADDLE)；
+  ~PaddleSegModel()；
+
+  public string ModelName()；
+
+  public SegmentationResult Predict(Mat img)；
+
+  public List<SegmentationResult> BatchPredict(List<Mat> imgs)；
+
+  public bool Initialized()；
+
+  // below are underlying C api
+  private IntPtr fd_paddleseg_model_wrapper;
+  [DllImport("fastdeploy.dll",
+             EntryPoint = "FD_C_CreatePaddleSegModelWrapper")]
+  private static extern IntPtr FD_C_CreatePaddleSegModelWrapper(
+      string model_file, string params_file, string config_file,
+      IntPtr fd_runtime_option_wrapper, ModelFormat model_format);
+  [DllImport("fastdeploy.dll",
+             EntryPoint = "FD_C_DestroyPaddleSegModelWrapper")]
+  private static extern void
+  FD_C_DestroyPaddleSegModelWrapper(IntPtr fd_paddleseg_model_wrapper);
+  [DllImport("fastdeploy.dll",
+             EntryPoint = "FD_C_PaddleSegModelWrapperPredict")]
+  private static extern bool
+  FD_C_PaddleSegModelWrapperPredict(IntPtr fd_paddleseg_model_wrapper,
+                                     IntPtr img,
+                                     ref FD_SegmentationResult fd_segmentation_result);
+
+  [DllImport("fastdeploy.dll", EntryPoint = "FD_C_DestroySegmentationResult")]
+  private static extern void
+  FD_C_DestroySegmentationResult(ref FD_SegmentationResult fd_segmentation_result);
+
+  [DllImport("fastdeploy.dll",
+             EntryPoint = "FD_C_PaddleSegModelWrapperInitialized")]
+  private static extern bool
+  FD_C_PaddleSegModelWrapperInitialized(IntPtr fd_paddleseg_model_wrapper);
+  [DllImport("fastdeploy.dll",
+             EntryPoint = "FD_C_PaddleSegModelWrapperBatchPredict")]
+  private static extern bool
+  FD_C_PaddleSegModelWrapperBatchPredict(IntPtr fd_paddleseg_model_wrapper,
+                                          ref FD_OneDimMat imgs,
+                                          ref FD_OneDimSegmentationResult results);
+
+}
+```
+
+
+具体实现可以参考文件 csharp/fastdeploy/vision/segmentation/model.cs 。
+
+3. 提供可视化函数C# API
+
+打开文件fastdeploy/vision/visualize/visualize.h，里面有对于不同类型模型推理结果进行可视化的函数。这里用C#对照写一个可视化SegmentationResult的API在Visualize类中。同样，会调用底层的C接口实现具体功能，因此需要声明对应的C接口。
+
+```c#
+public class Visualize {
+  public static Mat VisSegmentation(Mat im,
+                                    SegmentationResult segmentation_result,
+                                    float weight = 0.5);
+
+  [DllImport("fastdeploy.dll", EntryPoint = "FD_C_VisSegmentation")]
+  private static extern IntPtr
+  FD_C_VisSegmentation(IntPtr im, ref FD_SegmentationResult fd_segmentation_result, float weight);
+}
+```
+
+4. 创建example, 测试所添加的C# API
+
+在examples目录下，根据所接入的模型的类别，在对应的文件夹下新增目录名csharp，里面创建csharp的示例代码和CMakeLists.txt，编译测试，确保使用新增的C# API能够正常工作。
diff --git a/docs/en/faq/develop_c_sharp_api_for_a_new_model.md b/docs/en/faq/develop_c_sharp_api_for_a_new_model.md
@@ -0,0 +1,157 @@
+English | [中文](../../cn/faq/develop_c_sharp_api_for_a_new_model.md)
+
+# Adds C# API to models
+
+## Introduction
+
+The core code library of FastDeploy is implemented based on C++ development. In order to provide C# interface for users to call, C interface needs to be used as a bridge for communication. Usually when adding C# API for a model, C API is needed to implement first, then develop C# API. C# will call C interface, and C interface will call C++ interface to implement specific functions. About how to add C API for a model, please refer to document [Adds C API to models](./develop_c_api_for_a_new_model.md).
+
+According to FastDeploy’s current implementation structure, adding an API for a model usually involves following three parts:
+
+- Model
+
+Model interface, provide users with functions for model creation and loading, prediction.
+
+- Result
+
+Inference result of model
+
+- Visualization
+
+Function for visualizing inference result
+
+Since C# is an object-oriented programming language,  C# interface forms of each part need to be consistent with corresponding interfaces of C++, giving users the same usage experience. Since C# interface is implemented by calling underlying C interface, it is necessary to understand how C# calls C interface and relationship between data structures of both. Please refer to [C# Marshal](https://learn.microsoft.com/en-us/dotnet/framework/interop/marshalling-data-with-platform-invoke).
+
+## Implementation process
+
+The following describes how to implement C# API for ppseg series models as an example of how to implement C# API under the current framework.
+
+1. Provide a data structure that represents segmentation results
+
+Open file fastdeploy/vision/common/result.h, which defines data structures for different types of model prediction results, find SegmentationResult, and use C# structure to represent the following data structure
+
+```c++
+struct FASTDEPLOY_DECL SegmentationResult : public BaseResult {
+  std::vector<uint8_t> label_map;
+  std::vector<float> score_map;
+  std::vector<int64_t> shape;
+  bool contain_score_map = false;
+  ResultType type = ResultType::SEGMENTATION;
+}
+```
+
+Define C# SegmentationResult structure in csharp/fastdeploy/vision/result.cs
+
+```c#
+public class SegmentationResult{
+  public List<byte> label_map;
+  public List<float> score_map;
+  public List<long> shape;
+  public bool contain_score_map;
+  public ResultType type;
+}
+```
+
+This structure is used by users in C# API, but in order to use it when calling C interface, we need to use C# to define corresponding structure defined in C as internal use, and C# structure defined in csharp/fastdeploy/types_internal_c.cs that maps with structure in C language. For example, define FD_SegmentationResult structure as follows
+
+```c#
+[StructLayout(LayoutKind.Sequential)]
+public struct FD_SegmentationResult {
+  public FD_OneDimArrayUint8 label_map;
+  public FD_OneDimArrayFloat score_map;
+  public FD_OneDimArrayInt64 shape;
+  [MarshalAs(UnmanagedType.U1)]
+  public bool contain_score_map;
+  public FD_ResultType type;
+}
+```
+
+Next, we need to define two functions that are responsible for converting between the SegmentationResult and FD_SegmentationResult structures. These two functions need to be defined in the ConvertResult class.
+
+```c#
+public class ConvertResult {
+  public static SegmentationResult
+  ConvertCResultToSegmentationResult(FD_SegmentationResult fd_segmentation_result);
+  public static FD_SegmentationResult
+  ConvertSegmentationResultToCResult(SegmentationResult segmentation_result);
+}
+```
+
+2. Provide C# API for model interface
+
+Open the file fastdeploy/vision/segmentation/ppseg/model.h, which defines the C++ interface for the segmentation model, namely fastdeploy::vision::segmentation::PaddleSegModel class. Define a class in C# to implement these interfaces.
+Generally, there are two parts in the C# class declaration. The first part is the interface for users, and the second part is the declaration of C interfaces to be called.
+
+```c#
+public class PaddleSegModel {
+
+  public PaddleSegModel(string model_file, string params_file,
+                         string config_file, RuntimeOption custom_option = null,
+                         ModelFormat model_format = ModelFormat.PADDLE)；
+  ~PaddleSegModel()；
+
+  public string ModelName()；
+
+  public SegmentationResult Predict(Mat img)；
+
+  public List<SegmentationResult> BatchPredict(List<Mat> imgs)；
+
+  public bool Initialized()；
+
+  // below are underlying C api
+  private IntPtr fd_paddleseg_model_wrapper;
+  [DllImport("fastdeploy.dll",
+             EntryPoint = "FD_C_CreatePaddleSegModelWrapper")]
+  private static extern IntPtr FD_C_CreatePaddleSegModelWrapper(
+      string model_file, string params_file, string config_file,
+      IntPtr fd_runtime_option_wrapper, ModelFormat model_format);
+  [DllImport("fastdeploy.dll",
+             EntryPoint = "FD_C_DestroyPaddleSegModelWrapper")]
+  private static extern void
+  FD_C_DestroyPaddleSegModelWrapper(IntPtr fd_paddleseg_model_wrapper);
+  [DllImport("fastdeploy.dll",
+             EntryPoint = "FD_C_PaddleSegModelWrapperPredict")]
+  private static extern bool
+  FD_C_PaddleSegModelWrapperPredict(IntPtr fd_paddleseg_model_wrapper,
+                                     IntPtr img,
+                                     ref FD_SegmentationResult fd_segmentation_result);
+
+  [DllImport("fastdeploy.dll", EntryPoint = "FD_C_DestroySegmentationResult")]
+  private static extern void
+  FD_C_DestroySegmentationResult(ref FD_SegmentationResult fd_segmentation_result);
+
+  [DllImport("fastdeploy.dll",
+             EntryPoint = "FD_C_PaddleSegModelWrapperInitialized")]
+  private static extern bool
+  FD_C_PaddleSegModelWrapperInitialized(IntPtr fd_paddleseg_model_wrapper);
+  [DllImport("fastdeploy.dll",
+             EntryPoint = "FD_C_PaddleSegModelWrapperBatchPredict")]
+  private static extern bool
+  FD_C_PaddleSegModelWrapperBatchPredict(IntPtr fd_paddleseg_model_wrapper,
+                                          ref FD_OneDimMat imgs,
+                                          ref FD_OneDimSegmentationResult results);
+
+}
+```
+
+For more details, please refer to file 'csharp/fastdeploy/vision/segmentation/model.cs'.
+
+3. Provide C# API for visualization function
+
+Open the file fastdeploy/vision/visualize/visualize.h, which has functions for visualizing inference results of different types of models. Here, we write a C# API to visualize SegmentationResult in the Visualize class. Similarly, it will call the underlying C interface to implement specific functions, so you need to declare corresponding C interfaces.
+
+```c#
+public class Visualize {
+  public static Mat VisSegmentation(Mat im,
+                                    SegmentationResult segmentation_result,
+                                    float weight = 0.5);
+
+  [DllImport("fastdeploy.dll", EntryPoint = "FD_C_VisSegmentation")]
+  private static extern IntPtr
+  FD_C_VisSegmentation(IntPtr im, ref FD_SegmentationResult fd_segmentation_result, float weight);
+}
+```
+
+4. Create example to test added C# API
+
+In examples directory, according to category of model, create new directory named csharp in corresponding folder.Create csharp sample code and CMakeLists.txt inside, then compile and test it, to ensure that the added C# API can work normally.