Replies: 11 comments
-
|
Thanks for the interest! 💜 Short version: I think it should be pretty easy! Honestly the biggest problem if you were to do this yourself would probably be the lack of documentation. I've never used TensorRT before (I'm not much of an machine learning guy), but looking through the API suggests this library would be trivial for Biohazrd to translate. I have a (yet-to-be-published) tool that automatically assesses how compatible a given library is with Biohazrd, and it had a favorable outlook. (If you want to look at the report, I've uploaded it here) To summarize the results:
I did not check anything in the `plugin directory since I am under the impression that these APIs are not meant to be used directly. If this assessment is incorrect let me know and I can look over them too. (Although at a glance they look like they'd require me to install the CUDA SDK.) Outside of that, this stood out to me:
Does TensorRT rely on one of these technologies in order to be useful? Would bindings for one of these need to be made for .NET too or do such bindings already exist? (Assuming there aren't already .NET equivalents to any of those.) For my own reference (because NVIDIA loves to bury this stuff), here's the TensorRT API reference: https://docs.nvidia.com/deeplearning/tensorrt/api/c_api/index.html |
Beta Was this translation helpful? Give feedback.
-
I will note that currently only Windows x64 is supported. Linux x64 is currently planned/sponsored, but everything else is mostly up in the air. (Linux ARM64 is on the very distant roadmap. Anything else would likely require sponsorship.) Some libraries generated by Biohazrd only support .NET 5 due to the use of C#9 function pointers, and TensorRT would be one of them. In theory supporting older .NET versions is possible, but I'm not particularly keen on supporting them without a strong incentive. Biohazrd is extensible enough that you could probably add legacy |
Beta Was this translation helpful? Give feedback.
-
|
@PathogenDavid thank you very much for the very detailed and thoughtful reply. My apologies for my tardiness. :) This sounds great. Some answers and comments below. You can see examples of usage in e.g.:
This shouldn't be needed I think.
Good to mention this. Since I need to use this given my models are in ONNX format, it is unfortunately a requirement.
Don't need to do that. But need to load/use the
Shouldn't be a problem.
No not at all, TensorRT is useful in it self. I prefer using it via https://github.com/microsoft/onnxruntime but this does not support all the use cases via that. TensorRT is mainly for running "inference" of models, that is after training. It usually provides big performance improvements compared to running inference based using standard CUDA.
TensorRT only supports x64 anyway 😅 So no problem.
Bindings to more ML libraries and frameworks for .NET would be great overall! This is lacking a lot for .NET. For some reason a lot of these libraries have bindings to python and this without a C API and apparently this is easier to do for some reason. I don't know why that is the case? If there was a C API everything would be easier, but a lot of ML libraries don't have that. Luckily, ONNX Runtime does, though.
Yes, that I expected, but function pointers are more a C# 9 thing that a specific .NET 5 thing I think. If you wrap these with normal calls this should be .NET Standard 2.0 compatible I think. But this can be addressed after. Do you think it is possible to progress based on this? Could I do or help with something here to try to go forward with this and explore this? :) |
Beta Was this translation helpful? Give feedback.
-
|
@PathogenDavid I found it interesting the code contains #ifdef SWIG
inline IParser* createParser(nvinfer1::INetworkDefinition* network,
nvinfer1::ILogger* logger)
{
return static_cast<IParser*>(
createNvOnnxParser_INTERNAL(network, logger, NV_ONNX_PARSER_VERSION));
}
#endif // SWIG |
Beta Was this translation helpful? Give feedback.
-
|
It's only defined once so not sure that is true... https://github.com/NVIDIA/TensorRT/search?q=SWIG |
Beta Was this translation helpful? Give feedback.
-
No problem!
Well the good news is a Microsoft engineer gave me their fix for the issue blocking this file. Since that fix won't land in Visual Studio until 16.9 (2019 current is 16.7 so 16.9 is probably the 2021 Q1) I'm going to investigate working around this issue from Biohazrd's end until their fix is released. If you want to work on TensorRT before I develop that fix, you can work around the issue using this slightly modified
Nope, that won't be affected.
Cool, just wanted to make sure since I'm not a machine learning guy. I didn't want to tell you TesnorRT would work when some obvious-to-ML-people dependency was also implied by the question.
Cool, just wanted to be up front about it since I don't have a formal roadmap up yet.
Biohazrd may definitely be the answer there!
I don't think it's that it's necessarily easier to do. It's more a lot of academic types like Python and machine learning is still in that stage where the main users are academic types. There's basically three ways to do interop to C/C++ from non-C/C++ languages:
(.NET doesn't really have 3, but one could argue FCall functions is kinda similar. They only exist to support the runtime though.) Option 3 is pretty labor-intensive, but it means C++ developers get to keep writing C++ and it doesn't require a C layer. The Python parts of TensorRT are closed-source, but if you poke at things this is what they're using.
(Answering this out of order, but...) It would not surprise me if the SWIG strategy for Python involves generating a native library as mentioned above. That brief mention of it does suggest that could be what they're using for the Python bindings. SWIG does support C#, but personally I wasn't too impressed with the codegen or the strategies it uses. If I thought SWIG was a good binding generator I probably wouldn't have been working on Biohazrd for so long 🤡
The biggest issue with making C APIs for C++ libraries is you can't really surface anything besides POD structs on the API. (You can try, but compilers will warn you this is a bad idea and you may or may not run into issues.) At best this constrains how the C++ side of the library is written at worst it requires maintaining a much thicker C API layer and in some cases hurts performance.
True, but officially this isn't a supported scenario as far as .NET is concerned. If I remember right, getting function pointers to managed functions was something that required some degree of runtime support. Biohazrd doesn't need this out of the box, but there's reasons you might want to in your library. Downlevel .NET is not a priority for me or my sponsor, but if someone does try to use a Biohazrd-generated library in downlevel .NET I'd definitely be interested to hear how it went.
Definitely! Things are still a little rough (for example I don't have NuGet packages yet and there's basically no documentation), but if you're brave you can try to make a generator based off of InfectedImGui or InfectedPhysX. It'd be helpful for me to get feedback from someone else trying to use the library since all of my feedback thus far has been people reading example code I've sent them. I'd be willing to get on Discord or Teams to answer questions you might run into as you go along. If you aren't feeling quite so brave and you aren't in a rush, I could probably find some time soon to get a basic generator going for TensorRT. (I don't mind doing it since the library has such a simple API, but I'm definitely interested in hearing your feedback.) |
Beta Was this translation helpful? Give feedback.
-
I agree on that. We use CNTK via SWIG generated bindings and there are not exactly optimal in all cases. It works, though. So had thought about using SWIG for Tensorrt too. But SWIG depends on generating a C API in a new library/dll which is annoying, hence why I thought of Biohazrd. 👍
No problem, we will see 👍
I'm not in a rush so this sounds great! And I also thought with this simple API it might be good example/use case. Please do not hesitate to contact me if there is anything at all. Thanks! |
Beta Was this translation helpful? Give feedback.
-
Turns out this plan might've been for the best. I ended up finding a few missing features in Biohazrd that were pretty easy for me to solve, but I would've felt guilty if you had run into them. So in the end it turned out to be a good exercise of Biohazrd regardless. I've gotten the generator done, and I've ported the MNIST sample as well as the necessary portions of the common sample code. The only real unexpected roadblock was the fact that Biohazrd doesn't support namespaces yet. (Things can be in namespaces, but it translates everything into a global namespace.) This hasn't been an issue with PhysX or Dear ImGui, but TensorRT has conflicts between its own namespaces. (For example I'm currently investigating a crash in the sample that is caused because I've given you access to the WIP repository (You have to accept access, check your email.) If you have some time I'd definitely appreciate it if you could look over the ported sample code and see if something sticks out. I'm pretty sure there's just a typo somewhere that I'm not noticing, so another set of eyes would probably help. (Unfortunately this part of the code somehow ended up with almost all of the Off the top of my head, here's the main tasks left:
The MNIST sample is almost entirely a mechanical translation of the C++ sample, so minimal effort went into C#ifying it. I put a bit more effort into C#ifying the common sample code, but for the most part it's the same. |
Beta Was this translation helpful? Give feedback.
-
Awesome! :) Since the example isn't the
Haven't seen this before, and just to be clear I don't have a lot of experience using the raw Tensorrt API myself, sorry. Mainly used in via ONNX Runtime or the Overall, what I find it great is that it appears all the config options are available which is the primary driver for my interest in this. In ONNX Runtime one can't define this on your own, it uses a single optimization parameter instead but that means certain things can't be configured. The I haven't had time to look into this in detail, unfortunately. I, of course, can see, as you already noted, that this is very raw with no C#'ification around life time management etc. But it looks promising to me. |
Beta Was this translation helpful? Give feedback.
-
Yeah, I wanted to start with this sample since NVIDIA seems to present it as the Hello World of the library. That sample will come next.
No worries. I have basically zero practical experience with machine learning, so I have no idea how portable the terminology is.
No problem, I just wanted to give you an update. If you find some time to help debug the sample I'd of course appreciate it. (I know you only care about the ONNX bits, but want to be sure the sample is broken because of an error I made while re-writing it and not because of an error within Biohazrd.)
Yup. Also I feel like I didn't make it super clear earlier: The are plans for how Biohazrd will be able to help* you wrap libraries in a safer manner, but it currently doesn't provide much support out of the box. (Right now the focus is on the unsafe interop layer.) You can of course implement the safe layer yourself (or even use Biohazrd to generate it in your own style), but I'm still shaping how Biohazrd "recommends" (and helps) you do this. (*Part of the reason this I use the word "help" here is I don't believe there's a one size fits all approach for wrapping native interop. For example, an approach that works well for TensorRT wouldn't work well for PhysX.) |
Beta Was this translation helpful? Give feedback.
-
|
I developed a workaround for the
Changes are live on the private InfectedTensorRT repo if you want to check them out. I haven't had time to look at the non-ONNX sample again to figure out why it's broken. |
Beta Was this translation helpful? Give feedback.
-
Hi, this project looks really interesting. 👍 I want to use Tensorrt from c#, but as is unfortunately often the case there is no C API so simple p/invoke is not an option. How easy do you think it would be to wrap this using this project? For Windows.
Beta Was this translation helpful? Give feedback.
All reactions