Shared object support

[n-eiling]

Adds support for launching kernels from shared objects loaded during runtime using dlopen. As this is how pytorch uses CUDA, this should enable pytorch support to Cricket.
This involved adding support for decoding the fatbinary metadata before the embedded cubin ELF in binaries compiled by nvcc. Cricket becomes able to extract the cubin from a binary, send it via RPC to the server, where it will be executed using the driver APIs cuModuleLoadData.

• decode fatbinary
• extract cubin
• send cubin to server
• add registry for tranferred cubins and kernel functions so Cricket is able to identify them when launching kernels
• switch over old kernel launching functionality to always use the new registry instead of relying on kernel locations being the same on client and server
• use libelf to read kernel infos instead of relying on cuobjdump which does not support in-memory ELFs
• read parameter infos using libelf.
• enable reading CUDA elfs with debugging infos and compressed elfs
• Test with minimal pytorch (deactivated some features, no kernel compression)
• Test with default pytorch (no kernel compression)
• Fix compression and test with default pytorch (with kernel compression)
• fix CI
• large-scale test / verification
  • YOLOv5
• cuDNN implementation

This also makes LD_PRELOADing on the server side not necessary anymore, as we now also extract and send cubins for normal applications.

This is work in progress. Addresses #6

[nravic]

Just saw this! Thanks for taking it on haha, was about to start this weekend. I'd love to help with this effort, let me know if there's anything I can do.

[jin-zhengnan]

@n-eiling When will the entire test be completed？ I am looking forward to this！

[n-eiling]

I updated my todo list. There are still some open issues that need adressing. CUDA relies on the .nv.info section for information regarding kernel parameter sizes and offsets. I used to parse them using cuobjdump, but this doesn't support in-memory ELFs - only files.

[wangtianxia-sjtu]

oh really? hm I haven't looked into nvrtc yet. Currently I am trying to get YOLOv5 working. Do you know some other useful pytorch examples I could use?

Thanks for the reply. I just learn from a paper from Microsoft where they claim that they intercept nvrtcCompileProgram call to extract kernel info.

I do not have any examples which will actually use nvrtc APIs. But I have found that in some versions of Pytorch, it will generate some kernels with nvrtc when you run import torch. And we can find in pytorch source that it will use nvrtc.

[n-eiling]

I will merge this because the branch has diverged quite a bit and the original PR feature is working well. For pytorch, I still have some issues with the cudnnBackend API, which I will work on on a different branch.