Adds Python bindings to Caliper with pybind11

[ilumsden]

This PR adds initial Python bindings to Caliper using the pybind11 library. The resulting Python library (named pycaliper) contains bindings for the following Caliper APIs:

• cali::ConfigManager
• cali::Variant
• cali::Annotation
• cali_attribute functions
• cali_begin_* functions
• cali_set_* functions
• cali_end function
• cali_*_byname functions
• cali_[begin | end]_region functions
• cali_[begin | end]_comm_region functions
• cali_[begin | end]_phase functions
• cali_set_global_*_byname functions
• cali_make_loop_iteration_attribute
• cali_init
• cali_is_initialized
• cali_attr_type enum
• cali_attr_properties enum

These bindings have been tested using the annotations added to Hatchet and Thicket in LLNL/hatchet#142 and LLNL/thicket#195.

Remaining Work:

These bindings are complete and could be merged as-is. However, it would be ideal if I added docstrings to the Python module before merge.

[daboehme]

Great work, thanks @ilumsden! In addition to the documentation it would be great to add an example like the c/cxx/fortran-example apps in the examples/apps directory for the Python annotations. A unit test would also be great. The Caliper unit tests are actually run through Python already so it shouldn't be too difficult to add; check out the tests/ci_app_tests directory. I'm happy to help if needed.

[ilumsden]

Outstanding work:

• Double check if unit tests are running and working
• Add docstrings to Python bindings

[ilumsden]

@daboehme this PR is now ready for review.

[daboehme]

Hi @ilumsden, I added a few fixes in a PR to your branch.

I can't run the tests via make test because the pycaliper module is not found. Did you ever get that to work?

It works for me when I add the installed location to PYTHONPATH and run the ci_test_py_ann program or py-example directly. It still doesn't work through the test_python_api.py script, even with PYTHONPATH set.

I got a type mismatch error with the config_preset call, I had to comment that out in the test. Doesn't seem to like the lambda.

I'd also like to move the sources to src/interface/, where the Fortran/C bindings already are.

[ilumsden]

@daboehme thanks for the feedback. I've moved the bindings to src/interface, and I've fixed the tests so that PYTHONPATH will be set correctly. Only problem right now is that my way of setting the path uses configure_file, meaning the Python test files will not be automatically updated by make.

I also fixed a few bugs in the test file where I was not using the Python API correctly. The test should now work. I'm currently messing with the GitHub Actions runner to see if I can get that to work.

[daboehme]

So it looks like the github actions test failed because apt didn't find the python-pybind11 package.

The attribute properties should always be passed in as an int since we can combine multiple cali_attr_properties flags.

Not sure if we should keep the variant in the Python interface. That's getting a bit too low level. Looks like you use it in the Annotation class and for constructing attributes with extra metadata. Do you have any specific use cases for these? If not I think we can take these out and minimize the interface.

[daboehme]

The opt argument should be an int. That's because you can combine multiple cali_attr_properties flags via OR. Same in all other functions that take it.

[ilumsden]

On the C++ side, cali_attr_properties is an unscoped enum. As a result, per the C++ standard, it acts as an implementation-defined integral with size less than or equal to int. That means it's perfectly acceptable to apply an OR to multiple cali_attr_properties values.

Beyond the C++ stuff, it's actually better to use cali_attr_properties for the Python bindings because pybind11 does some auto-conversion stuff between Python and C++ types that could get messy if I just accepted an int.

One alternative that would play nice with everything is to export the properties as an int in pybind11 and then wrap the values in a proper Python enum. In that case, the Python interface could be a subclass of enum.IntFlag, which would provide a native way of doing OR logic in Python that would still convert to int when passed to C++.

[ilumsden]

Ignore my comment regarding IntFlag. That won't really be possible because Python does not support method overloading.

[daboehme]

Looks like instrumentation.cpp still has a #include "variant.h"

[ilumsden]

My last commit fixes that.

[ilumsden]

Oh, wait. The CPP file. Give me a sec to fix that.

[ilumsden]

Ok. Everything should be working with this new commit.

I just tested it on Corona.

[ilumsden]

@daboehme now that CI is passing, I just rebased this branch. It should now be ready for "final" review and (hopefully) merge.