Python bindings with PyO3

[alecandido]

Hi @cschwan, I was reading about PyO3, and after a while I started thinking that it may be not so hard to provide bindings for pineappl using it.

I don't know how much you care, but I decided to make a try, just in my spare time. The goal it's to port a minimal subset of things, just a proof of concept, to expand after to the full one.

It should not take so much, but of course it's not my main business, so it will take its time. Do you mind if I try?

I also thought it was a good way to familiarize a little more with pineappl, since soon it will be useful (both because of yadism, but also because I am going to work on pineko too, and I know you are the expert on pineappl, but it won't hurt to know something more).

If you agree maybe you can suggest me which one could be "a minimal set".

[cschwan]

Hi @alecandido , I don't mind at all, and in fact I think that would be very useful. Go ahead, and let me know if you need to know anything.

For a minimal set of structs and methods, just have a look at #50 , in particular the file https://github.com/N3PDF/pineappl/pull/50/files#diff-4a3c785edeb3672939e989f1aedce3a888747e39fabed535fba4a7909f13f7b2. This file more or less contains everything important from the user side.

Don't forget to change the author of pineappl_py to yourself.

[alecandido]

First, small, trade off due to python + rust interaction:

• maturin, the zero-configuration tool for building and deploying pyo3 packages, it's enforcing a specific layout, and it is perfectly fine
• unfortunately, the name of the PyPI package will be the same of the rust package:
  • in both the case of a pure rust package and a mixed python/rust one you are free to choose (the same) unrestricted name for the import package (i.e. the name you use when you do in python import my_package)
  • but there is another name involved: the one you use to do pip install my-package, and it's the distribution package

And now the problem: while the current python package it's named pineappl (and it makes sense), now we can't choose that name any longer, using maturin, because, since it's extracted from Cargo.toml package.name, picking pineappl would conflict with the other one (the original) in the workspace.

Maybe we can overcome this little pain by using setuptools_rust, but it's far more verbose and requires many more files (i.e. at least a couple), rather than the simple standard/enforced layout of maturin.

Is it worth?
(we can call pineappl_py the distr package and still do import pineappl`, i.e. what I'm currently doing)

[cschwan]

It's probably best to rename it, since keeping the name of the other python package suggest they're compatible, which they probably won't.

[alecandido]

You're right, we can think a little on the name (pineappl_py, pineappl-py, pyneappl, ...), but it's just the installation one for distribution package, so you will simply find pineappl_py when you search on PyPI, slightly uglier but it has the further advantage of being explicit: it's not shipping the rust lib only (possible for PyPI but useless) but also the proper bindings, even if it's of course expected from a PyPI package...

And it will keep the complexity low, without any extra than bindings themselves.

[cschwan]

Do we actually need a separate crate for the python bindings? It might be beneficial to put everything into the main crate, preferably only when the python feature is activated. In this way there's less code duplication and adding a method is easy (?!).

[alecandido]

You're right, doing it in the main crate would be easier, but when I decided to split I made two considerations:

1. maybe python it's not so special, so why is the C one split and python should fit in the main one? And in this way the python would be rather more coupled, because the development of python wrappers it's somehow interleaved in the library itself
2. I didn't want to mess up with the library myself, because in this way I was completely sure that I wasn't ruining anything at all

On the other hand there are some pros:

• there are examples of other projects doing the same
• maturin itself it's supporting this layout
• I probably need some python code for making proper classes (because of limitations of pyo3 and the struct-class mismatch) and for this I had to make a pineappl package (it would be pineappl/pineappl/pineappl inside the main crate)

Nevertheless being outside the main crate I need to wrap all the structs and functions to annotate them, and it's a pain in the neck, so it would be much better to use the main crate just for this reason.

What do you think?

[cschwan]

If it isn't master you can do whatever you like. I'm more concerned that whenever I add new Rust code in the future, I'll have to update lots of code in other places. This already happens for the C API, which definitely doesn't support all methods the Rust interface has. I was already thinking about merging the pineappl_capi into pineappl. But for the time being we should focus on adding the functionality.

[alecandido]

Fine. Let's say that since I started this way I'll try to complete just the proof of concept, with the goal of running import-yadism-yaml.py with the new wrapper.

As soon as I get there we can merge in dis and than decide the best layout. Just not to get lost in changes.

[cschwan]

Agreed!

[felixhekhorn]

@cschwan

change create c'tors in Python?

you were thinking of https://github.com/N3PDF/pineappl/blob/fe5b83989144c4360db156daf7a12134d5ac86c4/pineappl_py/pineappl/grid.py#L23, right?
then we can't because we need the c'tor with the raw wrapper object here: https://github.com/N3PDF/pineappl/blob/fe5b83989144c4360db156daf7a12134d5ac86c4/pineappl_py/pineappl/grid.py#L52

[felixhekhorn]

do we need PyBinRemapper?

it is still used here: https://github.com/N3PDF/yadism/blob/be995caf3a7f76dafd06336ba83fc570bc31f0a3/src/yadism/output.py#L236
so I guess the answer is yes ... or is there some other way?

[felixhekhorn]

add tests
add more documentation

actually, where? Rust@pineappl, Rust@pineappl_py, Python@pineappl_py ?

[felixhekhorn]

check README

I made a proposal in e8e1809 - please take a look

[cschwan]

add tests
add more documentation

actually, where? Rust@pineappl, Rust@pineappl_py, Python@pineappl_py ?

What I'd like to see is tests of the Python interface, so tests written in Python!