python-fortran-cpp template

This repository is a minimal example to make a python package that makes use of fortran (f2py) and C++ (cython) extensions with a setup.py file.

It also contains other patterns such as script installation, with the excellent argparse, and ipython notebooks, as example of user interaction.

With these various tools at hand, jungling between git-managed packages and normal work folders (NOT managaed by git, but keeping track of the git-version used), the work becomes very productive and reproducible. And excellent to share work in a team.

Dependencies

This template was tested in python 2.7 with:

• numpy : 1.9.2 (f2py)
• cython : 0.22 (c++ extension)
• ipython : 3.1.0 (ipython notebook)

Install

Install the pacakge on your system:

python setup.py install

Usage

Then you may:

• execute myscript.py from anywhere on your system (to see excellent argparse at work)

    myscript.py --help
    myscript.py --version
    myscript.py -a 1 2 3 -b 2 2 2
    myscript.py -a 1 2 3 -b 2 2 2 -o c_div
    myscript.py -a 1 2 3 -b 2 2 2 -o f_mult
  

• copy the notebook template in your work directory, and start working...

Then feel free to use these pattern for your real work by editing these files at your convenience. Do not forget:

• setup.py
• mypackage/__init__.py

Workflow

You need to install everytime you want the changes to be accessible from the notebook or other components that depend on mypackage. For that reason, it is better to only add functions and pieces of code that you consider stable.

In the development process, it is more convenient to work in the notebook and only add the functions to the actual package when you think you won't edit them every 5 min. Then your next notebook will be thinner, as it only need to from mypackage import my_func instead of having the whole body inside.

Note, if you need to modify / update some piece of packaged code later, you can always use the magicc command %load mymodule.py to have the content of the file loaded in the notebook cell, then edit out the unnecessary part, and expand / debug the bit of code you are interested in from within the notebook, without having to re-install the whole package at every new change. When you are happy with the changes, copy back into the actual package, make a git commit etc..(of course, this assumes the bit you want to edit is not a dependency for other parts of the code)

Notes on packaging

Packaging is useful to have your code importable from everywhere (as any package installed with pip) and to cleanly separate base functionality that you do not modify too often from daily work that will be done preferentially in the notebook, or anywhere on the disk with various input data, notes, output figures etc..., which you do not want to have tracked in git but instead archived.

The built-in packaging in python 2.7 is distutil:

setup(name = 'mypackage',
  description       = "example package using c++ and fortran",
  author            = "Your Name",
  packages = ["mypackage"],  # also add subpackages !!
  scripts = ["scripts/myscript.py"],  # add scripts to be called globally
  )

Extensions written are added via a ext_modules parameter. f2py and cython have highly simplified the way of programming extensions, by providing their own setup function and an Extension subclass (for f2py) or by providing a user-defined build_ext parameter (cython).

The way they do that does not seem to be compatible, so if both are to be used in the same pacakge, this needs to be done with two separate setup calls.

So in this example, we first install the python package without any extension (first) setup, then install the cython and f2py extensions as subpackages, with two additional setup calls.

... extension : fortran + f2py

The simplest to use (if you know fortran). Basically, as long as your fortan code only have simple types as input/output (scalars, arrays, strings) (no derived types!!), and make use of the intent(in) / intent(out) qualifiers, you do not need to do anything more than use numpy-extended Extension class and setup function:

from numpy.distutils.core import Extension
from numpy.distutils.core import setup

flib = Extension(name = 'mypackage.flib',
                 extra_compile_args = ['-O3'],
                 sources = ['src_fortran/mymodule.f90'], # you may add several module files under the same extension
                 )

setup(
    ext_modules = [flib]
    )

... extension : c/c++ + cython

In addition to c/c++ source files, it is necessary to add a definition indicating the c++ header:

cdef extern from "path/to/my_header.h":
    int array_div(double *a, double *b, double *c, int n);

(yes, it is redundant since this info is already present in my_header.h I am happy to hear your suggestion for less redundant alternatives)

And a wrapper in cython (see cython/*pyx file)

When this is done, the setup.py part is not more difficult than f2py:

from distutils.extension import Extension
from Cython.Distutils import build_ext

clib = Extension("mypackage.clib",  # indicate where it should be available !
                      sources=["cython/my_func.pyx",
                               "src_cpp/my_func.cpp",
                               ],
                      extra_compile_args=["-O3", "--std=c++11", "-ffast-math", "-Wall"],
                      language="c++")

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [clib]
    )

Limitations of f2py fortran

Just a few things out of my own experience with f2py (please refer to the official doc for more exhaustive information).

• use ìntent(in/out/inout) mentions
• use simple input/output arguments in subroutines and functions. This means in particular, no derived type, no allocatable arrays. f90wrap seem to relax this constraint, but not sure this will work for packaging.
• the approach of globally defining integer, parameter :: dp = kind(0.d0) and then using real(dp) instead of double precision, as encouraged on fortran90.org, does not work with f2py. You should use old-fashioned double precision (or plain real(8), which is more confusing to me than just double precision).
• using private module with only a few public methods/functions does not work when wrapping f2py. f2py blindly attemps to wrap everything it finds in the module, and a subsequent use moodule, only: my_private_func will fail... So keep everything public.
• Avoid optional arguments. present function for optional arguments does not work properly. An optional floating point argument will have the value 0 even though it is not actually provided. Again f90wrap provides a work-around, but I do not see clearly how to write a custom setup.py file using f90wrap, so I will leave it for now.
• A work around for some of these limitations is to create a new module that import only the relevant functions that you want to see wrap, and write a new, f2py-compliant fortran function/subroutine... If you are writing a wrapper anyway, you may alternatively use iso_c_binding as described below. Otherwise, again the f90wrap should be tested as a command-line tool to re-write your sources into something compatible.

iso_c_binding for fortran : cython and ctypes

It is also possible to make your fortran code c-compatible, using the iso_c_binding module in fortran, by following instructions on fortran90.org. This involves more changes to your source code than f2py would need, but then your code should also be callable from C++. And the C++ wrapping techniques (such as cython or ctypes) become available, as described there.

Credits

Much of the cython part was inspired by the dbg package.