Support Cython pure Python mode

[dnicolodi]

Cython supports a pure Python mode where code annotations are used to add Cython types to otherwise valid Python code. This allows to have modules that can be imported as Python modules or compiled into extension modules. The filename extension for this modules is .py to allow them to be imported by Python. Currently, Meson does not recognize .py files as valid source code for Cython.

Should the .py extension be added to the list of extensions handled by Cython? The drawback of doing so is that someone my accidentally pass Python source code files to a compilation target and get surprising results instead than an error.

[eli-schwartz]

It is something I have thought about in the past. I think it would be neat but not sure the best way to do it.

It's of course relatively straightforward to circumvent the issue by hand-rolling the transpile step using custom targets and meson.get_compiler('cython'). IIRC scipy still uses custom targets for some pyx cases that require more manual handling than meson knows how to do yet, so there's actually precedent.

[Freed-Wu]

Comes from #8693

I have another method:

py.extension_module(
  '__init__',
  [
    configure_file(input: '__init__.py', output: '__init__.pyx', configuration: conf_data),
  ],
  dependencies: deps,
  subdir: 'pyrime',
  install: true,
  cython_args: ['-3I' + meson.current_build_dir()],
)

A full example is here

[Freed-Wu]

I guess change https://github.com/mesonbuild/meson/blob/master/mesonbuild/compilers/compilers.py#L69 is OK.

[rgommers]

I'm not sure that associating .py with Cython is healthy. There are a number of other tools that do a very similar job to Cython and also work with .py files, like Pythran, Mypyc and Transonic. When packages use more than one such tool, a .py association with one of them is going to be a little odd.

IIRC there are also potential backwards compat issues, because .py files now get translated into order dependencies.

My sense is that supporting this via custom_target / generator will be better - and the documentation at https://mesonbuild.com/Cython.html can be improved to mention this.

[eli-schwartz]

I'm not sure that associating .py with Cython is healthy. There are a number of other tools that do a very similar job to Cython and also work with .py files, like Pythran, Mypyc and Transonic.

Transonic is new to me. It and pythran seem like quite reasonable things to want to do in addition to cython.

mypyc is a dead end, I'm afraid. Despite years of interest they are still impossibly heavily bound to the use of setuptools as a low-level implementation detail of their transpilation pipeline. They also require running exhaustive full-project type checking and then non-reproducibly generate the transpiled C source, so repeatedly rerunning setuptools.setup() build_ext will repeatedly recompile the entire project (with every single module in your project in one file, imported via capsules!) based on an arbitrary content hash (by which I mean the unified module that contains the entire project uses a hash as the import name for all the stubs to derive a capsule from).

I don't see any interest from them in supporting external build system integration.

My sense is that supporting this via custom_target / generator will be better - and the documentation at https://mesonbuild.com/Cython.html can be improved to mention this.

Well, a possible/plausible enhancement opportunity is to add direct support for specifying a generator, e.g.

py.install_sources(
    'foo.py',
    'bar.py',
    cythonize: get_option('python.can_cythonize'),
)

[rgommers]

Hmm, interesting idea. Tacking it only py.install_sources loses the ability to use c_args, cython_args and other extension module handling though. Not sure how many of the "optional extension" users would care.

Another option that came to mind as a merger of the two ideas above. Add a transpiler: 'cython' (extensible to 'pythran' & co in the future) to py.extension_module, and if that option is used, treat .py files as sources for that transpiler. Not sure if it's better or not.

[eli-schwartz]

That sounds pretty neat, actually.