c api next level manifesto

Taking the C API to the Next Level

Python's C extension API has historically been one of it's strengths. A huge ecosystem of high-performance scientific and numerical software has been built on top of it. It got us this far, but now it's holding us back. It's time to take it to the next level.

What needs to change

1. Don't return borrowed references.
2. Don't steal references.
3. Don't expose reference counting as part of the API.
4. Don't rely on pointers for object identity.
5. Don't expose the memory layout of Python objects as part of the API.
6. Expose Python (the language), not a specific Python implementation version.
7. Expose constructs generally useful in C, not constructs specific to a Python implementation version.
8. Provide an explicit execution context.

All abstractions leak, but the current C API leaks like a sieve!

Constraints

1. The existing ecosystem of C extensions should continue to work.
2. Performance of existing C extensions run on CPython should be no worse.
3. There should be a sensible migration path for big C extensions (e.g. numpy).
4. There should be a way to easily pick up common errors introduced by migrating.

In short, we should avoid making the lives of maintainers and users of C extensions difficult.

Next steps

Excitingly, we believe all of these goals are achievable within the constraints. We're currently working on one way to achieve them and we'd love input and feedback and wecome contributions of all kinds!