c api next level manifesto

Taking the C API to the Next Level

Python's C extension API has historically been one of it's strengths. A huge ecosystem of high-performance scientific and numerical software has been built on top of it. It got us this far, but now it's holding us back. It's time to take it to the next level.

The next level

If everything below is achievable -- and we believe it is! -- we'll arrive in a wonderful new future where Python implementations can experiment with all sorts of amazing new features such as:

• tracing garbage collectors
• nurseries for short-lived objects
• sub-interpreters with separate contexts
• specialised implementations of lists
• removing the GIL
• avoiding the boxing of primitive types
• just-in-time compilation

... and many other things you can imagine that we haven't!

It might take awhile to reach this future, but the sooner we get started, the sooner we'll arrive!

What needs to change

1. Don't return borrowed references.
2. Don't steal references.
3. Don't expose reference counting as part of the API.
4. Don't expose the location of objects in memory.
5. Don't rely on pointers for object identity.
6. Don't expose the memory layout of Python objects as part of the API.
7. Don't expose static types.
8. Expose Python (the language), not a specific Python implementation version.
9. Expose constructs generally useful in C, not constructs specific to a Python implementation version.
10. Provide an explicit execution context.

All abstractions leak, but the current C API leaks like a sieve!

Constraints

1. The existing ecosystem of C extensions should continue to work.
2. Performance of existing C extensions run on CPython should be no worse.
3. There should be a sensible migration path for big C extensions (e.g. numpy).
4. There should be a way to easily pick up common errors introduced by migrating.

In short, we should avoid making the lives of maintainers and users of C extensions difficult.

The good news

Excitingly, we believe all of these goals are achievable within the constraints. We're currently working on one way to achieve them and we'd love input and feedback and welcome contributions of all kinds!

For a more detailed explanation of what needs to change, see What needs to change and why.