How often are KernelFunctionOperations needed in practice?

[tomchor]

Some context to my question:

When I started using Oceananigans, there were many "medium-complexity" abstract operation expressions that couldn't be compiled into GPU code. Plus, (I think) we weren't aware that we could nest ComputedFields to achieve a more complex result. This motivated us to create KernelComputedFields (which now have been replaced by the simpler KernelFunctionOperations), and motivated me to create Oceanostics to save useful kernels.

Nowadays, however, abstract operations are much better (thanks to @glwagner), and we know that we can nest abstract operations (computed Fields really) to calculate a complex result. So generally when needing to calculate a complex diagnostic we're faced with two options:

1 - Nesting computed Fields of low/medium complexity. This is very easy to do and produces readable code, but it's inefficient.
2 - Using KernelFunctionOperations (either by hand, which isn't trivial at all, or through Oceanostics if that operation is there). Downside here is that it's harder, but upside is that it's more efficient.

However, in my experience abstract operations are so good now (i.e. are able to compile so many operations into GPU code) that I generally don't need to nest more than 2 abstract operations to get any diagnostic that I want. In fact, when comparing time-to-solution for simulations that need complex diagnostics, I find that the added time by using nested abstract operations is negligible when comparing to the same simulation using KernelFunctionOperations. And the code with abstract operations is generally more readable.

Of course, if you need to calculate many complex diagnostics very often, then the added time might pile up, but I suspect that this situation very rarely happens.

I was just wondering if you guys have also compared nested computed Fields against KFOs, and what are your thoughts on the issue. Do you end up using KernelFunctionOperations a lot?

[glwagner]

I don't personally use them much. But I guess there's a few other problems it solves other than compilation difficulties:

3. When some calculation isn't easily expressed via AbstractOperation. This is sort of the case for the parameterization implemented in How to create and save field-derived fields (e.g Richardson Number) and how to use it for a Turbulence Closure #2344 (because of the conditional statement). Though this may at some point be implementable with ConditionalOperation.
4. For calculating exact tendency terms like advective flux using internal Oceananigans functions. If we want to output the exact advective flux used in a model we have to use something like KernelFunctionOperation, eg

uc = KernelFunctionOperation{Face, Center, Center}(advective_tracer_flux_x, grid, computed_dependencies=(model.advection, model.velocities.u, model.tracers.c))

I think it's true though that in some ideal limit we would use abstract operations for everything, including the internal model implementation...

[glwagner]

An update on this from my end: it looks like the PTX / compilation errors are a lot more prevalent on ImmersedBoundaryGrid, so use cases for KernelFunctionOperation may increase when immersed boundaries become more prevalent. We'll see... !