Quantity API

[andrewgsavage]

There has been some interest in a Quantity or Units API, similar to the array API standard.

hgrecco/pint#2101
astropy/astropy#8210
astropy/astropy#13460
astropy/astropy-APEs#91
https://pydims.github.io/pydims/developer/index.html

A Quantity API that standardises methods and attributes would make it easier to write code that supports multiple unit libraries. At present this is difficult and has lead to multiple integration packages being written to support units, eg pint-xarray and xarray-quantity.

Although the implementations of the unit libraries differ, they share the same core concepts of a Quantity and Unit. I echo the suggestion to create a Protocol, similiar to https://github.com/nstarman/units/tree/main/src/units/api However I think the first version of a Quantity API should have a smaller scope to make it easier to agree on and adopt.

At time of writing, support for the array api is in development. An initial implementation suggests standardising the following methods would allow an implementation to be used across multiple libraries.

Quantity.__init__(value, units)
Quantity.units / .unit
Quantity.magnitude / .value
Quantity.m_as(units) / .to_value
Unit.__pow__
Unit.__add__
Unit.__sub__
Unit.__mul__
Unit.__div__

Unit.__sub__ is used to get a delta unit, eg for temperature. Quantity.__sub__ could be used instead.

That's not many functions at all!
pint-pandas looks like it'd also need Unit(unit_string) and I've used Quantity.to(Unit).magnitude instead of .m_as. It also uses pint's errors and formatting but that's quite complex.

Equally I think this could be easy for the unit libraries to implement; for example pint would add Quantity.value that returns Quantity.magnitude, astropy would add Quantity.units that returns Quantity.unit and so on (attribute names tbd!), there's no need to depreciate the current attributes. It would be good to change documentation to use the standardised methods to encourage their use.

I'm not against standardising other classes or methods like Systems or <<, I'd just rather do so in a future version.

[lucascolley]

Slightly off-topic, but I was wondering about a potential name for a standardised project. How about pyquantity (perhaps stylised as PyQuantity)? It seems like the standardised quantity/unit API and support for functions in the array API standard could belong outside of the astropy org.

[andrewgsavage]

Or QuantiPy! Yes I agree it should live outside astropy to signal it is common across libraries. Perhaps it would be a fit for https://data-apis.org/ ?

…

On Sun, 5 Jan 2025, 00:13 Lucas Colley, ***@***.***> wrote: Slightly off-topic, but I was wondering about a potential name for a standardised project. How about pyquantity (perhaps stylised as PyQuantity)? It seems like the standardised quantity/unit API and support for functions in the array API standard could belong outside of the astropy org. — Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEMLEEAMFXDULH4MCA4MV32JB2LBAVCNFSM6AAAAABUSSR52OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZRGQ2DOOBYGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[lucascolley]

Or QuantiPy!

Haha I did check that, but there is https://pypi.org/project/QuantiPy/

Perhaps it would be a fit for data-apis.org ?

I would be open to that, although the "standard" set by the work on the array API standard is quite high, and may be a little overkill for the task at hand. It might make more sense to get some rough prototypes together first and see how collaboration goes. Then if it looks like there are difficult decisions to make, we can look to following the data-driven approach and https://github.com/data-apis/governance/blob/main/process_document.md.

[SimonHeybrock]

I feel that a Units API should be favored over a Quantity API. Or at the very least, both should exist. The motivation for this is the (downsides of) nesting. People also want named dimensions (Xarray, NamedArray, Scipp, PyDims, ...), masks, and more (vectors, bin-edge arrays, ...). If we define, say,

Quantity.__init__(value, units)

then each of the extra components such as masks or dimension names may need to be constructed and accessed in a nested manner. We'd get something like

arr = MaskedArray(NamedArray(('x',), mask), NamedArray(('x',), Quantity(np.arange(5), 'meter')))
# with value accessed as
arr.value.value.value

which is obviously a problem. One could try to add special accessors at different level, but I think this will quickly get out of hand. I have experimented with something like this a while ago (see https://github.com/scipp/scippx and https://discuss.scientific-python.org/t/multiple-duck-array-layers-composability-implementations-and-usability-concerns/552) and my conclusion was that it probably is not a workable solution.

Thus, the approach I favor is what I have outlined in PyDims: Have API standards on the lowest level for the various components (mainly array and unit, but there may be more such as coords), then integrate all components in a single step. Now, integration in a single step does not imply that this would be a single library doing the integration. People will always have different and partially conflicting requirements so I would assume we would have multiple such libraries coexist, such as a plain Quantity library in addition to others that also add dimension names or masks.

At least this is how for my thought process went. It does leave important questions open though. For example, one would like to use SciPy and friends with units but it is not clear to me how to achieve this.

[nstarman]

I agree about the need for both a Quantity and Units API and also that they should probably be separate, but developed in dialogue with one another. In the various versions I've coded up (https://github.com/nstarman/units, https://github.com/nstarman/astropy-APEs/blob/units-quantity-2.0/APE25/report.pdf) I proposed a (pseudocode with illustrative names and attributes)

class QuantityAPI(Protocol[Value]):
    value: Value
    unit: ...
    def to_unit(self, unit): ...

class ArrayAPI(Protocol): ...

class QuantityArrayAPI(ArrayAPI , QuantityAPI[ArrayAPIType], Protocol): ...

So long as we all agree on a common Quantity API, and likewise a common Units API, then it's totally fine to have a proliferation of implementing libraries with different focuses, e.g. non-array values like python scalars or lists, or specific Q subclasses like longitudes, or masked values, or named fields. If there's eventually a Mask API then a masked Q class would be the intersection type, like QuantityArrayAPI is for the QuantityAPI and ArrayAPI.

[lucascolley]

@SimonHeybrock at least while we are in array API compatible territory, I think there is a solution, but I haven't tried it yet. Idea:

• in all creation functions (asarray, arange etc.) allow wrapping via a kwarg (e.g. units=..., mask=...) and accept arbitrary **kwargs
• pass **kwargs to the level below

I think this pattern should work as long as you have array API compatible layers throughout, except maybe for the top layer, which only needs to consume standard arrays and provide the asarray interface. So I think something like this should work:

from pint_array import pint_namespace, UnitRegistry
from marray import marray_namespace
# XXX: imagine that `dask_namespace` exists
from dask.array import dask_namespace
import cupy as cp

ureg = UnitRegistry()

dacp = dask_namespace(cupy)
pmdacp = pint_namespace(marray_namespace(dacp))
x = pmdacp.arange(4, chunks=2, mask=dacp.asarray([False, True, False, True]), units=ureg.pint)

[SimonHeybrock]

@lucascolley That solves the manual initialization, but the problem goes deeper, I think. Accessing nested "value" properties, unclear or incompatible order of nesting, and creation of derived objects in binary operations.

[lucascolley]

Accessing nested "value" properties

That does seem like a problem. Perhaps solvable if each array carries a .info (or something unique) attribute which can propagate these properties to the top level?

unclear order of nesting

That shouldn't be a problem, at least with standard-compatible wraps. x.__array_namespace__().__name__ should describe the precise ordering of nesting.

incompatible order of nesting

Perhaps the onus could be on users to not mix incompatible namespaces? The "users" here are still library authors rather than end users.

creation of derived objects in binary operations

could you give an example?

[SimonHeybrock]

creation of derived objects in binary operations

could you give an example?

As a very simple case, something we do quite often is compare two quantity-arrays to obtain a mask. That is,

mask = arr1 < arr2

should return an object that does not have a unit (or unit=None), but that has everything else (dims, coords, ...). That is, one might need to be able to remove a particular "layer" from the "stack" of array libraries.

Footnote: I don't think using unit='dimensionless' is correct for masks.

[lucascolley]

Footnote: I don't think using unit='dimensionless' is correct for masks.

I can see why it feels wrong, but practically, are there any problems?

[SimonHeybrock]

Footnote: I don't think using unit='dimensionless' is correct for masks.

I can see why it feels wrong, but practically, are there any problems?

We (Scipp) have done that in the past but eventually decided to distinguish dimensionless from unit-less. This was not just for masks but also (memory) index arrays, etc. Unfortunately we were not so good at documenting the full reasoning for the decision at the time. I think it is was mainly about being able to detector user errors better. It was also weird to have units for say, an array of strings or other Python objects, so we wanted unit-less anyway.

Edit: One example of the problems we have had, albeit a bit cryptic, reading it now: scipp/scipp#2396

[mhvk]

Astropy and Quantity2 also remove the unit when booleans are involved.

Indeed, I've argued for astropy (and may do so here) that Quantity should only take real or complex for the value - integer quantities make essentially no sense.

[neutrinoceros]

integer quantities make essentially no sense.

Preach. It comes up regularly in unyt's bug tracker that allowing quantity arrays to have an int dtype breaks end applications in surprising ways, and I don't really have an argument for why this is supported in the first place.

[nstarman]

The Array API doesn't police how dtypes are defined / used. It has a minimum set of dtypes (https://data-apis.org/array-api/latest/API_specification/data_types.html) but many libraries define more dtypes, e.g. for ML. I think it's literally impossible / highly impractical / way too much work for us to enumerate for all python array libraries what subset of dtypes are allowed. Especially for the Quantity API I don't think this makes much sense. Individual implementing libraries can choose to take on this Sisyphean task, but for the Quantity API it's probably best to say that values are e.g. the Array API and leave it at that.