DMF implementation status

See also: DMF (Data Mapping File) spec and DMF Implementation notes

Status as of mid-2024

Here I’d like to reply to one point not covered before - immediate usability DMF. As a short and quick reply - unfortunately no, you can not use it with stock upstream NUT at the moment, and not for USB when customized code is involved. That said, it is a long-term hope that one day this would be possible. Each recent NUT release milestone had some work that chipped away the technical barriers to including DMF into the main project.

For a longer version, see below.

History of 42ITy and DMF

DMF was designed and developed, fairly well tested and then a bit abandoned in the 42ITy fork of NUT, which went back to using upstream NUT for their appliances.

The developed code base however, for a long time, acts as the source of backports for features and vendor device nuances into upstream NUT to benefit everyone. That codebase is available in the FTY branch, including DMF along with many other changes (thousands of commits) so separating them into cleanly visible and revisable increments, and digesting in the upstream code, aligning with protocol and other changes that happened over the years, style and tests/quality gates, is quite an effort.

Conversely, merging back the upstream NUT into the FTY codebase (to achieve green CI survival) was an effort that took a few years, since the baseline was NUT v2.7.4 and codebases diverged significantly in some spots - so there were quite a few warnings found that needed addressing.

Currently the FTY branch is relatively recently synchronised with NUT, at least buildable and passes CI tests, including the DMF aspect. And since the codebases converged again recently - now as stuff gets upstreamed here and there, a mere resync and git diff FTY..master helps see how much is left, what commits were missed or typos added, and what can be picked up next :)

Goals of DMF

DMF per se is an effort to separate the binary driver from data mappings, in cases where logic remains the same and just the data points differ from device to device (e.g. SNMP OIDs to query for information about different aspects of UPS or ePDU state). This should well apply to USB HID and maybe to nutdrv_qx; a recently added concept of hwmon driver (talking to sysfs nodes) is also a good candidate.

DMF in limbo (so far)

Initial development in the 42ITy project provided a separate library for DMF general foundations, and build-time changes to the snmp-ups driver (and nut-scanner snmp mode) to use mappings from XML files rather than built into the driver binary once and forever until a rebuild. This allows for smaller binaries on one hand, and for field-maintainable mappings (edit XML, add another) to support new devices without changing NUT packages and programs (of course, PRs would be welcome to add such changes to the upstream library to be included in future releases). In fact, the few differences for the core driver are hidden by ifdef (so there are two binary builds, with built-in mappings and with DMF support), and the nut-scanner tool is dually-capable in the same build. In fact, to test the theory, NUT codebase in the branch provides scripts that convert *-mib.c tables into equivalent DMF XML files, so a copy of snmp-ups-dmf driver program with the bunch of text resource files is equivalent out of the box to a monolithic snmp-ups binary built from same codebase, but more extensible in the field afterwards.

Part of "abandonment" of this approach was the uncertainty of how correct the big wad of new code it is (hard to guarantee with that older NUT baseline which naturally had thousands of warnings so lots of diagnostic noise) and suspicions that it had memory leaks so maybe had issues with long-term stability (not proven to be definitely the root cause of some issues, as far as I am aware). Those were issues entangled with semi-commercial project priorities (rush to market - pick the lowest hanging fruits, backlog the rest) so it was left on a backburner, awaiting a revival in upstream NUT.

Another part was more technical, a sort of stalemate in design: many mapping tables in NUT involve data conversions (e.g. date formats, temperature units, integer milliVolts to floating-point Volts, text labels for numeric enum values, etc.) for data transfers from a device or writes back to it. In the current C codebase (*-mib.c, *-hid.c) there are helper methods to which we can point from the tables. An equivalent for DMF, with mappings conveyed by text files, involved adding LUA scripts with a LUA-C bridge to pass the data from binary driver to scripting context and back. This worked reasonably well; the problem was having two potentially different implementations of the helpers (C and LUA) for mappings provided with NUT - and no good way (at least back then) to either compare that they behave identically or to eliminate one (likely the native C…​ but then add LUA as a requirement for NUT builds everywhere and provide the LUA implementations of helpers via files or built-in strings). This aspect stalled in discussions - not an insurmountable problem, but no single apparently good solution (at least fitting the commercial project’s constraints) was finalized.

Probably a good way forward here is to track the C helper methods in the NUT codebase in separate files (or wrapped by macros) so that scripting can identify their existence. Then we can make sure the same set of code is available in LUA, and that all of these are covered by unit tests. This part seems fairly easy to automate, and can lead to a decent library of helper implementations provided by NUT with DMF-capable releases. Probably some duplicate efforts can also be identified this way. Note also that the "problem" originally concerns the subdrivers provided by NUT as C mapping tables and the requested ability to provide functionally identical DMF clones. Discrepancies of C and LUA behaviors are not a problem for drivers where DMF resource files are the only code base that defines the device interactions.

On the other hand, if we end up with a well defined library of C implementations of popular/common helpers for the mappings, maybe it would be viable to (re-)use known C method names from LUA with minimal one-line scripts (or specially processed XML tags) pointing to the C library methods built into a driverbinary right away - also simplifying the field support for new devices that rely on same concepts as something that was already handled earlier.

Summary

To summarize - currently we have a branch (which is not yet upstreamed, but finally is relatively close to that) with the core DMF library that can store and process the mappings and LUA helpers, and a consumer of that library for SNMP mapping purposes, further used by snmp-ups driver and nut-scanner tool, and scripts that convert C mapping tables to DMF XML files. There are some nuances about ensuring and testing their equivalence. Once the approach is deemed unambiguously good, further sets of consumers can be added for several other categories of drivers where logic is cleanly separable from data mappings - most of these are largely C tables already.