Fuzzy EUM Lookup

[stephankistner]

Hi!

Great work - I find it very exciting to see some community support for python and MIKE. We've developed our own "MIKE-IO" concept over the past 4 years, but now that there's an official version we will slowly start to migrate over to a more official one.

One of the concepts that we used was fuzzy lookups when writing Dfs files (FuzzyWuzzy module). This was especially advantageous when you want to do very quick I/O processing:

E.g, if we have a wind time series that looks like this (ASCII, Excel, or other similar format):

DateTime	Wind Spd	Wind Dir
2020-01-01 00:00	10.2	220
2020-01-01 01:00	11.1	230
...	...	...

The item names are already defined in the columns so we utilize a Fuzzy-lookup that finds the closest match of "Wind Spd" and "Wind Dir" in the EUM database (either local user or generic version). As for the units: 90% of the time we are working with standard units, so we can find that from our EUM database as well.

So then it's as simple as:

df = pd.read_excel("Wind_Timeseries.xlsx") 
df.to_dfs0("Wind_Timeseries.dfs0")

The accuracy of the lookup naturally depends on your input, so sometimes it necessitates "forcing" the item names to result in a more accurate search term:

df.to_dfs0("Wind_Timeseries.dfs0", items = ["Wind speed", "Wind Dir"])

Likewise for non-conventional units:

df.to_dfs0("Wind_Timeseries.dfs0", items = ["Wind speed", "Wind Dir"], units =["knots", "radians"])

The fuzzy-lookup is not unique to our Dfs0-I/O and is universally applied to Dfs1,2,3,U filetypes.

At the same time we also accept EUM integers EUM strings for the cases that you really want to be sure ;)

Another advantage of this approach is that we did not have to manually rewrite the long lists or dictionaries of EUM Items and Quantities: all of these are directly read from the .ubg files - as used in Mike Zero.

I'd happily send over the script - or at the very least a Notebook. It just needs a lot of refactoring, so it's not quite ready to share with the world.

Stephan Kistner
Coastal Engineer
PRDW Consulting Port and Coastal Engineers

[ecomodeller]

Thanks for this suggestion. The goal with MIKE IO is to working with MIKE files as easy as possible.

Are you aware that all EUM types are available as enums? Thus, if you know the first character, you can use tab completion. It is also possible to search with a string.

But all possibilities to make this even easier are welcome.

[stephankistner]

Yes, enums are very useful.

But for my example it will save you a line or 2 of code (still not much, I know).

The motivation behind this implementation was that you can spend time on processing and analysis rather than worrying about the correct read and write parameters - to simplify IO as you mentioned. But the Fuzzy lookup is a niche functionality with its limitations.

[jsmariegaard]

I like your ideas, @stephankistner. It's great to have these features that help you do what you want quicker 👍