Create unit table and unit conversion table

[han-f]

Create a table which contains common units with full name and abbreviation
Create another table with conversion between units (columns: from; to; factor)

[han-f]

Add a column note to the second table to include additional information if needed.

[han-f]

Created both tables in model_draft. Not all of the items in it are "pure" units though. Some are "things" that are sometimes translated into another thing, e.g. CH4 emissions measured in UNIT X into CO2 emission equivalents measured UNIT X. For such conversions the note column provides further information.

[stap-m]

I am wondering, if we could make a (seperate?) table with convertions like "giga to mega" and "kilo to milli" etc. and then combine it with factors for conversions like "Wh to Joule" etc. It would reduce the complexity of the tables and we are not running risk of forgetting some conversions.

[han-f]

Thanks - that sounds like a good approach, especially from a data related perspective. I made only one table with conversion since I wanted to give a user the possibility to immediately convert from one unit to another by applying only one factor (as a kind of service). But I am open to also include this third table as an option. What do others think?

[stap-m]

Further, convertions for units of time (s, min, h, day, year) are probably necessary, maybe also units of lengths (e.g. miles to km).

[MGlauer]

How do we manage conversions that are not products (e.g. Fahrenheit <-> celsius)?

[stap-m]

How do we manage conversions that are not products (e.g. Fahrenheit <-> celsius)?

I was also wondering about that. One possibility would be to fix special units, e.g. all temperatures in Kelvin. But maybe there are better ideas.
Do we find that problem with other units, too?

What about the conversion of currencies?

[han-f]

Re: currencies: I would refrain from adding conversions as these are dynamic (i.e. base years, inflation rates, exchange rates, the price of which basket of commodities to use to deflate etc..), and there are many approaches for different contexts.

[han-f]

@wingechr and I discussed a bit about currency conversion. An idea may be to add a separate table that includes conversions from one currency to another according to a specific rule which needs to be documented within a column or as explanatory text somewhere. Since currency conversion is highly dynamic, this would be a table that requires amendments as discussed in issue #337 and investigated exemplarily in here

[l-emele]

I saw that you also included conversion factors to convert t CH4 in t CO2eq, however these are not constant values but depending on which IPCC assesment report is used as source. Your values are probably from the IPCC AR5, but in modeling currently the IPCC AR4 values are used.

I would therefore suggest to delete this conversion, as there are more than one way for this calculation. As this is related to GWP values it might be useful to have a separate GWP table in the database. Should we create an issue to discuss this?

[han-f]

I have added a description of the source of the GWP conversion in the metadata accompanying the table. First, I had it included as a separate "comment" column, but thought this would be tedious. Having a separate GWP table may be a good idea. Let's think about it.

[l-emele]

Another observation:

The table contains a lot of slightly off conversion values, examples are:

• From PWh to GJ: 3599999999.9999714 instead of exactly 3600000000
• From ktoe to GJ: 41867.9999999993 instead of 41868
  I've done a lot of unit conversions and in my experience even such slight deviations can sometime cause serious trouble.

This can be avoided if the table is constructed using the defining exact relationships:

• 1 MWh = 3.6 GJ as 10^6 W * 60 * 60 s = 3.6*10^9 J
• 4.1868 MJ = 1 Mcal by definition
• 1 toe = 41.868 GJ = 10 Mcal by definition

Additionally there are some conversion values where the values are zero (ids 28, 29 and 53). I assume that there were some values but are now rounded to zero due to limited storage precision.

P.S. For convenience here the link to the table: https://openenergy-platform.org/dataedit/view/model_draft/szenariendb_conversion

[han-f]

I had saved the values as "numeric" under the impression this would keep the detail intact; I am not sure which other postgres data type to use (see screenshot below). May this just be a displaying problem on the OEP, while all the detail is still there?

Which type of data would I need to choose in the upload skript to display 10^9 for example?
I thus suggest you take it from here, building on your extensive experience in unit conversion. This probably means also reworking the metadata submitted to review here OpenEnergyPlatform/data-preprocessing#29

https://www.postgresql.org/docs/9.5/datatype-numeric.html

[Ludee]

I usually use double precision in postgres. It is the worry-free option until 14 decimal digits (covers 99% of all data).
I remember a discussion that there is no equivalent data type in python and thus in the API and oedialect.

[l-emele]

We just uploaded new data tables.

This is the review issue of the units and units conversions table: OpenEnergyPlatform/data-preprocessing#29

But I just saw that the data precision problem is not yet solved.

[wingechr]

The problem with lost precision occured on our side when loading the data from excel into pandas dataframes and from there to json for the api upload and has been resolved on the client side.