dp run 0.4.4. and 0.4.5

[ulfmueller]

We just succesfully ran a new dp run 0.4.4. which included some fixes for the hv-pf part.
The versioning of the mviews is still running. A new features shall include data of all (also old) versions within those mviews. Therefore this script shall take some time.

I'll inform you when it is done.

Afterwards we would run a clean-run which will be versioned as 0.4.5. Explanation: Since we had to fix a few minor bugs on-the-run it would be nice to have a so-called clean-run which just runs through the entire process without having any troubles.

[ulfmueller]

fyi:
The mview versioning is still ongoing. But it is totally independent from all tables which are created in the data processing.

We want to start the clean run now (v0.4.5) which will make many ego-specific model_draft tables less reliable the next approx. 2 days. We will start it tonight if no objections arise.

[nesnoj]

Please notify as soon as this is done as we're up to perform urgent calculations..

[ulfmueller]

I'll inform you asap. Right now it is at this rea script.

[ulfmueller]

A small update, the dp is here right now:
2018-08-15 06:21:10 Execute 'rea/ego_dp_rea_m4.sql' ...
2018-08-15 11:23:41 ...successfully done in 18151.02 seconds.
2018-08-15 11:23:41 Execute 'rea/ego_dp_rea_m5.sql' ...

I assume it'll be done by tonight.

[birgits]

For me as a non-dp-specialist - I assume that that means that the mviews for RES scenarios are still not available in model_draft or versioned schema?

[ulfmueller]

Yes for the model_draft. But the last script is running already 10 hours. When that one is finished the model_draft mviews are back which should be earlier than tonight. I'll inform you.
For the versioned schema, what is the status there @wolfbunke ?

[wolfbunke]

Hi for the schema supply you can use "Status Quo" already. The Creation of "NEP 2035" is still running since 45 hours and may need 3-9 more hours by my rough estimation (No. version x 9h).

[wolfbunke]

For the "ego 100" scenario I will maybe remove older versions because the run time will be 18h x No. version, if it is ok for you?

[nesnoj]

What do you mean by "No. version x 9h" ?

[wolfbunke]

We have several different versions (v0.3.0, ... v04.4) per version the script needs 9 hours

[wolfbunke]

The Script for NEP 2035 is still running since 68 hours. I counted the versions and there are 11. With my estimation (11x9h) it will take 31 more hours.

For the eGo 100 scenario I like to limited the number of version. Could you please name me version no. which are required? I will place them first and create an additional mview with all versions, which can be renamed later on.

[wolfbunke]

Unfortunately, the script is still running since 142 hours.

[ulfmueller]

And at the same time, the dp run 0.4.5 is still running. It is inexplicably and unexpectedly stuck here. That means there are still no new model_draft mviews available. I feel very sorry for that delay. We run the mysteriously not finishing script directly from sql hoping to finish the run soon.

In order to relieve the peoples' pressure for the need on the pp mviews asap I would suggest to stop the script and re-run it but only creating it for a limited amount of versions. I talked to @birgits , she likes to have 0.4.2 since the current dingo grids are generated for this version. Additionally I suggest to take the currently newest one i.e. 0.4.4.
Once these are generated (for 2 versions it should take 22 hours for the NEP scenario) we can run the mview creation for all scenarios aside on a mview which has a different name. After that is finished it can be renamed and replace the reduced mview.
Do you agree?

[nesnoj]

Sounds reasonable.
Except for the server duty which slows down out im- and exports, it does not affect our current runs as we use dingo (sq views) only.

[ulfmueller]

@nesnoj ah ok, so you are using only the sq views which already exist. Which means you are not having any mview thirst right now?

[nesnoj]

Right, no thirst except for ☕

[wolfbunke]

@ulfmueller and me talked about the run time and have decided to cancel the query. The next version will only include v0.4.2 and v0.4.4.

[wolfbunke]

v0.4.2 and v0.4.4 are online 🚀

[ulfmueller]

clean run 0.4.5 is finished as well - except for the mview versioning.

Meaning that the model_draft mviews are available.

The mviews have to be updated in order to include also the 0.4.5. This should be done this afternoon.

[wolfbunke]

The Mview includes all version for SQ and 0.4.4 and 0.4.5 for NEP 2015 and eGo100