fixes for various null values

[log1-c]

fixes #621
refs #551 (comment)

[Al2Klimov]

You seem to have re-written the whole file. Why?

[log1-c]

no, I just changed more of the int64 to sql.NullInt64 and used the file from #551 as base. Not sure why github recognizes different (encoding? tabs vs space? i don't know) whitespace.

[Al2Klimov]

tabs vs space?

I guess this is the problem.

[lippserd]

@log1-c Can you please reformat your changes and use tabs instead of spaces? go uses tabs. You may use gofmt to fix it.

[log1-c]

@log1-c Can you please reformat your changes and use tabs instead of spaces? go uses tabs. You may use gofmt to fix it.

done :)

[julianbrost]

It looks like this PR includes the changes from #551 (but in new commits) and these changes (only convert.go) seem to be what's new in this PR.

[julianbrost]

I don't see how these additional changes can fix the error stated in #551 (comment):

pq: null value in column "downtime_end" of relation "sla_history_downtime" violates not-null constraint

This PR additionally changes that some columns can read NULL from the IDO database (but then doesn't check for .Valid), but if that was the problem, it should have died before even attempting to write to the new database, which is where the error happens.

Can you please share a row from the IDO icinga_downtimehistory table that triggered this error?

[log1-c]

Can you please share a row from the IDO icinga_downtimehistory table that triggered this error?

Sure. Here are some of the entries where scheduled_end_time is NULL.

... WHERE scheduled_end_time IS NULL;
+--------------------+-------------+---------------+-----------+---------------------+---------------+-----------------------------+----------------------+-----------------+----------+----------+----------------------+--------------------+-------------+-------------------+------------------------+-----------------+----------------------+---------------+--------------+--------------+------------------------------------------------------------------------------------+--------------------+
| downtimehistory_id | instance_id | downtime_type | object_id | entry_time          | author_name   | comment_data                | internal_downtime_id | triggered_by_id | is_fixed | duration | scheduled_start_time | scheduled_end_time | was_started | actual_start_time | actual_start_time_usec | actual_end_time | actual_end_time_usec | was_cancelled | is_in_effect | trigger_time | name                                                                               | endpoint_object_id |
+--------------------+-------------+---------------+-----------+---------------------+---------------+-----------------------------+----------------------+-----------------+----------+----------+----------------------+--------------------+-------------+-------------------+------------------------+-----------------+----------------------+---------------+--------------+--------------+------------------------------------------------------------------------------------+--------------------+
|                177 |           1 |             2 |       362 | 2021-02-17 08:33:26 | icingaadmin   | This is a downtime.         |                   58 |               0 |        0 |     9000 | NULL                 | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!1481c127-e50b-45ec-8e83-b084a471c778                                 |                303 |
|                178 |           1 |             1 |       717 | 2021-02-17 08:33:26 | icingaadmin   | This is a downtime.         |                   59 |               0 |        0 |     9000 | NULL                 | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!cpu!99259d63-b7a4-4c9b-9077-0118799258af                             |                303 |
|                179 |           1 |             1 |       674 | 2021-02-17 08:33:26 | icingaadmin   | This is a downtime.         |                   60 |               0 |        0 |     9000 | NULL                 | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!disk!a587994c-04c7-4b48-ad40-b115f6c22685                            |                303 |
|                180 |           1 |             1 |       731 | 2021-02-17 08:33:26 | icingaadmin   | This is a downtime.         |                   61 |               0 |        0 |     9000 | NULL                 | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!memory!1256c4ba-4f43-4043-9f3d-a01e864fabca                          |                303 |
|                181 |           1 |             1 |      4884 | 2021-02-17 08:33:26 | icingaadmin   | This is a downtime.         |                   62 |               0 |        0 |     9000 | NULL                 | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!rdp-x224-status!f1e4081e-bfeb-43ac-85b8-3276038d8752                 |                303 |
|                182 |           1 |             1 |      5757 | 2021-02-17 08:33:26 | icingaadmin   | This is a downtime.         |                   63 |               0 |        0 |     9000 | NULL                 | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-WinDefendATP!c8728bde-7e44-43c2-a780-a19241b98208            |                303 |
|                448 |           1 |             1 |      4884 | 2021-02-17 13:21:57 | scalingscript | This is a scaling downtime. |                    6 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!rdp-x224-status!ac2740c8-dc72-4248-ad6d-f9a31edff5e7                 |                303 |
|                449 |           1 |             1 |      5757 | 2021-02-17 13:21:57 | scalingscript | This is a scaling downtime. |                    7 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-WinDefendATP!869687b8-6c05-4e98-9c32-8d0c09918709            |                303 |
|                450 |           1 |             1 |      5470 | 2021-02-17 13:21:57 | scalingscript | This is a scaling downtime. |                    8 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-dcomlaunch!2f2d25ae-1dcb-4c9a-b7fc-d2420625559a              |                303 |
|                451 |           1 |             1 |      5498 | 2021-02-17 13:21:57 | scalingscript | This is a scaling downtime. |                    9 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-eventlog!8c56cbe6-5214-41a3-b7f7-ad3c68ae3104                |                303 |
|                452 |           1 |             1 |      5613 | 2021-02-17 13:21:57 | scalingscript | This is a scaling downtime. |                   10 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-gpsvc!b1c27b56-aadb-4ec3-8345-db331a6e0875                   |                303 |
|                453 |           1 |             1 |      5689 | 2021-02-17 13:21:57 | scalingscript | This is a scaling downtime. |                   11 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-lanmanserver!f98db88b-dfcd-4cc0-8e76-7ba08e3051bb            |                303 |
|                454 |           1 |             1 |      5622 | 2021-02-17 13:21:57 | scalingscript | This is a scaling downtime. |                   12 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-lanmanworkstation!74b8c476-26a8-4352-8eeb-59e9a82a1f18       |                303 |
|                455 |           1 |             1 |      5720 | 2021-02-17 13:21:57 | scalingscript | This is a scaling downtime. |                   13 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-lmhosts!585a267a-15ef-4e9e-bd63-135a0f843439                 |                303 |
|                456 |           1 |             1 |      5671 | 2021-02-17 13:21:57 | scalingscript | This is a scaling downtime. |                   14 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-mpssvc!3f14898e-63a0-44e3-b4da-969265ec9bf6                  |                303 |
|                536 |           1 |             1 |      5794 | 2021-02-17 15:06:30 | scalingscript | This is a scaling downtime. |                   54 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-schedule!bed38bcb-4816-4569-aaf7-0ccd018a1e37                |                303 |
|                537 |           1 |             1 |      5562 | 2021-02-17 15:06:30 | scalingscript | This is a scaling downtime. |                   55 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-windefend!74b317bd-b1e4-4bec-8d02-fcbaca4c922e               |                303 |
|                538 |           1 |             1 |      5612 | 2021-02-17 15:06:30 | scalingscript | This is a scaling downtime. |                   56 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-winmgmt!5d916041-1436-4102-9d2d-04e3c489f457                 |                303 |
|                539 |           1 |             1 |      5749 | 2021-02-17 15:06:30 | scalingscript | This is a scaling downtime. |                   57 |               0 |        0 | 31536000 | 2021-02-17 09:36:30  | NULL               |           0 | NULL              |                      0 | NULL            |                    0 |             0 |            0 | NULL         | d1-ts1001!service-winrm!2fea8dc5-f670-4217-86c4-5429b52cb04b                   |                303 |
+--------------------+-------------+---------------+-----------+---------------------+---------------+-----------------------------+----------------------+-----------------+----------+----------+----------------------+--------------------+-------------+-------------------+------------------------+-----------------+----------------------+---------------+--------------+--------------+------------------------------------------------------------------------------------+--------------------+

``

[julianbrost]

Interesting, I just ran into the pq: null value in column "downtime_end" of relation "sla_history_downtime" violates not-null constraint error myself when testing something else. I don't have any rows with scheduled_end_time IS NULL though, so the cause might actually be something different.

MariaDB [icingaido]> SELECT * FROM icinga_downtimehistory WHERE scheduled_end_time IS NULL;
Empty set (0.125 sec)

[julianbrost]

We believe that what you wanted to fix should be fixed by #623. The rows you shared in #613 (comment) all have was_started=0 and is_fixed=0 which indicates that these were flexible downtimes that never triggered, i.e. the checkable never entered a problem state during the downtime window.

That fix will be included in Icinga DB 1.1.1 which will be released this week, please report back if you still experience any issues with that version.