Wrong editor time displayed in all time summary

[bestlinuxgamers]

Describe the bug
The editor time displayed on the main page of the summary (in my case for Android Studio) is oddly low. If you click on a project that mainly used this editor, the time is higher than on the main page. This is strange as it should contain all the data from the subprojects.

System information

• Wakapi version 2.10.4
• Operating system
  • If Linux: which distro? Ubuntu 22.04.3 LTS
  • If Docker: which image and tag? ghcr.io/muety/wakapi:2.10.4
• Database Postgres

Global Editor chart (All Time):

Editor Chart of a project (All Time)

[muety]

Can you please try to go to Settings -> Danger Zone, then Regenerate summaries and see if that helps?

[bestlinuxgamers]

I've tried that (and waited for about 20 Minutes). Now the charts are a complete mess. AndroidStudio has even less time and "idea" which I use far more often is basically not visible. My languages chart doesn't really look correct either.

It seems that old data was not taken into account while generating the charts.

[muety]

Do you see any error or warning messages in the server's console? Also, are the stats more accurate if you add &recompute=true as a URL parameter on the summaries page?

[bestlinuxgamers]

I can't spot any errors in the log. There are multiple weird lines like:

[wakapi]   | 2024-01-30T20:34:33.141703737Z [INFO ] starting with postgres database
[wakapi]   | 2024-01-30T20:34:33.150874894Z [INFO ] potentially running migration '20201103-rename_language_mappings_table'
[wakapi]   | 2024-01-30T20:34:33.155044674Z [INFO ] potentially running migration '20201106-migration_cascade_constraints'
[wakapi]   | 2024-01-30T20:34:33.156473067Z [INFO ] no need to migrate '20201106-migration_cascade_constraints'
[wakapi]   | 2024-01-30T20:34:33.156479067Z [INFO ] potentially running migration '20210202-fix_cascade_for_alias_user_constraint'
[...]

And it goes on like that for ~40 lines. I don't know if this is the expected behaviour.

And this warning is logged:
2024-01-30T20:34:33.141680437Z [WARN ] failed to parse reverse proxy ip

&recompute=true makes the charts much more accurate. They seem fine now. But if I remove this argument the old charts are still being displayed.

[muety]

Those log messages are fine. Something seems to be wrong with your summary calculation. Small discrepancies of a few minutes are common, but hours is too much. Would you be open to send me a dump of your database (with all sensitive information like passwords, etc. removed) as a private message? I'm aware that's a lot to ask for, but would probably be the easiest way to debug this.

[bestlinuxgamers]

In general, I would be willing to send a dump. However, since my instance has multiple users, I would need to limit the data to my user account and I would also like to anonymize the data a bit.
How would it be possible to get in touch with you? Do you have a Discord server?

[muety]

Yes, you can remove all users and their data from the dump. Best contact me via e-mail (see my GH profile).

[bestlinuxgamers]

What data exactly do you need from the various tables? Which tables can I delete and which do I need to anonymize?

[bestlinuxgamers]

I've sent you the dump via email.

[ouuan]

This also happens in my instance. And I can confirm 2.10.2 is fine and 2.10.3 is buggy.

[ouuan]

git bisect shows that the first bad commit is 791f5cc

[muety]

@ddadaal Any immediate ideas about that?

[ddadaal]

I can confirm this problem also exists in my MSSQL instance. Editor, Languages, OS and machine time are wrong, but the sum of numbers of all of these 4 items are identical, and the Projects and Labels seem to have correct numbers. It seems the same set of data is missing when these 4 items are calculated.

I checked my DB and found out all of the existing SummaryItem has type 0, which means SummaryProject in the code https://github.com/muety/wakapi/blob/master/models/duration.go#L72. On the other hand, it means the SummaryItems of all other types are missing in the db.

This commit 791f5cc removes foreign constraints and adds gorm:"-" tag in exactly these 4 columns, which was meant to prevent gorm from creating extraneous foreign keys. The one constraint in Projects is not changed, and so the Project result is correct.

I assume it is because the SummaryItem of other types are not persisted in the db because it is explicitly ignored by gorm:"-" tag. It also explains why recompute helps because when the recompute exists, all data are calculated live, not from db.

A quick fix would be to manually persist the other SummaryItem when persisting Summary. I've submitted a PR for it #603 and have deployed the fix into my instance. I will let it run for a day or two and see the result. I am pretty occupied recently so feedback might be delayed, so if anyone is interested in it, please also give it a try.

[muety]

I should have spotted that bug in the review of #592 😐. Thanks a lot for reacting so quickly, @ddadaal! 🙌 Tested your hotfix locally and seems to be working fine. Will merge the PR, deploy it and let it run for a short while.

[ddadaal]

No! It's not fixed yet! It doesn't persist entities correctly. The result is correct because no summary or summaryitem is persisted and the data are always calculated live. I have a working fix that does persist data and and I will push it later.

[muety]

Realized that just now. I'm getting errors like:

[ERROR] failed to save summary (n1try, 2024-02-01 00:00:00 +0100 CET, 2024-02-02 00:00:00 +0100 CET) - unsupported data type: [0xc001b80120]: Table not set, please set it like: db.Model(&user) or db.Table("users")

Will wait for your fix. Thanks a lot again!

[ouuan]

The release note should remind users to clear and regenerate summaries.

[tippfehlr]

yay it works again 🎉