Feedback on Migration to aw-server-rust

[nicolae-stroncea]

Just made the move to aw-server-rust. Very impressed by the change. The transition went smoothly, and the queries are blazingly fast. When I used to wait maybe 30-40 seconds for a monthly review, it now takes 10 seconds. Congrats on a huge milestone for AW!

Couple of observations:

1. My previous db size was 264MB, and its now 213MB, that's an 8% decrease in size. I checked the logs, the bucket sizes and everything seems to match. Is a decrease of this size expected? Furthermore, does aw-server-rust use sqlite by default compared to peewee for aw-server?

2. Might be a good idea to put a notice for users to hard-refresh their page in the release notes once aw-server-rust becomes the default. Some of the buttons weren't responding for me and I was getting 404's. Only once I did a hard reset, did web-ui function correctly.

3. Some people may not be aware that aw-server and aw-server-rust store events in different databases(I definitely wasn't), so if they use them interchangeably they will lose some events. This obviously won't happen when aw-server-rust becomes the default, but in the meanwhile, possibly consider adding a disclaimer to the release

4. Looking at the logs, there seems to be an issue with sending events from firefox-watcher:

[2020-06-25][21:29:52][rocket::rocket][INFO] OPTIONS /api/0/buckets/aw-watcher-web-firefox/heartbeat?pulsetime=80:
[2020-06-25][21:29:52][_][INFO] Verifying origin: moz-extension://a8cf0cd0-2dad-427e-b3d0-ab9326727db2
[2020-06-25][21:29:52][_][INFO] Origin has a regex match? true
[2020-06-25][21:29:52][_][ERROR] No matching routes for OPTIONS /api/0/buckets/aw-watcher-web-firefox/heartbeat?pulsetime=80.
[2020-06-25][21:29:52][_][WARN] Responding with 404 Not Found catcher.
[2020-06-25][21:29:52][_][INFO] CORS Fairing: Turned missing route OPTIONS /api/0/buckets/aw-watcher-web-firefox/heartbeat?pulsetime=80 into an OPTIONS pre-flight request
[2020-06-25][21:29:52][_][INFO] Response succeeded.
[2020-06-25][21:29:52][rocket::rocket][INFO] POST /api/0/buckets/aw-watcher-web-firefox/heartbeat?pulsetime=80 application/json; charset=utf-8:
[2020-06-25][21:29:52][_][INFO] Verifying origin: moz-extension://a8cf0cd0-2dad-427e-b3d0-ab9326727db2
[2020-06-25][21:29:52][_][INFO] Origin has a regex match? true
[2020-06-25][21:29:52][_][INFO] Matched: POST /api/0/buckets/<bucket_id>/heartbeat?<pulsetime> (bucket_events_heartbeat)
[2020-06-25][21:29:52][_][INFO] Outcome: Success
[2020-06-25][21:29:52][_][INFO] Response succeeded.

Logs for Chrome look to be fine.
System: Fedora 32, Firefox 77.0.1(most up to date)

[johan-bjareholt]

When I used to wait maybe 30-40 seconds for a monthly review, it now takes 10 seconds.

Still 10 seconds? Sounds like we still have room for improvement ;)
I assume that you are not running a debug build, if you are that makes a huge difference in performance.
I want it to be fast enough so a yearly summary won't be close to the per-request 30s timeout for most computers.

My previous db size was 264MB, and its now 213MB, that's an 8% decrease in size. I checked the logs, the bucket sizes and everything seems to match. Is a decrease of this size expected? Furthermore, does aw-server-rust use sqlite by default compared to peewee for aw-server?

Interesting, I had no idea. Peewee is a wrapper around sqlite and now we just use raw sqlite, not sure what the difference might be actually.

Might be a good idea to put a notice for users to hard-refresh their page in the release notes once aw-server-rust becomes the default. Some of the buttons weren't responding for me and I was getting 404's. Only once I did a hard reset, did web-ui function correctly.

Huh, did not know that either. Good point.

Some people may not be aware that aw-server and aw-server-rust store events in different databases(I definitely wasn't), so if they use them interchangeably they will lose some events. This obviously won't happen when aw-server-rust becomes the default, but in the meanwhile, possibly consider adding a disclaimer to the release

Did the automatic database migration work for you or did you manually export from aw-server-python and import again in aw-server-rust?
I don't think the migration has been tested very thoroughly in the "wild", would be interesting to hear if anyone have had any bugs with that before making it the default.

Looking at the logs, there seems to be an issue with sending events from firefox-watcher:

Yeah, the ERROR and WARN is not an issue because it's handled by the CORS fairing afterwards, I've tried to suppress that log and asked upstream how to be able to do so but no luck yet.

[johan-bjareholt]

About the DB size that could also simply be because of database fragmentation. If you import everything in order afterwards there will be less fragmentation.

[johan-bjareholt]

Regarding the annoying logs, that's a duplicate of the following issue:

#42

[nicolae-stroncea]

Still 10 seconds? Sounds like we still have room for improvement ;)
I assume that you are not running a debug build, if you are that makes a huge difference in performance.
I want it to be fast enough so a yearly summary won't be close to the per-request 30s timeout for most computers.

I'm running 0.92 release, with nothing extra. The biggest reason for the delay is the "Loading of the Browser Domains". It takes a disproportionate amount of time for some reason. I reran it, and the entire page for the month loaded within the first 5 seconds (Top applications, Window Titles, Categories), and then it took 25 more seconds for the Browser Domains to be displayed(literally 5 times more). Same issue occurs in aw-server, so I'm wondering it this is an issue on the aw-webui front.

@ErikBjare mentioned in ActivityWatch/activitywatch/issues/445 that he plans on merging the window and browser queries together. Would that fix this issue?

About the DB size that could also simply be because of database fragmentation. If you import everything in order afterwards there will be less fragmentation.

I think another reason might be the change in some types: since SQLITE now uses 2 Integers instead of 1 Datetime + 1 Decimal to show events. Too lazy to do the math and find the size of each type but even if there's a 1 byte difference when you have 500k+ events, it adds up really quickly. Either way, the space savings are another thing to promote in future releases when it becomes the default :)

Did the automatic database migration work for you or did you manually export from aw-server-python and import again in aw-server-rust?
I don't think the migration has been tested very thoroughly in the "wild", would be interesting to hear if anyone have had any bugs with that before making it the default.

So I just did a pkill aw-, then launched aw-server-rust + all watchers, so aw-server-rust did the migration automatically. I have maybe 1 million events, I think it took maybe 5-10 minutes(very rough estimates, I don't actually remember). Does the code account for a user suddenly shutting off aw-server-rust, and what happens in that case?

You said the migration hasn't been tested thoroughly in the 'wild'. What are your thoughts on something like this to test it:

If there are some tests for integrity in place (I remember the logs indicated the buckets were successfully imported), you could have users run the migration, capture the output in a file (easiest way might be to give them a script they can run locally), and post it in a 'Migration` Issue on ActivityWatch. In the issue we can ask for 5 reports(more would be even better) for each OS platform. I think given the amount of people using AW finding 15 people willing to do this would be very straightforward. 5 for each platform should good enough to find most of the big issues regarding the migration.

Other tests the users could do(above would of course be the most important one), is:

• Launch aw-webui play around with it, see if everything works
• Run aw-server and aw-server-rust side by side, and see if there is any glaring issues(to run side by side perhaps have aw-server-rust migration happen on --testing).

[johan-bjareholt]

I'm running 0.92 release, with nothing extra. The biggest reason for the delay is the "Loading of the Browser Domains". It takes a disproportionate amount of time for some reason. I reran it, and the entire page for the month loaded within the first 5 seconds (Top applications, Window Titles, Categories), and then it took 25 more seconds for the Browser Domains to be displayed(literally 5 times more). Same issue occurs in aw-server, so I'm wondering it this is an issue on the aw-webui front.

aw-server can only do one query at a time, so the first 5 seconds is usually the window query and then the browser query runs. So if the browser query takes 25 seconds thats 25-5=20 seconds. So 4 times more I guess which is still very bad.

@ErikBjare mentioned in ActivityWatch/activitywatch/issues/445 that he plans on merging the window and browser queries together. Would that fix this issue?

I was the one to fix that and is on master now so it will be available in next release. The time for all queries will be faster because they are now a single query (both the window query and browser query fetched events from the afk and window buckets which was unnecessary), but that also means that the time to only get the window titles will be longer since the result for both will arrive at the same time.

So I just did a pkill aw-, then launched aw-server-rust + all watchers, so aw-server-rust did the migration automatically. I have maybe 1 million events, I think it took maybe 5-10 minutes(very rough estimates, I don't actually remember). Does the code account for a user suddenly shutting off aw-server-rust, and what happens in that case?

Damn, that's a lot of events! I have around 250k on my laptop (which is not my primary computer, but anyway). Takes a little less than a minute to import though.
Would be possible to let the webui start first though and then import buckets and possibly have a "ready" api which can give a reason for why aw-server is not ready yet in the web-ui.

You said the migration hasn't been tested thoroughly in the 'wild'. What are your thoughts on something like this to test it:

If there are some tests for integrity in place (I remember the logs indicated the buckets were successfully imported), you could have users run the migration, capture the output in a file (easiest way might be to give them a script they can run locally), and post it in a 'Migration` Issue on ActivityWatch. In the issue we can ask for 5 reports(more would be even better) for each OS platform. I think given the amount of people using AW finding 15 people willing to do this would be very straightforward. 5 for each platform should good enough to find most of the big issues regarding the migration.

Other tests the users could do(above would of course be the most important one), is:

• Launch aw-webui play around with it, see if everything works
• Run aw-server and aw-server-rust side by side, and see if there is any glaring issues(to run side by side perhaps have aw-server-rust migration happen on --testing).

Yeah, shouldn't be too hard to test it on all platforms and to test it by some different people with different data in their buckets.

The issue would only be if an issue occurs that the user might not want to provide their buckets to show what broke the import. But I'm not sure what kind of data could possibly break that so that might not even be an issue.

[nicolae-stroncea]

@johan-bjareholt it's interesting that you have a lot less events than me, since I would imagine you started tracking with AW long before I did. My current buckets contain data starting from August last year

[johan-bjareholt]

@nicolae-stroncea Not that strange considering that it's from my laptop that is not my primary computer. I'll likely have more events on my personal desktop and likely the most on my work desktop.

[johan-bjareholt]

I just checked my work desktop, it has around 1.4 million events and I've been running activitywatch on it for about 3 years.