improvement(results): update Argus Client

[soyacz]

Updated Argus Client to support best results tracking and Argus data validation.

Refs: https://github.com/scylladb/qa-tasks/issues/1244

Testing

• - perf-simple-querry

PR pre-checks (self review)

• I added the relevant backport labels
• I didn't leave commented-out/debugging code

Reminders

• Add New configuration option and document them (in sdcm/sct_config.py)
• Add unit tests to cover my changes (under unit-test/ folder)
• Update the Readme/doc folder relevant to this change (if needed)

[soyacz]

Actually, there's an error in Argus client, it makes ValidationRules mandatory, which wasn't intended.
I think I'm going to fix it first and then come back when done.

[soyacz]

waiting for scylladb/argus#463

[fruch]

@soyacz can you clarify the status of this one ? if it's for review, what exactly we should be looking at ?

[soyacz]

@soyacz can you clarify the status of this one ? if it's for review, what exactly we should be looking at ?

Yes, it's for review. Upgrading client will add possibility for Argus to automatically mark results as passed/failed if "ValidationRules" are provided.
Because latency calculator/perf simple query regression checker is doing it already, we don't need to add them.
Before merge, I'll modify this PR to include some validation rules for SM testing @mikliapko recently introduced.

[soyacz]

@mikliapko please review if these numbers seem reasonable - 10% longer duration than 'best result' will mark test as failed.

[mikliapko]

Looks reasonable to me

[mikliapko]

Doesn't directly related to this PR, just a general question about this approach - for SM it would be a good improvement to show not only SUT timestamp but SM version as well. Is it feasible to implement?

[mikliapko]

What is related to SM part - LGTM

[soyacz]

Doesn't directly related to this PR, just a general question about this approach - for SM it would be a good improvement to show not only SUT timestamp but SM version as well. Is it feasible to implement?

I'd need to understand the use case first.
Varying both Scylla and SM versions makes hard to track regressions - it will never be clear if it comes from SM or from Scylla itself. Think if it wouldn't it be better to pin one.
The other thing, I don't know how to show it on graph?

[mikliapko]

Doesn't directly related to this PR, just a general question about this approach - for SM it would be a good improvement to show not only SUT timestamp but SM version as well. Is it feasible to implement?

I'd need to understand the use case first. Varying both Scylla and SM versions makes hard to track regressions - it will never be clear if it comes from SM or from Scylla itself. Think if it wouldn't it be better to pin one. The other thing, I don't know how to show it on graph?

In the graphs that we have for Manager, we use SUT timestamp of Manager. So, there is nothing here related to Scylla SUT.

From my point of view, it'd be nice to have SUT + version to improve readability since in Manager restore benchmark we want to compare with previous releases and it's hard to keep in mind to which release specific SUT date corresponds.

But it's not a "show-stopper", for sure. Just a small improvement that could have made these Manager graphs more readable.

[soyacz]

Doesn't directly related to this PR, just a general question about this approach - for SM it would be a good improvement to show not only SUT timestamp but SM version as well. Is it feasible to implement?

I'd need to understand the use case first. Varying both Scylla and SM versions makes hard to track regressions - it will never be clear if it comes from SM or from Scylla itself. Think if it wouldn't it be better to pin one. The other thing, I don't know how to show it on graph?

In the graphs that we have for Manager, we use SUT timestamp of Manager. So, there is nothing here related to Scylla SUT.

From my point of view, it'd be nice to have SUT + version to improve readability since in Manager restore benchmark we want to compare with previous releases and it's hard to keep in mind to which release specific SUT date corresponds.

But it's not a "show-stopper", for sure. Just a small improvement that could have made these Manager graphs more readable.

There's a plan to introduce comparison between tests. In that case, if each release had own restore benchmark jobs it should be possible to compare across them.
Also, you could use 'rows' for Scylla version - but this shouldn't change often as each row will be represented as separate series on one graph, so we don't want to have 10 lines, after 10 tests (rather points in that case). E.g. if each row would be marked as Scylla version e.g. 2024.1.3 then there would be 2024.1.3 series on graph, after switch to 2024.1.4 another series would appear (breaking connection line between 2024.1.3 series, which is good IMHO clearly showing impact of Scylla version change). Then SUT can be still bound to SM build date, showing impact of SM changes on restore timings.

[soyacz]

@fruch Are we merging this? Or maybe switching to some idea from discussion in https://github.com/scylladb/qa-tasks/issues/1766?