Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added workflow elapsed time metric #32

Merged
merged 4 commits into from
Mar 19, 2024
Merged

Added workflow elapsed time metric #32

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1c35ed3
Added workflow elapsed time metric
fheinecke Mar 19, 2024
3cf79d3
Updated docs
fheinecke Mar 19, 2024
326f630
Addressed PR comments
fheinecke Mar 19, 2024
4ad95bd
Record job/step metrics for multiple attempts
fheinecke Mar 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
### Added
- Added workflow elapsed time metric

### Changed
- Renamed `gha_workflow_run_time_seconds` to `gha_workflow_run_runner_seconds`

## [0.0.9] - 2024-03-18
### Fixed
Expand Down
23 changes: 19 additions & 4 deletions collector.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,15 @@ import (
)

var (
workflowRunTimeVec = prometheus.NewCounterVec(
workflowRunRunnerTimeVec = prometheus.NewCounterVec(
fheinecke marked this conversation as resolved.
Show resolved Hide resolved
prometheus.CounterOpts{
Name: "gha_workflow_run_time_seconds",
Name: "gha_workflow_run_runner_seconds",
fheinecke marked this conversation as resolved.
Show resolved Hide resolved
},
[]string{"repo", "ref", "event_type", "workflow"},
)
workflowRunElapsedTimeVec = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "gha_workflow_run_elapsed_seconds",
fheinecke marked this conversation as resolved.
Show resolved Hide resolved
},
[]string{"repo", "ref", "event_type", "workflow"},
)
Expand Down Expand Up @@ -67,7 +73,8 @@ type Collector struct {
}

func NewCollector(cfg *CLI) *Collector {
prometheus.MustRegister(workflowRunTimeVec)
prometheus.MustRegister(workflowRunRunnerTimeVec)
prometheus.MustRegister(workflowRunElapsedTimeVec)
prometheus.MustRegister(jobRunTimeVec)
prometheus.MustRegister(stepRunTimeVec)
prometheus.MustRegister(workflowRunCountVec)
Expand Down Expand Up @@ -199,6 +206,13 @@ func (c *Collector) collectJobs(ctx context.Context, repo string, run *github.Wo
}

func countJobs(run *github.WorkflowRun, jobs []*github.WorkflowJob) {
// The tool doesn't currently account for more than one run attempt.
// This brings a multitude of issues that are near-impossible to
// account for due to GH's API design.
if run.GetRunAttempt() > 1 {
return
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should still count the step and job times since they all complete and can provide meaningful data. But we should not count them towards the workflow counters.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that that this really screws up any query that depends on both a "time" metric and a "count" metric. For example, I'm writing a query that gets the average runner time of a workflow. This is basically increase(gha_workflow_runner_seconds / gha_workflow_run_count{conclusion = "success") over an interval. If we count successful runs without counting the run time (in the case of more than one run attempt where a later run was successful) then the average runner time metric is reported as lower than it actually is.

However I'm not sure if dropping metrics for workflow runs with more than one attempts is the right approach. Doing so may screw up some other metric and/or query.

What do you think?

Copy link
Collaborator

@camscale camscale Mar 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not suggesting we increase the workflow run count and not the runner time. I'm suggesting we do neither when the attempt > 1 (or is not successful). Just that in this case there the run attempt is > 1, we still tally up the job and step time and counts. If you are comparing across job/step and workflow there may be a mismatch but I could not think of anything you'd want to compare that way.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh I see. I'll make that change.

}

workflowName := path.Base(run.GetPath())
workflowName = strings.TrimSuffix(workflowName, path.Ext(workflowName))
repo := run.GetRepository().GetName()
Expand Down Expand Up @@ -244,7 +258,8 @@ func countJobs(run *github.WorkflowRun, jobs []*github.WorkflowJob) {
return
}

workflowRunTimeVec.WithLabelValues(repo, ref, eventType, workflowName).Add(workflowRunTime.Seconds())
workflowRunRunnerTimeVec.WithLabelValues(repo, ref, eventType, workflowName).Add(workflowRunTime.Seconds())
workflowRunElapsedTimeVec.WithLabelValues(repo, ref, eventType, workflowName).Add(run.GetUpdatedAt().Sub(run.GetCreatedAt().Time).Seconds())
}

func makeRef(run *github.WorkflowRun) string {
Expand Down
15 changes: 13 additions & 2 deletions metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,16 +43,27 @@ A `counter` of the run time of jobs in a workflow. This is a sum of the
Same as `gha_step_run_time_seconds` but without the `step` label.


## `gha_workflow_run_time_seconds`
## `gha_workflow_run_runner_seconds`

A `counter` of the run time of a workflow. This is a sum of the
A `counter` of the runner run time of a workflow. This is a sum of the
`gha_job_run_time_seconds` for a particular job instance (job ID in GHA).

### Labels

Same as `gha_job_run_time_seconds` but without the `job` label.


## `gha_workflow_run_elapsed_seconds`

A `counter` of the elapsed run time of a workflow. This is a "wall clock
time", or how much time passed between when a run was created and when it
ended.

### Labels

Same as `gha_job_run_time_seconds` but without the `job` label.


## `gha_step_run_count`

A `counter` of the number of runs of a step by conclusion.
Expand Down
Loading