Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add run_attempt as an optional key column for github_actions_repository_workflow_run get calls #464

Merged
merged 2 commits into from
Jan 2, 2025

Conversation

tsibley
Copy link
Contributor

@tsibley tsibley commented Dec 18, 2024

builds upon #463, but that PR can be considered separately from this one

This is the bare minimum to allow previous run attempts to be queried, which is useful for comparisons between attempts. User queries must explicitly iterate over a run's multiple attempts as necessary, e.g. by parsing previous_attempt_url in a with recursive CTE or simply presuming the series of attempts is generate_series(1, run_attempt).

A future improvement for this plugin might be exposing a new table, e.g. github_actions_repository_workflow_run_attempt, with the same list key columns of repository_full_name and id, and handling the iteration internally. GitHub's API unfortunately provides no listing endpoints for run attempts.

…itory_workflow_run`

The workflow run represented by this table/the GitHub API endpoint is
aggregate of all the attempts of the run, with details that are a
combination of the first attempt and the last attempt of the run.  When
dealing with runs that have multiple attempts, it's useful to know how
many attempts there have been and where to find the previous attempt.
tsibley added a commit to nextstrain/status that referenced this pull request Dec 19, 2024
Includes the following changes:

  - <turbot/steampipe-plugin-github#463>
  - <turbot/steampipe-plugin-github#464>
  - <turbot/steampipe-plugin-github#465>

for additional functionality around run attempts (to be used in the next
commit) and improved efficiency listing runs (automatically!).

The hope is that I can get my changes upstreamed and we can stop using a
customized and vendored version, but I'm not sure if that's likely or
not.
@misraved misraved self-requested a review December 19, 2024 15:25
KeyColumns: []*plugin.KeyColumn{
{Name: "repository_full_name", Require: plugin.Required, Operators: []string{"="}},
{Name: "id", Require: plugin.Required, Operators: []string{"="}},
{Name: "run_attempt", Require: plugin.Optional, Operators: []string{"="}, CacheMatch: "exact"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tsibley is there a specific reason for adding CacheMatch: "exact" to this optional qual? It is typically used for columns that are populated using FromQuals.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@misraved, @tsibley,

From my understanding, we don’t need to include CacheMatch: "exact" for these optional qualifiers here. This is because the run_attempt optional qualifier is part of the Get config, not the List config.

According to the table design, the Get API call should always return a single row. In this table, the unique identifier is the id column, which is unique per row.

Query, result:

List Config execution:

> select id, run_attempt from github_actions_repository_workflow_run where repository_full_name = 'turbot/steampipe-plugin-aws' and  run_attempt > 2
+-------------+-------------+
| id          | run_attempt |
+-------------+-------------+
| 11666032178 | 3           |
| 10667532782 | 3           |
| 9872917837  | 5           |
| 8801296800  | 3           |
| 8753110746  | 3           |
| 8296554622  | 3           |
| 8065062549  | 4           |
| 8076825093  | 3           |
| 7887624205  | 3           |
| 7358331805  | 4           |
+-------------+-------------+

Get Config execution:

> select id, run_attempt from github_actions_repository_workflow_run where repository_full_name = 'turbot/steampipe-plugin-aws' and id = 7358331805 and run_attempt = 4
+------------+-------------+
| id         | run_attempt |
+------------+-------------+
| 7358331805 | 4           |
+------------+-------------+

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ParthaI id is unique per row only because List can only return the latest run_attempt. A single id has multiple run attempts, though, and so the Get call needs to consider it as part of the key.

@misraved Without CacheMatch: "exact", there are false positive cache hits on a Get after a List when accessing previous run attempts.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ParthaI id is unique per row only because List can only return the latest run_attempt. A single id has multiple run attempts, though, and so the Get call needs to consider it as part of the key.

Apologies for the misunderstanding, @tsibley. I was unaware of the API behaviour. It makes perfect sense to include CacheMatch: "exact" for the run_attempt optional key column qualifier. Thank you for clarifying!

Query result WITHOUT CacheMatch: "exact":

> select id, run_attempt from github_actions_repository_workflow_run where repository_full_name = 'turbot/steampipe-plugin-aws' and id = 7358331805
+------------+-------------+
| id         | run_attempt |
+------------+-------------+
| 7358331805 | 4           |
+------------+-------------+

Time: 1.7s. Rows returned: 0. Rows fetched: 1. Hydrate calls: 0.
> select id, run_attempt from github_actions_repository_workflow_run where repository_full_name = 'turbot/steampipe-plugin-aws' and id = 7358331805 and run_attempt = 1
+----+-------------+
| id | run_attempt |
+----+-------------+
+----+-------------+

Time: 20ms. Rows returned: 0. Rows fetched: 1 (cached). Hydrate calls: 0.

Query result WITH CacheMatch: "exact":

> select id, run_attempt from github_actions_repository_workflow_run where repository_full_name = 'turbot/steampipe-plugin-aws' and id = 7358331805
+------------+-------------+
| id         | run_attempt |
+------------+-------------+
| 7358331805 | 4           |
+------------+-------------+

Time: 1.7s. Rows returned: 0. Rows fetched: 1. Hydrate calls: 0.
> select id, run_attempt from github_actions_repository_workflow_run where repository_full_name = 'turbot/steampipe-plugin-aws' and id = 7358331805 and run_attempt = 1
+------------+-------------+
| id         | run_attempt |
+------------+-------------+
| 7358331805 | 1           |
+------------+-------------+

Time: 451ms. Rows returned: 0. Rows fetched: 1. Hydrate calls: 0.

Additionally, could you please use CacheMatch: query_cache.CacheMatchExact instead of CacheMatch: "exact"? For reference, you can check this example: AWS CloudControl Resource Table.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ve swapped out the string literal for the symbol. Thanks for the pointer to it. I’d originally looked for it (quickly) but didn’t find it and the explanatory code comments used the string literal so I used that too.

…itory_workflow_run` get calls

This is the bare minimum to allow previous run attempts to be queried,
which is useful for comparisons between attempts.  User queries must
explicitly iterate over a run's multiple attempts as necessary, e.g. by
parsing `previous_attempt_url` in a `with recursive` CTE or simply
presuming the series of attempts is `generate_series(1, run_attempt)`.

A future improvement for this plugin might be exposing a new table, e.g.
`github_actions_repository_workflow_run_attempt`, with the same list key
columns of `repository_full_name` and `id`, and handling the iteration
internally.  GitHub's API unfortunately provides no listing endpoints
for run attempts.
@tsibley tsibley force-pushed the trs/workflow-run-attempt-get branch from f2417d9 to fc3ed5f Compare December 31, 2024 16:29
@misraved misraved merged commit cac0408 into turbot:main Jan 2, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants