-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add run_attempt
as an optional key column for github_actions_repository_workflow_run
get calls
#464
Conversation
…itory_workflow_run` The workflow run represented by this table/the GitHub API endpoint is aggregate of all the attempts of the run, with details that are a combination of the first attempt and the last attempt of the run. When dealing with runs that have multiple attempts, it's useful to know how many attempts there have been and where to find the previous attempt.
Includes the following changes: - <turbot/steampipe-plugin-github#463> - <turbot/steampipe-plugin-github#464> - <turbot/steampipe-plugin-github#465> for additional functionality around run attempts (to be used in the next commit) and improved efficiency listing runs (automatically!). The hope is that I can get my changes upstreamed and we can stop using a customized and vendored version, but I'm not sure if that's likely or not.
KeyColumns: []*plugin.KeyColumn{ | ||
{Name: "repository_full_name", Require: plugin.Required, Operators: []string{"="}}, | ||
{Name: "id", Require: plugin.Required, Operators: []string{"="}}, | ||
{Name: "run_attempt", Require: plugin.Optional, Operators: []string{"="}, CacheMatch: "exact"}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tsibley is there a specific reason for adding CacheMatch: "exact"
to this optional qual? It is typically used for columns that are populated using FromQuals
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From my understanding, we don’t need to include CacheMatch: "exact"
for these optional qualifiers here. This is because the run_attempt
optional qualifier is part of the Get
config, not the List
config.
According to the table design, the Get
API call should always return a single row. In this table, the unique identifier is the id
column, which is unique per row.
Query, result:
List Config execution:
> select id, run_attempt from github_actions_repository_workflow_run where repository_full_name = 'turbot/steampipe-plugin-aws' and run_attempt > 2
+-------------+-------------+
| id | run_attempt |
+-------------+-------------+
| 11666032178 | 3 |
| 10667532782 | 3 |
| 9872917837 | 5 |
| 8801296800 | 3 |
| 8753110746 | 3 |
| 8296554622 | 3 |
| 8065062549 | 4 |
| 8076825093 | 3 |
| 7887624205 | 3 |
| 7358331805 | 4 |
+-------------+-------------+
Get Config execution:
> select id, run_attempt from github_actions_repository_workflow_run where repository_full_name = 'turbot/steampipe-plugin-aws' and id = 7358331805 and run_attempt = 4
+------------+-------------+
| id | run_attempt |
+------------+-------------+
| 7358331805 | 4 |
+------------+-------------+
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ParthaI id
is unique per row only because List can only return the latest run_attempt
. A single id has multiple run attempts, though, and so the Get call needs to consider it as part of the key.
@misraved Without CacheMatch: "exact"
, there are false positive cache hits on a Get after a List when accessing previous run attempts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ParthaI id is unique per row only because List can only return the latest run_attempt. A single id has multiple run attempts, though, and so the Get call needs to consider it as part of the key.
Apologies for the misunderstanding, @tsibley. I was unaware of the API behaviour. It makes perfect sense to include CacheMatch: "exact"
for the run_attempt
optional key column qualifier. Thank you for clarifying!
Query result WITHOUT CacheMatch: "exact"
:
> select id, run_attempt from github_actions_repository_workflow_run where repository_full_name = 'turbot/steampipe-plugin-aws' and id = 7358331805
+------------+-------------+
| id | run_attempt |
+------------+-------------+
| 7358331805 | 4 |
+------------+-------------+
Time: 1.7s. Rows returned: 0. Rows fetched: 1. Hydrate calls: 0.
> select id, run_attempt from github_actions_repository_workflow_run where repository_full_name = 'turbot/steampipe-plugin-aws' and id = 7358331805 and run_attempt = 1
+----+-------------+
| id | run_attempt |
+----+-------------+
+----+-------------+
Time: 20ms. Rows returned: 0. Rows fetched: 1 (cached). Hydrate calls: 0.
Query result WITH CacheMatch: "exact"
:
> select id, run_attempt from github_actions_repository_workflow_run where repository_full_name = 'turbot/steampipe-plugin-aws' and id = 7358331805
+------------+-------------+
| id | run_attempt |
+------------+-------------+
| 7358331805 | 4 |
+------------+-------------+
Time: 1.7s. Rows returned: 0. Rows fetched: 1. Hydrate calls: 0.
> select id, run_attempt from github_actions_repository_workflow_run where repository_full_name = 'turbot/steampipe-plugin-aws' and id = 7358331805 and run_attempt = 1
+------------+-------------+
| id | run_attempt |
+------------+-------------+
| 7358331805 | 1 |
+------------+-------------+
Time: 451ms. Rows returned: 0. Rows fetched: 1. Hydrate calls: 0.
Additionally, could you please use CacheMatch: query_cache.CacheMatchExact
instead of CacheMatch: "exact"
? For reference, you can check this example: AWS CloudControl Resource Table.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I’ve swapped out the string literal for the symbol. Thanks for the pointer to it. I’d originally looked for it (quickly) but didn’t find it and the explanatory code comments used the string literal so I used that too.
…itory_workflow_run` get calls This is the bare minimum to allow previous run attempts to be queried, which is useful for comparisons between attempts. User queries must explicitly iterate over a run's multiple attempts as necessary, e.g. by parsing `previous_attempt_url` in a `with recursive` CTE or simply presuming the series of attempts is `generate_series(1, run_attempt)`. A future improvement for this plugin might be exposing a new table, e.g. `github_actions_repository_workflow_run_attempt`, with the same list key columns of `repository_full_name` and `id`, and handling the iteration internally. GitHub's API unfortunately provides no listing endpoints for run attempts.
f2417d9
to
fc3ed5f
Compare
builds upon #463, but that PR can be considered separately from this one
This is the bare minimum to allow previous run attempts to be queried, which is useful for comparisons between attempts. User queries must explicitly iterate over a run's multiple attempts as necessary, e.g. by parsing
previous_attempt_url
in awith recursive
CTE or simply presuming the series of attempts isgenerate_series(1, run_attempt)
.A future improvement for this plugin might be exposing a new table, e.g.
github_actions_repository_workflow_run_attempt
, with the same list key columns ofrepository_full_name
andid
, and handling the iteration internally. GitHub's API unfortunately provides no listing endpoints for run attempts.