-
Notifications
You must be signed in to change notification settings - Fork 204
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run k6 after staging deployments #4991
Conversation
Latest k6 run output1
Footnotes
|
85df194
to
f64845a
Compare
.github/workflows/ci_cd.yml
Outdated
- name: Trigger staging k6 load test | ||
uses: convictional/[email protected] | ||
with: | ||
owner: WordPress | ||
repo: openverse | ||
token: ${{ secrets.ACCESS_TOKEN }} | ||
workflow_file_name: k6.yml | ||
wait_interval: 60 | ||
client_payload: | | ||
{ | ||
"namespace": "frontend", | ||
"scenario": "all", | ||
"service_url": "https://staging.openverse.org/", | ||
"report": true | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This "propagates failures" so if the load testing workflow fails, it will mark the staging deployment as failed (which will send a note in our alerts channel about a failed CI/CD run).
This might prove to be a mistake, if these tests are flaky in any way. But, we can try it for now, and easily add the propagate_failure: false
option to the action.
We can also choose not to propagate the failure and instead use Grafana's alerting system to send a message when a load test run is uploaded with bad stats (failures).
c2000c2
to
8b6333d
Compare
5cdd576
to
83d0fea
Compare
Based on the contributor urgency of this PR, the following reviewers are being gently reminded to review this PR: @obulat Excluding weekend1 days, this PR was ready for review 17 day(s) ago. PRs labelled with contributor urgency are expected to be reviewed within 3 weekday(s)2. @sarayourfriend, if this PR is not ready for a review, please draft it to prevent reviewers from getting further unnecessary pings. Footnotes
|
83d0fea
to
91ad5ed
Compare
Co-authored-by: Sara <[email protected]>
Signed-off-by: Olga Bulat <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@obulat the workflow runs are failing, turns out |
Yes, turns out that was the reason for not including |
@sarayourfriend, the tests are running and passing now 🎉 We also had to increase the CPU resources for staging to prevent the task from restarting during the load tests. We might want to reduce the number of VUs in the tests or the number of requests to prevent the CPU load from going too high (the memory stays below 50%). I also thought that we might add a process whereby the task with higher resources is deployed when a PR is merged, and after the staging k6 passes, another task with lower values is deployed. But this seems quite complicated to get it right |
For the sake of long-term cost control and simplicity, I'd recommend scaling the staging load tests down until they reliably pass, rather than staging tasks up. Reducing the VUs or the number of iterations both sound like good options. You could also try using the On that note: I'd also recommend intentionally merging a memory leak and seeing whether this new testing mode catches that 🙂. Trying the same with something CPU bound would also be good. |
* Add workflow for running k6 on live urls * Use generic service_url variable for frontend scenarios * Set testing variables in workflow * Remove JS template string syntax from Python template string * Fix missing k6 envvar option for signing secret * Add step to trigger load test with staging deploy * Remove testing input overrides * Update .github/workflows/ci_cd.yml Co-authored-by: Sara <[email protected]> * Add comment Signed-off-by: Olga Bulat <[email protected]> --------- Signed-off-by: Olga Bulat <[email protected]> Co-authored-by: Olga Bulat <[email protected]>
Fixes
Fixes https://github.com/WordPress/openverse-infrastructure/issues/1031 by @obulat
Description
This PR:
FRONTEND_URL
to use a genericservice_url
variable so prepare for also having API tests in k6... this allows us not to need to change the variable name based on the other inputs.ls
recipe to the k6 justfile for listing the namespaces and scenarios.Testing Instructions
Review the testing runs here: https://github.com/WordPress/openverse/actions/workflows/k6.yml
Note that the three failures were due to issues with passing the signing token. That was fixed by https://github.com/WordPress/openverse-infrastructure/pull/1060 and by correctly passing the token in the workflow 🙂
If you have access to Openverse's Grafana, you can also see those runs here: https://openverse.grafana.net/a/k6-app/tests/926189
Those runs were triggered in this PR by me setting the workflow to dispatch on
pull_request
and hard-coding input parameters. I've removed those hard coded parameters and put the workflow back to dispatch.The only remaining thing to test is the staging deployment bit that triggers this workflow, and of course we can dispatch the workflow manually from the repository to test that further as well (e.g., to test the inputs).
Otherwise, there is not much real testing possible in this PR.
Checklist
Update index.md
).main
) or a parent feature branch.ov just catalog/generate-docs
for catalogPRs) or the media properties generator (
ov just catalog/generate-docs media-props
for the catalog or
ov just api/generate-docs
for the API) where applicable.Developer Certificate of Origin
Developer Certificate of Origin