[occm] API requests & Loadbalancer reconciliation metrics #1178

seanschneeweiss · 2020-08-31T18:55:34Z

What this PR does / why we need it:
This PR implements metrics for loadbalancer reconciliations:

cloudprovider_openstack_reconcile_duration_seconds
cloudprovider_openstack_reconcile_total
cloudprovider_openstack_reconcile_errors_total

and metrics for OpenStack API requests:

openstack_api_request_duration_seconds
openstack_api_requests_total
openstack_api_request_errors_total

Which issue this PR fixes(if applicable):
fixes #912

Special notes for reviewers:

Metrics are exposed to https://localhost:10258/metrics.
I would recommend to add the following arguments when running OCCM.

--authorization-always-allow-paths=/metrics
--secure-port=10258

Release note:

[openstack-cloud-controller-manager] Added metrics support for loadbalancer reconciliations and OpenStack API requests.

_{Sean Schneeweiss [email protected], Daimler TSS GmbH, legal info/Impressum}

k8s-ci-robot · 2020-08-31T18:55:42Z

Hi @seanschneeweiss. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

theopenlab-ci · 2020-08-31T18:59:00Z

Build succeeded.

cloud-provider-openstack-format : SUCCESS in 2m 13s

theopenlab-ci · 2020-08-31T18:59:35Z

Build succeeded.

cloud-provider-openstack-sanity-test-csi-cinder : SUCCESS in 2m 58s

theopenlab-ci · 2020-08-31T18:59:37Z

Build succeeded.

cloud-provider-openstack-unittest : SUCCESS in 3m 29s

theopenlab-ci · 2020-08-31T19:05:44Z

Build failed.

cloud-provider-openstack-acceptance-test-csi-cinder : POST_FAILURE in 8m 46s

theopenlab-ci · 2020-08-31T19:06:38Z

Build failed.

cloud-provider-openstack-e2e-test-csi-cinder : POST_FAILURE in 9m 47s

theopenlab-ci · 2020-08-31T19:09:59Z

Build failed.

cloud-provider-openstack-acceptance-test-csi-manila : POST_FAILURE in 11m 58s

theopenlab-ci · 2020-08-31T19:12:14Z

Build succeeded.

cloud-provider-openstack-acceptance-test-lb-octavia : SUCCESS in 16m 03s

theopenlab-ci · 2020-08-31T19:22:24Z

Build failed.

cloud-provider-openstack-multinode-csi-migration-test : FAILURE in 24m 56s

theopenlab-ci · 2020-08-31T21:12:53Z

Build succeeded.

cloud-provider-openstack-acceptance-test-e2e-conformance : SUCCESS in 2h 12m 14s

jichenjc · 2020-09-01T06:22:25Z

/ok-to-test

seanschneeweiss · 2020-09-01T07:26:31Z

/retest

seanschneeweiss · 2020-09-01T07:41:27Z

openlab/cloud-provider-openstack-multinode-csi-migration-test is failing maybe because of:

Events:
  Type     Reason       Age                  From                                                               Message
  ----     ------       ----                 ----                                                               -------
  Warning  FailedMount  56s (x10 over 5m5s)  kubelet, ubuntu-xenial-citynetwork-citynetwork-openlab-0000145850  MountVolume.SetUp failed for volume "pods-cloud-data" : hostPath type check failed: /var/lib/cloud/data is not a directory
  Warning  FailedMount  47s                  kubelet, ubuntu-xenial-citynetwork-citynetwork-openlab-0000145850  Unable to attach or mount volumes: unmounted volumes=[pods-cloud-data], unattached volumes=[csi-cinder-node-sa-token-d8qlb kubelet-dir pods-cloud-data pods-probe-dir secret-cinderplugin socket-dir registration-dir]: timed out waiting for the condition

openlab/cloud-provider-openstack-e2e-test-csi-cinder and openlab/cloud-provider-openstack-acceptance-test-csi-cinder are failing with:

/bin/bash: line 84: /var/lib/cloud/data/instance-id: No such file or directory

openlab/cloud-provider-openstack-acceptance-test-csi-manila is failing with:

Setting up the hosts
... this takes a few seconds (logs at logs/devstack-gate-setup-host.txt.gz)
./safe-devstack-vm-gate-wrap.sh: line 578: /tmp/ansible/bin/ara: No such file or directory
gzip: /home/zuul/workspace/logs/ara: No such file or directory
*** FAILED with status: 127

ramineni · 2020-09-02T08:48:57Z

/test cloud-provider-openstack-acceptance-test-csi-cinder

k8s-ci-robot · 2020-09-02T08:48:59Z

@ramineni: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

/test pull-cloud-provider-openstack-check
/test pull-cloud-provider-openstack-test

Use /test all to run all jobs.

In response to this:

/test cloud-provider-openstack-acceptance-test-csi-cinder

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

theopenlab-ci · 2020-09-02T08:58:39Z

Build failed.

cloud-provider-openstack-acceptance-test-csi-cinder : POST_FAILURE in 9m 13s

ramineni · 2020-09-02T10:02:40Z

/test /test cloud-provider-openstack-acceptance-test-csi-cinder

k8s-ci-robot · 2020-09-02T10:02:41Z

@ramineni: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

/test pull-cloud-provider-openstack-check
/test pull-cloud-provider-openstack-test

Use /test all to run all jobs.

In response to this:

/test /test cloud-provider-openstack-acceptance-test-csi-cinder

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ramineni · 2020-09-02T10:10:42Z

/test cloud-provider-openstack-acceptance-test-csi-cinder

k8s-ci-robot · 2020-09-02T10:10:44Z

@ramineni: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

/test pull-cloud-provider-openstack-check
/test pull-cloud-provider-openstack-test

Use /test all to run all jobs.

In response to this:

/test cloud-provider-openstack-acceptance-test-csi-cinder

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

theopenlab-ci · 2020-09-02T10:22:43Z

Build succeeded.

cloud-provider-openstack-acceptance-test-csi-cinder : SUCCESS in 11m 33s

ramineni · 2020-09-02T10:35:01Z

@seanschneeweiss Fixed the following failure in openlab

/bin/bash: line 84: /var/lib/cloud/data/instance-id: No such file or directory

theopenlab-ci · 2020-09-18T12:00:34Z

Build succeeded.

cloud-provider-openstack-format : SUCCESS in 2m 15s

theopenlab-ci · 2020-09-18T12:01:07Z

Build succeeded.

cloud-provider-openstack-sanity-test-csi-cinder : SUCCESS in 2m 59s

theopenlab-ci · 2020-09-18T12:01:13Z

Build succeeded.

cloud-provider-openstack-unittest : SUCCESS in 3m 16s

theopenlab-ci · 2020-09-18T12:12:59Z

Build succeeded.

cloud-provider-openstack-acceptance-test-csi-cinder : SUCCESS in 11m 19s

theopenlab-ci · 2020-09-18T12:21:14Z

Build succeeded.

cloud-provider-openstack-e2e-test-csi-cinder : SUCCESS in 19m 03s

theopenlab-ci · 2020-09-18T12:32:58Z

Build succeeded.

cloud-provider-openstack-acceptance-test-lb-octavia : SUCCESS in 13m 39s

theopenlab-ci · 2020-09-18T12:44:36Z

Build succeeded.

cloud-provider-openstack-acceptance-test-keystone-authentication-authorization : SUCCESS in 25m 08s

theopenlab-ci · 2020-09-18T12:46:22Z

Build succeeded.

cloud-provider-openstack-acceptance-test-csi-manila : SUCCESS in 25m 54s

theopenlab-ci · 2020-09-18T13:16:56Z

Build succeeded.

cloud-provider-openstack-multinode-csi-migration-test : SUCCESS in 43m 00s

theopenlab-ci · 2020-09-18T14:34:48Z

Build succeeded.

cloud-provider-openstack-acceptance-test-e2e-conformance : SUCCESS in 2h 10m 26s

lingxiankong · 2020-09-22T11:01:36Z

/lgtm

@seanschneeweiss Do you mind adding some doc for this?

seanschneeweiss · 2020-09-22T14:06:51Z

@seanschneeweiss Do you mind adding some doc for this?

@lingxiankong Should the documentation be added to a new markdown file e.g. /docs/metrics.md or to the existing /docs/using-openstack-cloud-controller-manager.md? Should I create a separate PR? Is there any existing documentation related to this, or maybe a template/guideline?

jichenjc · 2020-09-24T03:46:47Z

pkg/cloudprovider/providers/openstack/metrics/metrics.go

+limitations under the License.
+*/
+
+package metrics


I think we might want to move this to a generic location so that other component can benefit it later on

Thank you everyone for reviewing and approving this PR.
Refactoring / moving this to a generic location is handled within #1236.

jichenjc · 2020-09-24T03:49:33Z

/approve

I think we can do a followup PR and not blocking this..

k8s-ci-robot · 2020-09-24T03:49:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jichenjc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jichenjc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jichenjc · 2020-09-24T03:49:56Z

@seanschneeweiss please help to create a new PR related the doc and we can follow up discussion there, thanks

ramineni · 2020-10-05T09:19:50Z

@seanschneeweiss Do you mind adding some doc for this?

@lingxiankong Should the documentation be added to a new markdown file e.g. /docs/metrics.md or to the existing /docs/using-openstack-cloud-controller-manager.md? Should I create a separate PR? Is there any existing documentation related to this, or maybe a template/guideline?

@seanschneeweiss @sbueringer I think this is missed. Could you add the doc on how to use this . Adding new file docs/metrics.md sounds good to me , and link it from the main doc. You could receive more comments once you propose the PR. @lingxiankong do you have any other suggestions?

lingxiankong · 2020-10-05T09:47:33Z

Agree with @ramineni, besides, I'd like to see several things in the doc:

What metrics are supported? Which binaries are related?
How to config (if applicable) to make the metrics work? How to get metrics?
Metric examples

seanschneeweiss · 2020-10-06T15:38:29Z

I will create the PR soon, thank you for the feedback.

…#1178) * [occm] move metrics to seperate folder By moving the metrics definition to a seperate folder it can be used by pkg/cloudprovider/providers/openstack and pkg/util/openstack. Signed-off-by: Sean Schneeweiss <[email protected]> * [occm] API request and reconcile metrics This commit implements metrics for OpenStack API calls and for loadbalancer reconciliations. Signed-off-by: Sean Schneeweiss <[email protected]> * [occm] rename extension metric to network_extension

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 31, 2020

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 31, 2020

k8s-ci-robot requested review from dims and FengyunPan2 August 31, 2020 18:55

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 1, 2020

seanschneeweiss requested a review from lingxiankong September 21, 2020 16:23

k8s-ci-robot assigned lingxiankong Sep 22, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 22, 2020

jichenjc reviewed Sep 24, 2020

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 24, 2020

jichenjc mentioned this pull request Sep 24, 2020

[cinder-csi-plugin] metrics are not accessible #913

Closed

1 task

k8s-ci-robot merged commit eebfac2 into kubernetes:master Sep 24, 2020

seanschneeweiss mentioned this pull request Oct 20, 2020

[occm] Documentation for occm metrics #1283

Merged

sbueringer mentioned this pull request May 6, 2021

Expose gophercloud requests metrics kubernetes-sigs/cluster-api-provider-openstack#856

Closed

This was referenced May 18, 2021

REQUEST: New membership for seanschneeweiss kubernetes/org#2719

Closed

REQUEST: New membership for seanschneeweiss kubernetes/org#2725

Closed

[occm] API requests & Loadbalancer reconciliation metrics #1178

[occm] API requests & Loadbalancer reconciliation metrics #1178

Conversation

seanschneeweiss commented Aug 31, 2020 • edited by lingxiankong Loading

k8s-ci-robot commented Aug 31, 2020

theopenlab-ci bot commented Aug 31, 2020

theopenlab-ci bot commented Aug 31, 2020

theopenlab-ci bot commented Aug 31, 2020

theopenlab-ci bot commented Aug 31, 2020

theopenlab-ci bot commented Aug 31, 2020

theopenlab-ci bot commented Aug 31, 2020

theopenlab-ci bot commented Aug 31, 2020

theopenlab-ci bot commented Aug 31, 2020

theopenlab-ci bot commented Aug 31, 2020

jichenjc commented Sep 1, 2020

seanschneeweiss commented Sep 1, 2020

seanschneeweiss commented Sep 1, 2020

ramineni commented Sep 2, 2020

k8s-ci-robot commented Sep 2, 2020

theopenlab-ci bot commented Sep 2, 2020

ramineni commented Sep 2, 2020

k8s-ci-robot commented Sep 2, 2020

ramineni commented Sep 2, 2020

k8s-ci-robot commented Sep 2, 2020

theopenlab-ci bot commented Sep 2, 2020

ramineni commented Sep 2, 2020

theopenlab-ci bot commented Sep 18, 2020

theopenlab-ci bot commented Sep 18, 2020

theopenlab-ci bot commented Sep 18, 2020

theopenlab-ci bot commented Sep 18, 2020

theopenlab-ci bot commented Sep 18, 2020

theopenlab-ci bot commented Sep 18, 2020

theopenlab-ci bot commented Sep 18, 2020

theopenlab-ci bot commented Sep 18, 2020

theopenlab-ci bot commented Sep 18, 2020

theopenlab-ci bot commented Sep 18, 2020

lingxiankong commented Sep 22, 2020

seanschneeweiss commented Sep 22, 2020

jichenjc Sep 24, 2020

Choose a reason for hiding this comment

seanschneeweiss Sep 24, 2020

Choose a reason for hiding this comment

jichenjc commented Sep 24, 2020

k8s-ci-robot commented Sep 24, 2020

jichenjc commented Sep 24, 2020

ramineni commented Oct 5, 2020

lingxiankong commented Oct 5, 2020

seanschneeweiss commented Oct 6, 2020

seanschneeweiss commented Aug 31, 2020 •

edited by lingxiankong

Loading