Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] When disabling metrics - chart is not removing of prometheus annotations #647

Open
itamar-marom opened this issue Feb 17, 2025 · 3 comments · May be fixed by #652
Open

[Bug] When disabling metrics - chart is not removing of prometheus annotations #647

itamar-marom opened this issue Feb 17, 2025 · 3 comments · May be fixed by #652
Labels
bug Something isn't working

Comments

@itamar-marom
Copy link

What are you really trying to do?

I'm trying to monitor temporal through DataDog autodiscovery option (through annotations)

Describe the bug

When I'm disabling all metrics annotations there are still some services - the headless services, that still holds the metrics annotations.
Looking at the code I see that there are prometheus labels statically there and indeed cannot be removed through configuration.

Image

Environment/Versions

  • OS and processor: EKS amazon linux
  • Temporal Version: Chart version 0.54
  • Are you using Docker or Kubernetes or building Temporal from source? Kubernetes

Additional context
@itamar-marom itamar-marom added the bug Something isn't working label Feb 17, 2025
@robholland
Copy link
Contributor

Does this cause issues in some way?

@itamar-marom
Copy link
Author

itamar-marom commented Feb 19, 2025 •
edited
Loading

@robholland
I have a datadog agent that has a the prometheus scraping feature enabled (for other workloads in my cluster). For temporal I'm working with DataDog's temporal integration.
In this current state the datadog agent scrapes tepmoral twice. and in datadog scraping with the prometheus annotations counts as a custom metrics - which costs lost of money

@robholland
Copy link
Contributor

I understand, we'll get that fixed.

@robholland robholland linked a pull request Feb 25, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Turn off service annotations when requested. temporalio/helm-charts
2 participants
@robholland @itamar-marom