Skip to content

Add runbook for PrometheusSDRefreshFailure #89

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add runbook for PrometheusSDRefreshFailure #89

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions content/runbooks/prometheus/PrometheusSDRefreshFailure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# PrometheusSDRefreshFailure

## Meaning

Prometheus fails to refresh its service discovery (SD) targets. This might indicate connectivity issues to SD providers (such as Kubernetes, Consul, DNS), misconfiguration, or internal errors within Prometheus.

## Impact

Missing or stale service discovery information leads new targets not being scraped and old target are not removed from the configuration, resulting in invalid metrics and alerts.

## Diagnosis

The root causes for this issue can be pretty diverse, but here are some hints where to start:
- Check Prometheus logs for specific SD-related error messages.
- Ensure your Prometheus SD configuraion is formally correct.
- Verify network connectivity between Prometheus and SD endpoints.
- Based on the info from the steps before, check whether there are recent changes in Prometheus' configuration, the SD endpoints, network policies, or authentication credentials.

## Mitigation

Fix network issues that prevent Prometheus from reaching SD endpoints.

Check whether DNS resolution is working correctly in the cluster.

Check credentials and permissions are correct.

As a last resort, remove configuration for failing SD endpoints.