Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vagrant demo prometheus file is missing. #122

Closed
thatsk opened this issue Sep 4, 2021 · 11 comments
Closed

vagrant demo prometheus file is missing. #122

thatsk opened this issue Sep 4, 2021 · 11 comments
Labels
triage/rotten closed due to lack of information for too long, rejected feature...

Comments

@thatsk
Copy link

thatsk commented Sep 4, 2021

https://github.com/kumahq/kuma-demo/blob/master/vagrant/metrics/app/prometheus/prometheus.yaml#L6

this file is missing and that's why Prometheus is not able to scrape anything because the rule file is missing.
@thatsk
Copy link
Author

thatsk commented Sep 6, 2021

@jpeach any idea why this is not working?

@thatsk
Copy link
Author

thatsk commented Sep 7, 2021 

vagrant@metrics:~$ sudo journalctl -u kuma-prometheus-sd -f
-- Logs begin at Tue 2021-09-07 04:49:01 UTC. --
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.772Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        creating a gRPC client for Monitoring Assignment Discovery Service (MADS) server ...        {"streamID": 22674}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.773Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        starting an xDS stream ...        {"streamID": 22674}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.773Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        closing a connection ...        {"streamID": 22674}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.773Z        ERROR        kuma-prometheus-sd.run.xds_sd.discoverer        xDS stream terminated with an error        {"streamID": 22674, "error": "failed to start an xDS stream: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"\nfailed to start an xDS stream\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds/v1.(*streamDiscoverer).Run\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/v1/stream.go:55\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds.(*discoverer).Run.func1\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/xds.go:57\nruntime.goexit\n\t/home/circleci/go/src/runtime/asm_amd64.s:1371"}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.774Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        creating a gRPC client for Monitoring Assignment Discovery Service (MADS) server ...        {"streamID": 22675}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.774Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        starting an xDS stream ...        {"streamID": 22675}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        closing a connection ...        {"streamID": 22675}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        ERROR        kuma-prometheus-sd.run.xds_sd.discoverer        xDS stream terminated with an error        {"streamID": 22675, "error": "failed to start an xDS stream: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"\nfailed to start an xDS stream\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds/v1.(*streamDiscoverer).Run\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/v1/stream.go:55\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds.(*discoverer).Run.func1\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/xds.go:57\nruntime.goexit\n\t/home/circleci/go/src/runtime/asm_amd64.s:1371"}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        creating a gRPC client for Monitoring Assignment Discovery Service (MADS) server ...        {"streamID": 22676}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        starting an xDS stream ...        {"streamID": 22676}

@thatsk
Copy link
Author

thatsk commented Sep 7, 2021

looks like we are not able to generate /var/run/kuma-prometheus-sd/kuma.file_sd.json this file in vagrant provisioning script

@thatsk thatsk mentioned this issue Sep 7, 2021
Closed
@kunalsingthakur
Copy link

@jpeach can you take look at this

@jpeach
Copy link
Contributor

jpeach commented Sep 20, 2021

IIUC this service should produce the kuma.file_sd.json

@kunalsingthakur
Copy link

@jpeach its showing this logs

-- Logs begin at Tue 2021-09-07 04:49:01 UTC. --
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.772Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        creating a gRPC client for Monitoring Assignment Discovery Service (MADS) server ...        {"streamID": 22674}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.773Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        starting an xDS stream ...        {"streamID": 22674}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.773Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        closing a connection ...        {"streamID": 22674}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.773Z        ERROR        kuma-prometheus-sd.run.xds_sd.discoverer        xDS stream terminated with an error        {"streamID": 22674, "error": "failed to start an xDS stream: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"\nfailed to start an xDS stream\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds/v1.(*streamDiscoverer).Run\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/v1/stream.go:55\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds.(*discoverer).Run.func1\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/xds.go:57\nruntime.goexit\n\t/home/circleci/go/src/runtime/asm_amd64.s:1371"}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.774Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        creating a gRPC client for Monitoring Assignment Discovery Service (MADS) server ...        {"streamID": 22675}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.774Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        starting an xDS stream ...        {"streamID": 22675}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        closing a connection ...        {"streamID": 22675}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        ERROR        kuma-prometheus-sd.run.xds_sd.discoverer        xDS stream terminated with an error        {"streamID": 22675, "error": "failed to start an xDS stream: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"", "errorVerbose": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:5676: connect: connection refused\"\nfailed to start an xDS stream\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds/v1.(*streamDiscoverer).Run\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/v1/stream.go:55\ngithub.com/kumahq/kuma/app/kuma-prometheus-sd/pkg/discovery/xds.(*discoverer).Run.func1\n\t/home/circleci/project/app/kuma-prometheus-sd/pkg/discovery/xds/xds.go:57\nruntime.goexit\n\t/home/circleci/go/src/runtime/asm_amd64.s:1371"}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        creating a gRPC client for Monitoring Assignment Discovery Service (MADS) server ...        {"streamID": 22676}
Sep 07 05:02:12 metrics kuma-prometheus-sd[3408]: 2021-09-07T05:02:12.775Z        INFO        kuma-prometheus-sd.run.xds_sd.discoverer        starting an xDS stream ...        {"streamID": 22676}```

@kunalsingthakur
Copy link

somehow this is not discovering
https://github.com/kumahq/kuma-demo/blob/master/vagrant/metrics/kuma/kuma-prometheus-sd.service#L8
thats why it's not producing a file

@jpeach
Copy link
Contributor

jpeach commented Sep 20, 2021

Oh that makes sense. I guess it's configures for the wrong host/port :-(

@kunalsingthakur
Copy link

kunalsingthakur commented Sep 21, 2021 via email

@github-actions github-actions bot added the triage/stale Inactive for some time. It will be triaged again label Nov 19, 2021
@github-actions
Copy link
Contributor

This issue was inactive for 30 days it will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant please comment on it promptly or attend the next triage meeting.

@lahabana
Copy link
Contributor

lahabana commented Mar 9, 2022

Closing in favour of the more generic #145

@lahabana lahabana closed this as completed Mar 9, 2022
@lahabana lahabana added triage/rotten closed due to lack of information for too long, rejected feature... and removed triage/stale Inactive for some time. It will be triaged again labels Mar 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
triage/rotten closed due to lack of information for too long, rejected feature...
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jpeach @lahabana @thatsk @kunalsingthakur