Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relay Agent Fails to Start For Clusters Upgraded from 0.2.0 With fingerprint mismatch #77

Open
2 tasks done
estein9825 opened this issue Aug 16, 2023 · 0 comments
Open
2 tasks done
Labels
bug Something isn't working new Needs triage
Milestone

Comments

@estein9825
Copy link

Expected vs actual behavior

If you attempt to upgrade a paralus server from 0.2.0 with relay version 0.1.2 to 0.2.5 paralus with relay 0.1.5 (starting with 0.1.4), you'll get the error:

"level":"info","ts":"2023-08-16T23:25:59.118Z","caller":"agent/agent.go:171","msg":"Relay Agent::relay agent namespace: paralus-system  fingerprint: 4d63c263-0e39-4451-aa97-6882e339fc30  "}
{"level":"info","ts":"2023-08-16T23:25:59.119Z","caller":"agent/agent.go:394","msg":"Relay Agent::config: &{TemplateToken:cifi3glc3m5b406jcbc0 TemplateName: Scheme:https Mode: Addr:console.paralus.iherbtest.net:443 ClientID:cisqaj3ppcveb4n2bcrg ClientIP:10.7.228.141 Name:relay-agent-557f56bb69-x2rcc PrivateKey:[] CSR:[] Certificate:[] CACertificate:[] ServerHost: ServerPort:0 Fingerprint:4d63c263-0e39-4451-aa97-6882e339fc30} "}
[POST /v2/sentry/bootstrap/{templateToken}/register][500] bootstrapRegisterBootstrapAgentInternalServerError  map[code:2 message:fingerprint mismatch for token cisqaj3ppcveb4n2bcrg]
{"level":"error","ts":"2023-08-16T23:25:59.159Z","caller":"agent/agent.go:397","msg":"Relay Agent::failed to register relay agent error: [POST /v2/sentry/bootstrap/{templateToken}/register][500] bootstrapRegisterBootstrapAgentInternalServerError  map[code:2 message:fingerprint mismatch for token cisqaj3ppcveb4n2bcrg]  ","stacktrace":"github.com/paralus/relay/pkg/agent.registerRelayAgent\n\t/build/pkg/agent/agent.go:397\ngithub.com/paralus/relay/pkg/agent.handleRelayNetworks\n\t/build/pkg/agent/agent.go:606"}

The only way to fix this is to go to the database and update the fingerprint value in the sentry_bootstrap_agent table for the specific token row with the uid from the paralus-system namespace for that client (also found within the log)

Steps to reproduce the bug

  1. Create a cluster in paralus 0.2.0 or 0.2.1 (so the relay agent is at 0.1.3 or lower)
  2. Bootstrap it in and make sure it works
  3. Upgrade the paralus to 0.2.3 or higher (so relay agent is 0.1.4 or higher)
  4. Try to have the client connect and it will fail with the above issue

(optional) If you have ideas on why the bug happens or how it can be solved, please provide it here

This appears to be happening because an existing cluster does not automatically update the fingerprint if it's already bootstrapped into the database. You have to do a new bootstrapping, or update the value manually

  • I've described the bug, included steps to reproduce it, and included my environment setup with all customizations.
  • I'm using the latest version of the project.
@estein9825 estein9825 added bug Something isn't working new Needs triage labels Aug 16, 2023
@nirav-rafay nirav-rafay added this to the v0.1.8 milestone Jan 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working new Needs triage
Projects
None yet
Development

No branches or pull requests

2 participants