Guided installation of an additional service on a node: how to avoid that the install script to (re)install teleport?

[gramunno]

Hello,

I've a simple test setup with one VM running the proxy and the auth services and a VM as a teleport cluster node.

On this node I have installed a db service from scratch. When I want to install the ssh service on the same node (resource type named (Server) Ubuntu 14.04+), at a certain step I am requested to download and execute a script that tries to install teleport - and if it is already installed, it requests to completely remove it, disrupting the service(s) previously installed.

This problem also occurs if I first install ssh and then the db service.

I do not want to reinstall teleport at every service installation as I want to have multiple services (eg ssh, db, windows desktop,...) installed on the same VM.

How do I install the services on a node without reinstalling teleport every time?

Thanks in advance

[webvictim]

You'll want to manually edit the /etc/teleport.yaml configuration file to enable each service individually. Example for the SSH service:

ssh_service:
  enabled: yes
  # replace with labels you would like set, if needed
  labels:
    env: dev

You'll also need to generate a new token valid for the Node and Db services - you can also edit an existing token in the web UI to add both services under the "Join Tokens" section, then copy the token and replace it in the config file:

teleport:
  join_params:
    method: token
    token_name: <token-value-here>

After this, restart the Teleport service (sudo systemctl restart teleport) and both services should be active.

[gramunno]

Thanks for the answer.

As for the creation of a new token, unfortunately for the method iam in the web UI there is a box called AWS rule with a field account ID, but I don't have any AWS account. If I select the method gcp, there it appears another box with other requested parameters whose fields I don't know how to fill in...

Are there alternatives?

[webvictim]

There is a workaround for this.

When you run the "Enroll New Resource" wizard, the link it outputs for you to run on the server actually contains an alphanumeric join token which is valid for 4 hours. Once you see this link on screen, you can exit the wizard and go back to the "Join Tokens" screen - you should then see the same alphanumeric token and be able to edit its associated services there, then copy and use the token as described earlier.

[gramunno]

There is a workaround for this.

Thanks!

So the whole procedure (which I tested) to enable, e.g., the db service would be:

1. First steps of the wizard Enroll New Resource (Guided - (Self-Hosted) MySQL/MariaDB) until "Manually Deploy a Database Service": click on generate command

2. Apply the workaround described above and copy the token to the clipboard

When you run the "Enroll New Resource" wizard, the link it outputs for you to run on the server actually contains an alphanumeric join token which is valid for 4 hours. Once you see this link on screen, you can exit the wizard and go back to the "Join Tokens" screen - you should then see the same alphanumeric token and be able to edit its associated services there, then copy and use the token as described earlier.

You'll want to manually edit the /etc/teleport.yaml configuration file to enable each service individually.

Example for the DB service:

db_service:
  enabled: yes
  # replace with labels you would like set, if needed
  labels:
    env: dev

Paste the copied token value into the config file

teleport:
  join_params:
    method: token
    token_name: <token-value-here>

After this, restart the Teleport service (sudo systemctl restart teleport) and both services should be active.

4. delete the db resource (needed to restart with the whole wizard)

sudo tctl rm db/<db-resource-name>

5. Restart the wizard Enroll New Resource (Guided - (Self-Hosted) MySQL/MariaDB)

The step "Manually Deploy a Database Service" won't appear anymore, so teleport won't be reinstalled and the previous services/resources (e.g. SSH) will be kept.

[webvictim]

Yes, that's pretty much it. Once you have one database set up, you can also just follow the docs to add a new database rather than using the wizards.

https://goteleport.com/docs/enroll-resources/database-access/enroll-self-hosted-databases/

Each database registered under a db_service needs some additional parameters like protocol, uri etc. See the full configuration reference here for an example: https://goteleport.com/docs/reference/config/#database-service

[gramunno]

Yes, that's pretty much it. Once you have one database set up, you can also just follow the docs to add a new database rather than using the wizards.

https://goteleport.com/docs/enroll-resources/database-access/enroll-self-hosted-databases/

Each database registered under a db_service needs some additional parameters like protocol, uri etc. See the full configuration reference here for an example: https://goteleport.com/docs/reference/config/#database-service

Thanks a lot!

[gramunno]

Now that on a cluster node I got both the DB and SSH services enabled with the related resources visible from the web UI, I tried the opposite of the previous procedure, i.e. with just the database resource available, adding again the SSH resource because also the web UI wizard for SSH (Enroll New Resource -> (Server) Ubuntu 14.04+) requires to launch a script that wants to (re)install teleport on the node.

First of all I tried to delete the SSH resource with the command sudo tctl rm node/<node_name>: despite the <node_name> was present in both the web UI and the output of tsh ls it didn't work. Anyway deleting by UUID worked fine (sudo tctl rm node/<UUID> with the UUID obtained via tsh ls -v). The error message was ERROR: key /nodes/default/<node_name>. Any idea? Anyway this is not the core of the problem I'm reporting here.

Let's see the attempted procedure:

1. Enroll New Resource -> (Server) Ubuntu 14.04+: the link to the script that would (re)install teleport is shown with the join token embedded.

2. Leave the wizard and go to the Join Token section

3. Edit the token with the value seen at step 1 and add the role 'db' to keep the previous service

4. edit the config file on the node and replace the auth_token value with the one of step 1.

Without restarting teleport - weird - teleport takes the new configuration and:

a) the SSH resource appears on the web UI and is visible when executing tsh ls -v and the UUID is the same as the one deleted - weird as well...

b) If I restart the wizard [Enroll New Resource -> (Server) Ubuntu 14.04+], the link with the script that wants to (re)install teleport is still
there (while with the procedure for DB described above the link with the script didn't show up anymore after using the workaround)

So in the end the previous resource reappears and I cannot execute again the wizard (without the need to launch the script that (re)install teleport), e.g. to set up a different user account when connecting to the SSH server.

How to fix this? Is there another workaround? Or a sequence of CLI commands that can bypass the wizard?

[webvictim]

The error message was ERROR: key /nodes/default/<node_name>. Any idea? Anyway this is not the core of the problem I'm reporting here.

Yes, node deletion is keyed by UUID rather than name as you found. It's relatively rare that people want to delete nodes manually - they will automatically be deleted after being disconnected for 10-15 minutes.

How to fix this? Is there another workaround? Or a sequence of CLI commands that can bypass the wizard?

Are you deleting the contents of /var/lib/teleport each time you restart this process?

What a join token actually does is bootstrap the first-time trust between the Teleport agent and the auth/proxy server. After this is done, the agent uses a TLS certificate to connect to the server instead, so the token is no longer required. This TLS certificate is cached under /var/lib/teleport/proc, so if you're restarting the same agent which has already been joined to the cluster, any new token you provide won't be used and the old TLS identity will be used.

It's also worth noting that each service (ssh_service, db_service etc) has its own unique certificate stored. As such, if you add a new service, the existing certificate will be used for the old service which was previously joined, while the token will be used for the new service which is joining. After this initial setup, each service will use its certificate.

You can look at the startup messages for the agent process to determine whether you're using a TLS certificate or a token - it will either say:

• Connecting to the cluster with TLS client certificate (using existing certificate under /var/lib/teleport)
• Joining the cluster with a secure token (using token from config file)

[gramunno]

The error message was ERROR: key /nodes/default/<node_name>. Any idea? Anyway this is not the core of the problem I'm reporting here.

Yes, node deletion is keyed by UUID rather than name as you found. It's relatively rare that people want to delete nodes manually - they will automatically be deleted after being disconnected for 10-15 minutes.

OK, good to know

How to fix this? Is there another workaround? Or a sequence of CLI commands that can bypass the wizard?

Are you deleting the contents of /var/lib/teleport each time you restart this process?

When I tried to run the wizard, yes, I deleted /var/lib/teleport (and uninstalled teleport) as the script wanted me to do so.
Instead in the procedure with an interrupted wizard that I described in my previous comment , I did not deleted the content of /var/lib/teleport .

What a join token actually does is bootstrap the first-time trust between the Teleport agent and the auth/proxy server. After this is done, the agent uses a TLS certificate to connect to the server instead, so the token is no longer required. This TLS certificate is cached under /var/lib/teleport/proc, so if you're restarting the same agent which has already been joined to the cluster, any new token you provide won't be used and the old TLS identity will be used.

It's also worth noting that each service (ssh_service, db_service etc) has its own unique certificate stored. As such, if you add a new service, the existing certificate will be used for the old service which was previously joined, while the token will be used for the new service which is joining. After this initial setup, each service will use its certificate.

You can look at the startup messages for the agent process to determine whether you're using a TLS certificate or a token - it will either say:

• Connecting to the cluster with TLS client certificate (using existing certificate under /var/lib/teleport)
• Joining the cluster with a secure token (using token from config file)

Thanks for the answer. Now it is clear why in my attempt with the interrupted wizard I got the same UUID when I re-instantiated the node service.

Anyway I've found a way to instantiate the node service without using the wizard, simply by using the command tctl tokens add --ttl=60m --type=node and copying the output token value into the config file and restarting teleport. Of course, as I previously deleted the node service but did not deleted /var/lib/teleport, the node service got again the old UUID, as you explained.