Enhance cluster-wide known hosts support

[ypid]

Currently, the role allows to specify a list of FQDN to scan over the network for there public host-key. Each host does this on it’s own. This is nice as it saves the public host keys once.

But we can/should do better because:

• Currently an adversary could MITM the first public host key scan (and following connections) and then can MITM all ssh connections
• The approach of scanning should only be used for non-DebOps hosts to optimize run time behavior in large environments and to allow to include all hosts by default.

I would propose that the public host key fingerprints get captured on the Ansible controller. They could be saved under secret/sshd/{{ ansible_domain }}/{{ ansible_fqdn }}/public_host_key_fingerprint. This would allow to distribute all known (and validated) fingerprints to all hosts in the same domain by default and to keep them up-to-date. Use the default OpenSSH host-key fingerprint file format.

An alternative to that could be to use the already setup PKI also for sshd but I have no idea how good supported that is. I think this approach with ssh public host keys would still be worth implementing.

Things to checkout:

• https://packages.debian.org/jessie/openssh-known-hosts

[drybjed]

Sounds like an interesting approach, that would also lower a chance of a DDoS of the external services, although I would leave the current approach as a possibility.

Going with the secret/ directory sounds nice, I'm not sure if another method could be done without setting up some kind of database beforehand. You should use a separate variable for ansible_domain, maybe something like sshd__domain to select different set of directories.

Or perhaps, just go with all/group/host approach similar to how debops.pki distributes Root CA certificates. It will mesh nicely with the Ansible inventory structure, with groups of hosts. Copy the corresponding files stored in secret/ to /etc/ssh/ssh_known_hosts.d/ directory and assemble them.

I would keep the debops.pki and debops.sshd PKI infrastructures separate, X.509 certificates and SSH keys don't mix all that well, and debops.pki already does some crazy things, no need to complicate it even more.

[ypid]

Thanks for the feedback. I will look into this when I have time. For now this issue is open for discussion and to not forget the idea 😉

lower the chance of a DDoS of the external services

I am not quite getting it. Can you explain?

I would leave the current approach as a possibility.

I totally agree. The current implementation can still be used for non-DebOps hosts.

I would keep the debops.pki and debops.sshd PKI infrastructures separate

Sounds good. No problem with that 😉

[drybjed]

As for the DDoS, if you have a few hosts and you want to add, say, github.com to list of know hosts, each host checks the fingerprint at the same time (the same place in the playbook). That might look a bit like a DDoS om the remote side, I imagine. Not much to be flagged as a bad guy, but some automated firewall might block access to ssh because of that.

[ypid]

Ah now I am getting it :) Yes, that could also be optimized by maybe letting the Ansible controller do the scanning once.

[drybjed]

That might be an issue with Ansible Controller and remote hosts use different set of DNS nameservers, potentially seeing different set of hosts. How likely it is? I'm not sure.

[ypid]

Something to think about. I guess the best way to make the delegate_to configurable and default to the Ansible controller.

[ypid]

Seems that recent versions of Ansible (noticed in v2.4 but was probably introduced before) have ansible_ssh_host_key_ed25519_public as Ansible facts. A quick search also brought up a template to distribute the host keys to nodes: https://github.com/gc3-uzh-ch/ansible-playbooks/blob/master/roles/common/templates/etc/ssh/ssh_known_hosts.j2

Something like this could be done easily. This way, you only must ensure you have an authentic connection from the Ansible controller to all nodes (which you will need to verify manually anyway) and then can distribute the trust relationship to the nodes.

Also, I updated my initial comment from 2016-05-15 above. I don’t think anymore that a custom file below secret/ would be a good idea and would propose to go with the default OpenSSH host-key fingerprint file format. This should integrate with the OpenSSH configuration mechanism and just maintain one host-key fingerprint on the Ansible controller. It needs to handle non-default host-key file places as can be configured, example:

Host *.pubsrv.ypid.de
    UserKnownHostsFile ~/.ssh/p-adm/known_hosts.d/pubsrv.ypid.de

This would also allow us to easily rotate/regenerate host keys without manually rechecking the host key. I have this in my playbook for bootstrapping hosts based on a VM template. The template has a known fingerprint which I check after copying the template. Then, I delete the host keys and regenerate them via dpkg-reconfigure.

Rotating host keys is supported by ssh with the UpdateHostKeys config option, but currently, when ControlPersist is enabled, UpdateHostKeys has no effect (stays disabled). So this could also act UpdateHostKeys.

[drybjed]

The template you linked to is interesting, especially that Ansible provides these facts as built-in, so they are always available during Ansible execution. However, instead of templating the whole file, I would go with lineinfile approach. Check in a loop if a given inventory_hostname isn't "us", and if so, if the host fingerprint is available, add or update it based on the hostvars[item].ansible_fqdn and the host's IP addresses using the know hosts format. The ssh_known_hosts syntax supports some additional options which could also be supported by the role. I'm not sure about hashing of the entries - it depends if the Ansible known_hosts module supports the additional options or not. But if we keep to the cluster nodes, I don't think that hashing them is necessary.

This way we don't need to keep any state anywhere and the know hosts lists on each remote host can be expanded/updated during normal Ansible operation - just run debops service/sshd against the hosts you want to sync. This of course handles only the fingerprints of hosts in the cluster.

[ypid]

I would not hash them, ref: http://blog.joeyhewitt.com/2013/12/openssh-hashknownhosts-a-bad-idea/. I disabled it myself, ref: https://github.com/ypid/dotfiles/blob/master/ssh/config_ypid_defaults#L19-L20

I would still argue to keep state on the Ansible controller, it needs/has this anyway and then this could also be used to rotate host keys, see UpdateHostKeys above. Your approche would require to run against all host at once. I rarely if ever do this.

[drybjed]

I suspect that we are now talking about slightly different things:

• I think that you are talking about adding SSH fingerprints of hosts from a list, that could be either hosts in the DebOps cluster or external host not managed by your DebOps installation;

• I'm thinking about maintaining SSH host fingerprints specifically of the hosts in the current DebOps cluster, so that any ssh connections between cluster hosts can work automatically, without the need to accept the host fingerprint in a separate step for a given UNIX user account;

In any case, even if you keep the list of SSH fingerprints in a file on Ansible Controller, you still need to connect to all cluster hosts at least once to update it. And if you need to refresh the list because one of the cluster nodes changed, you still need to connect to all of them to keep the list consistent. Therefore doing it via lineinfile instead of a single template drops the need for a stateful file on Ansible Controller. Perhaps this could even be done as a completely separate role, say debops.ssh_known_hosts, so that it's "cheap" to run it via cron against all DebOps hosts.

I'm suddenly reminded about keeping the SSH host fingerprints in DNS which would alleviate the above need to keep the ssh_known_hosts file updated, similarly how the introduction of the DNS system made the hosts.txt file obsolete... But that requires DNSSEC to make any sort of sense, so it's a long way away for now.

[ypid]

All right, simplicity probably speaks for your solution, feel free to give it a try. I agree that a separate role would be good, debops.ssh_known_hosts sounds good.

I think that you are talking about adding SSH fingerprints of hosts from a list, that could be either hosts in the DebOps cluster or external host not managed by your DebOps installation;

Not directly. I had the ideal of a unified approach which:

1. Distributes host keys to all nodes without the need to run against all your nodes at once. Consider you are deploying a new host. I don’t run ansible-playbook against all of the others hosts just to sync host keys for this one host.
2. Handles host keys rotations for the Ansible controller.
3. Supports the distribution of host keys of others hosts not managed by DebOps to DebOps managed nodes.

The idea is that you have your known hosts file on the Ansible controller anyway and it could be reused for this.