connection reset after sshkeys removed

[hudecof]

Hi,

tested on debian 9.

If you remove the ssh keys, the next attempt to connect failed. So the role is not able to finish. I enabled pipelining and check the ControlMaster, but generaly I clould not relay in this.

(ansible-env)air-2:ovirt hudecof$ ansible-playbook seal.yml --skip-tags=poweroff -k
SSH password:

PLAY [all] ***********************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************
ok: [template-debian-9]

TASK [rhevm-qe-automation.ansible-role-seal : Load a variable file based on the OS type.] ****************************
ok: [template-debian-9] => (item=/Users/hudecof/devel/_tmp/ovirt/roles/rhevm-qe-automation.ansible-role-seal/vars/defaults.yml)

TASK [rhevm-qe-automation.ansible-role-seal : Flag the system for reconfiguration] ***********************************
changed: [template-debian-9]

TASK [rhevm-qe-automation.ansible-role-seal : Fetch SSH keys to be removed] ******************************************
ok: [template-debian-9]

TASK [rhevm-qe-automation.ansible-role-seal : remove ssh host keys] **************************************************
changed: [template-debian-9] => (item={u'uid': 0, u'woth': False, u'mtime': 1506501341.612, u'inode': 147266, u'isgid': False, u'size': 419, u'roth': False, u'isuid': False, u'isreg': True, u'gid': 0, u'ischr': False, u'wusr': True, u'xoth': False, u'rusr': True, u'nlink': 1, u'issock': False, u'rgrp': False, u'path': u'/etc/ssh/ssh_host_ed25519_key', u'xusr': False, u'atime': 1506501342.972, u'isdir': False, u'ctime': 1506501341.612, u'wgrp': False, u'xgrp': False, u'dev': 2049, u'isblk': False, u'isfifo': False, u'mode': u'0600', u'islnk': False})
changed: [template-debian-9] => (item={u'uid': 0, u'woth': False, u'mtime': 1506501341.612, u'inode': 154464, u'isgid': False, u'size': 104, u'roth': True, u'isuid': False, u'isreg': True, u'gid': 0, u'ischr': False, u'wusr': True, u'xoth': False, u'rusr': True, u'nlink': 1, u'issock': False, u'rgrp': True, u'path': u'/etc/ssh/ssh_host_ed25519_key.pub', u'xusr': False, u'atime': 1506501341.62, u'isdir': False, u'ctime': 1506501341.612, u'wgrp': False, u'xgrp': False, u'dev': 2049, u'isblk': False, u'isfifo': False, u'mode': u'0644', u'islnk': False})
changed: [template-debian-9] => (item={u'uid': 0, u'woth': False, u'mtime': 1506501341.564, u'inode': 147261, u'isgid': False, u'size': 1675, u'roth': False, u'isuid': False, u'isreg': True, u'gid': 0, u'ischr': False, u'wusr': True, u'xoth': False, u'rusr': True, u'nlink': 1, u'issock': False, u'rgrp': False, u'path': u'/etc/ssh/ssh_host_rsa_key', u'xusr': False, u'atime': 1506501342.968, u'isdir': False, u'ctime': 1506501341.564, u'wgrp': False, u'xgrp': False, u'dev': 2049, u'isblk': False, u'isfifo': False, u'mode': u'0600', u'islnk': False})
changed: [template-debian-9] => (item={u'uid': 0, u'woth': False, u'mtime': 1506501341.564, u'inode': 147263, u'isgid': False, u'size': 404, u'roth': True, u'isuid': False, u'isreg': True, u'gid': 0, u'ischr': False, u'wusr': True, u'xoth': False, u'rusr': True, u'nlink': 1, u'issock': False, u'rgrp': True, u'path': u'/etc/ssh/ssh_host_rsa_key.pub', u'xusr': False, u'atime': 1506501341.572, u'isdir': False, u'ctime': 1506501341.564, u'wgrp': False, u'xgrp': False, u'dev': 2049, u'isblk': False, u'isfifo': False, u'mode': u'0644', u'islnk': False})
changed: [template-debian-9] => (item={u'uid': 0, u'woth': False, u'mtime': 1506501341.588, u'inode': 147264, u'isgid': False, u'size': 227, u'roth': False, u'isuid': False, u'isreg': True, u'gid': 0, u'ischr': False, u'wusr': True, u'xoth': False, u'rusr': True, u'nlink': 1, u'issock': False, u'rgrp': False, u'path': u'/etc/ssh/ssh_host_ecdsa_key', u'xusr': False, u'atime': 1506501342.968, u'isdir': False, u'ctime': 1506501341.588, u'wgrp': False, u'xgrp': False, u'dev': 2049, u'isblk': False, u'isfifo': False, u'mode': u'0600', u'islnk': False})
changed: [template-debian-9] => (item={u'uid': 0, u'woth': False, u'mtime': 1506501341.588, u'inode': 147265, u'isgid': False, u'size': 184, u'roth': True, u'isuid': False, u'isreg': True, u'gid': 0, u'ischr': False, u'wusr': True, u'xoth': False, u'rusr': True, u'nlink': 1, u'issock': False, u'rgrp': True, u'path': u'/etc/ssh/ssh_host_ecdsa_key.pub', u'xusr': False, u'atime': 1506501341.596, u'isdir': False, u'ctime': 1506501341.588, u'wgrp': False, u'xgrp': False, u'dev': 2049, u'isblk': False, u'isfifo': False, u'mode': u'0644', u'islnk': False})

TASK [rhevm-qe-automation.ansible-role-seal : reset hostname on RHEL/CentOS <= 6] ************************************
skipping: [template-debian-9]

TASK [rhevm-qe-automation.ansible-role-seal : reset hostname] ********************************************************
No handlers could be found for logger "paramiko.transport"
fatal: [template-debian-9]: UNREACHABLE! => {"changed": false, "msg": "[Errno 54] Connection reset by peer", "unreachable": true}
	to retry, use: --limit @/Users/hudecof/devel/_tmp/ovirt/seal.retry

PLAY RECAP ***********************************************************************************************************
template-debian-9          : ok=5    changed=2    unreachable=1    failed=0

[KKoukiou]

@hudecof thanks for pointing it out..
So you 're right, I use ssh-pipelining option as well to keep alive the ssh connection.
I will add it to documentation for now and in the meantime I 'll try to think of some solution.

[hudecof]

this do not helped me anyway

[ssh_connection]

pipelining = True

[KKoukiou]

Hey so, this is because ssh pipelining actually helps to open ssh connection once per ansible task IIUC.
The problem here is however between different tasks, one that removes the ssh keys and the next one, which can not be executed because ssh connection gets lost.

However, SSH ControlMaster setting on you OpenSSH is actually supposed to do the authentication only once, when the TCP connection is opened for target, and thereafter all your extra SSH sessions are sent down that connection.

I didn't test it get yet, but I believe it should work.
I will let you know later today if this will solve this issue.

[KKoukiou]

@hudecof Hey,

So I managed to reproduce your issue by changing the default ansible.cfg file to disable the ControlMaster option from the ssh_args. In this way, the connection gets indeed broken after removing the ssh keys.
So, for me the role works, using the defaults ansible config file, since there is ControlMaster=auto for the ssh.
I am using ansible 2.4.0.0 and default config file located in /etc/ansible/ansible.cfg

Did you by any chance change these values in the configuration file?

[bndabbs]

I am still getting the disconnect issue even after updating the ControlMaster option. I am running 2.4.1 on MacOS.

My workaround was to move the offending task to the step right before poweroff. The poweroff step fails, but that's easy enough to handle manually.

[KKoukiou]

@bndabbs thanks for using the role. Maybe it would be useful to post your ansible.cfg file so that we can try to reproduce. The role isn't tested on MacOS yet from my side at all..

[Hy3n4]

Hi,

the same issue here.
MacOS 10.14.1 (18B75)
Target OS: CentOS 7, Ubuntu 18.04 LTS
ansible: version 2.4.3.0 (installed via HomeBrew)

Tried to set -o ControlMaster=auto -o ControlPersist=60s but it is pretty same as default value.
Tried to set USE_PERSISTENT_CONNECTIONS=True but with that settings i can't even connect to target server via SSH

Anything else I should try? I could use @bndabbs workaround but I am sure that this worked before. Not sure in which versions of ansible or seal role.

Thanks