[Remote filesystems /w Docker] Another Certbot instance is already running

[sertys3]

Dear all,
thanks for the wonderful work on dnsrobocert. Implemented the project recently into my deployment orchestration. But ran into an issue that others might face.
Since i'm keeping certificates on a glusterfs filesystem mounter over NFS for HA between orchestrators, saw some issues when mounting the volume over /etc/letsencrypt .
Looks like without dwelving into too much detail keeping lock files on remote filesystems is not the best idea. Once i moved the volume to the NFS share certbot was unable to validate ownership of the lockfile( either in /etc/letsencrypt/.certbot.lock or /etc/letsencrypt/logs/.certbot.lock ) and spitted errors like :
Another Certbot instance is already running

The workaround for me was to mount the individual subfolders in /etc/letsencrypt(in docker-compose.yml) like so :
dnsrobocert:
image: adferrand/dnsrobocert
container_name: dnsrobocert
environment:
CONFIG_PATH: /etc/letsencrypt/dnsrobocert.yml
volumes:
- ./configs/letsencrypt/archive:/etc/letsencrypt/archive
- ./configs/letsencrypt/live:/etc/letsencrypt/live
- ./configs/letsencrypt/keys:/etc/letsencrypt/keys
- ./configs/letsencrypt/accounts:/etc/letsencrypt/accounts
- ./configs/letsencrypt/dnsrobocert.yml:/etc/letsencrypt/dnsrobocert.yml
restart: always

This way the lock files are held on the container storage and certbot no longer complains.

Hope this helps the community.
Regards,
Daniel

[adferrand]

Hello Daniel !

Thanks for the report. Indeed, certbot uses lock files to ensure that only instance of certbot is running. For the record, I am part of the dev team of certbot, and I had in mind for a long time some improvements to do for lock files, and more generally for situations where certbot is acting more like a library.

But at this time, there is no way to customize that, so your workaround is the best so far.

If you want, you can open an issue on the certbot repository to report about bad behavior of lock files on remote filesystem, it would give more concrete arguments to start some refactorings here.

[qqilihq]

Interesting topic -- I just noticed (by accident), that DNSroboCert creates mentioned log file while running certificate renewal.

Our setup is similar to @sertys3 -- we share the data and certificates via NFS so that it's available on our clustered reverse proxy instances.

So far, I just made sure, that only one single DNSroboCert instance was running cluster-wide. However, with that lock file, this would probably not even be needed, as DNSroboCert would synchronize itself, which in turn would simplify our setup and make it more robust.

@sertys3 mentions “lock files on remote filesystems is not the best idea”. Does this still apply? Any experience with running DNSroboCert in a clustered mode or is this something to avoid?