Add retry to Verification process

[WadeBarnes]

Occasionally a verification can run into an issue and fail on the first attempt. If attempted again a sort time later it will succeed.

For the sake of robustness add a configurable retry and delay to the verification process so verifications are retried if they fail the first time.

Example failure:

Verifying backup ...
  |  
  | Settings:
  | - Database: postgres=vc-authn-database/vc_authn
  | - Backup file: /backups/daily/2021-01-28/vc-authn-database-vc_authn_2021-01-28_01-00-00.sql.gz
  |  
  | Already clean ...
  |  
  | waiting for server to start.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
  | The server failed to start within 0h:10m:0s.

In this case the local server used to verify the backup did not start in the allotted time (and as you can see it was given plenty of time to startup). This was the first verification of four this server performs daily. The subsequent three verifications competed successfully with the local server starting in around 2 minutes (slower than expected, but still acceptable for the purpose of automated verification).

Verifying backup ...
--
  |  
  | Settings:
  | - Database: postgres=vc-authn-wallet/vc_authn_agent_wallet
  | - Backup file: /backups/daily/2021-01-28/vc-authn-wallet-vc_authn_agent_wallet_2021-01-28_01-00-03.sql.gz
  |  
  | Already clean ...
  |  
  | waiting for server to start....................................................................................................
  | The server started in 0h:1m:45s.
  |  
  |  
  | Restoring from backup ...
  | Restoring '/backups/daily/2021-01-28/vc-authn-wallet-vc_authn_agent_wallet_2021-01-28_01-00-03.sql.gz' to '127.0.0.1/vc_authn_agent_wallet' ...
  | waiting for server to shut down.... done
  | server stopped
  | Cleaning up ...
  |  
  | Successfully verified backup: /backups/daily/2021-01-28/vc-authn-wallet-vc_authn_agent_wallet_2021-01-28_01-00-03.sql.gz
  | The restored database contained 4 tables, and is 7783 kB in size.
  |  
  | Elapsed time: 0h:1m:55s - Status Code: 0
  |  
  | Verifying backup ...
  |  
  | Settings:
  | - Database: postgres=wallet-lsbc/agent_lsbc_wallet
  | - Backup file: /backups/daily/2021-01-28/wallet-lsbc-agent_lsbc_wallet_2021-01-28_01-00-07.sql.gz
  |  
  | Already clean ...
  |  
  | waiting for server to start..................................................................................................
  | The server started in 0h:1m:43s.
  |  
  |  
  | Restoring from backup ...
  | Restoring '/backups/daily/2021-01-28/wallet-lsbc-agent_lsbc_wallet_2021-01-28_01-00-07.sql.gz' to '127.0.0.1/agent_lsbc_wallet' ...
  | waiting for server to shut down.... done
  | server stopped
  | Cleaning up ...
  |  
  | Successfully verified backup: /backups/daily/2021-01-28/wallet-lsbc-agent_lsbc_wallet_2021-01-28_01-00-07.sql.gz
  | The restored database contained 4 tables, and is 7783 kB in size.
  |  
  | Elapsed time: 0h:2m:58s - Status Code: 0
  |  
  | Verifying backup ...
  |  
  | Settings:
  | - Database: postgres=wallet-prime/agent_prime_wallet
  | - Backup file: /backups/daily/2021-01-28/wallet-prime-agent_prime_wallet_2021-01-28_01-00-11.sql.gz
  |  
  | Already clean ...
  |  
  | waiting for server to start......................................................................................................................
  | The server started in 0h:2m:4s.
  |  
  |  
  | Restoring from backup ...
  | Restoring '/backups/daily/2021-01-28/wallet-prime-agent_prime_wallet_2021-01-28_01-00-11.sql.gz' to '127.0.0.1/agent_prime_wallet' ...
  | waiting for server to shut down.... done
  | server stopped
  | Cleaning up ...
  |  
  | Successfully verified backup: /backups/daily/2021-01-28/wallet-prime-agent_prime_wallet_2021-01-28_01-00-11.sql.gz
  | The restored database contained 4 tables, and is 7927 kB in size.