If proxy server goes down, it won't retry.

[tacLog]

Hey all,
Some one noticed in this ticket that they way we suggest this to be configured will result in it preminatly disconnecting if the proxy server goes down.

From the redsocks docs:

// various ways to handle proxy failure
	//  close -- just close connection (default)
	//  forward_http_err -- forward HTTP error page from proxy as-is
	// on_proxy_fail = close;
	```

By default the connect just closes and drops forever. Never retrying. We should probably at least warn of this behavior and suggest the use of a monitoring process like this one:

``` python
import requests
import time

endpoint = "<endpoint>" # Change this to the URL you want to check the connection to
max_retries = 5
retry_count = 0

while True:
    try:
        response = requests.get(endpoint)
        if response.status_code == 200:
            retry_count = 0
        else:
            retry_count += 1
    except:
        retry_count += 1

    if retry_count >= max_retries:
        # Run the command to restart all containers
        import subprocess
        appId = "<appId>" # Replace with the actual appId
        cmd = f"curl -X POST --header 'Content-Type:application/json' --data '{{\"appId\": {appId}}}' '$BALENA_SUPERVISOR_ADDRESS/v1/restart?apikey=$BALENA_SUPERVISOR_API_KEY'"
        subprocess.call(cmd, shell=True)
        retry_count = 0

    time.sleep(30)

I could be wrong here as I didn't test this failure myself, but I thought it would be good to track this.
-Thomas

[jellyfish-bot]

[taclog] This has attached https://jel.ly.fish/29190029-f226-4d42-a8ee-fca4ce0235c6