TRON-2210: Add a tool that can sync from pod state to tron w/ tronctl

[jfongatyelp]

This adds a pretty simple script that will:

1. search for any completed pods in the tron namespace given a KUBECONFIG
2. attempt to match those pods with any existing action runs that are in a state that can be updated
3. update the state w/ tronctl if the pod & tron's states do not match.

Tron will only allow us to transition from a few known states, so this script only allows using tronctl fail or tronctl succeed if the tron action run was either lost/unknown or 'stuck' at running or starting.

Right now, I allow passing both a tron-url + tronctl-wrapper so that this could be run from outside the tron server itself, but defaults assume we'll be running on a local tron server.

Still adding some tests but thought I should get feedback now in case I need to shift gears.

Validation

Currently, one reliable way to get an actual tron action run into an unknown state is to restart tron w/in a short period of the pod having been started.

Forcing a few of these cases in tron-spark-pnw-devc, the script correctly updates the 'unknown' action runs' states:

Action state scheduled for paasta-contract-monitor.k8s_bad.41.sleep_with_retries not modifiable, no action taken
Action state failed for paasta-contract-monitor.k8s_bad.40.sleep not modifiable, no action taken
paasta-contract-monitor.k8s_bad.40.sleep_with_retries state unknown needs updating to fail
Dry-Run: Would run ['tronctl', 'fail', 'paasta-contract-monitor.k8s_bad.40.sleep_with_retries']
Action state failed for paasta-contract-monitor.k8s_bad.39.sleep not modifiable, no action taken
paasta-contract-monitor.k8s_bad.39.sleep_with_retries state unknown needs updating to fail
Dry-Run: Would run ['tronctl', 'fail', 'paasta-contract-monitor.k8s_bad.39.sleep_with_retries']
Action state failed for paasta-contract-monitor.k8s_bad.38.sleep not modifiable, no action taken
paasta-contract-monitor.k8s_bad.38.sleep_with_retries state running needs updating to fail
Dry-Run: Would run ['tronctl', 'fail', 'paasta-contract-monitor.k8s_bad.38.sleep_with_retries']
Action state failed for paasta-contract-monitor.k8s_bad.37.sleep not modifiable, no action taken
paasta-contract-monitor.k8s_bad.37.sleep_with_retries state running needs updating to fail
Dry-Run: Would run ['tronctl', 'fail', 'paasta-contract-monitor.k8s_bad.37.sleep_with_retries']
Action state failed for paasta-contract-monitor.k8s_bad.36.sleep not modifiable, no action taken
paasta-contract-monitor.k8s_bad.36.sleep_with_retries state running needs updating to fail
Dry-Run: Would run ['tronctl', 'fail', 'paasta-contract-monitor.k8s_bad.36.sleep_with_retries']
Action state cancelled for paasta-contract-monitor.k8s_bad.35.sleep not modifiable, no action taken

[nemacysts]

not a bad idea :)

[jfongatyelp]

Turns out we don't actually include run_limit in job/job run API responses :/ so this would need to pull in from other sources (either config api or reading the configs some other way), so I think I'm going to leave it as is for now (we can always run w/ a higher num_run if we need to).

[nemacysts]

yea, we can probably read from /nail/etc/services - but it's probably fine to leave as-is, i can't think of anything getting anywhere near their run_limit in terms of concurrently running jobs

[nemacysts]

not doing anything wrong - i'm not quite sure why this method doesn't automatically add in /api for us