stale node entries in controller registry

[pohly]

When a node has registered with the controller and then exists (either during a normal shutdown of that node or because the driver dies unexpectedly), the registry has a stale entry.

We need a strategy for dealing with those stale entries. Right now, the controller will try to connect to the dead nodes in perpetuity (?).

[okartau]

Does the controller keep track when/if node is alive and good, and can it easily count connection attempts? The controller could remove stale entries based on some policy, if counting failed attempts/measuring "dead" time is easily doable.

[pohly]

I don't think it currently does, but something like that should be easily doable.

[pohly]

Resolved via PR #838