updated ClientComplete to do nothing if node is already complete

[keithwiersema]

This is to remedy an issue where multiple ClientComplete events can cause invalid transitions on the server state. Instead of attempting the transition, ignore these messages if the node is already complete (this is how it is handled in NodeComplete).

[wlapham]

🥇

[Nayshins]

LGTM

[minostro]

@keithwiersema Do we know why this is happening? should we at least log these events?

[keithwiersema]

@minostro updated to include logging when this occurs. We believe this may be a race condition where redis is double-processing events - as a stop gap, I propose that we make the ClientComplete status idempotent to get rid of the noise we are getting in our system.

[minostro]

@keithwiersema I have a conflicting view on this problem. If the team is not 100% sure of what's going on, I would advise against making changes to the open source version of Backbeat. Instead, I would make changes in the private repo the team has for it, so experimentation can be done faster without impacting any other teams who are using the public version. Unfortunately, I don't have the time and the context to add any significant value to this discussion. I'll lean on @sydneycodes who is the last person which I know had context on Backbeat.