LeaderElector error on leader change

[kbredemeier]

Environment

• Bonny version 1.4.0
• Elixir & Erlang/OTP versions (elixir --version): 1.14/OTP25.3
• Operating system: Docker Container derived from elixir:1.14-alpine
• K8s: AKS v1.28.3

Current behavior

I'm occasionally seeing the following error:

** (CaseClauseError) no case clause matching: {:error, %K8s.Client.APIError{message: "etcdserver: leader changed", reason: "Failure"}}

Backtrace:
lib/bonny/operator/leader_elector.ex:200 Bonny.Operator.LeaderElector.acquire_or_renew/2	
lib/bonny/operator/leader_elector.ex:86 Bonny.Operator.LeaderElector.handle_info/2	
gen_server.erl:1077 :gen_server.try_handle_info/3	
gen_server.erl:1165 :gen_server.handle_msg/6	
proc_lib.erl:241 :proc_lib.init_p_do_apply/3

I'm not sure what's causing this issue for me but google indicates that this might be caused by some snapshot running every 2h on AKS clusters resulting in this leader change.

Expected behavior

Not sure, maybe this should just log a warning 🤷

[mruoss]

Hmm... this is new to me. Does the operator recover afterwards?

[kbredemeier]

Hmm... this is new to me. Does the operator recover afterwards?

I'm not sure yet. I was not able to inspect this before someone restarted the pod, just found it in our error reporting tool.
But I do run into an issue where the operator stops receiving add events, but I'm not sure this is related. Still investigating... this happens very rarely.