Add support for annotations on loadbalancer service

[mddeff]

Expected Behaviour

Unable to apply custom annotations to specifically the nginx loadBalancer service.

Current Behaviour

Because of how certain load balancers handle annotations, they must be present at resource creation time, otherwise they'll stay in a failed state.

Possible Solution

Expose annotations for the service in values.yml and apply if defined. Suggested change to type: service definition in stack/templates/nginx.yaml (completely untested):

apiVersion: v1
kind: Service
metadata:
  labels:
    app: {{ .Values.stack.name }}
  name: {{ .Values.stack.name }}
  namespace: {{ .Release.Namespace | quote }}
  annotations:
+ {{- with .Values.stack.service.annotations }}
+ {{- toYaml . | nindent 4 }}
+ {{- end }}

(with supporting additions to chart values.yml as well)

Context

Some k8s load balancers (like metallb) allow sharing of IP addresses between services. This is beneficial when you're trying to reduce the IP space requirements for a capability that may consist of multiple helm charts (with their own incumbent loadBalancer services.

Your Environment

K3s with KlipperLB disabled and MetalLB deployed.

Discussion welcome!