TCP routing via ingress-per-unit fails if application is scaled

[hemanthnakkina]

Bug Description

Deployed an application with single unit and exposed TCP port via Traefik using ingress-per-unit interface.
All looks good and can send traffic over Loadbalancer IP and tcp port.

Scaled the application and then see issues in sending traffic over Loadbalancer IP and tcp port. Can see traefik instance is in Inactive state.

To Reproduce

1. Build the charm https://github.com/canonical/traefik-k8s-operator/tree/main/tests/integration/testers/tcp
2. Modify the attached bundle file test_tcp_router.txt to change tcp-tester charm path
3. Pre-requisite juju+microk8s with controller on microk8s (with storage and metallb addons enabled)
4. juju add-model test
5. juju deploy ./test_tcp_router.txt
6. nc -vz 9999
7. juju scale-application tcp-tester 2
8. nc -vz 9999 # Fails

Bundle file:
test_tcp_router.txt

Environment

juju bootstrapped on microk8s

juju 3.5/stable
microk8s 1.28-strict/stable

Relevant log output

unit-traefik-0: 03:54:09 INFO juju.cmd running containerAgent [3.5.3 63d460f9ee6c7c710131961390687e7a0ab90470 gc go1.21.12]
unit-traefik-0: 03:54:09 INFO juju.cmd.containeragent.unit start "unit"
unit-traefik-0: 03:54:09 INFO juju.worker.upgradesteps upgrade steps for 3.5.3 have already been run.
unit-traefik-0: 03:54:09 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301
unit-traefik-0: 03:54:09 INFO juju.api cannot resolve "controller-service.controller-microk8s.svc.cluster.local": lookup controller-service.controller-microk8s.svc.cluster.local: operation was canceled
unit-traefik-0: 03:54:09 INFO juju.api connection established to "wss://10.152.183.186:17070/model/c76eb599-00ae-4ec2-87dd-63015328229c/api"
unit-traefik-0: 03:54:09 INFO juju.worker.apicaller [c76eb5] "unit-traefik-0" successfully connected to "10.152.183.186:17070"
unit-traefik-0: 03:54:09 INFO juju.api cannot resolve "controller-service.controller-microk8s.svc.cluster.local": lookup controller-service.controller-microk8s.svc.cluster.local: operation was canceled
unit-traefik-0: 03:54:09 INFO juju.api connection established to "wss://10.152.183.186:17070/model/c76eb599-00ae-4ec2-87dd-63015328229c/api"
unit-traefik-0: 03:54:09 INFO juju.worker.apicaller [c76eb5] "unit-traefik-0" successfully connected to "10.152.183.186:17070"
unit-traefik-0: 03:54:09 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-traefik-0: 03:54:09 INFO juju.worker.logger logger worker started
unit-traefik-0: 03:54:09 INFO juju.worker.leadership traefik/0 promoted to leadership of traefik
unit-traefik-0: 03:54:09 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-traefik-0: 03:54:09 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-traefik-0: 03:54:09 INFO juju.worker.caasupgrader unblocking abort check
unit-traefik-0: 03:54:09 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-traefik-0
unit-traefik-0: 03:54:09 INFO juju.worker.uniter unit "traefik/0" started
unit-traefik-0: 03:54:09 INFO juju.worker.uniter resuming charm install
unit-traefik-0: 03:54:09 INFO juju.worker.uniter.charm downloading ch:amd64/focal/traefik-k8s-194 from API server
unit-traefik-0: 03:54:09 INFO juju.downloader downloading from ch:amd64/focal/traefik-k8s-194
unit-traefik-0: 03:54:09 INFO juju.downloader download complete ("ch:amd64/focal/traefik-k8s-194")
unit-traefik-0: 03:54:09 INFO juju.downloader download verified ("ch:amd64/focal/traefik-k8s-194")
unit-traefik-0: 03:54:18 INFO juju.worker.uniter hooks are retried true
unit-traefik-0: 03:54:18 INFO juju.worker.uniter found queued "install" hook
unit-traefik-0: 03:54:19 INFO unit.traefik/0.juju-log Running legacy hooks/install.
unit-traefik-0: 03:54:20 INFO unit.traefik/0.juju-log Kubernetes service 'traefik' patched successfully
unit-traefik-0: 03:54:20 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-traefik-0: 03:54:21 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app' expected but not received.
unit-traefik-0: 03:54:21 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app_name' expected in snapshot but not found.
unit-traefik-0: 03:54:21 INFO unit.traefik/0.juju-log ingress-per-unit:1: Kubernetes service 'traefik' patched successfully
unit-traefik-0: 03:54:21 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app_name' expected in snapshot but not found.
unit-traefik-0: 03:54:22 INFO juju.worker.uniter.operation ran "ingress-per-unit-relation-created" hook (via hook dispatching script: dispatch)
unit-traefik-0: 03:54:23 INFO juju.worker.uniter.operation ran "peers-relation-created" hook (via hook dispatching script: dispatch)
unit-traefik-0: 03:54:23 INFO juju.worker.uniter found queued "leader-elected" hook
unit-traefik-0: 03:54:24 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-traefik-0: 03:54:25 INFO unit.traefik/0.juju-log Kubernetes service 'traefik' patched successfully
unit-traefik-0: 03:54:25 INFO unit.traefik/0.juju-log tracing not enabled: skipping tracing config
unit-traefik-0: 03:54:26 ERROR unit.traefik/0.juju-log failed processing the ingress relation <ops.model.Relation ingress-per-unit:1>: 'provider is not ready: ingress for <ops.model.Relation ingress-per-unit:1> wiped.'
unit-traefik-0: 03:54:26 ERROR unit.traefik/0.juju-log The setup of some ingress relation failed, see previous logs
unit-traefik-0: 03:54:30 INFO juju.worker.uniter.operation ran "traefik-pebble-ready" hook (via hook dispatching script: dispatch)
unit-traefik-0: 03:54:31 INFO juju.worker.uniter.operation ran "configurations-storage-attached" hook (via hook dispatching script: dispatch)
unit-traefik-0: 03:54:31 WARNING unit.traefik/0.juju-log No 'certificates' relation found. Cannot generate csr.
unit-traefik-0: 03:54:32 ERROR unit.traefik/0.juju-log failed processing the ingress relation <ops.model.Relation ingress-per-unit:1>: 'provider is not ready: ingress for <ops.model.Relation ingress-per-unit:1> wiped.'
unit-traefik-0: 03:54:32 ERROR unit.traefik/0.juju-log The setup of some ingress relation failed, see previous logs
unit-traefik-0: 03:54:32 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-traefik-0: 03:54:32 INFO juju.worker.uniter found queued "start" hook
unit-traefik-0: 03:54:33 INFO unit.traefik/0.juju-log Running legacy hooks/start.
unit-traefik-0: 03:54:34 ERROR unit.traefik/0.juju-log failed processing the ingress relation <ops.model.Relation ingress-per-unit:1>: 'provider is not ready: ingress for <ops.model.Relation ingress-per-unit:1> wiped.'
unit-traefik-0: 03:54:34 ERROR unit.traefik/0.juju-log The setup of some ingress relation failed, see previous logs
unit-traefik-0: 03:54:34 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
unit-traefik-0: 03:54:35 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app' expected but not received.
unit-traefik-0: 03:54:35 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app_name' expected in snapshot but not found.
unit-traefik-0: 03:54:35 INFO unit.traefik/0.juju-log ingress-per-unit:1: Kubernetes service 'traefik' patched successfully
unit-traefik-0: 03:54:35 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app_name' expected in snapshot but not found.
unit-traefik-0: 03:54:35 INFO unit.traefik/0.juju-log ingress-per-unit:1: tracing not enabled: skipping tracing config
unit-traefik-0: 03:54:41 INFO juju.worker.uniter.operation ran "ingress-per-unit-relation-joined" hook (via hook dispatching script: dispatch)
unit-traefik-0: 03:54:41 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app' expected but not received.
unit-traefik-0: 03:54:41 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app_name' expected in snapshot but not found.
unit-traefik-0: 03:54:42 INFO unit.traefik/0.juju-log ingress-per-unit:1: Kubernetes service 'traefik' patched successfully
unit-traefik-0: 03:54:42 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app_name' expected in snapshot but not found.
unit-traefik-0: 03:54:42 INFO juju.worker.uniter.operation ran "ingress-per-unit-relation-changed" hook (via hook dispatching script: dispatch)
unit-traefik-0: 03:58:57 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app' expected but not received.
unit-traefik-0: 03:58:57 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app_name' expected in snapshot but not found.
unit-traefik-0: 03:58:57 ERROR unit.traefik/0.juju-log ingress-per-unit:1: Kubernetes service patch failed: Service "traefik-lb" is invalid: spec.ports[3]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:9999, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0}
unit-traefik-0: 03:58:57 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app_name' expected in snapshot but not found.
unit-traefik-0: 03:58:57 INFO unit.traefik/0.juju-log ingress-per-unit:1: tracing not enabled: skipping tracing config
unit-traefik-0: 03:59:01 ERROR unit.traefik/0.juju-log ingress-per-unit:1: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 1199, in <module>
    main(TraefikIngressCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/main.py", line 548, in main
    manager.run()
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/main.py", line 527, in run
    self._emit()
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/main.py", line 516, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/main.py", line 147, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-traefik-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 548, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-traefik-0/charm/lib/charms/traefik_k8s/v1/ingress_per_unit.py", line 338, in _handle_relation
    self.on.data_provided.emit(relation)  # type: ignore
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-traefik-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 548, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 679, in _handle_ingress_data_provided
    self._process_status_and_configurations()
  File "/var/lib/juju/agents/unit-traefik-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 548, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 601, in _process_status_and_configurations
    self._update_ingress_configurations()
  File "/var/lib/juju/agents/unit-traefik-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 548, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 616, in _update_ingress_configurations
    self._clear_all_configs_and_restart_traefik()
  File "/var/lib/juju/agents/unit-traefik-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 548, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 511, in _clear_all_configs_and_restart_traefik
    self._restart_traefik()
  File "/var/lib/juju/agents/unit-traefik-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 548, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 1064, in _restart_traefik
    self.traefik.restart()
  File "/var/lib/juju/agents/unit-traefik-0/charm/src/traefik.py", line 630, in restart
    self._container.restart(self.service_name)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/model.py", line 2226, in restart
    self._pebble.restart_services(service_names)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/pebble.py", line 2065, in restart_services
    return self._services_action('restart', services, timeout, delay)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/pebble.py", line 2090, in _services_action
    raise ChangeError(change.err, change)
ops.pebble.ChangeError: cannot perform the following tasks:
- Start service "traefik" (cannot start service: exited quickly with code 1)
----- Logs from task 1 -----
2024-09-23T03:59:01Z INFO Most recent service output:
    (...)
    time="2024-09-23T03:58:57Z" level=error msg="accept tcp [::]:9999: use of closed network connection" entryPointName=sunbeam-tcp-tester-0
    time="2024-09-23T03:58:57Z" level=error msg="Error while starting server: accept tcp [::]:9999: use of closed network connection" entryPointName=sunbeam-tcp-tester-0
    time="2024-09-23T03:58:57Z" level=error msg="accept tcp [::]:80: use of closed network connection" entryPointName=web
    time="2024-09-23T03:58:57Z" level=error msg="close tcp [::]:80: use of closed network connection" entryPointName=web
    time="2024-09-23T03:58:57Z" level=error msg="accept tcp [::]:8082: use of closed network connection" entryPointName=diagnostics
    time="2024-09-23T03:58:57Z" level=debug msg="Entry point web closed" entryPointName=web
    time="2024-09-23T03:58:57Z" level=debug msg="Entry point sunbeam-tcp-tester-0 closed" entryPointName=sunbeam-tcp-tester-0
    time="2024-09-23T03:58:57Z" level=debug msg="Waiting 10s seconds before killing connections." entryPointName=websecure
    time="2024-09-23T03:58:57Z" level=error msg="close tcp [::]:8082: use of closed network connection" entryPointName=diagnostics
    time="2024-09-23T03:58:57Z" level=debug msg="Entry point diagnostics closed" entryPointName=diagnostics
    time="2024-09-23T03:58:57Z" level=error msg="accept tcp [::]:443: use of closed network connection" entryPointName=websecure
    time="2024-09-23T03:58:57Z" level=error msg="close tcp [::]:443: use of closed network connection" entryPointName=websecure
    time="2024-09-23T03:58:57Z" level=debug msg="Entry point websecure closed" entryPointName=websecure
    time="2024-09-23T03:58:57Z" level=info msg="Server stopped"
    time="2024-09-23T03:58:57Z" level=info msg="Shutting down"
    time="2024-09-23T03:59:01Z" level=info msg="Configuration loaded from file: /etc/traefik/traefik.yaml"
    time="2024-09-23T03:59:01Z" level=info msg="Traefik version v2.11.0 built on 2024-03-14_05:12:45PM"
    time="2024-09-23T03:59:01Z" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"diagnostics\":{\"address\":\":8082\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"sunbeam-tcp-tester-0\":{\"address\":\":9999\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"sunbeam-tcp-tester-1\":{\"address\":\":9999\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"file\":{\"directory\":\"/opt/traefik/juju\",\"watch\":true}},\"metrics\":{\"prometheus\":{\"buckets\":[0.1,0.3,1.2,5],\"addEntryPointsLabels\":true,\"addRoutersLabels\":true,\"addServicesLabels\":true,\"entryPoint\":\"diagnostics\"}},\"ping\":{\"entryPoint\":\"diagnostics\",\"terminatingStatusCode\":503},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"}}"
    time="2024-09-23T03:59:01Z" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
    2024/09/23 03:59:01 traefik.go:80: command /bin/traefik error: error while building entryPoint sunbeam-tcp-tester-1: error preparing server: error opening listener: listen tcp :9999: bind: address already in use
2024-09-23T03:59:01Z ERROR cannot start service: exited quickly with code 1
-----
unit-traefik-0: 03:59:01 ERROR juju.worker.uniter.operation hook "ingress-per-unit-relation-joined" (via hook dispatching script: dispatch) failed: exit status 1
unit-traefik-0: 03:59:01 INFO juju.worker.uniter awaiting error resolution for "relation-joined" hook
unit-traefik-0: 03:59:06 INFO juju.worker.uniter awaiting error resolution for "relation-joined" hook
unit-traefik-0: 03:59:07 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app' expected but not received.
unit-traefik-0: 03:59:07 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app_name' expected in snapshot but not found.
unit-traefik-0: 03:59:07 ERROR unit.traefik/0.juju-log ingress-per-unit:1: Kubernetes service patch failed: Service "traefik-lb" is invalid: spec.ports[3]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:9999, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0}
unit-traefik-0: 03:59:07 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app_name' expected in snapshot but not found.
unit-traefik-0: 03:59:08 INFO juju.worker.uniter.operation ran "ingress-per-unit-relation-joined" hook (via hook dispatching script: dispatch)
unit-traefik-0: 03:59:09 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app' expected but not received.
unit-traefik-0: 03:59:09 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app_name' expected in snapshot but not found.
unit-traefik-0: 03:59:09 ERROR unit.traefik/0.juju-log ingress-per-unit:1: Kubernetes service patch failed: Service "traefik-lb" is invalid: spec.ports[3]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:9999, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0}
unit-traefik-0: 03:59:09 WARNING unit.traefik/0.juju-log ingress-per-unit:1: 'app_name' expected in snapshot but not found.
unit-traefik-0: 03:59:10 INFO juju.worker.uniter.operation ran "ingress-per-unit-relation-changed" hook (via hook dispatching script: dispatch)
unit-traefik-0: 04:00:05 ERROR unit.traefik/0.juju-log Kubernetes service patch failed: Service "traefik-lb" is invalid: spec.ports[3]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:9999, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0}
unit-traefik-0: 04:00:05 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-traefik-0: 04:04:29 ERROR unit.traefik/0.juju-log Kubernetes service patch failed: Service "traefik-lb" is invalid: spec.ports[3]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:9999, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0}
unit-traefik-0: 04:04:30 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-traefik-0: 04:09:47 ERROR unit.traefik/0.juju-log Kubernetes service patch failed: Service "traefik-lb" is invalid: spec.ports[3]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:9999, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0}
unit-traefik-0: 04:09:48 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-traefik-0: 04:15:24 ERROR unit.traefik/0.juju-log Kubernetes service patch failed: Service "traefik-lb" is invalid: spec.ports[3]: Duplicate value: core.ServicePort{Name:"", Protocol:"TCP", AppProtocol:(*string)(nil), Port:9999, TargetPort:intstr.IntOrString{Type:0, IntVal:0, StrVal:""}, NodePort:0}
unit-traefik-0: 04:15:25 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)

Additional context

/etc/traefik/traefik.yml exposes multiple entrypoints with same tcp port number (one for each unit)

entryPoints:
  diagnostics:
    address: :8082
  sunbeam-tcp-tester-0:
    address: :9999
  sunbeam-tcp-tester-1:
    address: :9999

Not sure if https://doc.traefik.io/traefik/routing/entrypoints/#reuseport option helps