tracing-relation-changed hook failing on instrumenting workload

[mmkay]

Issue

In some cases, tracing-relation-changed hook fails on relating tempo with traefik in tracing protocol. It doesn't seem to happen every time so this might be a race condition. Most likely this is related to calls for grpc endpoint in instrumenting the workload here.

Reproduction steps

juju deploy tempo-k8s tempo --channel edge
juju deploy cos-lite tempo --channel edge --trust
(wait until the model settles)
jhack imatrix fill

In some attempts, I see:

Model  Controller  Cloud/Region        Version  SLA          Timestamp
test   microk8s    microk8s/localhost  3.1.7    unsupported  15:10:06+01:00

App           Version  Status   Scale  Charm             Channel  Rev  Address         Exposed  Message
alertmanager  0.26.0   active       1  alertmanager-k8s  edge     103  10.152.183.187  no       
catalogue              active       1  catalogue-k8s     edge      33  10.152.183.225  no       
grafana       9.5.3    active       1  grafana-k8s       edge     105  10.152.183.167  no       
loki          2.9.4    active       1  loki-k8s          edge     121  10.152.183.197  no       
prometheus    2.49.1   active       1  prometheus-k8s    edge     170  10.152.183.198  no       
tempo         ...      active       1  tempo-k8s         edge      27  10.152.183.224  no       
traefik       2.10.5   waiting      1  traefik-k8s       edge     170  10.211.88.121   no       installing agent

Unit             Workload  Agent  Address       Ports  Message
alertmanager/0*  active    idle   10.1.157.124         
catalogue/0*     active    idle   10.1.157.125         
grafana/0*       active    idle   10.1.157.70          
loki/0*          active    idle   10.1.157.107         
prometheus/0*    active    idle   10.1.157.93          
tempo/0*         active    idle   10.1.157.66          
traefik/0*       error     idle   10.1.157.99          hook failed: "tracing-relation-changed"

juju debug-log output:

unit-traefik-0: 15:09:04 WARNING unit.traefik/0.juju-log tracing:37: 'app' expected but not received.
unit-traefik-0: 15:09:04 WARNING unit.traefik/0.juju-log tracing:37: 'app_name' expected in snapshot but not found.
unit-traefik-0: 15:09:04 WARNING unit.traefik/0.juju-log tracing:37: `TracingEndpointRequirer.otlp_http_endpoint` is deprecated. Use `TracingEndpointRequirer.get_endpoint('otlp_http') instead.`
unit-traefik-0: 15:09:04 ERROR unit.traefik/0.juju-log tracing:37: no receiver found with protocol='otlp_http'
unit-traefik-0: 15:09:04 WARNING unit.traefik/0.juju-log tracing:37: `TracingEndpointRequirer.otlp_grpc_endpoint` is deprecated. Use `TracingEndpointRequirer.get_endpoint('otlp_grpc') instead.`
unit-traefik-0: 15:09:04 ERROR unit.traefik/0.juju-log tracing:37: no receiver found with protocol='otlp_grpc'
unit-traefik-0: 15:09:04 ERROR unit.traefik/0.juju-log tracing:37: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 1034, in <module>
    main(TraefikIngressCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/main.py", line 456, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/main.py", line 144, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/framework.py", line 853, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/framework.py", line 943, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-traefik-0/charm/lib/charms/tempo_k8s/v2/tracing.py", line 780, in _on_tracing_relation_changed
    self.on.endpoint_changed.emit(relation, data.host, [i.dict() for i in data.receivers])  # type: ignore
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/framework.py", line 853, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-traefik-0/charm/venv/ops/framework.py", line 943, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-traefik-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 532, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 370, in _on_tracing_endpoint_changed
    self._configure_tracing()
  File "/var/lib/juju/agents/unit-traefik-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 532, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "./src/charm.py", line 436, in _configure_tracing
    elif endpoint := self._tracing.otlp_grpc_endpoint():
  File "/var/lib/juju/agents/unit-traefik-0/charm/lib/charms/tempo_k8s/v2/tracing.py", line 838, in otlp_grpc_endpoint
    return self.get_endpoint("otlp_grpc")
  File "/var/lib/juju/agents/unit-traefik-0/charm/lib/charms/tempo_k8s/v2/tracing.py", line 826, in get_endpoint
    raise ProtocolNotRequestedError(protocol, relation)
charms.tempo_k8s.v2.tracing.ProtocolNotRequestedError: ('otlp_grpc', <ops.model.Relation tracing:37>)
unit-traefik-0: 15:09:04 ERROR juju.worker.uniter.operation hook "tracing-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1

[PietroPasotti]

closing as duplicate of #304 (which technically was opened later but is already linked to a PR closing it)