Restarting nats stops stream processing forever

[matrixbot]

This issue was originally created by @recht at matrix-org/dendrite#3177.

Background information

• Dendrite version or git SHA: 0.13.1
• SQLite3 or Postgres?: Postgres
• Running in Docker?: Yes
• go version: 1.19.7
• Client used (if applicable):

Description

• What is the problem: Dendrite stops processing events if nats is restarted.
• Who is affected: Everybody
• How is this bug manifesting: Restart the entire nats cluster in a non-rolling fashion. This will trigger a nats: Server Shutdown error which exits the JetStreamConsumers, but leaves the rest of the system running. Only remediation seems to be to restart the server.
• When did this first appear:

Steps to reproduce

• Start Dendrite
• Kill all nats servers simultaneously
• Messages are no longer processed

Ideally Dendrite would reconnect once nats is available again. Alternatively the service should exit, or at the very list trigger a signal somewhere that streams are no longer being processed.

[S7evinK]

Should be fixed with #3485