Add prometheus metrics

[danielballan]

Draw inspiration from https://docs.dask.org/en/latest/setup/prometheus.html. Exposing gc metrics in particular seems a good idea.

[dylanmcreynolds]

I love this.

[dylanmcreynolds]

What about also exposing time spent processing requests?

[danielballan]

Might also be use to count socket and file system reads.

[danielballan]

I wonder if it’s appropriate to use Python audit hooks for that kind of performance tracing.

[danielballan]

We needed to start capturing metrics urgently to make sense of some user reports, so I rushed something into main. Needs docs and refinement.

[danielballan]

The socket and file usage can be observed through a generic node_exporter. I don’t think we need the Tiled application to do anything special there.

Still needs docs.

[danielballan]

Old gist that is relevant: https://gist.github.com/danielballan/1bfb0576781c5671e3b97af2ebafcfa1

[danielballan]

Evidently podman-compose does not support --scale. It seems to silently ignore it, as reported here: containers/podman-compose#267