While going though some recent work where additional prometheus metrics were being added, I realised that I had some firm expectations that I hadn’t communicated. This post is to outline these.
Only workers should register metrics.
The machine manifolds config has a PrometheusRegisterer and this should be passed into the manifold for any worker that is wanting to add metrics.
The manifold should pass this into the worker config, and the worker should register the metrics in the main loop, and unregister before the main loop is done.
func (w *someworker) loop() error {
// Assuming that w.metrics supports the prometheus.Collector interface.
w.config.PrometheusRegisterer.Register(w.metrics)
defer w.config.PrometheusRegisterer.Unregister(w.metrics)
// real work...
}
It is fine for other parts of the system to define metrics, and expose a prometheus.Collector interface, but the actual registration of them should be done in the workers.