[DO NOT MERGE] Proposal for Upkeep State Telemetry

[EasterTheBunny]

Relying on log output for debugging has been challenging at best and error prone at worst. This is frequently due to the fluctuating nature of full text log output as changes occur in the code base. On top of that, log collection at a full debug level for simple state telemetry is highly wasteful as most of the log output doesn't directly relate to state changes.

This proposal introduces distinct states of a specific upkeep that are indicated as being altered at specific points in the program. The approach replaces the default logger with a special telemetry logger that adds telemetry logging functions.

To view potential output of the telemetry logging, run the following in a local terminal:

$ make simulator && ./bin/simulator --simulate --verbose -f ./tools/simulator/plans/only_log_trigger.json

Then view ./simulation_plan_logs/simulation.log for "Status Delays per Work ID". The output attempts to indicated the time taken for an upkeep to reach specific states. This allows for evaluation of the time spent at each point in the process.

[amirylm]

The general idea of reporting the states of upkeeps is good, but we shouldn't create such coupling with the logger.
Why not starting with prometheus collectors, or at least follow their architecture where the collectors are independent, decoupled from the components that uses them.

We can have collectors (and states) exposed from a single package, use prometheus to collect the metrics and to query it from the simulator as part of the test.

cc @ogtownsend

[EasterTheBunny]

The general idea of reporting the states of upkeeps is good, but we shouldn't create such coupling with the logger. Why not starting with prometheus collectors, or at least follow their architecture where the collectors are independent, decoupled from the components that uses them.

We can have collectors (and states) exposed from a single package, use prometheus to collect the metrics and to query it from the simulator as part of the test.

cc @ogtownsend

It does make sense to decouple from the logger and I assume the components that need the telemetry collector would require it in their constructors along with a logger?

My only issue with your proposal of using prometheus is the potential coupling with a 3rd party. As long as we structure the collector dependency in an abstract way that essentially wraps prometheus, I would be good with that. I think that is what you are saying in your last statement, but I just want to make sure.

[amirylm]

abstract way that essentially wraps prometheus

That is usually a common pattern, but we should follow guidelines we have in the core node which seems to be adding prometheus dependency (github.com/prometheus/client_golang/prometheus[/promauto])