Integrating with the metrics crate

[bonsairobo]

In the interest of making it easier to debug Bevy apps, I think we should consider adding metrics directly inside of the bevy_* crates that can be consumed by an arbitrary monitoring backend. This is precisely the design of the metrics crate (and ecosystem).

While much can be done with metrics outside of Bevy proper (see bevy_metrics_dashboard), we can have a much richer set of metrics by adding instrumentation within Bevy. For example, there is probably no way to define a counter for how many commands are being run, or how many systems are being run, outside of the Bevy source code. The cost of adding and maintaining metrics is very low; because there is no plumbing involved. All metrics are written into a global recorder that is invisible to the code being instrumented (similar to how log and tracing work). Recorders can be defined arbitrarily and installed on-demand. It should also be possible to "compile out" metrics in builds that don't need them by using cargo features; though it remains to be seen that metrics are even heavy enough to necessitate this.

I believe this approach would be strictly better than the existing bevy_diagnostic option. For one, bevy_diagnostic only supports writing metrics from within the Bevy ECS. metrics allows writing metrics from anywhere; which means that you get access to all metrics in your dependency tree, regardless of if they have access to the ECS. You just have to take the lightweight metrics dependency (under 2K LOC); it's mostly just a set of facade macros for updating metrics by key/label and describing them to the recorder.

Let me know what you think! I want to start using metrics more in Bevy apps that I work on, and if this proposal seems agreeable, I would be willing to make contributions for defining metrics inside Bevy.

[alice-i-cecile]

This seems like a sensible idea. Questions:

• Would this be a full replacement for bevy_diagnostics?
• Would all be feature-flagged in the same way that `tracing currently is?
• How does metrics contrast to and integrate with tracing?
• What steps would we need to take to start supporting this?

[bonsairobo]

Would this be a full replacement for bevy_diagnostics?

AFAICT there would be no reason to use bevy_diagnostics over metrics, but there would be some work to replace any visualizations or logging that the diagnostic plugin does.

It looks like one notable API difference is that DiagnosticMeasurement has a timestamp, while metrics would do this differently by timestamping an entire "snapshot" of multiple metrics at once.

That said, I have very little experience with bevy_diagnostic beyond reading the API docs.

Would all be feature-flagged in the same way that `tracing currently is?

Yea I don't see any reason to treat it differently from tracing. Though I'm guessing metrics will incur less runtime overhead.

How does metrics contrast to [...] tracing?

While there is some overlap in use cases, they do fundamentally different things. tracing is event-based while metrics tracks numbers or buckets of numbers (plus their metadata). So when you use an event or span in tracing, you are appending to a log somewhere. When you use a metric, you are just overwriting a single number. You only get a stream of metrics by virtue of implementing a sampler on the backend. This makes metrics theoretically "lighter weight" than traces/logs.

There is a subtle exception in the case of histogram metrics. A histogram is implemented as a "value bucket," so it really does append a number into an atomic list.

The other main difference is that metrics has metadata per metric and tracing has metadata per event. The data cardinality of tracing is theoretically higher than metrics. For this reason, I'm expecting there to be less performance impact, but we should measure it if we start adding metrics to any tight loops.

How does metrics [...] integrate with tracing?

I might not have enough knowledge to fully answer this question. I think in theory it would be possible to correlate a time series of metrics with a trace stream, but I don't know if there is any tooling to support this.

The APIs for writing backends for metrics and tracing are totally distinct, so there is no innate integration between them, but you could probably implement a central recorder/subscriber than ingests data from both sources. I'm not sure if this is the right approach though. It might be too early to start designing along this train of thought, and instead we can just see what gets built after metrics have been added to Bevy. I know I will be building on top of it anyway. There might also be some desire to plot metrics directly in the Bevy editor.

Personally, I see the main value of metrics as having real-time feedback directly in your application. It could also be used for telemetry.

I found this crate that apparently makes it easier to keep the metadata of tracing spans and metrics in sync (if that's even something we want).

What steps would we need to take to start supporting this?

I would start by adding the metrics dependency and then adding metrics for whatever already exists in the bevy_diagnostic plugins. We don't need Bevy to implement its own metrics registry yet; that can happen in "userspace" for now. But eventually it might be worth having a first-party metrics recorder just to integrate with the Bevy editor or something. Or maybe that's what bevy_diagnostic could evolve into.

In general, we should probably be somewhat judicious about the set of metrics being added, since it's easy to add too much and have redundant or even misleading data. It might be worth writing down some guidelines about how to add meaningful metrics.

[superdump]

One thing we want to do with diagnostics is to allow them to be nested/have paths. So for example take measurements of how long various render passes take to execute. Presumably we could do this by convention in the key of the metric. So like “view/pass name” or so?

[bonsairobo]

Yea you can choose a path delimiter convention. metrics doesn't have any kind of namespacing feature by default (yet). See this discussion.

I doubt you would need deep nesting though.

I'm not sure what the best choice of delimiter is. :: is familiar, but maybe it's misleading, since it could incorrectly imply module paths. / seems like a decent choice.

measurements of how long various render passes take to execute

If you are hoping to aggregate latency by scope, you would need to do that manually. Shouldn't be too hard. I'm not sure if tracing has something like this or not. I guess if you don't care to record the log of events for many frames, it would be cheaper to just keep a time series of latency metrics.

I have a working example of browsing a namespace tree of metrics in bevy_metrics_dashboard. It just sorts metric names, then uses a pretty simple recursive parser to create the tree.

[superdump]

I see that metrics has a Prometheus exporter. That’s cool, and useful for longer-term measurement given that Prometheus would request metric values from the exposed Prometheus endpoint quite sparsely, like every 30s for example.

One thing that Prometheus recommends is that measurements not be aggregated up front and that totals are logged for counts of things. So instead of measuring a rate of something, you measure the total of it instead. This allows Prometheus to do correct aggregation and if for some reason fetching measurements is missed, you can still calculate a rate across a gap in measurements because you know how long it was between measurements and you know what the totals were either side. So you only lose resolution of measurements rather than having a gap.

I’m not sure how we would reconcile the low frequency and high frequency observation of measurements through just the metrics crate. Measuring the frame time of a frame once every 30s doesn’t give much insight as there were lots of other frames during that period. But maybe summing frame times during the proof doesn’t make sense either. However, maybe tracking the quantity of allocations and the amount of data allocated during a 30s period is interesting.

Not suggesting these thoughts are blockers necessarily, rather exploring whether/what we might want to do for fine, high frequency metrics to be visualised rapidly/in real time and coarse, low frequency metrics that we’d want to observe for longer runs of a bevy application - imagine someone playing through an entire scene that takes 30 minutes or an hour and you want to record what they did and measure metrics to later analyse and identify where problem areas were or something.

[bonsairobo]

Honestly I don't think low-frequency metrics are the biggest selling point for Bevy. There are maybe a few metrics that are interesting to track at low frequency in telemetry, like frame time, memory, CPU, and bandwidth usage. For example, it might be interesting to use telemetry to see trends in frame time according to which level or scene users are playing in.

I'm much more interested in using it for per-frame sampling. I find value in having a dashboard in-process for interactive debugging, but I could also see wanting to record metrics along with a trace (or maybe as a lightweight alternative to a trace) when reproducing an issue. Then you can view the metrics "offline" in full detail to tweak the visualization as much as necessary.

[naomijub]

I would suggest https://github.com/madesroches/micromegas for this issue. It is a telemetry stack (in early stages) focus on game engines. So it is definitely targeted for what we may need.