Message tracing using Opentelemetry.

[hairyhum]

In complex distributed data pipeline it helps to have some sort of tracing mechanism to see where and when messages went.
There are some tools to help with that, most recently emerging in OpenTelemetry project emerging from OpenTracing project

It's still a bit young and may lack instrumentation, but something to look at when thinking of debugging and tracing complex routes.

If messages could carry some minimal information about span references, we can instrument Ockam Nodes to support sending traces. This can be opt-in and relatively cheap on the data size.

[thomcc]

Yeah, we already use https://github.com/tokio-rs/tracing in some capacity, and I intend to move us to use it more. It's a rust logging/tracing framework that takes heavy influence from OpenTracing/OpenTelemetry, and includes direct compatibility with that as an part of the core/officially-supported set of tools (https://github.com/tokio-rs/tracing/tree/master/tracing-opentelemetry).

[hairyhum]

If we can use the "generic metadata" bit in all messages as described in https://github.com/ockam-network/proposals/pull/87 we can put the telemetry span reference in a metadata field there.

[hairyhum]

For future integration with AWS x-ray: OpenTelemetry is supported by AWS x-ray, but requires a specific ID generator.