Mingration from OpenTracing: the baggageItem/context propagation.

[SimoneGiusso]

With OpenTracing, for context propagation, I have this very simple utility class:

public MyClass {

    public static final MY_KEY = "myKey";

    public static void setBaggageItem(String value) {
        Optional.ofNullable(GlobalTracer.get().activeSpan()).ifPresent(span -> span.setBaggageItem(MY_KEY, value));
    }

    private static Optional<String> getBaggageItem(String key) {
        return Optional.ofNullable(GlobalTracer.get().activeSpan()).map(span -> span.getBaggageItem(MY_KEY));
    }

}

what is the simplest way to translate this with OpenTelemetry? I had a look at this doc and I'm a bit surprise how is this more difficult with OpenTelemetry. Moreover there is the concept of "carrier". It means that probably I would have to pass the httpExchange on both methods in the context propagation if propagation is done with HttpHeaders. I'm missing something or context propagation is became far more complex compare to OpenTracing?

[mateuszrzeszutek]

In OTel Baggage is not mutable; to set an entry you have to do sth like this:

Baggage baggage = Baggage.current().toBuilder()
    .put(MY_KEY, value)
    .build();
try (Scope ignored = baggage.makeCurrent()) {
  // code that uses the modified baggage
}

The way it's propagated is very similar to Span, really.

[mateuszrzeszutek]

baggage.makeCurrent() will make my value propagated?

It will add the baggage that you've constructed to the current context, and make the resulting context current.

    public static void setBaggageItem(String value) {
        Baggage baggage = Baggage.current().toBuilder().put(MY_KEY, value).build();
        baggage.makeCurrent();
    }

You have to close the Scope returned by makeCurrent(), otherwise you'll cause context leaks and most likely break context propagation in your application.

[SimoneGiusso]

I'll try to test this then:

public MyClass {

    public static final MY_KEY = "myKey";

    public static void setBaggageItem(String value) {
        Baggage baggage = Baggage.current().toBuilder().put(MY_KEY, value).build();
        baggage.makeCurrent().close();
    }

    private static String getBaggageItem(String key) {
        return Baggage.current() != Baggage.empty() ? Baggage.current().getEntryValue(key) : NONE;
    }

}

[mateuszrzeszutek]

This is not gonna work either:

    public static void setBaggageItem(String value) {
        Baggage baggage = Baggage.current().toBuilder().put(MY_KEY, value).build();
        baggage.makeCurrent().close();
    }

You have to treat baggage the same way you treat a span: the OpenTelemetry implementation of it is fundamentally different from the OpenTracing one, you won't be able to set the baggage in one static void method.

[SimoneGiusso]

This is where my problems come. As I said I want to propagate the baggage over multiple services, and currently I'm using opentracing and the code above, as explained in https://opentracing.io/docs/best-practices/ (Using Baggage / Distributed Context Propagation):

# client side
span.context.set_baggage_item('auth-token', '.....')

# server side (one or more levels down from the client)
token = span.context.get_baggage_item('auth-token')

The problem I have with OpenTelemetry is that the API (as shown in this example) requires a carrier. I'm not sure what to pass here, because currently I'm not using any carrier.

Moreover, in different parts of the code, I'm just calling the above static methods to just set or retrieve the propagated baggage. It seems quite hard in this case to migrate.

Furthermore I do not see the point to have the carrier object as mandatory in the API. In this example, we explicitly set the header on the client side (URLConnection::setRequestProperty;) and extract it on the server side:

public String get(HttpExchange carrier, String key) {
          if (carrier.getRequestHeaders().containsKey(key)) {
            return carrier.getRequestHeaders().get(key).get(0);
          }
          return "";
        }

What should keep me to avoid just using the URLConnection and HttpExchange in my code, without using OpenTelemetry (which requires more code to just doing the same thing)?

In a nutshell, OpenTracing has a more abstract and easier API to achieve the same thing: propagate context between services (using HTTP headers).

Is there any chance to have the same abstraction in Opentelemetry in the near future?

[jkwatson]

The OpenTelemetry W3CBaggagePropagator is responsible for propagating baggage from the Context to/from the wire format that you're using.

[SimoneGiusso]

Hello @jkwatson @mateuszrzeszutek

In the end I had to change a bit the code but the baggage is propagated. So basically in service 1, as suggest by @mateuszrzeszutek, I have these two static utility methods:

    public static <T> T addCallerToCurrentBaggage(String caller, Supplier<T> supplier) {
        Baggage baggage = Baggage.current().toBuilder()
            .put(ROOT_CALLER, caller)
            .build();
        try (Scope ignored = baggage.makeCurrent()) {
            return supplier.get();
        }
    }

    public static String getRootCaller() {
        return getBaggageItem(ROOT_CALLER).orElse(NONE);
    }

If for example the supplier make an http call to service 2 the baggage will be (magically) propagated. In Service 2 I have to call getRootCaller(). That's it. This is what I want.

I have still one problem though. I'm using also reactive (Reactor project). When a use addCallerToCurrentBaggage, if the code invoked by the supplier, is executed from different threads, even inside the same Service I lose the information set in the baggage.

So if a set the baggage in thread 1, in thread 2 I do not have it. How can I propagate the baggage in this case?

[mateuszrzeszutek]

Are you using the reactor instrumentation? Either as part of the javaagent, or as a standalone library instrumentation.
The reactor instrumentation propagates context captured on subscription.

[SimoneGiusso]

Yes I am using reactor. I added the java agent to automatically instrument the application but I also do some manual instrumentation. Everything works. The only problem is when I manually instrumenting with the above code. Baggage is not propagate among threads.

[mateuszrzeszutek]

How are you using the addCallerToCurrentBaggage() function?
Is it

Mono mono = prepareMono();
addCallerToCurrentBaggage("asdf", () -> mono.block());

or

Mono mono = addCallerToCurrentBaggage("asdf", () -> prepareMono());
mono.block();

? Can you post a repro scenario of your issue?

[SimoneGiusso]

    public Mono<Map<String, Resource>> fetchResourse(@RequestBody payload, ServerHttpRequest request) {
        return ReactiveSecurityContextHolder.getContext()
            .flatMap(securityContext -> addCallerToCurrentBaggage(isService(securityContext) ? getService(securityContext) : thisServiceName,
                () -> resourceService.fetchResource(ids))
            );
    }

I can try to write a small webflux app to reproduce the issue.