Customization Callback?

[aselder]

I'm looking at the OTel libraries after having used Spandex for a while.

One really nice feature that Spandex had was the ability to pass a callback function in the config which would allow the user to customize the meta data. We used that for things like adding whether a testing cookie was set or not, or adding location metadata that Akamai passed in as headers.

The other callback was a filter_traces call back. For example, we had a function that return false (do not trace) for OPTIONS request and also request to our k8s healthcheck endpoint, because we didn't want those cluttering up the logs.

Are callback like this something that is of wide interest?

[tsloughter]

The latter, filter_traces should be covered by each instrumentation library itself. So if you're using Phoenix the OPTIONS requests and healthcheck endpoint would be configured to not have spans created by an option to OpenTelemetryPhoenix.

I don't think this exists yet, but it is the pattern used by other libraries, and other languages otel http libraries.

I'm not sure I understand the first part. Is this callback passed to the start span function or a global callback that is called on every span start?

[aselder]

For the filter trace, yup, should be handled by OpenTelemetryPhoenix. Since that's part of the repo, thought this was the right spot for the discussion. If not, let me know where I should go.

At least the SpandexPhoenix world, it was a function provided as part of the config, that was called as part of the completion callback which updated the top level span.

https://hexdocs.pm/spandex_phoenix/SpandexPhoenix.html
https://hexdocs.pm/spandex_phoenix/SpandexPhoenix.Telemetry.html#install/1

Here's an example from our code.

def customize_metadata(conn) do
    service = Map.get(conn.private, :service, :elixir)

    tenant =
      case conn.assigns[:tenant] do
        nil -> "none"
        t -> t.domain
      end

    tags =
      [
        "http.request_ip": conn.remote_ip |> :inet.ntoa() |> to_string(),
        "http.host": conn.host,
        tenant: tenant,
        test_cookie: test_cookie?(conn.cookies) |> to_string()
      ] ++ audience_tag(conn) ++ geo_tags(conn.private[:akamai_geoloc])

    meta =
      conn
      |> SpandexPhoenix.default_metadata()
      |> Keyword.put(:service, service)
      |> Keyword.put(:resource, compute_resource(conn))
      |> Keyword.put(:tags, tags)

    case conn.status in 500..599 do
      true -> Keyword.put(meta, :error, error?: true)
      false -> meta
    end
    ```

[tsloughter]

Ah! The customize metadata is also about Phoenix. Sorry, it wasn't clear you were talking about Phoenix specifically.

[tsloughter]

This also makes me think we should have a "Spandex parity" issue to track what needs to be done to ensure Spandex users are able to smoothly transition.

[btkostner]

Adding our use case:

Currently phoenix traces will show exceptions from Ecto.get!, even though they are caught by our fallback controller. It would be nice to ignore these exceptions (not set as an error or record the exception) because they are handled.

[bryannaegele]

Exceptions cannot be removed from a span or intercepted within a span, so I'm afraid there's nothing we can do about that. If you want to isolate around that call, you will need another span around it. If you want to catch the exception, you can mark the span as ok but if your fallback controller then errors, we're not allowed to override you having market that span ok.

You can also mark the span as ok in the fallback controller. As a general design side note, I don't suggest using fallback controllers for all your error handling as it complicates things like this.