- Table of Contents
One can use opentracing-spring-haystack-web-starter or opentracing-spring-haystack-cloud-starter to instrument spring boot or spring web applications and send tracing information to Opentracing complicant Haystack server, distributed tracing platform.
These libraries in turn use opentracing-spring-haystack-starter which helps build the
io.opentracing.Tracer
instance required by underlying io.opentracing.contrib : opentracing-spring-web-starter.
This section provides steps required to quickly configure your spring application to be wired using Opentracing's spring integration to Haystack. If you need additional information, please read the subsequent sections in this documentation
Add the following dependency to your application
<dependency>
<groupId>com.expedia.www</groupId>
<artifactId>opentracing-spring-haystack-web-starter</artifactId>
<version>${opentracing-spring-haystack-web-starter.version}</version>
</dependency>
Alternately, one can use opentracing-spring-haystack-cloud-starter
instead. It is convenience starter that includes opentracing-spring-haystack-starter, opentracing-spring-web-starter and opentracing-spring-cloud-starter. This allows one to take advantage of opentracing instrumentation provided for Spring Cloud.
<dependency>
<groupId>com.expedia.www</groupId>
<artifactId>opentracing-spring-haystack-cloud-starter</artifactId>
<version>${opentracing-spring-haystack-cloud-starter.version}</version>
</dependency>
opentracing-spring-cloud-starter provides ready instrumentation of various known OSS projects like Redis, Hystrix etc.,
To access an instance of opentracing.io.Tracer
to instrument a Spring application, one can add the following dependencies
<dependency>
<groupId>com.expedia.www</groupId>
<artifactId>opentracing-spring-haystack-starter</artifactId>
<version>${opentracing-spring-haystack-starter.version}</version>
</dependency>
Enable @ComponentScan
on the package com.expedia.haystack.opentracing.spring.starter
to configure the Tracer
bean
Optionally, add the following to get metrics recorded in JMX
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-jmx</artifactId>
<version>${io-micrometer.version}</version>
</dependency>
Add the following to the properties or yaml file of the application being instrumented (this is just a sample. change the name of the application, host name/port of the agent etc)
spring:
application:
name: springbootsample
opentracing:
haystack:
dispatchers:
logger:
name: span-logger
agent:
enabled: true
host: haystack-agent
port: 35000
One can checkout the example project using this integration @ https://github.com/ExpediaDotCom/opentracing-spring-haystack-example
Check maven for latest versions of this library. At present, this library has been built and tested with Spring Boot 2.x
<dependency>
<groupId>com.expedia.www</groupId>
<artifactId>opentracing-spring-haystack-web-starter</artifactId>
<version>${opentracing-spring-haystack-web-starter.version}</version>
</dependency>
Adding this library autoconfigures an instance of Haystack Tracer using defaults mentioned below.
service-name
: Is read from configuration propertyspring.application.name
. If it is not provided then the value will be set tounnamed-application
dispatcher
: Spans are dispatched to Haystack using one or more Dispatcher instances. If none is configured or created, then a LoggerDispatcher is configured with "haystack" as the logger namemetrics
: This library depends on Micrometer's MeterRegistry to instrument the library itself. If no instance ofMeterRegistry
is present in the spring application, then it uses a built-in No-op implementation. Which means no metrics are recorded
One can also configure the tracer created by the library using few configuration properties and optional beans.
One can completely disable tracing with configuration property opentracing.haystack.enabled
. If the property is missing (default), this property value is assumed as true
.
opentracing:
haystack:
enabled: false
One can configure Dispatcher
in two ways: using configuration properties or by creating a spring bean.
Using configuration properties one can configure one or more of the following dispatchers. Configuring more than one dispatcher, creates a ChainedDispatcher
and sends a span to all of the configured dispatcher instances.
One can configure the name of the logger to use in a LoggerDispatcher by setting the following property in Spring Boot yaml or properties file
opentracing:
haystack:
dispatchers:
logger:
name: span-logger
Haystack provides a GRPC agent as a convenience to send protobuf spans to Haystack's kafka. One can configure grpc agent by simply enabling it in the configuration.
opentracing:
haystack:
dispatchers:
agent:
enabled: true
There are other properties available to further configure the grpc agent dispatcher
opentracing.haystack.dispatchers.agent.host=haystack-agent
opentracing.haystack.dispatchers.agent.port=35000
opentracing.haystack.dispatchers.agent.keep-alive-time-m-s=30
opentracing.haystack.dispatchers.agent.keep-alive-timeout-m-s=30
opentracing.haystack.dispatchers.agent.keep-alive-without-calls=true
opentracing.haystack.dispatchers.agent.negotiation-type=PLAINTEXT
Alternately, one can create a bean of type GrpcDispatcherFactory.
public interface GrpcDispatcherFactory {
Dispatcher create(MetricsRegistry metricsRegistry,
TracerSettings.AgentConfiguration agentConfiguration);
}
If available, this bean will be invoked with configuration properties defined to build a RemoteDispatcher with GrpcAgentClient.
@Bean
public GrpcDispatcherFactory grpcDispatcherFactory() {
return (metricsRegistry, config) ->
new RemoteDispatcher.Builder(metricsRegistry,
config.builder(metricsRegistry).build()).build();
}
Haystack also provides a http collector to ingest Json and Protobuf serialized spans over http.
One can configure a http dispatcher by adding the following endpoint configuration
opentracing:
haystack:
dispatchers:
http:
endpoint: http://localhost:8080/span
headers:
client-id: foo
client-key: bar
headers
property is optional. All properties defined under 'headers' will be sent as HTTP headers along with the serialized span data.
As in Grpc Agent, one can create a bean of type HttpDispatcherFactory. If available, that bean will be invoked to create a RemoteDispatcher with HttpClient
Instead of configuring dispatchers through properties, one can create a bean of type Dispatcher
in the application's spring context. This library will use that bean instead of creating one using configuration or defaults. One can see this in the integration test example.
As mentioned earlier, this library looks for a bean of type Micrometer's MeterRegistry. If present, it uses that to write all metrics from the library to the configured store. If not, the library will use a no-op implementation and no metrics will be written.
For example, adding the following two dependencies to the application will automatically create a MeterRegistry
bean and write the metrics to JMX
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-jmx</artifactId>
<version>${io-micrometer.version}</version>
</dependency>
Haystack Tracer's Builder class exposes a number of possible configurations that can be used to customize the Tracer
instance built by this library and used by Opentracing's Spring integration. To customize the Tracer instance, one can create a bean of type TracerCustomizer. This will be invoked when the library attempts to build an instance of Tracer. One can see this in the integration test.
Update the version of repo using:
./mvnw org.codehaus.mojo:versions-maven-plugin:2.5:set -DnewVersion=<version>-SNAPSHOT
Please don't remove snapshot from the version. The travis job is triggered when a new PR is merged in the master branch. Travis also updates the project version whenever a new release is tagged.
One can turn on the blob feature that writes the request+response payload in distributed file-system and also adds a span tag to dereference the store location. For more information, read here. One can also use haystack-agent as a blob forwarder to distributed file systems like AWS/S3.
haystack:
blobs:
enabled: true
store:
name: file # for local development
To use haystack-agent, use following:
haystack:
blobs:
enabled: true
store:
name: agent
host: haystack-agent
port: 35001
If one wants to blob conditionally based on certain properties on request/response, then one needs to implement the Blobable interface. For instance, one can control to blob only if there is a http failure. Check the BlobStoreTest example for quick reference. Please note, the blobs integration works both for client and server side span.