Distributed Tracing in an Axon Application

The Giftcard-demo demonstrates Axon Commands, Events and Queries either together or in a distributed fashion. Now with the tracing extension, these messages and context information can be visualised to get insight into the application behaviour.

Requirements to run the demo

You will need git, docker and maven. We will checkout out the code, build the application and run docker-compose with 2 containers. The first is Axon Server which be the event store and take care of message handling for the Application. The second container will be the Jaeger UI to visualize the traces.

For additional reading on Axon and Axon Server, please also check the reference documentation.

Distributed tracing, a must have….

Distributed tracing is a must have in application architectures based on microservices. Fortunately, observability of a distributed or microservices architecture is getting a lot of attention, and several community initiatives are evolving to be able to follow a request through the system and present a trace.

When using the Axon Framework in a distributed fashion with components either handling commands, queries or events, tracing becomes a necessity.

OpenTracing the Axon Framework

We decided to opt for the Spring OpenTracing contribution in the first release of Axon tracing. The alternative is to use Opencensus or the brave framework in combination with Zipkin, also supported by Spring Cloud Sleuth.

We felt OpenTracing was giving more bang for the buck out of the box, like the capability to do include DB traces. (But other specifications would also work out, and can be supported later on).

Axon Framework at this present date, doesn’t support OpenTracing (nor sleuth for that matter), so we decided to add Axon Framework instrumentation as an extension.

Propagating tracing information

The term propagation in tracing, means the passing of a span context from one system (microservice) to another one. Typically this is done using HTTP headers, which are ideal to carry this information. Before we move on, let’s get an overview of tracing.

[Source: https://opentracing.io/docs/overview/]

Definition:
“Span context: Trace information that accompanies the distributed transaction, including when it passes the service to service over the network or through a message bus. The span context contains the trace identifier, span identifier, and any other data that the tracing system needs to propagate to the downstream service.“

In the case of Axon, the framework defines several types of messages. The most notable ones are the Command-, Event- and the Query Message.

The Message interface itself in Axon, allows to add MetaData to any of these messages and this is the mechanism we use. The Span Context of an active trace, with the MetaData object.

An alternative is to use HTTP headers to propagate the Span Context, which we didn’t opt for, but would like to research further in the future. Axon Server for example could also be instrumented and record separate spans for each unit of work, like storing an event in the event store or related to the gRPC communication.  

Tracer Implementation

The extension-tracing has a dependency on opentracing-spring-tracer-configuration-starter , which provides Spring Boot Auto configuration, resolving the NoopTracer and wiring the GlobalTracer.

An application therefor has to add a Tracer implementation. We will see later how this is done in the GiftCard demo.

With the Opentracing API on the classpath of extension-tracing, we can now instrument Axon Framework with tracing capabilities and propagation at various points:

• Query and Command Gateway
• MessageDispatchInterceptor
• MessageHandlerInterceptor
• CorrelationDataProvider

With this instrumentation, we can chain synchronous and asynchronous commands and queries, all belonging to the same parent span. A request can be visualized and analysed across Axon clients, command handlers, query handlers and event handlers, when running together or decomposed and deployed as separate parts.

Tracing the GiftCard demo

Very little changes are required to make the GiftCard-demo support tracing.

This is what I changed in the pom.xml

<!-- Axon Tracing -->
<dependency>
  <groupId>org.axonframework.extensions.tracing</groupId>
  <artifactId>axon-tracing-spring-boot-starter</artifactId>
  <version>4.0-M1</version>
</dependency>

<dependency>
  <groupId>io.opentracing.contrib</groupId>
  <artifactId>opentracing-spring-jaeger-web-starter</artifactId>
  <version>1.0.1</version>
</dependency>

The second dependency is Jaeger support for OpenTracing. The tracer implementation will report traces to the Jaeger UI.

To run Axon Server and Jaeger UI, we also created a docker-compose file:

version: '3.4'

services:

 axonserver:
   image: axoniq/axonserver
   container_name: giftcard-axonserver
   ports:
     - 8024:8024
     - 8124:8124
   networks:
     - axonnet
     - default

 jaeger:
   image: jaegertracing/all-in-one:1.7
   container_name: jaegertracing
   ports:
     - 16686:16686
     - "6831:6831/udp"
     - "6832:6832/udp"
   networks:
     - axonnet
     - default

networks:
 axonnet:
   driver: bridge

Now that’s it really! You can check out this forked version of the GiftCard demo here:

1. Clone the repo with `git clone https://github.com/dzonekl/giftcard-demo.git`
2. Run `docker-compose -f docker/docker-compose.yml up`  to bring up Axon Server and Jaeger.
3. Follow the readme to run the application in 3 separate parts. (Basically run it 3 times, each with a different spring profile).
4. When running, issue a giftcard through the UI on localhost:8080.
5. Now navigate to the Jaeger UI, localhost:16686.

From the ‘Service’ pull down, the various recorded services are visible, select the ‘gc-ui’ service, which is the client initiating commands and queries and press ‘Find Traces’.

On the right, the various traces should show up.

Loading the initial page

Shows the following trace:

1. The initial POST to localhost:8080 from Vaadin
2. This will result in two queries being issued, CoundCardSummariesQuery and FetchCardSummariesQuery.

As can be observed, the trace is propagated from the gc-gui service to the gc-query service and child spans with operation name ‘Extracting’ are created containing Axon specific tracing tags for the Axon message.

(Note: The tags are not configurable for now, which could change).

Issuing a gift card

A new gift card can be issued in the UI, by providing a ‘card id’ and an amount.  

Creating a gift card is a IssueCmd fired by ‘gc-ui’, then handled by the service ‘gc-command’, to be applied by the GiftCard aggregate, and which will lead to an IssuedEvt handled by ‘gc-query’ service persisting the gif-card in a JPA repository.

Also here, we see that 3 different services are tied together by trace-id for one single request on the UI.

Notice also, the asynchronous nature. The command is handled async, the event is then processed while the command child span ended. This is eventual consistency visualized.

Other interactions

The GiftCard demo also supports bulk-issuing and redeeming of gift cards (Awesome!). You should be able to see the traces for it in Jaeger, minding you click the corresponding buttons and pass UI form validation.

Note on the Vaadin framework traces

The GiftCard demo uses the Vaadin framework which has some internal HTTP requests also showing up in Jaeger. These are not bound to Application handlers, so will actually be distracting.

Limitations of extension-tracing

Big disclaimer here, we focused on making extension-tracing working for the GiftCard demo simple cases. We still lack:

• Support for Subscription Queries
• Deadline Messages
• Saga’s

Additionally, it would be nice, if we have more control on the issued tags.

Finally, Opentracing is supported by many tracers , we only tested one of them. Feel free to report back on your experience with other tracers.