Spring Insight plugin for the Axon CQRS framework

by Joris Kuipers, November 13, 2012

Introduction

In a previous blog post we introduced the Spring Insight module that’s part of SpringSource’s tc Server developer edition and gives you, well, insight into what’s taking up the time to process requests in your applications. Insight does this by instrumenting your application at load time using AspectJ: a single call to your application will result in a so-called trace consisting of various operations, and for each of these operations it’s measured how long it takes to execute. An operation can provide additional details: for example, when sending a query to a database using a JDBC Connection the actual SQL statement including prepared statement bindings will be stored in the operation.

Spring Insight is not a profiler: it doesn’t create operations for each and every little thing that happens, but only for ‘significant’ events, like a Spring-MVC controller invocation, the start of a transaction or communication with a database. These events are called ‘operations’. It does allow assigning operations to so-called endpoints. An endpoint could be a controller method, or a method on a class annotated with one of Spring’s stereotype annotations like @Service or @Repository for example. For these endpoints Insight can provide statistics across traces, so that you can measure the health of these endpoints during the lifespan of your application based on configurable thresholds.

Insight’s plugin model

Spring Insight consists of a core plus a set of plugins that are shipped with the tool. These plugins define the actual operations and endpoints that Insight knows about and exposes. One of the nice things about Spring Insight is that the plugins use a public API that’s not restricted to the built-in plugins: you can build your own plugins to teach the tool new tricks.

Although you could do this on a per-application basis to expose metrics relevant to your particular app, you wouldn’t usually write a dedicated plugin for that. Insight already exposes several application-level methods as operations if they’re part of your stereotype-annotated Spring beans, and you can use their set of annotations to expose additional application-specific operations and endpoints easily .

Plugins are much more useful for framework developers: their framework might contain several methods that would be interesting to expose as Insight operations or even endpoints to show users what the framework is doing and how long that takes. Earlier this year, that’s exactly what we did for the Axon CQRS framework that’s being developed within Trifork.

This blog briefly discusses the plugin’s implementation. All source code has been added to the Axon Framework itself and is available on Github.

Update:
After publishing this blog entry, VMware has contacted us to host the source code for this plugin in their public GitHub repository, so that the plugin can be shipped out-of-the-box with the Spring Insight distribution. That means that the plugin sources are no longer found under the Axon repository. The link above has been updated to reflect this change.

The Axon Insight Plugin

Applications built with Axon create and handle commands and events. Commands are dispatched through a command bus to a single command handler, and the handling of a command typically leads to one or more events. These events are published to all interested event handlers for further consumption. There are some other relevant concepts like sagas, but for the purpose of explaining the plugin this is pretty much all you need to know about Axon’s model.

This means that Axon-related processing originates from commands. Both the dispatching and handling of a command are thus interesting operations to capture. To identify these sort of operations you have to write an AspectJ pointcut expression that matches the relevant method execution.

An Axon command handler method is typically identified by an annotation, but can also be the implementation of a CommandHandler interface. Moreover, the support for annotation-based handlers is actually provided by a framework implementation of this interface. This means we want to match executions of annotated methods and interface methods, but we should exclude the built-in CommandHandler implementation to avoid the creation of two operations for a single command handling event in the case of annotated methods.
Pointcuts allow you to express exactly this information in a concise manner. This is what the definition used by the plugin looks like:

public pointcut collectionPoint():
        execution(@org.axonframework.commandhandling.annotation.CommandHandler * *(*, ..)) ||
                (execution(* org.axonframework.commandhandling.CommandHandler.handle(*, org.axonframework.unitofwork.UnitOfWork))
                        && !within(org.axonframework.commandhandling.annotation.AnnotationCommandHandlerAdapter));

Based on that named pointcut you can then create an Operation instance. An operation contains information about the context it was captured in: this includes a type, label and source code location but can be extended with arbitrary additional information that’s relevant to your use case.

What’s interesting in the case of Axon is that the current version under development, version 2.0, has a different way of representing commands internally than the production 1.x versions. Prior to the 2.0 version a command was always a simple POJO, whereas starting with 2.0 the programming model still supports POJOs but internally a CommandMessage object is used. This CommandMessage can contain additional metadata next to the command object itself that it wraps as its payload. If your command handler accepts a CommandMessage rather than a POJO as its argument, it’s nice to let the plugin capture that metadata as part of the Operation that is created.

The plugin supports both Axon 1.x and 2.0, so it checks what version is being used and conditionally adds the extra info. Since most of the work of creating an Operation is boilerplate code (capturing the source location for a method invocation is always the same) it’s easiest to let your command creation aspect extend an Insight-provided base aspect. Here’s the full code (without imports) for the aspect that captures command handler invocations and creates the corresponding operations:

public aspect CommandHandlerOperationCollectionAspect extends MethodOperationCollectionAspect {

    public pointcut collectionPoint():
            execution(@org.axonframework.commandhandling.annotation.CommandHandler * *(*, ..)) ||
                    (execution(* org.axonframework.commandhandling.CommandHandler.handle(*, org.axonframework.unitofwork.UnitOfWork))
                            && !within(org.axonframework.commandhandling.annotation.AnnotationCommandHandlerAdapter));

    @Override
    protected Operation createOperation(JoinPoint jp) {
        Operation operation = super.createOperation(jp).type(AxonOperationType.COMMAND_HANDLER);
        Object[] args = jp.getArgs();
        if (!AxonVersion.IS_AXON_1X) {
            if (Axon20OperationUtils.processCommandMessage(args, operation)) {
                // we're done here
                return operation;
            }
        }
        Object command = args[0];
        operation.put("commandType", command.getClass().getName());
        return operation;
    }
}

In this case it’s fairly straightforward to deal with the two Axon versions in a single class (the Axon20OperationUtils handles the metadata part if the command is in fact a CommandMessage instance): for some other cases, like the actual dispatching of a command by the CommandBus, the plugin has dedicated aspects per version with pointcuts that are specific to either Axon 1.x or 2.0.

The plugin defines operations for the following events:

Command dispatching by the command bus
Command handling by a command handler
Event publishing by the event bus
Event handling by an event handler
Event handling by a saga

The result of doing this can be seen in the following screenshot of a trace taken from the Axon Trader sample application:

Axon Trader Insight screenshot

Note the event sourcing in action: to handle the remove contact command, an aggregate is first loaded from a snapshot and subsequent events obtained from the event store. Had the aggregate been cached already, then obviously the trace wouldn’t have shown these JDBC SELECT statements.

Endpoint analyzers

In addition to providing these Axon-specific operations, the plugin also defines several endpoint analyzers. Endpoint analyzers can be used to define new endpoints that will show up in the Spring Insight UI so that their health can be monitored across requests handled by the application. Not every operation needs to be associated with an endpoint: endpoints are often application components that perform some type of handling of dispatched requests in your application, like Spring-MVC controllers. That means that it’s interesting to define endpoints for command handlers, and potentially for (saga or regular) event handlers as well. Currently the plugin supports all three, although feedback from real-world usage might lead to the conclusion that defining endpoints for event handlers is a bit too fine grained.

The endpoint analyzer implementation for command and event handlers is similar enough that most of the code has been extracted into a common baseclass. I won’t discuss the details here, but when you start with Insight plugin development you’ll see that the code is very similar to that of the built-in plugins for e.g. Spring-MVC Controller support.

By having something like a command handler endpoint, you can monitor how long it takes to handle certain types of commands during the lifecycle of your Axon application. Especially when event handling is done synchronously this will give you a good feel for what parts of your application might contain a bottleneck. By then drilling down into representative traces that have been captured you can see where the actual time is being spent.

Here’s a screenshot of what this looks like for the Axon Trader; note how HTTP and Axon endpoints are both shown:

Conclusion

As you can see, you can gain a lot of insight into the workings and performance of your Axon-based applications using the Insight plugin with relatively little effort. The current plugin is an initial proof of concept and can surely be expanded and improved: of course we’d love to hear your feedback for improvements and other suggestions!