I used Spring’s RestTemplate to fetch some JSON, and you won’t believe what happened next!

Motivation is an interesting thing. What is it that drives us? What makes you want to do the things you do on a daily basis?
Over the course of my career I’ve noticed that irritation is one of my driving factors. At first you get slightly annoyed by something, and you can shake it off, but some things keep rubbing you the wrong way until you can’t stand it anymore and you finally address them, no matter what the costs are. You dive into internals of technology you never wanted to learn, go outside your comfort zone, and do whatever it takes to get back some peace of mind. It keeps you going all night long, together with Beer Driven Development (I’ve successfully flattened the curve of my Ballmer peak many years ago).

Recently I had a similar experience with how we use Jackson for unmarshalling JSON. 

What Jackson unmarshalling can do will leave you in awe!

We use Spring and Jackson on pretty much all of our projects. To unmarshall JSON responses returned from HTTP services we often model the response data as POJOs, but we like to make these immutable DTOs. 

An easy way to do that is to annotate the type’s constructor with @JsonCreator and to annotate the constructor arguments with @JsonProperty:

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;

public class SystemResponse {
  private String someField;
  private String someOtherField;

  @JsonCreator
  public SystemResponse(@JsonProperty("someField") String someField,
                        @JsonProperty("someOtherField") String someOtherField) {
    this.someField = someField;
    this.someOtherField = someOtherField;
  }

  // getters ommitted
}

The @JsonCreator isn’t needed if you only have one constructor, but I like having it in to indicate that Jackson is an intended client for this constructor. 
As you can see in the example, the names provided through the @JsonProperty annotation are the same as the argument names. 

My project had literally hundreds of these small classes, representing all sorts of responses. Every one of them specified the name of the JSON field as a String attribute of the @JsonProperty, while it’s exactly the same as the name of the constructor argument. I knew there must be some way to avoid this redundancy of specifying the JSON field names twice, which is especially tricky when things get renamed or copy/pasted to new types: you might accidentally misname a property, so that the constructor arg is always null because its annotation references a non-existent field. However, removing the annotation didn’t work, and I hadn’t investigated yet what was needed to make it work.

Until this weekend that is, when I was finally annoyed enough to have a look at what was needed to get rid of the annotations and duplication…

Can you guess what the solution is? 90% of readers had no idea!

It’s actually quite straightforward: Jackson provides a ParameterNamesModule that can extract the names from the constructor arguments for all code compiled with Java 8 or higher, as long as you pass the -parameters flag to javac. In addition to that, Spring Boot ensures that all ObjectMappers created via the Jackson2ObjectMapperBuilder bean have the module registered already if it’s on the classpath, which it is if you’re using the spring-boot-starter-json (directly or, more typically, transitively through spring-boot-starter-web). 

That means that all I needed to do was update our Gradle build script to pass the extra flag to javac. Something like this will do the trick:

compileJava {
  options.compilerArgs << '-parameters'
}

Colleagues won’t be able to resist you if you apply this simple trick!

Most of our projects still use Spring’s RestTemplate to perform HTTP requests, as migrating all the code to the newer WebClient doesn’t provide enough return on investment. 

When you create your RestTemplate using the RestTemplateBuilder provided as a bean by Spring Boot, you will benefit from a lot of configuration applied automatically to your template. Actually, I have a talk about that.

So, by simply using the RestTemplateBuilder our RestTemplate will automatically use a MappingJackson2HttpMessageConverter configured with an ObjectMapper that uses the required ParameterNamesModule.
We were already doing that, so after ensuring the -parameternames was passed to javac I went ahead and used some regexp magic to remove all the @JsonProperty annotations that specified the same name as their constructor arg. Then I committed 350+ changed files and patted myself on the back: itch scratched!

Is your code cheating on you? It is, if you see these logging messages!

We deployed the changes, and everything seemed to work OK. That is, until a colleague pointed out some warnings in the logs that suggested otherwise. They looked like this:

Failed to evaluate Jackson deserialization for type [[simple type, class mypackage.SomeResponseType]]: com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
Invalid type definition for type `mypackage.SomeResponseType`:
Argument #0 has no property name, is not Injectable:
can not use as Creator [constructor for mypackage.SomeResponseType, annotations: {interface com.fasterxml.jackson.annotation.JsonCreator=@com.fasterxml.jackson.annotation.JsonCreator(mode=DEFAULT)}]

WAT? I may not be known for writing code with 100% test coverage, but I definitely tested this and saw it working. Was the code compiled without the -parameters flag somehow? Were we missing the ParameterNamesModule? Was I to roll back the huge commit that I bragged smugly about, defeated by this error?

Of course not! Irritation-driven development isn’t my only mode of operation: I love solving technical puzzles as well, so I went on a quest and you won’t believe what I found!

Two things that are wrong with Spring Boot’s RestTemplate setup

When I tried to reproduce the problem locally, I took a use case from the logs that showed the error and ran it locally on my laptop. The response returned by my service was just as I expected, no problem there. Until I looked at the logs that service produced: it had the exact error that I was trying to reproduce! What was happening here?

This required a bit of debugging, but as it turns out, the RestTemplate created by the builder actually has two instances of a MappingJackson2HttpMessageConverter configured. The first one looks as expected, but the second one is configured with an ObjectMapper that does not have the ParameterNames module registered! That must be the problem, but how does that explain the observed behavior?

As it turns out, in order to prepare the request, the RestTemplate will pass the return type that you want to unmarshal your response to every configured message converter and ask it if it knows how to take some result and deserialize it to the given type. If the answer is yes, it will use the media type associated with these converters to construct an Accept header for the request. 

While it does this, the second instance of the MappingJackson2HttpMessageConverter has a look at the response type, inspects the @JsonCreator constructor and fails to derive property names for its constructor args, leading to that warning being logged. However, that converter won’t even be used, so the warning is completely pointless…

It’s a problem that Spring creates two instances of this converter. This sucks, but is a known issue that people just live with, most without even knowing. However, the second converter is configured with some default ObjectMapper not created through Boot’s configuration, and therefore only has two default modules configured and not the ParameterNamesModule! That’s a real issue, as now every request will result in a warning in our log that looks like a real error happening during the unmarshalling of a response.

This solution will change your life!

It’s not actually that easy to avoid having Spring creating the second JSON converter, so I opted for a solution that would just remove it before the RestTemplate is used. Spring Boot’s RestTemplateBuilder automatically applies any RestTemplateCustomizer beans that you’ve defined when building a RestTemplate, so that’s a good place to do some post-processing and fix the problem. 

Without further ado, here’s the bean that I created in our auto-configuration:

@Bean
RestTemplateCustomizer unwantedConvertersRemovingCustomizer() {
  return restTemplate -> {
    boolean foundFirstJsonConverter = false;
    for (var iter = restTemplate.getMessageConverters().listIterator(); iter.hasNext(); ) {
     HttpMessageConverter<?> converter = iter.next();
     if (converter instanceof MappingJackson2HttpMessageConverter) {
       if (foundFirstJsonConverter) {
         iter.remove();
       } else {
         foundFirstJsonConverter = true;
       }
     }
   }
 };
}

We know that the first converter is always the correct one, so just toss any additional JSON converters that pop up.

I would be interested to learn about solutions for this problem that readers might’ve come up with already: if you have something, make sure to like this post and subscribe! I mean, leave a comment below.