For observability this client has chosen Datadog<\/a>, which has built-in and deep Azure integration. In the case of Azure App Services, that means that the logs from your Apache HTTP Server, Apache Tomcat and <\/em>application are automatically sent to Datadog. For application logs that means “everything written to the console”. Sounds easy? Yes, but also too good to be true.<\/p>\n\n\n\n What happens is that every log line written to STDOUT is simply taken as-is and stuffed in a JSON document as a field called “resultDescription”, to which a whole bunch of other fields are added and shipped to Datadog. In Datadog there’s a default logging pipeline for logging coming from the so-called “azure.web” source, and eventually you end up with something like this: As you can see, we don’t have a log message, the level is wrong and effectively no info is taken from our application logging, because it\u2019s all wrapped up inside the Typically I’d try to remedy this at the side that’s producing <\/em>the logs: configure the Datadog agent, set up an observability pipeline<\/a>. However, in this case we’re leaning on built-in App Service integration and there’s no dedicated agent to configure. Although you can configure a Datadog side car in Azure App Service<\/a>, I’d prefer to do without that for every individual application (and also am not sure that our pipeline will be able to configure this, or if this is a UI-only feature). So, in this case I opted for a solution on the Datadog side: we need a custom logging pipeline.<\/p>\n\n\n\n With Datadog you can set up ingestion pipelines for your logging that teach the tool how to process incoming log statements. For well-known sources, integrations can set up predefined pipelines, but you can also define your own.\u00a0<\/p>\n\n\n\n As we\u2019re using the built-in Azure integration for our App Service-based applications, there is a default pipeline for logs with an \u201cAzure Web\u201d source. This one isn\u2019t sufficient for our case, though, as it doesn\u2019t know about the custom The trick is to define an additional pipeline that runs BEFORE the built-in one for the application logs. This is how I’ve done that:<\/p>\n\n\n\n This is what the definition containing the filter of that new pipeline looks like:<\/p>\n\n\n\n By filtering on category and the fx_version (note the wildcard!) we know that we’re matching console logs and that they come from a Java Tomcat web application (we don’t want to accidentally match applications from other teams here). The Azure App Service integration also sends HTTP server logs and Tomcat logs, and these do not use structured JSON logging and also do not result in a The most important goal is to let Datadog parse the JSON in the As shown in the screenshot above, the pipeline first contains a Grok processor that does exactly this, and then some processors that handle the resulting fields.<\/p>\n\n\n\n This is the Grok processor:<\/p>\n\n\n\n Under the Advanced Settings we configure the field, and then the rule itself becomes trivial. You can read about processing nested JSON here<\/a>. It’s very nice that we can indicate that the processor should apply to a specific field: I was worried that I’d had to come up with a complex Grok rule that would match the entire JSON message sent by the Azure integration, but as you can see there’s no need for that at all.<\/p>\n\n\n\n The following two processors map The next two remap The final two remap ECS keys<\/a> produced by a custom filter of our applications to the corresponding Datadog default attribute names<\/a> (again, without preserving the original fields).\u00a0<\/p>\n\n\n\n With this additional pipeline in place, our logging now looks as expected. The logger and thread name are picked up automatically as those fields have the expected names already, and the rest is handled by the pipeline:<\/p>\n\n\n\n As can be seen, Datadog understands that this is a DEBUG message, the service is now “JiraTalker” rather than “azure” and the logging contains the actual message. There’s no The nice thing about this solution is that it doesn’t require anything on the Azure side: no log files, no side cars, no API keys for Datadog, nothing. Everything is handled at the Datadog side. We are stuck with the resultDescription field itself, but if its size would become an issue then you could configure Datadog to not index it or you could remap some other small field onto it: it seems like there’s no way to simply drop a field using a logging pipeline processor.<\/p>\n\n\n\n A new year, a new blog! Well, the year isn’t actually new, but I haven’t blogged for a while so it’s about time. If you think that the title of this one is oddly specific then you have a point, although I like to think of it as “highly SEO optimized”.Another option would’ve been “Yo […]<\/p>\n","protected":false},"author":62,"featured_media":21209,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[88,337,31,1],"tags":[],"class_list":["post-21201","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-from-the-trenches","category-java","category-uncategorized"],"acf":[],"yoast_head":"\n
<\/p>\n\n\n\n![\"\"](\"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2025\/09\/image-1024x763.png\")
<\/figure>\n\n\n\nresultDescription<\/code> field.
Note that the value of that field is a single string with JSON: that’s because the Azure Datadog integration actually escapes the application’s JSON logging, as it doesn’t know (and cannot assume) that the logging uses JSON formatting. If it had included the JSON as-is, then Datadog would have shown nested fields under the resultDescription<\/code> instead. <\/p>\n\n\n\nThe Solution<\/h1>\n\n\n\n
You can also forego the whole Datadog integration and use Agentless logging<\/a>, but that doesn’t work with Log4J2 (which is what this client is using) and has other issues.<\/p>\n\n\n\nDatadog Logging Pipelines<\/h2>\n\n\n\n
resultDescription<\/code> field and its JSON contents.<\/p>\n\n\n\nDefining a custom logging pipeline<\/h2>\n\n\n\n
![\"\"](\"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2025\/09\/image-1.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2025\/09\/image-2.png\")
<\/figure>\n\n\n\nresultDescription <\/code>field, so we want to ensure we’re matching only our custom application logs that we know to contain JSON. <\/p>\n\n\n\nProcessors<\/h2>\n\n\n\n
resultDescription<\/code> and extract the contents as top-level fields of the log event. <\/p>\n\n\n\n![\"\"](\"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2025\/09\/image-4.png\")
<\/figure>\n\n\n\nmessage<\/code>\u00a0as the message and service.name<\/code>\u00a0as the service. The latter is added by our code to every log message to identity the Java service that’s producing the logging. <\/p>\n\n\n\n@timestamp<\/code> to time<\/code> and status<\/code> to level<\/code> without <\/em>preserving the original attributes: the reason is that the log event already contains those fields (but not with the value that we want) and the built-in azure.web pipeline already defines how to handle those fields. It’s better to reuse them and remove our own fields, that also means there’s less data to store in Datadog.<\/p>\n\n\n\nEffect<\/h2>\n\n\n\n
![\"\"](\"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2025\/09\/image-5.png\")
<\/figure>\n\n\n\n@timestamp<\/code> nor status<\/code> field, as they’ve been remapped.<\/p>\n\n\n\nConclusion<\/h2>\n\n\n\n
The log messages there also still contain the same fields as all other App Services logging, we just change the service name.\u00a0<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"