{"id":21018,"date":"2024-03-13T17:23:50","date_gmt":"2024-03-13T16:23:50","guid":{"rendered":"https:\/\/trifork.nl\/blog\/?p=21018"},"modified":"2024-03-15T15:16:29","modified_gmt":"2024-03-15T14:16:29","slug":"spring-boot-observability-database","status":"publish","type":"post","link":"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/","title":{"rendered":"Spring Boot Observability: Database Interactions"},"content":{"rendered":"\n<h2 class=\"wp-block-heading has-black-color has-text-color has-link-color wp-elements-8f42995bf4baa1c99f6f7e7dcbf36a7c\">Watch That Booty!<\/h2>\n\n\n\n<p>For Nederlandse Loterij, we develop a <a href=\"https:\/\/trifork.nl\/?cases=flexibility-and-scalability-for-millions-of-subscribers\" target=\"_blank\" rel=\"noreferrer noopener\">subscription system<\/a> that manages subscriptions for various lottery brands, creates invoices, handles the payments and eventually provisions the tickets.&nbsp;<\/p>\n\n\n\n<p>The system consists of ca. 10 different services, which all connect to a database, which in our case is a managed PostgreSQL running on AWS. This database makes a lot of metrics available out of the box.<\/p>\n\n\n\n<h1 class=\"wp-block-heading\">Who Are You? (who who, who who)<\/h1>\n\n\n\n<p>When you start to drill down into some of these metrics, you\u2019ll notice that metrics like <code>postgresql.queries.duration<\/code> and <code>postgresql.transactions.duration<\/code> contain a tag named \u201capp\u201d, which represents the application connecting to the DB. However, by default for the Boot apps this is set to \u201cPostgreSQL JDBC Driver\u201d:<\/p>\n\n\n\n<p><img loading=\"lazy\" decoding=\"async\" width=\"624\" height=\"337\" src=\"https:\/\/lh7-us.googleusercontent.com\/dNsSxDuCgmeCDSjEfsCQHZCTO-FE3GGe58QGC5_LcYEpcpCBWlFcMGAUyFWaujQtMjC1sZL5__aKiCIEEE_6msIxcjZQm1ptykvqaqw-trVIgxfyZDf-jhvFLGL_cWMeeSJ3m4R9WzVwmsa9kxDOAFA\"><\/p>\n\n\n\n<p>We have around 10 different services. One contains Batch jobs, others are handling OLTP concerns like creating a new subscription, or create business reports. We don\u2019t really want to see those lumped together as one app in the DB metrics.&nbsp;<br>Fortunately it\u2019s easy to change this: you can simply configure the database connection to use the application\u2019s service name instead. This is how we do that in YML:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: yaml; title: ; notranslate\" title=\"\">\nspring:\n datasource:\n   hikari:\n     data-source-properties:\n       # see https:\/\/jdbc.postgresql.org\/documentation\/use\/#connection-parameters\n       ApplicationName: ${spring.application.name}\n<\/pre><\/div>\n\n\n<p>With that simple change, it suddenly becomes possible to distinguish between the apps in the PostgreSQL metrics:<\/p>\n\n\n\n<p><img loading=\"lazy\" decoding=\"async\" width=\"780\" height=\"419\" src=\"https:\/\/lh7-us.googleusercontent.com\/Ce2o1R1fKhJ6YNrl1GvOxDErauzw2_siKTr3wlRuoDQMAet3i2tau4CG1j4s3bjQJJQXjX_hcnwmt_igHmpYkaOstpjCYwygxVxQ3eKqH8Pglo0EMtHi9DuYdUzwW91IXbhmlqnl8kngfHcuwY70X5Q\"><\/p>\n\n\n\n<p>These names are the same values that our application metrics use for a \u201cservice\u201d tag that we add. This is done by using a Micrometer meter registry customizer:<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: java; title: ; notranslate\" title=\"\">\n@Bean\nMeterRegistryCustomizer&lt;MeterRegistry&gt; commonTags(Environment env) {\n   String serviceName = env.getProperty(&quot;spring.application.name&quot;, &quot;unknown-service&quot;);\n   String hostName = Objects.requireNonNullElse(System.getenv(&quot;HOSTNAME&quot;), &quot;localhost&quot;);\n   return registry -&gt; registry.config().commonTags(\n       &quot;service&quot;, serviceName,\n       &quot;instance&quot;, hostName\n   );\n}\n<\/pre><\/div>\n\n\n<p>That means that we can now create a database calls-related dashboard with template variables to allow all widgets to filter on the selected service. In that dashboard we also include built-in metrics from the Hikari connection pool and the Spring Data JPA repositories.&nbsp;Note how this allows to see what info displayed in the widgets correlates to the same service, and how we can filter (best watched full-screen):<\/p>\n\n\n\n<figure class=\"wp-block-video alignwide\"><video controls src=\"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2024\/03\/Calls-to-the-database.mp4\"><\/video><\/figure>\n\n\n\n<p>Note: since our default tag is \u201cservice\u201d but the DB metrics use \u201capp\u201d instead, you can use a syntax in Datadog where you refer only to the selected template variable\u2019s value, rather than the name:value combo:<\/p>\n\n\n\n<p><img loading=\"lazy\" decoding=\"async\" width=\"780\" height=\"115\" src=\"https:\/\/lh7-us.googleusercontent.com\/5Tc8Rx-KwBNn50owlq2HYFhodGYVgynEwbWUIiFdceSUWbuoO4PkRdRu4Si1XMI46dZODPBwbewTiWkLA7TOjdBZN86qf_hwF_th4cb5JRBPukXBF0YrHQXIe_0vxVN-q1uyePrja3GSa4qH660OeNM\"><\/p>\n\n\n\n<h1 class=\"wp-block-heading\">Application-level Metrics<\/h1>\n\n\n\n<p>As shown in the video, our dashboard contains metrics from our Spring Data repositories and the Hikari connection pool. Let\u2019s discuss these in some detail.<\/p>\n\n\n\n<p>Spring Data has a metric called <code>spring.data.repository.invocations<\/code>. When you export this to Datadog, it derives a min, avg, max and count for that. These metrics provide \u201crepository\u201d and \u201cmethod\u201d tags that can be used to filter on the code executing the queries.<br>With this and the service tag that we add, it becomes trivial to show the average query execution time per service and the top 10 of slowest query methods:<\/p>\n\n\n\n<p><img loading=\"lazy\" decoding=\"async\" width=\"780\" height=\"165\" src=\"https:\/\/lh7-us.googleusercontent.com\/Tp12wA_PfgdBXrO88TajYukDS_PtMvNeFxrMe_wmFiyrvq9zvrVXD784Vrn5gQLTtCHeW5rHuR-xiWCb1QyzlHZvntNIA1NZbkJUdpPY4RHzX_24JYqM6uB6Vrxe9--fikwpPeEATlzgsFXwTuwEe_A\"><\/p>\n\n\n\n<p>By ensuring that the top list groups by method 1st but by service 2nd, you can hover over these methods and see the associated service, which then also gets highlighted in all the other graphs automatically.<br>Because the dashboard has the service as a template variable, filtering everything to show only the metrics for one particular service now becomes trivial as well.<\/p>\n\n\n\n<p>The Hikari metrics are also extremely useful. Not only can you see things like the actual number of connections in use, but you can also track how long connections are in use and what the wait time is for an available connection in case of contention.<br>Here we see an example where it clearly shows that our marketing app has some really long-running queries, but these do not consume all the available pooled connections so the wait time remains low:<\/p>\n\n\n\n<p><img loading=\"lazy\" decoding=\"async\" width=\"780\" height=\"480\" src=\"https:\/\/lh7-us.googleusercontent.com\/5HtpjiIVYdvzMtXUmMl7jipXMT_lczePnJaFcRZZ7a2ATrZBHg1E5yqWvcFguv9F04BAXv3dbKQWx1OcKTdLB9WIXAtZUUVezLcJdpdt330eHVQEmeKw7BRfNy-8PW59xQjGWulXUFFy-PEn_FMMvJ0\"><\/p>\n\n\n\n<p>Having insight into these sorts of metrics is super-valuable, esp. since it\u2019s so easy to combine information from your repositories, connection pool and database together for correlation.<\/p>\n\n\n\n<p>In the <a href=\"https:\/\/trifork.nl\/blog\/spring-boot-observability-spring-batch-jobs\/\" target=\"_blank\" rel=\"noreferrer noopener\">next blog<\/a>, I will show how we can use metrics as well as logging to improve the observability of Spring Batch jobs.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Watch That Booty! For Nederlandse Loterij, we develop a subscription system that manages subscriptions for various lottery brands, creates invoices, handles the payments and eventually provisions the tickets.&nbsp; The system consists of ca. 10 different services, which all connect to a database, which in our case is a managed PostgreSQL running on AWS. This database [&hellip;]<\/p>\n","protected":false},"author":62,"featured_media":21047,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"content-type":"","footnotes":""},"categories":[88,337,94],"tags":[73,561,70],"class_list":["post-21018","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-from-the-trenches","category-spring","tag-database","tag-observability","tag-spring"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v24.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Spring Boot Observability: Database Interactions - Trifork Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Spring Boot Observability: Database Interactions - Trifork Blog\" \/>\n<meta property=\"og:description\" content=\"Watch That Booty! For Nederlandse Loterij, we develop a subscription system that manages subscriptions for various lottery brands, creates invoices, handles the payments and eventually provisions the tickets.&nbsp; The system consists of ca. 10 different services, which all connect to a database, which in our case is a managed PostgreSQL running on AWS. This database [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/\" \/>\n<meta property=\"og:site_name\" content=\"Trifork Blog\" \/>\n<meta property=\"article:published_time\" content=\"2024-03-13T16:23:50+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-03-15T14:16:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2024\/03\/website-dashboard-line-icon-vector.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"200\" \/>\n\t<meta property=\"og:image:height\" content=\"200\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Joris Kuipers\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Joris Kuipers\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/\",\"url\":\"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/\",\"name\":\"Spring Boot Observability: Database Interactions - Trifork Blog\",\"isPartOf\":{\"@id\":\"https:\/\/trifork.nl\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2024\/03\/website-dashboard-line-icon-vector.jpg\",\"datePublished\":\"2024-03-13T16:23:50+00:00\",\"dateModified\":\"2024-03-15T14:16:29+00:00\",\"author\":{\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/265bd41e503f7176742258a927de598b\"},\"breadcrumb\":{\"@id\":\"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/#primaryimage\",\"url\":\"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2024\/03\/website-dashboard-line-icon-vector.jpg\",\"contentUrl\":\"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2024\/03\/website-dashboard-line-icon-vector.jpg\",\"width\":200,\"height\":200},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/trifork.nl\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Spring Boot Observability: Database Interactions\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/trifork.nl\/blog\/#website\",\"url\":\"https:\/\/trifork.nl\/blog\/\",\"name\":\"Trifork Blog\",\"description\":\"Keep updated on the technical solutions Trifork is working on!\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/trifork.nl\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/265bd41e503f7176742258a927de598b\",\"name\":\"Joris Kuipers\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/trifork.nl\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9ab8da0d60582bad84342d4602d23dbd?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/9ab8da0d60582bad84342d4602d23dbd?s=96&d=mm&r=g\",\"caption\":\"Joris Kuipers\"},\"sameAs\":[\"http:\/\/www.trifork.nl\"],\"url\":\"https:\/\/trifork.nl\/blog\/author\/jorisk\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Spring Boot Observability: Database Interactions - Trifork Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/","og_locale":"en_US","og_type":"article","og_title":"Spring Boot Observability: Database Interactions - Trifork Blog","og_description":"Watch That Booty! For Nederlandse Loterij, we develop a subscription system that manages subscriptions for various lottery brands, creates invoices, handles the payments and eventually provisions the tickets.&nbsp; The system consists of ca. 10 different services, which all connect to a database, which in our case is a managed PostgreSQL running on AWS. This database [&hellip;]","og_url":"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/","og_site_name":"Trifork Blog","article_published_time":"2024-03-13T16:23:50+00:00","article_modified_time":"2024-03-15T14:16:29+00:00","og_image":[{"width":200,"height":200,"url":"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2024\/03\/website-dashboard-line-icon-vector.jpg","type":"image\/jpeg"}],"author":"Joris Kuipers","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Joris Kuipers","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/","url":"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/","name":"Spring Boot Observability: Database Interactions - Trifork Blog","isPartOf":{"@id":"https:\/\/trifork.nl\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/#primaryimage"},"image":{"@id":"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/#primaryimage"},"thumbnailUrl":"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2024\/03\/website-dashboard-line-icon-vector.jpg","datePublished":"2024-03-13T16:23:50+00:00","dateModified":"2024-03-15T14:16:29+00:00","author":{"@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/265bd41e503f7176742258a927de598b"},"breadcrumb":{"@id":"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/#primaryimage","url":"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2024\/03\/website-dashboard-line-icon-vector.jpg","contentUrl":"https:\/\/trifork.nl\/blog\/wp-content\/uploads\/sites\/3\/2024\/03\/website-dashboard-line-icon-vector.jpg","width":200,"height":200},{"@type":"BreadcrumbList","@id":"https:\/\/trifork.nl\/blog\/spring-boot-observability-database\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/trifork.nl\/blog\/"},{"@type":"ListItem","position":2,"name":"Spring Boot Observability: Database Interactions"}]},{"@type":"WebSite","@id":"https:\/\/trifork.nl\/blog\/#website","url":"https:\/\/trifork.nl\/blog\/","name":"Trifork Blog","description":"Keep updated on the technical solutions Trifork is working on!","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/trifork.nl\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/265bd41e503f7176742258a927de598b","name":"Joris Kuipers","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/trifork.nl\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/9ab8da0d60582bad84342d4602d23dbd?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/9ab8da0d60582bad84342d4602d23dbd?s=96&d=mm&r=g","caption":"Joris Kuipers"},"sameAs":["http:\/\/www.trifork.nl"],"url":"https:\/\/trifork.nl\/blog\/author\/jorisk\/"}]}},"_links":{"self":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts\/21018","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/users\/62"}],"replies":[{"embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/comments?post=21018"}],"version-history":[{"count":22,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts\/21018\/revisions"}],"predecessor-version":[{"id":21061,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/posts\/21018\/revisions\/21061"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/media\/21047"}],"wp:attachment":[{"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/media?parent=21018"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/categories?post=21018"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/trifork.nl\/blog\/wp-json\/wp\/v2\/tags?post=21018"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}