Helm your way to Kubernetes with Spring Boot Admin

by Cristina Negrean, October 5, 2022

In this blog post I am experimenting with Helm, the package manager for Kubernetes by packaging codecentric’s Spring Boot Admin for out of the box real-time insights into a suite of Spring Boot Java services deployed to Amazon Elastic Kubernetes Service (EKS).

Spring Boot Admin gives a nice overview of what is deployed in your Kubernetes cluster, in one or more namespaces. Behind the scenes, its magic is given by the Spring Boot Actuator’s endpoints.

Applications Journal Actuator Endpoints Detected Event

One of the easiest ways to install Java Spring Boot applications on Amazon Elastic Kubernetes Service (EKS) is using Helm as it offers:

package management via a common blue-print called helm chart which is a collection of yaml files bundled together;
templating of dynamic configuration, think of possible Java options (java -X), service name, http port, any other configuration eligible for being externalised in order to make the helm chart re-usable, as well as application secrets;
easy override of dynamic configuration per environment and service. Example increase replica count for a stateless critical service in an environment under higher load;
release management by providing a history of charts installation within the EKS cluster.


➜ helm-charts (main) ✗ helm history spring-boot-admin
REVISION        UPDATED                         STATUS          CHART                   APP VERSION     DESCRIPTION     
1               Mon May  9 11:05:36 2022        superseded      spring-boot-admin-0.0.2 2.6.6           Install complete
2               Thu May 12 13:47:23 2022        superseded      spring-boot-admin-0.0.3 2.6.7           Upgrade complete
3               Thu May 12 15:02:49 2022        deployed        spring-boot-admin-0.0.3 2.6.7           Upgrade complete
➜ helm-charts (main) ✗ helm rollback spring-boot-admin 1
Rollback was a success! Happy Helming!
➜ helm-charts (main) ✗ helm history spring-boot-admin   
REVISION        UPDATED                         STATUS          CHART                   APP VERSION     DESCRIPTION     
1               Mon May  9 11:05:36 2022        superseded      spring-boot-admin-0.0.2 2.6.6           Install complete
2               Thu May 12 13:47:23 2022        superseded      spring-boot-admin-0.0.3 2.6.7           Upgrade complete
3               Thu May 12 15:02:49 2022        superseded      spring-boot-admin-0.0.3 2.6.7           Upgrade complete
4               Thu May 12 16:05:42 2022        deployed        spring-boot-admin-0.0.2 2.6.6           Rollback to 1

Creating the Spring Boot Admin Helm Chart

One of the reasons Helm is so popular is the ease of finding charts in open-sourced repositories. I even started my experiment with this Spring Boot Admin chart but then discovered that my installation use-case required a bit different Kubernetes objects to be created:

ClusterRole instead of a Role
ClusterRoleBinding instead of a RoleBinding
I’ve wanted to customise the ServiceAccount annotations to something in the trend: eks.amazonaws.com/role-arn: arn:aws:iam::awsAccountId:role/projectName/awsEnv/eks-spring-boot-admin
as well as I’ve implemented slight changes to the ConfigMap and Deployment

I’ve kept the container image to the one published into Red Hat’s Quay.io registry: quay.io/evryfs/spring-boot-admin:2.7.5 (now), quay.io/evryfs/spring-boot-admin:2.6.7 (then) as:

I liked that the maintainer is actively providing new versions in alignment with codecentric’s Spring Boot Admin releases
base image is using Java 17, at the moment of writing, just like the Spring Boot applications I was scraping for insights in the Dutch Lottery assignment, as part of the proof-of-concept
it packaged the dependencies I was interested in: Spring Cloud Kubernetes Discovery and Spring Boot Admin Server Starter containing both server and server UI components
wanted to keep it simple while researching its capabilities. It is though recommended to build & publish your own container image, when you desire Spring Boot Admin UI customisations or plan deploying it to production.

Note: One can use helm repo add to expand your list of trusted helm charts repositories and helm search repo to search charts in those repositories.

Updated Helm chart can be found here

Lessons Learnt and Chart Usage

Namespace Bound or Cluster-Wide?

In the Dutch Lottery assignment, I needed to scrape Spring Boot services being installed in two Kubernetes namespaces. With the initial Helm chart config value spring.cloud.kubernetes.discovery.all-namespaces set to true , the installation is discovering only services installed in one namespace: gateway-private. This namespace I’d use, by project convention, for all deployments of services that are not publicly exposed via Kubernetes Ingress or IngressRoute. Thus, it was not discovering any of the Spring Boot applications deployed in gateway-public namespace.

The reason for that is: Roles in Kubernetes are scoped, either bound to a specific namespace or cluster-wide. While a namespace bound Role is a safer practice, my only solution for the Spring Boot Admin Tool to discover all the services I was interested in, while creating a Role instead of a ClusterRole, was to install the Role and RoleBinding Kubernetes RBAC API objects in both namespaces. In which case I also need to install the Kubernetes ServiceAccount in both namespaces, as “User accounts” are for humans and “Service accounts” are for processes in Kubernetes, which run in pods. “User accounts” are intended to be global. “Service accounts” are “namespaced“. As the ServiceAccount is then used in the Deployment, I end up installing the whole Helm chart in both gateway-private and gateway-public namespaces. That would look, in my terminal, something in the trend:


➜ helm-charts(main) ✔ helm install --set namespace=gateway-private \
                     --set multi.namespaced=false \
                     spring-boot-admin-gw-private ./spring-boot-admin --debug

followed by:


➜ helm-charts (main) ✔ helm install --set namespace=gateway-public \
                     --set multi.namespaced=false \
                     spring-boot-admin-gw-public ./spring-boot-admin --debug

Now I have two Spring Boot Admin Kubernetes Service objects I need to port-forward into, use separate local ports, and two Spring Boot Admin UI browser tabs, in order to see insights for all microservices. That is cumbersome, and my requirement was to collect insights from all Spring Boot Java applications in one overview.

I ended up using the alternative solution, which is to create ClusterRole and ClusterRoleBinding Kubernetes Role-based access control (RBAC) API objects and leveraging chart usage:


➜ helm-charts(main) ✔ helm install --set namespace=gateway-private \
                     --set multi.namespaced=true \
                     spring-boot-admin ./spring-boot-admin --debug

Treat your pods according to their needs

Containers are just processes. In Kubernetes not all containers are equal. When you launch a pod in Kubernetes, a really nice and sophisticated piece of software called “scheduler” determines which host should be chosen to run it. If you describe your pod after it is running, you’ll notice a label called QoS Class.

While leveraging Spring Boot Admin helm chart, the QoS class that gets assigned to the pod is “BestEffort”. And yes, that’s the less prioritised class. The Spring Boot Admin pod would be between the first one(s) to be evicted, when host is running low on resources. This might be perfectly fine in a development environment, in production running one replica of the Spring Boot Admin process with “QoS Class: BestEffort” means your insights on your Spring Boot applications may come and go.

Discovery Based on Kubernetes Service Label

In each of the two namespaces, there is more running than just Spring Boot Java applications, so I needed to define a filter for which services to scrape, and ignore the rest.

This can be done using property (see ConfigMap listing below): spring.cloud.kubernetes.discovery.service-labels , a map is required here for label name and value.

# Source: spring-boot-admin/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: RELEASE-NAME-config
  labels:
    app: RELEASE-NAME
  namespace: gateway-private
data:
  application.yml: |-
    spring:
      application:
        name: admin-server
      boot:
        admin:
          context-path: '/admin'
          ui:
            title: 'Gateway Spring Boot Admin'
            brand: 'Gateway Spring Boot Admin'
      server:
        port: 8080
      logging:
        level:
          org.springframework.cloud.kubernetes: TRACE
          de.codecentric.boot.admin.discovery.ApplicationDiscoveryListener: DEBUG
      cloud:
        kubernetes:
          discovery:
            all-namespaces: true
            service-labels:
              type: gateway-base
            catalog-services-watch:
              enabled: true
              catalogServicesWatchDelay: 10000

I’ve named map key type and I added one value: gateway-base. However you can change these to whatever desired. You will do have to update as well the helm chart used to deploy your Java applications to add configured service label.

apiVersion: v1
kind: Service
metadata:
  name: {{ template "getApplicationFullName" . }}
  namespace: {{ .Values.namespace }}
  labels:
    type: {{ .Values.app.type }}

Note: Labels are key/value pairs that are attached to objects, such as Pods and Services, and are different from Kubernetes service types.

NAME                         TYPE        TYPE
eventhub-system              ClusterIP   gateway-base
game-process                 ClusterIP   gateway-base
game-system                  ClusterIP   gateway-base
inlane-process               ClusterIP   gateway-base
marketing-system             ClusterIP   gateway-base
player-process               ClusterIP   gateway-base
player-system                ClusterIP   gateway-base
responsible-gaming-system    ClusterIP   gateway-base
salesforce-sync-system       ClusterIP   gateway-base
spring-boot-admin            ClusterIP
subscription-process         ClusterIP   gateway-base
subscription-sales-process   ClusterIP   gateway-base
traefik-ingress-private      ClusterIP

As you can see in the Wallboard , there are no spring-boot-admin or traefik-ingress-private services being discovered.

UI Customisations

Note: It has been chosen not to expose the Spring Boot Admin UI externally. Instead one can use port forwarding inside the spring-boot-admin service to access the UI.

First time, I have installed the Helm chart, I was noticing an “unspecified” label in Spring Boot Admin UI “Wallboard” (image above), but also in the “Applications” overview feature. That is when your Spring Boot Application has no “build version” in the JSON output of “/actuator/info” endpoint.
As the project is leveraging Spring Boot improved image creation technique called layered jar, including information as build version or time would result in layer recreation, which might slow down the CI & CD pipeline.

So what if you would like to brand your installation of Spring Boot Admin UI a bit? Well it is possible!
Just a few examples of possible overrides in codecentric’s Spring Boot Admin:

“Page-Title” to be shown top navigation bar and overwriting property spring.boot.admin.ui.title via corresponding chart template value: ui.title
“Brand” to be shown in the navigation bar as brand image and changing default value of property spring.boot.admin.ui.brand. As this one is an image asset, it makes sense to build a custom Docker image that includes the image asset.

Below screenshot exemplifies both the "Page-Title" and "Build version" customisations in "Applications" screen.

Useful Features of the Spring Boot Admin

A former colleague wrote a Trifork blog post making a nice inventory of useful features in Spring Boot Admin. The features he describes in his article, section “The top cool features we like and use most” were relevant as well for the Dutch Lottery assignment, with the exception of the database migrations. This post is more focused on Helm and how I deployed Spring Boot Admin in EKS, nonetheless I would like to extend on that blog post with few other unmentioned useful features:

hunt down memory (even possible leak) issues in any of your Spring Boot Java application. You can use JDK’s Flight Recorder or VisualVM and serve it a Heap Dump you can download to your machine via Spring Boot Admin UI. As a Heap Dump contains sensitive data, you will need write, port-forward privileges in a production grade cluster, and knowledge when your Java service will be under high load, and possibly misbehaving.
list Scheduled Tasks, Caches, Circuit Breakers, database connection pool details, if your service integrates with any.
display application’s Request Mappings. While I prefer Swagger UI as a Rest API Documentation Tool, not all Spring Boot Java applications abide by the “REpresentational State Transfer” as an architectural style for distributed hypermedia systems,. Nonetheless they might provide some HTTP based APIs and having a list of those endpoints might offer insights on what the service does.
Notifications based on the “Event Journal” (see image below). In Kubernetes is pretty common for services to come and go, however if you have a scenario of a critical stateful deployment with only one replica, and you want to be notified of its lifecycle events, it’s possible to integrate Spring Boot Admin with monitoring tools like: Slack, PagerDuty, OpsGenie.