Self-Healing Applications with Kubernetes and Spring Boot

1. Introduction

In this tutorial, we’re going to talk about Kubernetes‘s probes and demonstrate how we can leverage Actuator‘s HealthIndicator to have an accurate view of our application’s state.

For the purpose of this tutorial, we’re going to assume some pre-existing experience with Spring Boot Actuator, Kubernetes, and Docker.

2. Kubernetes Probes

Kubernetes defines two different probes that we can use to periodically check if everything is working as expected: liveness and readiness.

2.1. Liveness and Readiness

With Liveness and Readiness probes, Kubelet can act as soon as it detects that something’s off and minimize the downtime of our application.

Both are configured the same way, but they have different semantics and Kubelet performs different actions depending on which one is triggered:

• Readiness – Readiness verifies if our Pod is ready to start receiving traffic. Our Pod is ready when all of its containers are ready
• Liveness – Contrary to readiness, liveness checks if our Pod should be restarted. It can pick up use cases where our application is running but is in a state where it’s unable to make progress; for example, it’s in deadlock

We configure both probe types at the container level:

apiVersion: v1
kind: Pod
metadata:
  name: goproxy
  labels:
    app: goproxy
spec:
  containers:
  - name: goproxy
    image: k8s.gcr.io/goproxy:0.1
    ports:
    - containerPort: 8080
    readinessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 1
      successThreshold: 1
    livenessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 20
      timeoutSeconds: 2
      failureThreshold: 1
      successThreshold: 1

There are a number of fields that we can configure to more precisely control the behavior of our probes:

• initialDelaySeconds – After creating the container, wait n seconds before initiating the probe
• periodSeconds – How often this probe should be run, defaulting to 10 seconds; the minimum is 1 second
• timeoutSeconds – How long we wait before timing out the probe, defaulting to 1 second; the minimum is again 1 second
• failureThreshold – Try n times before giving up. In the case of readiness, our pod will be marked as not ready, whereas giving up in case of liveness means restarting the Pod. The default here is 3 failures, with the minimum being 1
• successThreshold – This is the minimum number of consecutive successes for the probe to be considered successful after having failed. It defaults to 1 success and its minimum is 1 as well

In this case, we opted for a tcp probe, however, there are other types of probes we can use, too.

2.2. Probe Types

Depending on our use case, one probe type may prove more useful than the other. For example, if our container is a web server, using an http probe could be more reliable than a tcp probe.

Luckily, Kubernetes has three different types of probes that we can use:

• exec – Executes bash instructions in our container. For example, check that a specific file exists. If the instruction returns a failure code, the probe fails
• tcpSocket – Tries to establish a tcp connection to the container, using the specified port. If it fails to establish a connection, the probe fails
• httpGet – Sends an HTTP GET request to the server that is running in the container and listening on the specified port. Any code greater than or equal to 200 and less than 400 indicates success

It’s important to note that HTTP probes have additional fields, besides the ones we mentioned earlier:

• host – Hostname to connect to, defaults to our pod’s IP
• scheme – Scheme that should be used to connect, HTTP or HTTPS, with the default being HTTP
• path – The path to access on the web server
• httpHeaders – Custom headers to set in the request
• port – Name or number of the port to access in the container

3. Spring Actuator and Kubernetes Self-Healing Capabilities

Now that we have a general idea on how Kubernetes is able to detect if our application is in a broken state, let’s see how we can take advantage of Spring’s Actuator to keep a closer eye not only on our application but also on its dependencies!

For the purpose of these examples, we’re going to rely on Minikube.

3.1. Actuator and its HealthIndicators

Considering that Spring has a number of HealthIndicators ready to use, reflecting the state of some of our application’s dependencies over Kubernetes‘s probes is as simple as adding the Actuator dependency to our pom.xml:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

3.2. Liveness Example

Let’s begin with an application that will boot up normally and, after 30 seconds will transition to a broken state.

We’re going to emulate a broken state by creating a HealthIndicator that verifies if a boolean variable is true. We’ll initialize the variable to true, and then we’ll schedule a task to change it to false after 30 seconds:

@Component
public class CustomHealthIndicator implements HealthIndicator {

    private boolean isHealthy = true;

    public CustomHealthIndicator() {
        ScheduledExecutorService scheduled =
          Executors.newSingleThreadScheduledExecutor();
        scheduled.schedule(() -> {
            isHealthy = false;
        }, 30, TimeUnit.SECONDS);
    }

    @Override
    public Health health() {
        return isHealthy ? Health.up().build() : Health.down().build();
    }
}

With our HealthIndicator in place, we need to dockerize our application:

FROM openjdk:8-jdk-alpine
RUN mkdir -p /usr/opt/service
COPY target/*.jar /usr/opt/service/service.jar
EXPOSE 8080
ENTRYPOINT exec java -jar /usr/opt/service/service.jar

Next, we create our Kubernetes template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: liveness-example
spec:
  ...
    spec:
      containers:
      - name: liveness-example
        image: dbdock/liveness-example:1.0.0
        ...
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          timeoutSeconds: 2
          periodSeconds: 3
          failureThreshold: 1
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 20
          timeoutSeconds: 2
          periodSeconds: 8
          failureThreshold: 1

We’re using an httpGet probe pointing to Actuator’s health endpoint. Any change to our application state (and its dependencies) will be reflected on the healthiness of our deployment.

After deploying our application to Kubernetes, we’ll be able to see both probes in action: after approximately 30 seconds, our Pod will be marked as unready and removed from rotation; a few seconds later, the Pod is restarted.

We can see the events of our Pod executing kubectl describe pod liveness-example:

Warning  Unhealthy 3s (x2 over 7s)   kubelet, minikube  Readiness probe failed: HTTP probe failed ...
Warning  Unhealthy 1s                kubelet, minikube  Liveness probe failed: HTTP probe failed ...
Normal   Killing   0s                kubelet, minikube  Killing container with id ...

3.3. Readiness Example

In the previous example, we saw how we could use a HealthIndicator to reflect our application’s state on the healthiness of a Kubernetes deployment.

Let’s use it on a different use case: suppose that our application needs a bit of time before it’s able to receive traffic. For example, it needs to load a file into memory and validate its content.

This is a good example of when we can take advantage of a readiness probe.

Let’s modify the HealthIndicator and Kubernetes template from the previous example and adapt them to this use case:

@Component
public class CustomHealthIndicator implements HealthIndicator {

    private boolean isHealthy = false;

    public CustomHealthIndicator() {
        ScheduledExecutorService scheduled =
          Executors.newSingleThreadScheduledExecutor();
        scheduled.schedule(() -> {
            isHealthy = true;
        }, 40, TimeUnit.SECONDS);
    }

    @Override
    public Health health() {
        return isHealthy ? Health.up().build() : Health.down().build();
    }
}

We initialize the variable to false, and after 40 seconds, a task will execute and set it to true.

Next, we dockerize and deploy our application using the following template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: readiness-example
spec:
  ...
    spec:
      containers:
      - name: readiness-example
        image: dbdock/readiness-example:1.0.0
        ...
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 40
          timeoutSeconds: 2
          periodSeconds: 3
          failureThreshold: 2
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 100
          timeoutSeconds: 2
          periodSeconds: 8
          failureThreshold: 1

While similar, there are a few changes in the probes configuration that we need to point out:

• Since we know that our application needs around 40 seconds to become ready to receive traffic, we increased the initialDelaySeconds of our readiness probe to 40 seconds
• Similarly, we increased the initialDelaySeconds of our liveness probe to 100 seconds to avoid being prematurely killed by Kubernetes

If it still hasn’t finished after 40 seconds, it still has around 60 seconds to finish. After that, our liveness probe will kick in and restart the Pod.

4. Conclusion

In this article, we talked about Kubernetes probes and how we can use Spring’s Actuator to improve our application’s health monitoring.

The full implementation of these examples can be found over on Github.