Using Watch with the Kubernetes API

1. Introduction

In this tutorial, we'll continue to explore the Java Kubernetes API. This time, we'll show how to use Watches to efficiently monitor cluster events.

2. What are Kubernetes Watches?

In our previous articles covering the Kubernetes API, we've shown how to recover information about a given resource or a collection of them. This is fine if all we wanted was to get the state of those resources at a given point on time. However, given that Kubernetes clusters are highly dynamic in nature, this is usually not enough.

Most often, we also want to monitor those resources and track events as they happen. For instance, we might be interested in tracking pod life cycle events or deployment status changes. While we could use polling, this approach would suffer from a few limitations. Firstly, it would not scale well as the number of resources to monitor increases. Secondly, we risk losing events that happen to occur between polling cycles.

To address those issues, Kubernetes has the concept of Watches, which is available for all resource collection API calls through the watch query parameter. When its value is false or omitted, the GET operation behaves as usual: the server processes the request and returns a list of resource instances that match the given criteria. However, passing watch=true changes its behavior dramatically:

• The response now consists of a series of modification events, containing the type of modification and the affected object
• The connection will be kept open after sending the initial batch of events, using a technique called long polling

3. Creating a Watch

The Java Kubernetes API support Watches through the Watch class, which has a single static method: createWatch. This method takes three arguments:

• An ApiClient, which handles handles actual REST calls to the Kubernetes API server
• A Call instance describing the resource collection to watch
• A TypeToken with the expected resource type

We create a Call instance from any of the xxxApi classes available in the library using one of their listXXXCall() methods. For instance, to create a Watch that detects Pod events, we'd use listPodForAllNamespacesCall():

CoreV1Api api = new CoreV1Api(client);
Call call = api.listPodForAllNamespacesCall(null, null, null, null, null, null, null, null, 10, true, null);
Watch<V1Pod> watch = Watch.createWatch(
  client, 
  call, 
  new TypeToken<Response<V1Pod>>(){}.getType()));

Here, we use null for most parameters, meaning “use the default value”, with just two exceptions: timeout and watch. The latter must be set to true for a watch call. Otherwise, this would be a regular rest call. The timeout, in this case, works as the watch “time-to-live”, meaning that the server will stop sending events and terminate the connection once it expires.

Finding a good value for the timeout parameter, which is expressed in seconds, requires some trial-and-error, as it depends on the exact requirements of the client application. Also, it is important to check your Kubernetes cluster configuration. Usually, there's a hard limit of 5 minutes for watches, so passing anything beyond that will not have the desired effect.

4. Receiving Events

Taking a closer look at the Watch class, we can see that it implements both Iterator and Iterable from the standard JRE, so we can use the value returned from createWatch() in for-each or hasNext()-next() loops:

for (Response<V1Pod> event : watch) {
    V1Pod pod = event.object;
    V1ObjectMeta meta = pod.getMetadata();
    switch (event.type) {
    case "ADDED":
    case "MODIFIED":
    case "DELETED":
        // ... process pod data
        break;
    default:
        log.warn("Unknown event type: {}", event.type);
    }
}

The type field of each event tells us what kind of event happened to the object – a Pod in our case. Once we consume all events, we must do a new call to Watch.createWatch() to start receiving events again. In the example code, we surround the Watch creation and result processing in a while loop. Other approaches are also possible, such as using an ExecutorService or similar to receive updates in the background.

5. Using Resource Versions and Bookmarks

A problem with the code above is the fact that every time we create a new Watch, there's an initial event stream with all existing resource instances of the given kind. This happens because the server assumes that we don't have any previous information about them, so it just sends them all.

However, doing so defeats the purpose of processing events efficiently, as we only need new events after the initial load. To prevent receiving all data again, the watch mechanism supports two additional concepts: resource versions and bookmarks.

5.1. Resource Versions

Every resource in Kubernetes contains a resourceVersion field in its metadata, which is just an opaque string set by the server every time something changes. Moreover, since a resource collection is also a resource, there's a resourceVersion associated with it. As new resources are added, removed, and/or modified from a collection, this field will change accordingly.

When we make an API call that returns a collection and includes the resourceVersion parameter, the server will use its value as a “starting point” for the query. For Watch API calls, this means that only events that happened after the time where the informed version was created will be included.

But, how do we get a resourceVersion to include in our calls? Simple: we just do an initial synchronization call to retrieve the initial list of resources, which includes the collection's resourceVersion, and then use it in subsequent Watch calls:

String resourceVersion = null;
while (true) {
    if (resourceVersion == null) {
        V1PodList podList = api.listPodForAllNamespaces(null, null, null, null, null, "false",
          resourceVersion, null, 10, null);
        resourceVersion = podList.getMetadata().getResourceVersion();
    }
    try (Watch<V1Pod> watch = Watch.createWatch(
      client,
      api.listPodForAllNamespacesCall(null, null, null, null, null, "false",
        resourceVersion, null, 10, true, null),
      new TypeToken<Response<V1Pod>>(){}.getType())) {
        
        for (Response<V1Pod> event : watch) {
            // ... process events
        }
    } catch (ApiException ex) {
        if (ex.getCode() == 504 || ex.getCode() == 410) {
            resourceVersion = extractResourceVersionFromException(ex);
        }
        else {
            resourceVersion = null;
        }
    }
}

The exception handling code, in this case, is rather important. Kubernetes servers will return a 504 or 410 error code when, for some reason, the requested resourceVersion doesn't exist. In this case, the returned message usually contains the current version. Unfortunately, this information doesn't come in any structured way but rather as part of the error message itself.

The extraction code (a.k.a. ugly hack) uses a regular expression for this intent, but since error messages tend to be implementation-dependent, the code falls back to a null value. By doing so, the main loop goes back to its starting point, recovering a fresh list with a new resourceVersion and resuming watch operations.

Anyhow, even with this caveat, the key point is that now the event list will not start from scratch on every watch.

5.2. Bookmarks

Bookmarks are an optional feature that enables a special BOOKMARK event on events streams returned from a Watch call. This event contains in its metadata a resourceVersion value that we can use in subsequent Watch calls as a new starting point.

As this is an opt-in feature, we must explicitly enable it by passing true to allowWatchBookmarks on API calls. This option is valid only when creating a Watch and ignored otherwise. Also, a server may ignore it completely, so clients should not rely on receiving those events at all.

When comparing with the previous approach using resourceVersion alone, bookmarks allow us to mostly get away with a costly synchronization call:

String resourceVersion = null;
while (true) {
    // Get a fresh list whenever we need to resync
    if (resourceVersion == null) {
        V1PodList podList = api.listPodForAllNamespaces(true, null, null, null, null,
          "false", resourceVersion, null, null, null);
        resourceVersion = podList.getMetadata().getResourceVersion();
    }
    while (true) {
        try (Watch<V1Pod> watch = Watch.createWatch(
          client,
          api.listPodForAllNamespacesCall(true, null, null, null, null, 
            "false", resourceVersion, null, 10, true, null),
          new TypeToken<Response<V1Pod>>(){}.getType())) {
              for (Response<V1Pod> event : watch) {
                  V1Pod pod = event.object;
                  V1ObjectMeta meta = pod.getMetadata();
                  switch (event.type) {
                      case "BOOKMARK":
                          resourceVersion = meta.getResourceVersion();
                          break;
                      case "ADDED":
                      case "MODIFIED":
                      case "DELETED":
                          // ... event processing omitted
                          break;
                      default:
                          log.warn("Unknown event type: {}", event.type);
                  }
              }
          }
        } catch (ApiException ex) {
            resourceVersion = null;
            break;
        }
    }
}

Here, we only need to get the full list on the first pass and whenever we get an ApiException in the inner loop. Notice that BOOKMARK events have the same object type as other events, so we don't need any special casting here. However, the only field we care about is the resourceVersion, which we save for the next Watch call.

6. Conclusion

In this article, we've covered different ways to create Kubernetes Watches using the Java API client. As usual, the full source code of the examples can be found over on GitHub.