1. Overview

In this tutorial, we’ll explore the Java Apache Commons CLI library. It’s a framework that empowers developers by helping them build the command line interface (CLI) of existing software tools in an efficient and standard way.

The library can speed up the development of CLIs with its support for defining the CLI options and basic validation of them. It helps parse the command line arguments and their values. Finally, the argument values can be passed to the underlying services implementing the tool.

Notably, the Apache Commons CLI library is also used in several of Apache’s products, including Kafka, Maven, Ant, and Tomcat.

We’ll discuss a few important classes from Apache Commons CLI and then use them in sample programs to showcase its capabilities.

2. Key Concerns of a CLI

CLIs give an edge to tools by helping automate a series of tasks relevant to their domain. Moreover, in today’s world, it’s unimaginable for DevOps engineers to work without CLIs.

Apart from the challenge of the underlying implementation of the tool, all CLIs need to handle some basic requirements:

• Parse command line arguments, extract argument values, and pass them to the underlying services
• Display help information with a certain format
• Display version
• Handle missing required options
• Handle unknown options
• Handle mutually exclusive options

3. Important Classes

Let’s take a look at the important classes of the Apache Commons CLI library:

The classes Option, OptionGroup, and Options help define a CLI. The definitions of all the CLI options are wrapped into the Options class. The parse() method of the CommandLineParser class uses the Options class to parse the command line. In case of any deviation, appropriate exceptions are thrown by the parse() method. After parsing, the CommandLine class can be probed further to extract the values of the CLI options, if any.

Finally, the extracted values can be passed to the underlying services implementing the CLI tool.

Similar to the parse() method in the CommandLineParser class, the HelpFormatter also uses the Options Class to display the help text of a CLI tool.

4. Implementation

Let’s explore more about the Apache Commons CLI library classes and understand how they help create a CLI tool consistently and quickly.

4.1. Prerequisite Maven Dependency

Firstly, let’s add the necessary Maven dependency in the pom.xml file:

<dependency>
    <groupId>commons-cli</groupId>
    <artifactId>commons-cli</artifactId>
    <version>1.6.0</version>
</dependency>

4.2. Define, Parse, and Probe Command Line Arguments

Consider the command to connect to a PostgreSQL database using its psql CLI:

psql -h PGSERVER -U postgres -d empDB

Alternatively:

psql --host PGSERVER -username postgres -dbName empDB

Both commands require input parameters for the database server host, username, and database name. The first command uses short option names, while the second uses long option names. The username and dbName are required options, whereas the host is optional. If the host is missing, then by default, we consider the localhost as the host value.

Now, let’s define, parse, and probe the command line arguments:

@Test
void whenCliOptionProvided_thenParseAndExtractOptionAndArgumentValues() throws ParseException {
    Options options = new Options();
    Option hostOption = createOption("h", "host", "HOST", "Database server host", false);
    Option userNameOption = createOption("U", "username", "USERNAME", "Database user name", true);
    Option dbNameOption = createOption("d", "dbName", "DBNAME", "Database name to connect to", true);
    options.addOption(hostOption)
      .addOption(dbNameOption)
      .addOption(userNameOption);
    String[] commandWithShortNameOptions = new String[] { "-h", "PGSERVER", "-U", "postgres", "-d", "empDB" };
    parseThenProcessCommand(options, commandWithShortNameOptions, "h", "U", "d" );
    String[] commandWithLongNameOptions =  new String[] { "--username", "postgres", "--dbName", "empDB" };
    parseThenProcessCommand(options, commandWithShortNameOptions, "host", "username", "dbName" );
}

To define the options in the command, we created the Option objects corresponding to each of the input options by calling the method createOption():

Option createOption(String shortName, String longName, String argName, String description, boolean required) {
    return Option.builder(shortName)
      .longOpt(longName)
      .argName(argName)
      .desc(description)
      .hasArg()
      .required(required)
      .build();
}

We used the Option.Builder class to set the short name, long name, argument name, and description of the input options in the CLI. Additionally, we consider the -U and -d options defined earlier as mandatory with the help of the required() method in the builder class.

Finally, we pass the arguments with short name options and long name options, respectively, to the method parseThenProcessCommand():

void parseThenProcessCommand(Options options, String[] commandArgs, String hostOption,
    String usernameOption, String dbNameOption) throws ParseException {
    CommandLineParser commandLineParser = new DefaultParser();
    CommandLine commandLine = commandLineParser.parse(options, commandArgs);
    String hostname = commandLine.hasOption("h") ? commandLine.getOptionValue(hostOption) : "localhost";
    String username = commandLine.getOptionValue(usernameOption);
    String dbName = commandLine.getOptionValue(dbNameOption);
    if (commandLine.hasOption("h")) {
        assertEquals("PGSERVER", hostname);
    } else {
        assertEquals("localhost", hostname);
    }
    assertEquals("postgres", userName);
    assertEquals("empDB", dbName);
    createConnection(hostname, username, dbName);
}

Interestingly, the method can handle both commands with short names and long names of options. The CommandLineParser class parses the arguments, and then we retrieve their values by calling the getOptionValue() method of the CommandLine object. Since the host is optional, we call the hasOption() method in the class CommandLine to probe if it’s present. If it’s not present, we replace its value with the default localhost.

Finally, we pass on the values to the underlying services by calling the method createConnection().

4.3. Handle Missing Mandatory Options

In most CLIs, an error should be displayed when a mandatory option is missing. Assume that the mandatory host option is missing in the psql command:

psql -h PGSERVER -U postgres

Let’s see how to handle this:

@Test
void whenMandatoryOptionMissing_thenThrowMissingOptionException() {
    Options options = createOptions();
    String[] commandWithMissingMandatoryOption = new String[]{"-h", "PGSERVER", "-U", "postgres"};
    CommandLineParser commandLineParser = new DefaultParser();
    assertThrows(MissingOptionException.class, () -> {
        try {
            CommandLine commandLine = commandLineParser.parse(options, commandWithMissingMandatoryOption);
        } catch (ParseException e) {
            assertTrue(e instanceof MissingOptionException);
            handleException(new RuntimeException(e));
            throw e;
        }
    });
}

When we invoke the parse() method in the CommandLineParser class, it throws MissingOptionException indicating the absence of the required option d. Following this, we call a method handleException() to manage the exception.

Suppose the –d option is present, but its argument is missing:

psql -h PGSERVER -U postgres -d

Now, let’s see how to handle this:

@Test
void whenOptionArgumentIsMissing_thenThrowMissingArgumentException() {
    Options options = createOptions();
    String[] commandWithOptionArgumentOption = new String[]{"-h", "PGSERVER", "-U", "postgres", "-d"};
    CommandLineParser commandLineParser = new DefaultParser();
    assertThrows(MissingArgumentException.class, () -> {
        try {
            CommandLine commandLine = commandLineParser.parse(options, commandWithOptionArgumentOption);
        } catch (ParseException e) {
            assertTrue(e instanceof MissingArgumentException);
            handleException(new RuntimeException(e));
            throw e;
        }
    });
}

When the parse() method is invoked on the CommandLineParser, a MissingArgumentException is thrown due to the absence of an argument next to the -d option. Further down, we call handleException() to manage the exception.

4.4. Handle Unrecognized Options

Sometimes, while running commands, we provide unrecognized options:

psql -h PGSERVER -U postgres -d empDB -y

We provided an incorrect non-existent -y option. Let’s see how we handle it in the code:

@Test
void whenUnrecognizedOptionProvided_thenThrowUnrecognizedOptionException() {
    Options options = createOptions();
    String[] commandWithIncorrectOption = new String[]{"-h", "PGSERVER", "-U", "postgres", "-d", "empDB", "-y"};
    CommandLineParser commandLineParser = new DefaultParser();
    assertThrows(UnrecognizedOptionException.class, () -> {
        try {
            CommandLine commandLine = commandLineParser.parse(options, commandWithIncorrectOption);
        } catch (ParseException e) {
            assertTrue(e instanceof UnrecognizedOptionException);
            handleException(new RuntimeException(e));
            throw e;
        }
    });
}

The parse() method throws UnrecogniedOptionException when it encounters the unknown -y option. Later, we call handleException() to manage the runtime exception.

4.5. Handle Mutually Exclusive Options

Consider the command using cp to copy files in Unix platforms:

cp -i -f file1 file2

The -i option prompts before overwriting files; however, the -f option overwrites the files without prompting. Both these options are conflicting and hence should not be used together.

Let’s try implementing this validation:

@Test
void whenMutuallyExclusiveOptionsProvidedTogether_thenThrowAlreadySelectedException() {
    Option interactiveOption = new Option("i", false, "Prompts the user before overwriting the existing files");
    Option forceOption = new Option("f", false, "Overwrites the existing files without prompting");
    OptionGroup optionGroup = new OptionGroup();
    optionGroup.addOption(interactiveOption)
      .addOption(forceOption);
    Options options = new Options();
    options.addOptionGroup(optionGroup);
    String[] commandWithConflictingOptions = new String[]{"cp", "-i", "-f", "file1", "file2"};
    CommandLineParser commandLineParser = new DefaultParser();
    assertThrows(AlreadySelectedException.class, () -> {
        try {
            CommandLine commandLine = commandLineParser.parse(options, commandWithConflictingOptions);
        } catch (ParseException e) {
            assertTrue(e instanceof AlreadySelectedException);
            handleException(new RuntimeException(e));
            throw e;
        }
    });
}

First, we created the relevant Option objects using its constructor instead of the Option.Builder class. This is also another way of instantiating the Option class.

The OptionGroup class helps group mutually exclusive options. Hence, we added the two options to the OptionGroup object. Then, we added the OptionGroup object to the Options object. Finally, when we called the parse() method on CommandLineParser class, it raised AlreadySelectedException, indicating the conflicting options.

4.6. Display Help Text

Formatting the help text and displaying it on the terminal is a common concern for all the CLI tools. Hence, the Apache Commons CLI addresses this as well with the help of the HelpFormatter class.

Let’s consider the psql CLI as an example:

@Test
void whenNeedHelp_thenPrintHelp() {
    HelpFormatter helpFormatter = new HelpFormatter();
    Options options = createOptions();
    options.addOption("?", "help", false, "Display help information");
    helpFormatter.printHelp("psql -U username -h host -d empDB", options);
}

The printHelp() method in the class HelpFormatter uses the Options object that holds the definition of the CLI to display the help text. The first argument of the method generates the usage text of the CLI shown at the top.

Let’s look at the output generated by the HelpFormatter class:

usage: psql -U username -h host -d empDB
 -?,--help                  Display help information
 -d,--dbName <DBNAME>       Database name to connect to
 -h,--host <HOST>           Database server host
 -U,--username <USERNAME>   Database user name

5. Conclusion

In this article, we discussed the Apache Commons CLI library’s ability to help create CLIs quickly and efficiently in a standardized way. Moreover, the library is concise and easy to understand.

Notably, there are other libraries like JCommander, Airline, and Picocli that are equally efficient and worth exploring. Unlike Apache Commons CLI, all of them support annotation as well.

As usual, the code used is available over on GitHub.

       

1. Overview

We often solve mathematical problems in programming. Determining whether a number is a happy number is an interesting task.

In this tutorial, we’ll understand how a happy number is defined and explore how to implement a Java program to check whether a given number is a happy number.

2. Understanding the Happy Number

A happy number reaches 1 through repeated replacement by the sum of the squares of its digits. Conversely, a non-happy (sad) number gets caught in an infinite cycle without ever reaching 1.

As usual, a couple of examples can help us quickly understand the happy number definition:

Given number: 19
 19 -> 1^2 + 9^2 = 82
 82 -> 8^2 + 2^2 = 68
 68 -> 6^2 + 8^2 = 100
100 -> 1^2 + 0^2 + 0^2 = 1
  1

As the example above shows, we reached 1 in the end for the input number 19. Therefore, 19 is a happy number. Similarly, more happy number examples are 7, 10, 13, 23, etc.

However, if the input is 15, we’ll never reach 1:

Given number: 15                  
                                  
       15 -> 1^2 + 5^2 = 26       
       26 -> 2^2 + 6^2 = 40       
       40 -> 4^2 + 0^2 = 16       
+--->  16 -> 1^2 + 6^2 = 37       
|      37 -> 3^2 + 7^2 = 58       
|      58 -> 5^2 + 8^2 = 89       
|      89 -> 8^2 + 9^2 = 145      
|     145 -> 1^2 + 4^2 + 5^2 = 42 
|      42 -> 4^2 + 2^2 = 20       
|      20 -> 2^2 + 0^2 = 4        
|       4 -> 4^2 = 16             
+------16                         
                                  

As we can see, the process loops infinitely between 16 and 4 and never reaches 1. So, 15 is a sad number.

Following this rule, we can find more sad numbers, such as 4, 6, 11, 20, etc.

3. Implementing the Check Method

Now that we understand how a happy number is defined, let’s implement Java methods to check whether a given number is a happy number.

A number is sad if the sequence, which each sum of the digits’ square forms, contains a loop. In other words, given a number, if one step’s calculation result already exists in the sequence, it’s a sad number.

We can utilize a HashSet data structure to implement this logic in Java. This allows us to store each step’s result and efficiently check whether a newly calculated result already exists in the set:

class HappyNumberDecider {
    public static boolean isHappyNumber(int n) {
        Set<Integer> checkedNumbers = new HashSet<>();
        while (true) {
            n = sumDigitsSquare(n);
            if (n == 1) {
                return true;
            }
            if (checkedNumbers.contains(n)) {
                return false;
            }
            checkedNumbers.add(n);
        }
    }
    // ...
}

As the code above shows, the sumDigitsSquare() method performs the actual calculation and returns the result:

private static int sumDigitsSquare(int n) {
    int squareSum = 0;
    while (n != 0) {
        squareSum += (n % 10) * (n % 10);
        n /= 10;
    }
    return squareSum;
}

Next, let’s create a test to verify whether our isHappyNumber() method reports the expected result for different inputs:

assertTrue(HappyNumberDecider.isHappyNumber(7));
assertTrue(HappyNumberDecider.isHappyNumber(10));
assertTrue(HappyNumberDecider.isHappyNumber(13));
assertTrue(HappyNumberDecider.isHappyNumber(19));
assertTrue(HappyNumberDecider.isHappyNumber(23));
 
assertFalse(HappyNumberDecider.isHappyNumber(4));
assertFalse(HappyNumberDecider.isHappyNumber(6));
assertFalse(HappyNumberDecider.isHappyNumber(11));
assertFalse(HappyNumberDecider.isHappyNumber(15));
assertFalse(HappyNumberDecider.isHappyNumber(20));

4. Performance Analysis

First, let’s analyze the solution’s time complexity.

Let’s say we have a number n with k digits. So, we need k iterations to calculate the sum of digit squares. Further, since k = log n (logarithm at base 10), O(log n) is the time complexity for calculating the first result. Let’s call it result1. As the maximum digit is 9, the maximum value of result1 is 9^2 * log n.

If result1 isn’t 1, we must repeat calling sumDigitsSquare(). Then, the time complexity so far is O(log n) + O(log (9^2 * log n)) = O(log n) + O(log 81 + log log n). After removing the constant part, we get O(log n) + O(log log n).

Therefore, our total time complexity becomes O(log n) + O(log log n) + O(log log log n) + O(log log log log n) + … until a result reaches 1 or we detect a loop. As log n is the dominant part in this expression, the solution’s time complexity is O(log n).

Before we move to the space complexity analysis, let’s list the largest number n with k digits and the corresponding result1:

k     Largest n        result1
1     9                81
2     99               162
3     999              243
4     9999             324
5     99999            405
6     999999           486
7     9999999          567
8     99999999         648
9     999999999        729
10    9999999999       810
11    99999999999      891
12    999999999999     972
13    9999999999999    1053
       ...
1m    9..a million..9  81000000

As we can see, given the number n with more than three digits, repeatedly applying the sumDigitsSquare() operation can rapidly decrease n to a 3-digit number within a few steps.

For example, when k=1m, n is much larger than Java’s Long.MAX_VALUE. It only takes two steps to reach a number with less than three digits: 9..(1m)..9 -> 81000000 (9^2 * 1m = 81000000) -> 65 (8^2 + 1^2 = 65)

Hence, within Java’s int or long range, it’s reasonable to assume that n requires a constant C steps to reach a number with three or fewer digits. Consequently, our HashSet holds a maximum of C+243 results, yielding a space complexity of O(1).

While the space complexity of this method is O(1), it still requires space to store results for detecting cycles.

Let’s now aim to refine the solution to identify happy numbers without using extra space.

5. Improving the isHappyNumber() Method

Floyd’s cycle detection algorithm can detect cycles in a sequence. For example, we can apply this algorithm to check if a LinkedList contains a circle.

The idea is to maintain two-pointers, slow and fast. slow gets updated one step at a time, and fast is replaced every two steps. If they meet at 1, the given number is happy. Otherwise, if they meet but the value isn’t 1, a cycle is detected. Thus, the given number is sad.

Next, let’s implement the slow-fast logic in Java:

public static boolean isHappyNumberFloyd(int n) {
    int slow = n;
    int fast = n;
    do {
        slow = sumDigitsSquare(slow);
        fast = sumDigitsSquare(sumDigitsSquare(fast));
    } while (slow != fast);
 
    return slow == 1;
}

Let’s test the isHappyNumberFloyd() method with the same inputs:

assertTrue(HappyNumberDecider.isHappyNumberFloyd(7));
assertTrue(HappyNumberDecider.isHappyNumberFloyd(10));
assertTrue(HappyNumberDecider.isHappyNumberFloyd(13));
assertTrue(HappyNumberDecider.isHappyNumberFloyd(19));
assertTrue(HappyNumberDecider.isHappyNumberFloyd(23));
                                                   
assertFalse(HappyNumberDecider.isHappyNumberFloyd(4));
assertFalse(HappyNumberDecider.isHappyNumberFloyd(6));
assertFalse(HappyNumberDecider.isHappyNumberFloyd(11));
assertFalse(HappyNumberDecider.isHappyNumberFloyd(15));
assertFalse(HappyNumberDecider.isHappyNumberFloyd(20));

Finally, this solution’s time complexity remains O(log n). Since it requires no extra space, its space complexity is O(1).

6. Conclusion

In this article, we learned the concept of a happy number and implemented Java methods to determine whether a given number is happy.

As always, the complete source code for the examples is available over on GitHub.

       

1. Overview

In this tutorial, we’ll discuss a potent event streaming platform called Redpanda. It’s a competition to the de facto industry streaming platform Kafka and, interestingly, it’s also compatible with the Kafka APIs.

We’ll look at the key components, features, and use cases of Redpanda, create Java programs for publishing messages to Redpanda topics, and then read messages from it.

2. Redpanda vs. Kafka

Since the makers of Redpanda are claiming to be competition to Kafka, let’s compare them on a few of the important factors:

3. Redpanda Architecture

Redpanda’s architecture is not only simple but extremely easy to grasp. Interestingly, it has a single binary installation package that’s easy to install. This gives the developers a quick headstart, hence the reason for its popularity. Moreover, it delivers an extremely high-performing streaming platform with a great throughput.

3.1. Key Components and Features

Let’s dive into the key components and features of Redpanda that make it extremely robust and performant:

The control plane supports Kafka API for managing the broker, creating messaging topics, publishing and consuming messages, and much more. Hence, the legacy systems relying on Kafka can migrate to Redpanda with significantly less effort. However, there’s a different set of Admin APIs for managing and configuring the Redpanda cluster.

Redpanda supports tiered storage. This means we can configure it to offload or archive its data logs from its local cache to a cheaper object storage in the cloud. Also, on-demand from the consumers, the data is moved back to the local cache from the remote object storage in real time.

Redpanda has a Raft consensus algorithm implementation layer that replicates topic-partition data across its nodes. This feature prevents data loss in the event of a failure. Naturally, it guarantees high data safety and fault tolerance.

Redpanda has robust authentication and authorization support. It can authenticate external users and applications using methods such as SASL, OAuth, OpenID Connect (OIDC), basic authentication, Kerberos, and others. Additionally, it enables fine-grained access control over its resources through the Role Based Access Control (RBAC) mechanism.

Schemas are essential in defining the data exchanged between the Redpanda broker, consumers, and producers. Hence, the cluster has a Schema Registry. The Schema Registry API helps register and modify the schemas.

The HTTP Proxy (pandaproxy) API provides a convenient way to interact with Redpanda for basic data operations like listing topics and brokers, getting events, producing events, and much more.

Finally, Redpanda provides metric endpoints for its monitoring. These can be configured on Prometheus (monitoring tool) to pull important metrics and show them on Grafana dashboards.

3.2. Single Binary Installation Package

Redpanda’s installation package comprises a single binary, hence its installation is significantly simpler than Kafka. Unlike Kafka, it’s not dependent on a JVM or a cluster manager like Zookeeper. Due to these factors, operating Redpanda is remarkably easy.

It’s developed in C++ and has a compelling thread-per-core programming model that helps utilize the CPU cores, memory, and network optimally. Consequently, the hardware cost for its deployment is significantly reduced. This model also results in low latency and high throughput.

Redpanda’s cluster comprises multiple nodes. Each node can be either a data plane or a control plane. All these nodes need is a single binary package installed on them with the appropriate configurations. If the nodes have high-end computing power, they can play both roles without performance bottlenecks.

3.3. Management Tools

Redpanda provides two management tools, a Web Console and a CLI called Redpanda Keeper (RPK). The Console is a user-friendly web application that cluster administrators can use.

RPK is mostly used for low-level cluster management and tuning. However, the Console provides visibility into data streams and the capability to troubleshoot and manage the cluster.

4. Deployment

Redpanda supports Self-hosted and Redpanda Cloud deployment.

In Self-hosted deployment, customers can deploy the Redpanda cluster inside their private data centers or in their VPCs in the public cloud. It can be deployed on physical or virtual machines and Kubernetes. As a rule of thumb, each broker should have its dedicated node. Currently, RHEL/CentOS and Ubuntu operating systems are supported.

Additionally, AWS Simple Storage Service (S3), Azure Blob Storage (ABS), and Google Cloud Storage (GCS) can be used for supporting tiered storage.

Interestingly, customers can also opt for Redpanda Cloud for managed services. They can either have the whole cluster completely on Redpanda Cloud or choose to own the data plane running in their private data centers or public cloud accounts. The control plane remains on the Redpanda Cloud where monitoring, provisioning, and upgrades are all taken care of.

5. Key Use Cases

Unlike Kafka, Redpanda is an extremely robust streaming platform for developers because of its simple architecture and ease of installation. Let’s quickly look at the use case along the same lines:

In general, the participants in a streaming platform are:

• Source systems generate feeds
• Feeds could be monitoring events, metrics, notifications, and more
• Brokers in the cluster managing the topics
• Producers read feeds from source systems and publish them to the topics
• Consumers constantly poll on the subscribed topics
• Target Systems receive the transformed messages from the consumers

Redpanda guarantees the delivery of live feeds from various sources like monitoring tools, compliance and security platforms, IoT devices, and others to target systems with an incredibly 10x lower average latency.

It supports the consumer and producer model for processing live feeds or events from various sources. The producers are applications that read data from source systems and publish it to topics in the Redpanda cluster. The brokers in the cluster are highly reliable and fault-tolerant, guaranteeing message delivery.

The consumer applications subscribe to the topics in the cluster. Eventually, they read the data from the topics and, after further transforming the data, send them to various target systems like analytics platforms, NoSQL databases, relational databases, or other streaming platforms.

In Microservice architecture, Redpanda helps decouple microservices by facilitating asynchronous communication between them.

Consequently, it can play a substantial role across industries in developing:

• Observability platforms for event and log processing, reporting, troubleshooting, and auto-healing
• Real-time compliance and fraud-detection systems
• Real-time analytic dashboards and applications

6. Implement Redpanda Client With Kafka API

Notably, Redpanda supports the Kafka API. Hence, we’ll use the Kafka client to write programs that can interact with the Redpanda Stream.

For our examples, we’ve used Java Testcontainers to deploy a single-node Redpanda on a Windows desktop.

Furthermore, we’ll explore fundamental programs covering topic creation, message publishing, and message consumption. This is just for demonstration purposes and, hence, we won’t delve deeply into the Kafka API concepts.

6.1. Prerequisites

Before we begin, let’s import the necessary Maven dependency for the Kafka client library:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>3.6.1</version>
</dependency>

6.2. Create Topic

For creating a topic on Redpanda, we’ll first instantiate the AdminClient class from the Kafka client library:

AdminClient createAdminClient() {
    Properties adminProps = new Properties();
    adminProps.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, getBrokerUrl());
    return KafkaAdminClient.create(adminProps);
}

To set up the AdminClient, we got the broker URL and passed it to its static create() method.

Now, let’s see how we create a topic:

void createTopic(String topicName) {
    try (AdminClient adminClient = createAdminClient()) {
        NewTopic topic = new NewTopic(topicName, 1, (short) 1);
        adminClient.createTopics(Collections.singleton(topic));
    } catch (Exception e) {
        LOGGER.error("Error occurred during topic creation:", e);
    }
}

The createTopics() method of the AdminClient class takes in the NewTopic object as an argument for creating a topic.

Finally, let’s take a look at the createTopic() method in action:

@Test
void whenCreateTopic_thenSuccess() throws ExecutionException, InterruptedException {
    String topic = "test-topic";
    createTopic(topic);
    try(AdminClient adminClient = createAdminClient()) {
        assertTrue(adminClient.listTopics()
          .names()
          .get()
          .contains(topic));
    }
}

The program creates the topic test-topic successfully on Redpanda. We also validate the presence of the topic in the broker with the method listTopics() of the AdminClient class.

6.3. Publish Message to a Topic

Understandably, the most basic requirement of a producer application is publishing messages to a topic. For this purpose, we’ll use a KafkaProducer:

KafkaProducer<String, String> createProducer() {
    Properties producerProps = new Properties();
    producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, getBrokerUrl());
    producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    return new KafkaProducer<String, String>(producerProps);
}

We instantiated the producer by supplying essential properties like the broker URL and the StringSerializer class to the KafkaProducer constructor.

Now, let’s use the producer to publish the messages to a topic:

void publishMessage(String msgKey, String msg, String topic, KafkaProducer<String, String> producer)
    throws ExecutionException, InterruptedException {
    ProducerRecord<String, String> record = new ProducerRecord<>(topic, msgKey, msg);
    producer.send(record).get();
}

After creating the ProducerRecord object, we pass it to the send() method in KafkaProducer object to publish the message. The send() method operates asynchronously and, hence, we call the method get() to ensure blocking until the message is published.

Finally, now, let’s publish a message:

@Test
void givenTopic_whenPublishMsg_thenSuccess() {
    try (final KafkaProducer<String, String> producer = createProducer()) {
        assertDoesNotThrow(() -> publishMessage("test_msg_key_2", "Hello Redpanda!", "baeldung-topic", producer));
    }
}

First, we create the KafkaProducer object by invoking the method createProducer(). Then we publishe the message “Hello Redpanda!” to the topic baeldung-topic by calling the method publishMessage() that we covered earlier.

6.4. Consume Message From a Topic

As a next step, we’ll first create a KafkaConsumer before we can consume the messages from the stream:

KafkaConsumer<String, String> createConsumer() {
    Properties consumerProps = new Properties();
    consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
    consumerProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, getBrokerUrl());
    consumerProps.put(ConsumerConfig.GROUP_ID_CONFIG, "test-consumer-group");
    consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    return new KafkaConsumer<String, String>(consumerProps);
}

We instantiate the consumer by providing essential properties like the broker URL, StringDeSerializer class, and others to the KafkaConsumer constructor. Additionally, we ensure that the consumer would consume messages from the offset 0 (“earliest”).

Moving on, let’s consume some messages:

@Test
void givenTopic_whenConsumeMessage_thenSuccess() {
    try (KafkaConsumer<String, String> kafkaConsumer = createConsumer()) {
        kafkaConsumer.subscribe(Collections.singletonList(TOPIC_NAME));
        while(true) {
            ConsumerRecords<String, String> records = kafkaConsumer.poll(Duration.ofMillis(1000));
            if(records.count() == 0) {
                continue;
            }
            assertTrue(records.count() >= 1);
            break;
        }
    }
}

The method, after creating a KafkaConsumer object, subscribes to a topic. Then, it polls on it for every 1000 ms to read messages from it. Here, for demonstration, we’re coming out of the loop, but in the real world, applications continuously poll for the messages and then process them further.

7. Conclusion

In this tutorial, we explored the Redpanda Streaming platform. Conceptually, it’s similar to Apache Kafka but much easier to install, monitor, and manage. Additionally, with less compute and memory resources, it can achieve extremely high performance with high fault tolerance.

However, Redpanda still has a considerable distance to cover in terms of industry adoption when compared to Kafka. Additionally, the community support for Redpanda is not as strong as that for Kafka.

Finally, applications can migrate to Redpanda from Kafka with considerably less effort because it’s compatible with Kafka API.

As usual, the code used in this article is available over on GitHub.

       

1. Introduction

Servlet filters offer a powerful mechanism for intercepting and manipulating incoming requests. However, accessing Spring-managed beans within these filters can pose a challenge.

In this tutorial, we’ll explore various approaches to seamlessly obtain Spring beans within a Servlet filter, which is a common requirement in Spring-based web applications.

2. Understanding the Limitations of @Autowired in Servlet Filter

While Spring’s dependency injection mechanism, @Autowired, is a convenient way to inject dependencies into Spring-managed components, it doesn’t work seamlessly with Servlet filters. This is because Servlet filters are initialized by the Servlet container, typically before Spring’s ApplicationContext is fully loaded and initialized.

As a result, when the container instantiates a Servlet filter, the Spring context may not yet be available, leading to null or uninitialized dependencies when attempting to use @Autowired annotations. Let’s explore alternative approaches for accessing Spring beans within Servlet filters.

3. Setup

Let’s create a common LoggingService that will be autowired into our filter:

@Service
public class LoggingService {
    private final Logger logger = LoggerFactory.getLogger(this.getClass());
    public void log(String message,String url){
        logger.info("Logging Request {} for URI : {}",message,url);
    }
}

We’ll then create our filter, which will intercept incoming HTTP requests to log the HTTP method and URI details using the LoggingService dependency:

@Component
public class LoggingFilter implements Filter {
    @Autowired
    LoggingService loggingService;
    @Override
    public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) 
      throws IOException, ServletException {
        HttpServletRequest httpServletRequest=(HttpServletRequest)servletRequest;
        loggingService.log(httpServletRequest.getMethod(),httpServletRequest.getRequestURI());
        filterChain.doFilter(servletRequest,servletResponse);
    }
}

Let’s expose a RestController that returns a list of users:

@RestController
public class UserController {
    @GetMapping("/users")
    public List<User> getUsers(){
        return Arrays.asList(new User("1","John","john@email.com"),
          new User("2","Smith","smith@email.com"));
    }
}

We’ll set up our test to check if our LoggingService was autowired successfully into our filter:

@RunWith(SpringRunner.class)
@SpringBootTest
public class LoggingFilterTest {
    @Autowired
    private LoggingFilter loggingFilter;
    @Test
    public void givenFilter_whenAutowired_thenDependencyInjected() throws Exception {
        Assert.assertNotNull(loggingFilter);
        Assert.assertNotNull(getField(loggingFilter,"loggingService"));
    }
    private Object getField(Object target, String fieldName) throws NoSuchFieldException, IllegalAccessException {
        Field field = target.getClass().getDeclaredField(fieldName);
        field.setAccessible(true);
        return field.get(target);
    }
}

However, at this stage, it’s possible that the LoggingService won’t be injected into the LoggingFilter because the Spring context wasn’t available yet. We’ll explore the various options for resolving this issue in the following sections.

4. Using SpringBeanAutowiringSupport in Servlet Filter

Spring’s SpringBeanAutowiringSupport class provides support for dependency injection into non-Spring-managed classes such as Filters and Servlets. By using this class, Spring can inject dependencies, such as the LoggingService, which is a Spring-managed bean, into the LoggingFilter.

The init method is used to initialize a Filter instance, and we’ll override this method in the LoggingFilter to use SpringBeanAutowiringSupport:

@Override
public void init(FilterConfig filterConfig) throws ServletException {
    SpringBeanAutowiringSupport.processInjectionBasedOnServletContext(this,
      filterConfig.getServletContext());
}

The processInjectionBasedOnServletContext method uses the ApplicationContext associated with the ServletContext to perform the autowiring. It first retrieves the ApplicationContext from the ServletContext and then uses it to autowire the dependencies into the target object. This process involves inspecting the target object’s fields for @Autowired annotations and then resolving and injecting the corresponding beans from the ApplicationContext.

This mechanism allows non-Spring-managed objects, like filters and servlets, to benefit from Spring’s dependency injection capabilities.

5. Using WebApplicationContextUtils in Servlet Filter

WebApplicationContextUtils provides a utility method that retrieves the ApplicationContext associated with the ServletContext. The ApplicationContext contains all the beans managed by the Spring container.

Let’s override the init method of the LoggingFilter class:

@Override
public void init(FilterConfig filterConfig) throws ServletException {
    loggingService = WebApplicationContextUtils
      .getRequiredWebApplicationContext(filterConfig.getServletContext())
      .getBean(LoggingService.class);
}

We retrieve an instance of LoggingService from the ApplicationContext and assign it to the loggingService field of the filter. This approach is useful when we need to access Spring-managed beans in a non-Spring-managed component, such as a Servlet or a Filter, and we cannot use annotation-based or constructor injection.

It should be noted that this approach tightly couples the filter with Spring, which may not be ideal in some cases.

6. Using FilterRegistrationBean in Configuration

FilterRegistrationBean is used to register Servlet filters in the servlet container programmatically. It provides a way to configure the filter registration dynamically in the application’s configuration classes.

By annotating the method with @Bean and @Autowired, the LoggingService is automatically injected into the method, allowing it to be passed to the LoggingFilter constructor. Let’s set up the method for FilterRegistrationBean in our config class:

@Bean
public FilterRegistrationBean<LoggingFilter> loggingFilterRegistration(LoggingService loggingService) {
    FilterRegistrationBean<LoggingFilter> registrationBean = new FilterRegistrationBean<>();
    registrationBean.setFilter(new LoggingFilter(loggingService));
    registrationBean.addUrlPatterns("/*");
    return registrationBean;
}

We’ll then include a constructor in our LoggingFilter to support the above configuration:

public LoggingFilter(LoggingService loggingService) {
    this.loggingService = loggingService;
}

This approach centralizes the configuration of filters and their dependencies, making the code more organized and easier to maintain.

7. Using DelegatingFilterProxy in Servlet Filter

The DelegatingFilterProxy is a Servlet filter that allows passing control to Filter classes that have access to the Spring ApplicationContext.

Let’s configure the DelegatingFilterProxy to delegate to a Spring-managed bean named “loggingFilter”. FilterRegistrationBean is used by Spring to register the filter with the Servlet container when the application starts up:

@Bean
public FilterRegistrationBean<DelegatingFilterProxy> loggingFilterRegistration() {
    FilterRegistrationBean<DelegatingFilterProxy> registrationBean = new FilterRegistrationBean<>();
    registrationBean.setFilter(new DelegatingFilterProxy("loggingFilter"));
    registrationBean.addUrlPatterns("/*");
    return registrationBean;
}

Let’s use the same bean name for our filter defined earlier:

@Component("loggingFilter")
public class LoggingFilter implements Filter {
    // standard methods
}

This approach allows us to use Spring’s dependency injection to manage the loggingFilter bean.

8. Comparing Dependency Injection Approaches in Servlet Filter

The DelegatingFilterProxy approach differs from both SpringBeanAutowiringSupport and the direct use of WebApplicationContextUtils in how it delegates the filter’s execution to a Spring-managed bean, allowing us to use Spring’s dependency injection.

The DelegatingFilterProxy is better aligned with the typical Spring application architecture and allows for a cleaner separation of concerns. The FilterRegistrationBean approach allows more control over the dependency injection of the filter and centralizes the configuration of dependencies.

In contrast, SpringBeanAutowiringSupport and WebApplicationContextUtils are more low-level approaches that can be useful in certain scenarios where we need more control over the filter’s initialization process or want to access the ApplicationContext directly. However, they require more manual setup and do not provide the same level of integration with Spring’s dependency injection mechanism.

9. Conclusion

In this article, we’ve explored the different approaches to autowire Spring beans into Servlet Filters. Each approach has its advantages and limitations, and the choice of approach depends on the specific requirements and constraints of the application. Overall, they enable seamless integration of Spring-managed beans into Servlet filters, enhancing the flexibility and maintainability of applications.

As always, the code is available over on GitHub.

       

1. Introduction

When we test our code, we sometimes want to capture the parameters passed to our method.

In this tutorial, we’ll learn how to capture arguments in a Spock test using Stubs, Mocks, and Spies and check what we captured. We’ll also learn how to verify multiple calls to the same Mock with different arguments and assert the order of those calls.

2. The Subject of Our Test

First, we need a method that takes a single parameter or argument we want to capture.

So, let’s create an ArgumentCaptureSubject with a catchMeIfYouCan() method that takes a String and returns it with “Received ” prepended:

public class ArgumentCaptureSubject {
    public String catchMeIfYouCan(String input) {
        return "Received " + input;
    }
}

3. Preparing Our Data-Driven Test

We’ll start our tests with the typical use of a Stub and evolve it to capture arguments.

Let’s create a Stub of our class to return a stubbed response of “42” and invoke its catchMeIfYouCan() method:

def "given a Stub when we invoke it then we capture the stubbed response"() {
    given: "an input and a result"
    def input = "Input"
    def stubbedResponse = "42"
    and: "a Stub for our response"
    @Subject
    ArgumentCaptureSubject stubClass = Stub()
    stubClass.catchMeIfYouCan(_) >> stubbedResponse
    when: "we invoke our Stub's method"
    def result = stubClass.catchMeIfYouCan(input)
    then: "we get our stubbed response"
    result == stubbedResponse
}

We’ve used a simple Stub in this example as we’re not verifying any method invocations.

4. Capturing Arguments

Now we have our basic test, let’s see how to capture the argument we used to invoke our method.

First, we’ll declare a method-scoped variable to assign when we capture the argument:

def captured

Next, we’ll replace our static stubbedResponse with a Groovy Closure. Spock will pass our Closure a List of method arguments when our stubbed method is invoked.

Let’s create a simple Closure to capture the arguments list and assign it to our captured variable:

{ arguments -> captured = arguments }

For our assertion, we’ll assert that the first element in our list of captured arguments, at index 0, equals our input:

captured[0] == input

So, let’s update our test with our captured variable declaration, replace our stubbedResponse with our argument-capturing Closure, and add our assertion:

def "given a Stub when we invoke it then we capture the argument"() {
    given: "an input"
    def input = "Input"
    and: "a variable and a Stub with a Closure to capture our arguments"
    def captured
    @Subject
    ArgumentCaptureSubject stubClass = Stub()
    stubClass.catchMeIfYouCan(_) >> { arguments -> captured = arguments }
    when: "we invoke our method"
    stubClass.catchMeIfYouCan(input)
    then: "we captured the method argument"
    captured[0] == input
}

When we want to return a stubbedResponse as well as capture the arguments, we update our Closure to return it:

{ arguments -> captured = arguments; return stubbedResponse }
...
then: "what we captured matches the input and we got our stubbed response"
captured == input
result == stubbedResponse

Note that although we used “return” for clarity, it isn’t strictly necessary since Groovy closures return the result of the last executed statement by default.

When we’re only interested in capturing one of the arguments, we can capture the argument we want by using its index in the Closure:

{ arguments -> captured = arguments[0] }
...
then: "what we captured matches the input"
captured == input

In this case, our captured variable will be the same type as our parameter – a String.

5. Capturing With Spies

When we want to capture a value but also want the method to continue executing, we add a call to Spy‘s callRealMethod().

Let’s update our test to use a Spy instead of a Stub and use the Spy‘s callRealMethod() in our Closure:

def "given a Spy when we invoke it then we capture the argument and then delegate to the real method"() {
    given: "an input string"
    def input = "Input"
    and: "a variable and a Spy with a Closure to capture the first argument and call the underlying method"
    def captured
    @Subject
    ArgumentCaptureSubject spyClass = Spy()
    spyClass.catchMeIfYouCan(_) >> { arguments -> captured = arguments[0]; callRealMethod() }
    when: "we invoke our method"
    def result = spyClass.catchMeIfYouCan(input)
    then: "what we captured matches the input and our result comes from the real method"
    captured == input
    result == "Received Input"
}

Here, we’ve captured the input argument without impacting the method’s return value.

When we want to change the captured argument before passing it on to the real method, we update it inside the closure, then use Spy’s callRealMethodWithArgs to pass on our updated argument.

So, let’s update our Closure to prepend “Tampered: ” to our String before passing it on to the real method:

spyClass.catchMeIfYouCan(_) >> { arguments -> captured = arguments[0]; callRealMethodWithArgs('Tampered:' + captured) }

And let’s update our assertion to expect our tampered result:

result == "Received Tampered:Input"

6. Capturing Arguments Using an Injected Mock

Now that we’ve seen how to use Spock’s mocking framework to capture an argument, let’s apply this technique to a class with a dependency that we can mock.

First, let’s create an ArgumentCaptureDependency class that our subject can call with a simple catchMe() method that takes and modifies a String:

public class ArgumentCaptureDependency {
    public String catchMe(String input) {
        return "***" + input + "***";
    }
}

Now, let’s update our ArgumentCaptureSubject with a constructor that takes our ArgumentCaptureDependency. Let’s also add a callOtherClass method that takes no parameters and calls our ArgumentCaptureDependency‘s catchMe() method with a parameter:

public class ArgumentCaptureSubject {
    ArgumentCaptureDependency calledClass;
    public ArgumentCaptureSubject(ArgumentCaptureDependency calledClass) {
        this.calledClass = calledClass;
    }
    public String callOtherClass() {
        return calledClass.catchMe("Internal Parameter");
    }
}

Finally, let’s create a test like before. This time, let’s inject a Spy into our ArgumentCaptureSubject when we create it so that we can also callRealMethod() and compare the result:

def "given an internal method call when we invoke our subject then we capture the internal argument and return the result of the real method"() {
    given: "a mock and a variable for our captured argument"
    ArgumentCaptureDependency spyClass = Spy()
    def captured
    spyClass.catchMe(_) >> { arguments -> captured = arguments[0]; callRealMethod() }
    and: "our subject with an injected Spy"
    @Subject argumentCaptureSubject = new ArgumentCaptureSubject(spyClass)
    when: "we invoke our method"
    def result = argumentCaptureSubject.callOtherClass(input)
    then: "what we captured matches the internal method argument"
    captured == "Internal Parameter"
    result == "***Internal Parameter***"
}

Our test captured the internal argument “Internal Parameter“. In addition, our invocation of Spy‘s callRealMethod ensured that we didn’t impact the result of the method: “***Internal Parameter***”.

When we don’t need to return the real result, we can simply use a Stub or Mock.

Note that when we test Spring applications, we can inject our Mock using Spock’s @SpringBean annotation.

7. Capturing Arguments From Multiple Invocations

Sometimes, our code invokes a method multiple times, and we want to capture the value for each invocation.

So, let’s add a String parameter to the callOtherClass() method in our ArgumentCaptureSubject. We’ll invoke this with different parameters and capture them.

public String callOtherClass(String input) {
    return calledClass.catchMe(input);
}

We need a collection to capture the argument from each invocation. So, we’ll declare a capturedStrings variable as an ArrayList:

def capturedStrings = new ArrayList()

Now, let’s create our test and make it invoke our callOtherClass() twice, first with “First” as a parameter and then with “Second”:

def "given an dynamic Mock when we invoke our subject then we capture the argument for each invocation"() {
    given: "a variable for our captured arguments and a mock to capture them"
    def capturedStrings = new ArrayList()
    ArgumentCaptureDependency mockClass = Mock()
    and: "our subject"
    @Subject argumentCaptureSubject = new ArgumentCaptureSubject(mockClass)
    when: "we invoke our method"
    argumentCaptureSubject.callOtherClass("First")
    argumentCaptureSubject.callOtherClass("Second")
}

Now, let’s add a Closure to our Mock to capture the argument from each invocation and add it to our list. Let’s also have our Mock verify that our method was called twice by prefixing “2 *” to our statement: 

then: "our method was called twice and captured the argument"
2 * mockClass.catchMe(_ as String) >> { arguments -> capturedStrings.add(arguments[0]) }

Finally, let’s assert that we captured both arguments in the right order:

and: "we captured the list and it contains an entry for both of our input values"
capturedStrings[0] == "First"
capturedStrings[1] == "Second"

When we’re not concerned about the order, we can use List‘s contains method:

capturedStrings.contains("First")

8. Using Multiple Then Blocks

Sometimes, we want to assert a sequence of calls using the same method with different arguments but don’t need to capture them. Spock allows verifications in any order within the same then block, so it won’t matter what order we write them in. We can, however, enforce the order by adding multiple then blocks.

Spock verifies that assertions in one then block are met before assertions in the next then block.

So, let’s add two then blocks to verify that our method is invoked with the correct arguments in the right sequence:

def "given a Mock when we invoke our subject twice then our Mock verifies the sequence"() {
    given: "a mock"
    ArgumentCaptureDependency mockClass = Mock()
    and: "our subject"
    @Subject argumentCaptureSubject = new ArgumentCaptureSubject(mockClass)
    when: "we invoke our method"
    argumentCaptureSubject.callOtherClass("First")
    argumentCaptureSubject.callOtherClass("Second")
    then: "we invoked our Mock with 'First' the first time"
    1 * mockClass.catchMe( "First")
    then: "we invoked our Mock with 'Second' the next time"
    1 * mockClass.catchMe( "Second")
}

When our invocations occur in the wrong order, such as when we invoke callOtherClass(“Second”) first, Spock gives us a helpful message:

Wrong invocation order for:
1 * mockClass.catchMe( "First")   (1 invocation)
Last invocation: mockClass.catchMe('First')
Previous invocation:
	mockClass.catchMe('Second')

9. Conclusion

In this tutorial, we learned how to capture method arguments using Spock’s Stubs, Mocks, and Spies using Closures. Next, we learned how to use a Spy to change a captured argument before invoking the real method. We also learned how to collect arguments when our method is invoked multiple times. Finally, as an alternative to capturing the argument, we learned how to use multiple then blocks to check that our invocations occurred in the right sequence.

As usual, the source for this article can be found over on GitHub.

      

Related Stories

• Setting up and Using Spock With Gradle
• Using Groovy in Spring
• How to Remove a Prefix From Strings in Groovy

 

1. Introduction

The RGB color model is widely used in various applications and devices because it aligns well with how electronic displays work. ‘R’ in RGB stands for Red, ‘G’ for Green, and ‘B’ for Blue. Combining these three primary colors at varying intensities can produce a broad spectrum of colors.

In programming languages, including Java, a common practice is to represent an RGB color as a single integer, packing the three color components and, sometimes, an alpha (transparency) component into a 32-bit integer.

In this tutorial, we’ll look into methods of moving between these representations.

2. RGB Integer Representation

In a 32-bit integer representation of an RGB color, each color component is typically allocated 8 bits. The highest 8 bits are often used for the alpha channel (representing transparency), followed by red, green, and blue. The structure looks like this:

• Bits 24-31: Alpha (A)
• Bits 16-23: Red (R)
• Bits 8-15: Green (G)
• Bits 0-7: Blue (B)

3. RGB to Integer Conversion

We can create an integer representing an (A)RGB color by bitwise shifting individual components to their desired positions:

int alpha = 255; // Fully opaque
int red = 100;   
int green = 150; 
int blue = 200;  
int argb = (alpha << 24) | (red << 16) | (green << 8) | blue;

If we’re working in a context where transparency isn’t needed, we might want to omit the alpha channel:

int rgb = (red << 16) | (green << 8) | blue;

3.1. Converting with Clamping

Using that knowledge, we can write a function that takes RGB values and returns an integer representation. First, we can implement clamping to limit the input values to the appropriate range. Quitely keeping values in desired ranges can be very convenient while working with color transformations:

int clamp(int value, int min, int max) {
    return Math.max(min, Math.min(max, value));
}

Next, let’s implement a function that will use the clamp function to convert RGB values to integers:

int rgbToInt(<span class="hljs-type">int</span> alpha, int red, int green, int blue) {
    alpha = clamp(alpha, <span class="hljs-number">0</span>, <span class="hljs-number">255</span>);
    red = clamp(red, 0, 255);
    green = clamp(green, 0, 255);
    blue = clamp(blue, 0, 255);
    return (alpha << <span class="hljs-number">24</span>) | (red << 16) | (green << 8) | blue;
}

We can also implement a version without an alpha channel:

int rgbToInt(int red, int green, int blue) {
    red = clamp(red, 0, 255);
    green = clamp(green, 0, 255);
    blue = clamp(blue, 0, 255);
    return (red << 16) | (green << 8) | blue;
}

Finally, we can use the created function to convert RGB values:

assertEquals(0x00ABCDEF, rgbToInt(171, 205, 239))

4. RGB to Integer Conversion

Extracting RGB components from an integer representation is as easy as shifting bits to bring the desired part to the lowest 8 bits and then masking off redundant higher bits. For example, to extract the red channel, we need to shift the integer value 16 bits to the right:

int red = (argb >> 16)

Now, the lowest 8 bits represent our red value, but we still need to mask the rest. We can do that by using bitwise and operator:

int red = (argb >> 16) & 0xFF;

We can think about 0xFF number (255 in decimal) as a bit mask that has the first 8 bits equal 1 and all of the rest bits equal 0. So, in conjunction with the and operator, it’ll discard all but the first 8 bits. We can extract the rest of the RGB components in a similar fashion:

int alpha = (argb >> 24) & 0xFF;
int red = (argb >> 16) & 0xFF;
int green = (argb >> 8) & 0xFF;
int blue = argb & 0xFF;

5. Color Transformations

We can use the RGB color representation in various transformations to make the RGB color representation more practical.

5.1. Brightness Adjustment

Adjusting the brightness involves scaling the RGB components. Because the alpha channel has nothing to do with the brightness, we won’t scale it:

float scale = 0.8f; // darken by 20%
red = (int)(red * scale);
green = (int)(green * scale);
blue = (int)(blue * scale);
int newArgb = (alpha << 24) | (red << 16) | (green << 8) | blue;

5.2. Grayscale Conversion

To perform grayscale conversion, we’ll set all three color components to the same value. We can use the weighted average of the original colors:

int average = (int)(red * 0.299 + green * 0.587 + blue * 0.114);
int grayscaleArgb = (alpha << 24) | (average << 16) | (average << 8) | average;

Similarly to the previous example, the alpha channel is not affected because it doesn’t contain color information. We achieve different results by using different weights for each of the color components.

5.3. Color Inversion

We can also invert a color by flipping its RGB components. To do that, we need to subtract the value of each color component from 255:

red = 255 - red;
green = 255 - green;
blue = 255 - blue;
int invertedArgb = (alpha << 24) | (red << 16) | (green << 8) | blue;

6. Summary

In this article, we learned how to use bitwise operations to move between RGB and integer representations. We also look at examples of how RGB representation can be used for various color transformations.

       

1. Spring and Java

>> Function Calling in Java and Spring AI using the latest Mistral AI API [spring.io]

Integrating LLM with external tools: exploring advanced function calling in Java and Spring AI with the latest Mistral AI API.

>> The Ktor Roadmap for 2024 [blog.jetbrains.com]

Exciting new features to expect from Ktor in 2024: OpenTelemetry, gRPC, managed transactions, simplified dependency injection, and quite a bit more.

Also worth reading:

• >> Bootiful Spring Boot in 2024 (part 1) [spring.io]
• >> TornadoInsight: Harness the Power of TornadoVM from IntelliJ IDEA [foojay.io]

Webinars and presentations:

• >> A Bootiful Podcast: Cristian Schuszter on CERN [spring.io]
• >> Modern Java in Action [inside.java]

Time to upgrade:

• >> MongoDB Java Driver 5.0 is Released [mongodb.com]
• >> Spring AI 0.8.1 Released [spring.io]
• >> Quarkus 3.8.2 released – Maintenance release [quarkus.io]

2. Technical & Musings

>> PostgreSQL Index Types [vladmihalcea.com]

What index to choose? exploring different index types in PostgreSQL and use cases for each. 

>> Measuring Developer Productivity via Humans [martinfowler.com]

And a nuanced piece on the obviously difficult problem of measuring developer productivity, where there are no obvious answers.

Also worth reading:

• >> Using my new Raspberry Pi to run an existing GitHub Action [frankel.ch]
• >> Kanban – Not The Lazy Option [blog.scottlogic.com]
• >> Does Platform Engineering Solve the People Problem [blog.christianposta.com]
• >> Microservices Design Principles for Well-Crafted Architectures [foojay.io]
• >> What if we rotate pairs every day? [martinfowler.com]
• >> Hello eBPF: First steps with libbpf (5) [foojay.io]
• >> Programming with Nothing [code-cop.org]

3. Pick of the Week

And the Microsoft live conference is getting closer:

>> Microsoft JDConf 2024, on the 27th and 28th [jdconf.com]

       

1. Overview

Jsoup is an open-source library used to scrape HTML pages. It provides an API for data parsing, extraction, and manipulation using DOM API methods.

In this article, we will see how to parse an HTML table using Jsoup. We will be retrieving and updating data from the HTML table and also, adding and deleting rows in the table using Jsoup.

2. Dependencies

To use the Jsoup library, add the following dependency to the project:

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.17.2</version>
</dependency>

We can find the latest version of the Jsoup library in the Maven central repository.

3. Table Structure

To illustrate parsing HTML tables via jsoup, we will be using a sample HTML structure. The complete HTML structure is available in the code base provided in the GitHub repository mentioned at the end of the article. Here, we are showing a table with only two rows of data for representational purposes:

<table>
    <thead>
        <tr>
            <th>Name</th>
            <th>Maths</th>
            <th>English</th>
            <th>Science</th>
         </tr>
    </thead>
    <tbody>
        <tr>
            <td>Student 1</td>
            <td>90</td>
            <td>85</td>
            <td>92</td>
        </tr>
     </tbody>
</table>

As we can see, we are parsing the table with a header row with thead tag followed by data rows in the tbody tag. We are assuming that the table in the HTML document will be in the above format.

4. Parsing Table

Firstly, to select an HTML table from the parsed document, we can use the code snippet below:

Element table = doc.select("table");
Elements rows = table.select("tr"); 
Elements first = rows.get(0).select("th,td");

As we can see, the table element is selected from the document, and then, to get the row element, tr is selected from the table element. As there are multiple rows in the table, we have selected the th or td elements in the first row. By using these functions, we can write the below function to parse table data.

Here, we are assuming no colspan or rowspan elements are used in the table, and the first row is present with header th tags.

Following is the code for parsing the table:

public List<Map<String, String>> parseTable(Document doc, int tableOrder) {
    Element table = doc.select("table").get(tableOrder);
    Element tbody = table.select("tbody").get(0);
    Elements dataRows = tbody.select("tr");
    Elements headerRow = table.select("tr")
      .get(0)
      .select("th,td");
    List<String> headers = new ArrayList<String>();
    for (Element header : headerRow) {
        headers.add(header.text());
    }
    List<Map<String, String>> parsedDataRows = new ArrayList<Map<String, String>>();
    for (int row = 0; row < dataRows.size(); row++) {
        Elements colVals = dataRows.get(row).select("th,td");
        int colCount = 0;
        Map<String, String> dataRow = new HashMap<String, String>();
        for (Element colVal : colVals) {
            dataRow.put(headers.get(colCount++), colVal.text());
        }
        parsedDataRows.add(dataRow);
    }
    return parsedDataRows;
}

In this function, parameter doc is the HTML document loaded from the file, and tableOrder is the nth table element in the document. We are using List<Map<String, String>> to store a list of dataRows in the table under the tbody element. Each element of the list is a Map representing a dataRow. This Map stores the column name as a key and the row value for that column as a map value. Using a list of Maps makes it easy to access the retrieved data.

The list index represents row numbers, and we can get specific cell data by its map key.

We can verify if table data is retrieved correctly using the test case below:

@Test
public void whenDocumentTableParsed_thenTableDataReturned() {
    JsoupTableParser jsoParser = new JsoupTableParser();
    Document doc = jsoParser.loadFromFile("Students.html");
    List<Map<String, String>> tableData = jsoParser.parseTable(doc, 0);
    assertEquals("90", tableData.get(0).get("Maths")); 
}

From the JUnit test case, we can confirm that since we have parsed the text of all table cells and stored it in an ArrayList of HashMap objects, each element of the list represents a data row in the table. The row is represented by a HashMap with the key as the column header and cell text as the value. Using this structure, we can easily access table data.

5. Update Elements of the Parsed Table

To insert or update elements while parsing, we can use the below code on the td element retrieved from the row:

colVals.get(colCount++).text(updateValue);

or

colVals.get(colCount++).html(updateValue);

The function to update values in the parsed table would look like below:

public void updateTableData(Document doc, int tableOrder, String updateValue) {
    Element table = doc.select("table").get(tableOrder);
    Element tbody = table.select("tbody").get(0);
    Elements dataRows = tbody.select("tr");
    for (int row = 0; row < dataRows.size(); row++) {
        Elements colVals = dataRows.get(row).select("th,td");
        for (int colCount = 0; colCount < colVals.size(); colCount++) {
            colVals.get(colCount).text(updateValue);
        }
    }
}

In the above function, we are getting data rows from the tbody element of the table. The function traverses each cell of the table and sets its value to the parameter value, updatedValue. It updates all cells to the same value to demonstrate that cell values can be updated using Jsoup. We can update the individual cell values by specifying the row and column index for the data row.

The test below verifies the update function:

@Test
public void whenTableUpdated_thenUpdatedDataReturned() {
    JsoupTableParser jsoParser = new JsoupTableParser();
    Document doc = jsoParser.loadFromFile("Students.html");
    jsoParser.updateTableData(doc, 0, "50");
    List<Map<String, String>> tableData = jsoParser.parseTable(doc, 0);
    assertEquals("50", tableData.get(2).get("Maths"));
}

The JUnit test case confirms that the update operation updates all table cell values to 50. Here we are verifying data from the third data row of the Maths column.

Similarly, we can set desired values for specific cells of the table.

6. Adding Row to the Table

We can add a row to the table using the following function:

public void addRowToTable(Document doc, int tableOrder) {
    Element table = doc.select("table").get(tableOrder);
    Element tbody = table.select("tbody").get(0);
    Elements rows = table.select("tr");
    Elements headerCols = rows.get(0).select("th,td");
    int numCols = headerCols.size();
    Elements colVals = new Elements(numCols);
    for (int colCount = 0; colCount < numCols; colCount++) {
        Element colVal = new Element("td");
        colVal.text("11");
        colVals.add(colVal);
    }
    Elements dataRows = tbody.select("tr");
    Element newDataRow = new Element("tr");
    newDataRow.appendChildren(colVals);
    dataRows.add(newDataRow);
    tbody.html(dataRows.toString());
}

In the above function, we are getting the number of columns from the header row and the data rows from the tbody element of the table. After adding a new row to the dataRows list, we are updating the tbody HTML content with the dataRows.

We can verify row addition using the following test case:

@Test
public void whenTableRowAdded_thenRowCountIncreased() {
    JsoupTableParser jsoParser = new JsoupTableParser();
    Document doc = jsoParser.loadFromFile("Students.html");
    List<Map<String, String>> tableData = jsoParser.parseTable(doc, 0);
    int countBeforeAdd = tableData.size();
    jsoParser.addRowToTable(doc, 0);
    tableData = jsoParser.parseTable(doc, 0);
    assertEquals(countBeforeAdd + 1, tableData.size());
}

We can confirm from the JUnit test case that the addRowToTable operation on the table increases the number of rows in the table by 1. This operation adds a new row at the end of the list.

Similarly, we can add a row at any position by specifying the index while adding it to the row elements collection.

7. Delete the Row From the Table

We can delete a row from the table using the following function:

public void deleteRowFromTable(Document doc, int tableOrder, int rowNumber) {
    Element table = doc.select("table").get(tableOrder);
    Element tbody = table.select("tbody").get(0);
    Elements dataRows = tbody.select("tr");
    if (rowNumber < dataRows.size()) {
        dataRows.remove(rowNumber);
    }
}

In the above function, we are getting the tbody element of the table. From tbody, we are getting a list of dataRows. From the list of dataRows, we are deleting the row at the rowNumber position in the table. We can verify row deletion using the following test case:

@Test
public void whenTableRowDeleted_thenRowCountDecreased() {
    JsoupTableParser jsoParser = new JsoupTableParser();
    Document doc = jsoParser.loadFromFile("Students.html");
    List<Map<String, String>> tableData = jsoParser.parseTable(doc, 0);
    int countBeforeDel = tableData.size();
    jsoParser.deleteRowFromTable(doc, 0, 2);
    tableData = jsoParser.parseTable(doc, 0);
    assertEquals(countBeforeDel - 1, tableData.size());
}

The JUnit test case confirms that the deleteRowFromTable operation on the table reduces the number of rows in the table by 1.

Similarly, we can delete a row at any position by specifying the index while removing it from the row elements collection.

8. Conclusion

In this article, we have seen how we can use jsoup to parse HTML tables from HTML documents. Also, we can update table structure as well as table cell data. As always, the source for these examples is available over on GitHub.

       

1. Introduction

Java Persistence API (JPA) acts as a bridge between Java objects and relational databases, allowing us to persist and retrieve data seamlessly. In this tutorial, we’ll explore various strategies and techniques to effectively refresh and fetch entities after saving operations in JPA.

2. Understanding Entity Management in Spring Data JPA

In Spring Data JPA, entity management revolves around the JpaRepository interface, which serves as the primary mechanism for interacting with the database. Through the JpaRepository interface, which extends CrudRepository, Spring Data JPA offers a robust set of methods for entity persistence, retrieval, updating, and deletion.

Furthermore, the entityManager is automatically injected into these repository interfaces by the Spring container. This component is an integral part of the JPA infrastructure embedded within Spring Data JPA, facilitating interaction with the underlying persistence context and the execution of JPA queries.

2.1. Persistence Context

One crucial component within JPA is the persistence context. Imagine this context as a temporary holding area where JPA manages the state of retrieved or created entities.

It ensures that:

• Entities are unique: Only one instance of an entity with a specific primary key exists within the context at any given time.
• Changes are tracked: The EntityManager keeps track of any modifications made to entity properties within the context.
• Data consistency is maintained: The EntityManager synchronizes changes made within the context with the underlying database during transactions.

2.2. Lifecycle of JPA Entities

There are four distinct lifecycle stages of JPA entities: New, Managed, Removed, and Detached.

When we create a new entity instance using the entity’s constructor, it is in the “New” state. We can verify this by checking if the entity’s ID (primary key) is null:

Order order = new Order();
if (order.getId() == null) {
    // Entity is in the "New" state
}

After we persist the entity using a repository’s save() method, it transitions to the “Managed” state. We can verify this by checking if the saved entity exists in the repository:

Order savedOrder = repository.save(order);
if (repository.findById(savedOrder.getId()).isPresent()) {
    // Entity is in the "Managed" state
}

When we call the repository’s delete() method on a managed entity, it transitions to the “Removed” state. We can verify this by checking if the entity is no longer present in the database after deletion:

repository.delete(savedOrder);
if (!repository.findById(savedOrder.getId()).isPresent()) {
    // Entity is in the "Removed" state
}

Lastly, once the entity is detached using the repository’s detach() method, the entity is no longer associated with the persistence context. Changes made to a detached entity won’t be reflected in the database unless explicitly merged back into the managed state. We can verify this by attempting to modify the entity after detaching it:

repository.detach(savedOrder);
// Modify the entity
savedOrder.setName("New Order Name");

If we call save() on a detached entity, it re-attaches the entity to the persistence context and persists the changes to the database upon flushing the persistence context.

3. Saving Entities with Spring Data JPA

When we invoke save(), Spring Data JPA schedules the entity for insertion into the database upon transaction commit. It adds the entity to the persistence context, marking it as managed.

Here’s a simple code snippet demonstrating how to use the save() method in Spring Data JPA to persist an entity:

Order order = new Order();
order.setName("New Order Name");
repository.save(order);

However, it’s important to note that invoking save() doesn’t immediately trigger a database insert operation. Instead, it merely transitions the entity to the managed state within the persistence context. Thus, if other transactions read data from the database before our transaction commits, they might retrieve outdated data that doesn’t include the changes we’ve made but not yet committed.

To ensure that the data remains up-to-date, we can employ two approaches: fetching and refreshing.

4. Fetching Entities in Spring Data JPA

When we fetch an entity, we don’t discard any modifications made to it within the persistence context. Instead, we simply retrieve the entity’s data from the database and add it to the persistence context for further processing.

4.1. Using findById()

Spring Data JPA repositories offer convenient methods like findById() for retrieving entities. These methods always fetch the latest data from the database, regardless of the entity’s state within the persistence context. This approach simplifies entity retrieval and eliminates the need to manage the persistence context directly.

Order order = repository.findById(1L).get();

4.2. Eager vs. Lazy Fetching

In eager fetching, all related entities associated with the main entity are retrieved from the database at the same time as the main entity. By setting fetch = FetchType.EAGER on the orderItems collection, we instruct JPA to eagerly fetch all associated OrderItem entities when retrieving an Order:

@Entity
public class Order {
    @Id
    private Long id;
    @OneToMany(mappedBy = "order", fetch = FetchType.EAGER)
    private List<OrderItem> orderItems;
}

This means that after the findById() call, we can directly access the orderItems list within the order object and iterate through the associated OrderItem entities without requiring any additional database queries:

Order order = repository.findById(1L).get();
// Accessing OrderItems directly after fetching the Order
if (order != null) {
    for (OrderItem item : order.getOrderItems()) {
        System.out.println("Order Item: " + item.getName() + ", Quantity: " + item.getQuantity());
    }
}

On the other hand, by setting fetch = FetchType.LAZY, related entities will not retrieved from the database until they are explicitly accessed within the code:

@Entity
public class Order {
    @Id
    private Long id;
    @OneToMany(mappedBy = "order", fetch = FetchType.LAZY)
    private List<OrderItem> orderItems;
}

When we call order.getOrderItems(), a separate database query is executed to fetch the associated OrderItem entities for that order. This additional query is only triggered because we explicitly accessed the orderItems list:

Order order = repository.findById(1L).get();
if (order != null) {
    List<OrderItem> items = order.getOrderItems(); // This triggers a separate query to fetch OrderItems
    for (OrderItem item : items) {
        System.out.println("Order Item: " + item.getName() + ", Quantity: " + item.getQuantity());
    }
}

4.3. Fetching with JPQL

Java Persistence Query Language (JPQL) allows us to write SQL-like queries that target entities instead of tables. It provides flexibility for retrieving specific data or entities based on various criteria.

Let’s see an example of fetching orders by customer name and with the order date falling within a specified range:

@Query("SELECT o FROM Order o WHERE o.customerName = :customerName AND 
  o.orderDate BETWEEN :startDate AND :endDate")
List<Order> findOrdersByCustomerAndDateRange(@Param("customerName") String customerName, 
  @Param("startDate") LocalDate startDate, @Param("endDate") LocalDate endDate);

4.4. Fetching with Criteria API

The Criteria API in Spring Data JPA provides a reliable and flexible method to create queries dynamically. It allows us to construct complex queries safely using method chaining and criteria expressions, ensuring that our queries are error-free at compile time.

Let’s consider an example where we use the Criteria API to fetch orders based on a combination of criteria, such as customer name and order date range:

CriteriaBuilder criteriaBuilder = entityManager.getCriteriaBuilder();
CriteriaQuery<Order> criteriaQuery = criteriaBuilder.createQuery(Order.class);
Root<Order> root = criteriaQuery.from(Order.class);
Predicate customerPredicate = criteriaBuilder.equal(root.get("customerName"), customerName);
Predicate dateRangePredicate = criteriaBuilder.between(root.get("orderDate"), startDate, endDate);
criteriaQuery.where(customerPredicate, dateRangePredicate);
return entityManager.createQuery(criteriaQuery).getResultList();

5. Refreshing Entities with Spring Data JPA

Refreshing entities in JPA ensures that the in-memory representation of entities in the application stays synchronized with the latest data stored in the database. When other transactions modify or update entities, the data within the persistence context might become outdated. Refreshing entities allow us to retrieve the most recent data from the database, preventing inconsistencies and maintaining data accuracy.

5.1. Using refresh()

In JPA, we achieve entity refreshing using the refresh() method provided by the EntityManager. Invoking refresh() on a managed entity discards any modifications made to the entity within the persistence context. It reloads the entity’s state from the database, effectively replacing any modifications made since the entity was last synchronized with the database.

However, it’s important to note that Spring Data JPA repositories do not offer a built-in refresh() method.

Here’s how to refresh an entity using the EntityManager:

@Autowired
private EntityManager entityManager;
entityManager.refresh(order);

5.2. Handling OptimisticLockException

The @Version annotation in Spring Data JPA serves the purpose of implementing optimistic locking. It helps ensure data consistency when multiple transactions might try to update the same entity concurrently. When we use @Version, JPA automatically creates a special field (often named version) on our entity class.

This field stores an integer value representing the version of the entity in the database:

@Entity
public class Order {
    @Id
    @GeneratedValue
    private Long id;
    
    @Version
    private Long version;
}

When retrieving an entity from the database, JPA actively fetches its version. Upon updating the entity, JPA compares the version of the entity in the persistence context with the version stored in the database. If the versions of the entity differ, it indicates that another transaction has modified the entity, potentially leading to data inconsistency.

In such cases, JPA throws an exception, often an OptimisticLockException, to indicate the potential conflict. Therefore, we can call the refresh() method within the catch block to reload the entity’s state from the database.

Let’s see a brief demonstration of how this approach works:

Order order = orderRepository.findById(orderId)
  .map(existingOrder -> {
      existingOrder.setName(newName);
      return existingOrder;
  })
  .orElseGet(() -> {
      return null;
  });
if (order != null) {
    try {
        orderRepository.save(order);
    } catch (OptimisticLockException e) {
        // Refresh the entity and potentially retry the update
        entityManager.refresh(order);
        // Consider adding logic to handle retries or notify the user about the conflict
    }
}

Additionally, it’s worth noting that refresh() may throw javax.persistence.EntityNotFoundException if the entity being refreshed has been removed from the database by another transaction since it was last retrieved.

6. Conclusion

In this article, we learned the distinction between refreshing and fetching entities in Spring Data JPA. Fetching involves retrieving the latest data from the database when needed. Refreshing involves updating the entity’s state in the persistence context with the most recent data from the database.

By utilizing these approaches strategically, we can maintain data consistency and ensure that all transactions operate on the latest data.

As always, the source code for the examples is available over on GitHub.

       

1. Overview

In Kafka, consumers read messages from partitions. While reading messages, there are some concerns to consider, like determining which messages to read from the partitions or, preventing duplicate message reading or message loss in case of failure. The solution to these concerns is the use of offsets.

In this tutorial, we’ll learn about offsets in Kafka. We’ll see how to commit offsets to manage message consumption and discuss its methods and drawbacks.

2. What Is Offset?

We know that Kafka stores messages in topics, and each topic can have multiple partitions. Each consumer reads messages from one partition of a topic. Here, Kafka, with the help of offsets, keeps track of the messages that consumers read. Offsets are integers starting from zero that increment by one as the message gets stored.

Let’s say one consumer has read five messages from a partition. Then, based on configuration, Kafka marks the offset till 4 as committed(zero-based sequence). The consumer consumes messages with offsets 5 onwards the next time it attempts to read messages.

Without offsets, there is no way to avoid duplicate processing or data loss. That’s why it’s so crucial.

We can make an analogy with database storage. In a database, we commit after executing SQL statements to persist the changes. In the same way, after reading from the partition, we commit offsets to mark the position of the processed message.

3. Ways to Commit Offsets

There are four ways to commit offsets. We’ll look at each in detail and discuss their use cases, advantages, and disadvantages.

Let’s start by adding the Kafka Client API dependency in the pom.xml:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>3.6.1</version>
</dependency>

3.1. Auto Commit

This is the simplest way to commit offsets. Kafka, by default, uses auto-commit – at every five seconds it commits the largest offset returned by the poll() method. poll() returns a set of messages with a timeout of 10 seconds, as we can see in the code:

KafkaConsumer<Long, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(KafkaConfigProperties.getTopic());
ConsumerRecords<Long, String> messages = consumer.poll(Duration.ofSeconds(10));
for (ConsumerRecord<Long, String> message : messages) {
  // processed message
}

The problem with auto-commit is that there is a very high chance of data loss in case of application failure. When poll() returns the messages, Kafka may commit the largest offset before processing messages.

Let’s say poll() returns 100 messages, and the consumer processes 60 messages when the auto-commit happens. Then, due to some failure, the consumer crashes. When a new consumer goes live to read messages, it commences reading from offset 101, resulting in the loss of messages between 61 and 100.

Thus, we need other ways where this drawback isn’t present. The answer is manual commit.

3.2. Manual Sync Commit

In manual commits, whether sync or async,  it’s necessary to disable auto-commit by setting the default property (enabled.auto.commit property) to false:

Properties props = new Properties();
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

After disabling the manual commit, let’s now understand the use of commitSync():

KafkaConsumer<Long, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(KafkaConfigProperties.getTopic());
ConsumerRecords<Long, String> messages = consumer.poll(Duration.ofSeconds(10));
  //process the messages
consumer.commitSync();

This method prevents data loss by committing the offset only after processing the messages. However, it doesn’t prevent duplicate reading when a consumer crashes before committing the offset. Besides this, it also impacts application performance.

The commitSync() blocks the code until it completes. Also, in case of an error, it keeps on retrying. This decreases the throughput of the application, which we don’t want. So, Kafka provides another solution, async commit, that deals with these drawbacks.

3.3. Manual Async Commit

Kafka provides commitAsync() to commit offsets asynchronously. It overcomes the performance overhead of manual sync commits by committing offsets in different threads. Let’s implement an async commit to understand this:

KafkaConsumer<Long, String> consumer = new KafkaConsumer<>(props); 
consumer.subscribe(KafkaConfigProperties.getTopic()); 
ConsumerRecords<Long, String> messages = consumer.poll(Duration.ofSeconds(10));
  //process the messages
consumer.commitAsync();

The problem with the async commit is that it doesn’t retry in case of failure. It relies on the next call of commitAsync(), which will commit the latest offset.

Suppose 300 is the largest offset we want to commit, but our commitAsync() fails due to some issue. It could be possible that before it retries, another call of commitAsync() commits the largest offset of 400 as it is asynchronous. When failed commitAsync() retries and if it commits offsets 300 successfully, it will overwrite the previous commit of 400, resulting in duplicate reading. That is why commitAsync() doesn’t retry.

3.4. Commit Specific Offset

Sometimes, we need to take more control over offsets. Let’s say we’re processing the messages in small batches and want to commit the offsets as soon as messages are processed. We can use the overloaded method of commitSync() and commitAsync() that takes a map argument to commit the specific offset:

KafkaConsumer<Long, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(KafkaConfigProperties.getTopic());
Map<TopicPartition, OffsetAndMetadata> currentOffsets = new HashMap<>();
int messageProcessed = 0;
  while (true) {
    ConsumerRecords<Long, String> messages = consumer.poll(Duration.ofSeconds(10));
    for (ConsumerRecord<Long, String> message : messages) {
        // processed one message
      messageProcessed++;
      currentOffsets.put(
          new TopicPartition(message.topic(), message.partition()),
          new OffsetAndMetadata(message.offset() + 1));
      if (messageProcessed%50==0){
        consumer.commitSync(currentOffsets);
      }
    }
  }

In this code, we manage a currentOffsets map, which takes TopicPartition as key and OffsetAndMetadata as value. We insert the TopicPartition and OffsetAndMetadata of processed messages during message processing into the currentOffsets map. When the number of processed messages reaches fifty, we call commitSync() with the currentOffsets map to mark these messages as committed.

The behavior of this way is the same as sync and async commit. The only difference is that here we’re deciding the offsets to be committed not Kafka.

4. Conclusion

In this article, we learned about the offset and its importance in Kafka. Further, we explored the four ways to commit the offsets, both manual and automatic. Lastly, we analyzed their respective pros and cons. We can conclude that there is no definitive best way to commit in Kafka; rather, it depends on the specific use cases.

All the code examples used in this article are available over on GitHub.

       

1. Overview

In this tutorial, we’ll see different algorithms allowing us to find the largest number possible after removing the k digits of a number.

First, we’ll explain the problem. Then, we’ll see two different algorithms that suit our needs. Finally, we’ll discuss their complexities.

2. Problem Explanation

First, let’s explain what the goal of the algorithm is. We want to find the largest number possible after removing the k digits of a number.

For example, consider the number 286281. The number of elements we must remove is 2, so the largest possible number will be 8681. Let’s say we consider another value of k, i.e. 2, so the expected output would be 88.

3. Using Arithmetic

In this section, we’ll see the logic explanation and time and space complexity.

3.1. Logic

Let’s see the logic that would help achieve our goal using some arithmetic. We’ll use the method findLargestNumberUsingArithmetic(num, k) to implement our logic that returns the resultant number.

The function findLargestNumberUsingArithmetic(n,k) takes two parameters: $n$ (the original number) and $k$ (the number of digits to remove):

public static int findLargestNumberUsingArithmetic(int num, int k) {
    //...
    return num;
}

We have an outer loop that iterates k times, representing the number of digits to remove:

for (int j = 0; j < k; j++) {
    //...
}

For each iteration, it enters an inner loop to remove each digit once. The inner loop calculates the number formed after removing each digit once and compares it with the current maximum number:

while (num / i > 0) {
    int temp = (num / (i * 10)) * i + (num % i);
    i *= 10;
    result = Math.max(result, temp);
}
num = result;

After removing the digit, it updates the maximum number if a larger number is found.

After k iterations, it returns the remaining number, representing the largest number possible after removing k digits:

return num;

3.2. Time and Space Complexity

The code iterates through k iterations in the outer loop. Inside the outer loop, there’s a while loop that iterates through the digits of num. This loop executes for each digit of the num, which is approximately log base10 ​num times since we’re dividing num by 10 in each iteration. Therefore, the time complexity of the inner loop is O(K*log₁₀N).

We did not use any extra space, therefore the space complexity will be O(1).

4. Using Stack

In this section, we’ll see a more optimized approach to improve the complexity.

3.1. Logic

The approach involves using a stack to track the digits of the number while ensuring that the resulting number is maximized.

We’ll use the method findLargestNumberUsingStack(num, k) to implement our logic that returns the resultant number.

The function findLargestNumberUsingStack(num,k) takes two parameters: $num$ (the original number) and $k$ (the number of digits to remove).

Start with converting the number num into a character array or a string to iterate through its digits:

String numStr = Integer.toString(num); 
int length = numStr.length();

If the number of digits to remove is the same as the length of the input number, we have to return 0:

if (k == length) return 0;

Otherwise, initialize an empty stack to store the digits:

Stack<Character> stack = new Stack<>();

Now, iterate through each digit of the number num and follow the steps:

• While the stack is not empty, the current digit is greater than the top element of the stack, and the number of remaining digits to remove ($K$) is greater than 0, pop elements from the stack
• Push the current digit onto the stack

for (int i = 0; i < length; i++) {
    char digit = numStr.charAt(i);
    while (k > 0 && !stack.isEmpty() && stack.peek() < digit) {
        stack.pop();
        k--;
    }
    stack.push(digit);
}

If there are remaining digits to remove ($K$), pop elements from the stack to satisfy the condition:

while (k > 0) {
    stack.pop();
    k--;
}

Finally, construct the largest number from the digits remaining in the stack and return the result:

while (!stack.isEmpty()) {
    result.insert(0, stack.pop());
}
return Integer.parseInt(result.toString());

4.2. Time and Space Complexity

The algorithm iterates through each digit of the number once, performing constant-time operations within the loop. Therefore, the time complexity will be O(N).

The space required is primarily determined by the stack used to store the digits of the number. Therefore, the space complexity will be O(N).

5. Conclusion

In this article, we’ve examined algorithms for finding the largest number possible after removing the k digits of a number. We’ve seen how to achieve that in two ways: arithmetic and stack. We also discussed the time and space complexities of both algorithms, allowing us to choose one wisely according to our needs.

As usual, the complete code examples shown in this article are available over on GitHub.

       

1. Overview

When it comes to analyzing data in Java, calculating percentiles is a fundamental task in understanding the statistical distribution and characteristics of a numeric dataset.

In this tutorial, we’ll walk through the process of calculating percentiles in Java, providing code examples and explanations along the way.

2. Understanding Percentiles

Before discussing the implementation details, let’s first understand what percentiles are and how they’re commonly used in data analysis.

A percentile is a measure used in statistics indicating the value at or below which a given percentage of observations fall. For instance, the 50th percentile (also known as the median) represents the value below which 50% of the data points fall.

It’s worth noting that percentiles are expressed in the same unit of measurement as the input dataset, not in percent. For example, if the dataset refers to monthly salary, the corresponding percentiles will be expressed in USD, EUR, or other currencies.

Next, let’s see a couple of concrete examples:

Input: A dataset with numbers 1-100 unsorted
-> sorted dataset: [1, 2, ... 49, (50), 51, 52, ..100] 
-> The 50th percentile: 50
Input: [-1, 200, 30, 42, -5, 7, 8, 92]
-> sorted dataset: [-2, -1, 7, (8), 30, 42, 92, 200]
-> The 50th percentile: 8

Percentiles are often used to understand data distribution, identify outliers, and compare different datasets. They’re particularly useful when dealing with large datasets or when succinctly summarizing a dataset’s characteristics.

Next, let’s see how to calculate percentiles in Java.

3. Calculating Percentile From a Collection

Now that we understand what percentiles are. Let’s summarize a step-by-step guide to implementing the percentile calculation:

• Sort the given dataset in ascending order
• Calculate the rank of the required percentile as (percentile / 100) * dataset.size
• Take the ceiling value of the rank, as the rank can be a decimal number
• The final result is the element at the index ceiling(rank) – 1 in the sorted dataset

Next, let’s create a generic method to implement the above logic:

static <T extends Comparable<T>> T getPercentile(Collection<T> input, double percentile) {
    if (input == null || input.isEmpty()) {
        throw new IllegalArgumentException("The input dataset cannot be null or empty.");
    }
    if (percentile < 0 || percentile > 100) {
        throw new IllegalArgumentException("Percentile must be between 0 and 100 inclusive.");
    }
    List<T> sortedList = input.stream()
      .sorted()
      .collect(Collectors.toList());
    int rank = percentile == 0 ? 1 : (int) Math.ceil(percentile / 100.0 * input.size());
    return sortedList.get(rank - 1);
}

As we can see, the implementation above is pretty straightforward. However, it’s worth mentioning a couple of things:

• The validation of the percentile parameter is required ( 0<= percentile <= 100)
• We sorted the input dataset using the Stream API and collected the sorted result in a new list to avoid modifying the original dataset

Next, let’s test our getPercentile() method.

4. Testing the getPercentile() Method

First, the method should throw an IllegalArgumentException if the percentile is out of the valid range:

assertThrows(IllegalArgumentException.class, () -> getPercentile(List.of(1, 2, 3), -1));
assertThrows(IllegalArgumentException.class, () -> getPercentile(List.of(1, 2, 3), 101));

We used the assertThrows() method to verify if the expected exception was raised.

Next, let’s take a List of 1-100 as the input to verify whether the method can produce the expected result:

List<Integer> list100 = IntStream.rangeClosed(1, 100)
  .boxed()
  .collect(Collectors.toList());
Collections.shuffle(list100);
 
assertEquals(1, getPercentile(list100, 0));
assertEquals(10, getPercentile(list100, 10));
assertEquals(25, getPercentile(list100, 25));
assertEquals(50, getPercentile(list100, 50));
assertEquals(76, getPercentile(list100, 75.3));
assertEquals(100, getPercentile(list100, 100));

In the above code, we prepared the input list through an IntStream. Further, we used the shuffle() method to sort the 100 numbers randomly.

Additionally, let’s test our method with another dataset input:

List<Integer> list8 = IntStream.of(-1, 200, 30, 42, -5, 7, 8, 92)
  .boxed()
  .collect(Collectors.toList());
assertEquals(-5, getPercentile(list8, 0));
assertEquals(-5, getPercentile(list8, 10));
assertEquals(-1, getPercentile(list8, 25));
assertEquals(8, getPercentile(list8, 50));
assertEquals(92, getPercentile(list8, 75.3));
assertEquals(200, getPercentile(list8, 100));

5. Calculating Percentile From an Array

Sometimes, the given dataset input is an array instead of a Collection. In this case, we can first convert the input array to a List and then utilize our getPercentile() method to calculate the required percentiles.

Next, let’s demonstrate how to achieve this by taking a long array as the input:

long[] theArray = new long[] { -1, 200, 30, 42, -5, 7, 8, 92 };
 
//convert the long[] array to a List<Long>
List<Long> list8 = Arrays.stream(theArray)
  .boxed()
  .toList();
 
assertEquals(-5, getPercentile(list8, 0));
assertEquals(-5, getPercentile(list8, 10));
assertEquals(-1, getPercentile(list8, 25));
assertEquals(8, getPercentile(list8, 50));
assertEquals(92, getPercentile(list8, 75.3));
assertEquals(200, getPercentile(list8, 100));

As the code shows, since our input is an array of primitives (long[]), we employed Arrays.stream() to convert it to List<Long>. Then, we can pass the converted List to the getPercentile() to get the expected result.

6. Conclusion

In this article, we first discussed the underlying principles of percentiles. Then, we explored how to compute percentiles for a dataset in Java.

As always, the complete source code for the examples is available over on GitHub.

       

1. Introduction

Iterating over a List is a common operation in Java, but adding elements to it while iterating requires careful consideration to avoid exceptions and ensure the correctness of the code.

In this tutorial, we’ll discuss several methods for adding elements to a collection during iteration.

2. Utilizing the ListIterator Class

One common approach is a ListIterator, which provides bidirectional traversal and modification capabilities for lists.

2.1. String Scenario

Consider the following example, where we add the element JavaScript to the list of programming languages after encountering Python:

List<String> programmingLanguages = new ArrayList<>(List.of("Java", "Python", "C++"));
@Test
public void givenList_whenAddElementWithListIterator_thenModifiedList() {
    ListIterator<String> listIterator = programmingLanguages.listIterator();
    while (listIterator.hasNext()) {
        String language = listIterator.next();
        if (language.equals("Python")) {
            listIterator.add("JavaScript");
        }
    }
    assertIterableEquals(Arrays.asList("Java", "Python", "JavaScript", "C++"), programmingLanguages);
}

In the provided code, we initialize a List named programmingLanguages containing the strings (Java, Python, and C++). In addition, we iterate through the elements of the list using the listIterator.next() method.

When we encounter the element Python we dynamically insert the string JavaScript immediately after it, using listIterator.add(“JavaScript”).

Finally, the test asserts that the modified list matches the expected result, ensuring the successful addition of JavaScript after Python in the list.

2.2. Numeric Scenario

Let’s apply the ListIterator approach to a list of integers, adding double the value when encountering the number 2:

List<Integer> numbers = new ArrayList<>(List.of(1, 2, 3));
@Test
public void givenNumericalList_whenMultiplyElementWithListIterator_thenModifiedList() {
    ListIterator<Integer> listIterator = numbers.listIterator();
    while (listIterator.hasNext()) {
        int num = listIterator.next();
        if (num == 2) {
            listIterator.add(num * 10);
        }
    }
    assertIterableEquals(Arrays.asList(1, 2, 20, 3), numbers);
}

In this numeric scenario, we use the ListIterator to iterate through a list of integers. When the number 2 is encountered, the value is multiplied by 10 and dynamically added to the list.

3. Enhanced For Loop with a Copy

Another strategy involves creating a copy of the original list and iterating over it while modifying the original list.

3.1. String Scenario

Consider the following example, where we add the uppercase version of each word in the original list to the list itself:

@Test
public void givenStringList_whenAddElementWithEnhancedForLoopAndCopy_thenModifiedList() {
    List<String> copyOfWords = new ArrayList<>(programmingLanguages);
    for (String word : copyOfWords) {
        programmingLanguages.add(word.toUpperCase());
    }
    assertIterableEquals(Arrays.asList("Java", "Python", "C++", "JAVA", "PYTHON", "C++"), programmingLanguages);
}

Within the enhanced for loop, we iterate over each element of the copyOfWords list, convert the corresponding value to uppercase, and add it to the original list programmingLanguages.

Notably, this insertion process ensures that the original list is expanded with the uppercase versions of the existing words while maintaining the sequence integrity. In other words, the programmingLanguages list will contain the original elements followed by the newly added uppercase versions.

3.2. Numeric Scenario

Now, let’s apply the enhanced for-loop approach to a list of integers, adding each number multiplied by 2:

List<Integer> numbers = new ArrayList<>(List.of(1, 2, 3));
@Test
public void givenList_whenAddElementWithEnhancedForLoopAndCopy_thenModifiedList() {
    List<Integer> copyOfNumbers = new ArrayList<>(numbers);
    for (int num : copyOfNumbers) {
        numbers.add(num * 2);
    }
    assertIterableEquals(Arrays.asList(1, 2, 3, 2, 4, 6), numbers);
}

Here, we iterate, multiply each element multiplied by 2, and add it to the original list. Same as the string approach,

4. Utilizing the Java 8 Stream Approach

Java 8 Streams provide a concise way to add elements to a list during iteration.

4.1. String Scenario

Consider the following example, where we use Java 8 Streams to add the string JavaScript to the list of programmingLanguages:

@Test
public void givenStringList_whenConvertToUpperCaseWithJava8Stream_thenModifiedList() {
    programmingLanguages = programmingLanguages.stream().map(String::toUpperCase).collect(Collectors.toList());
    assertIterableEquals(Arrays.asList("JAVA", "PYTHON", "C++"), programmingLanguages);
}

In this code snippet, we utilize the map operation to transform each string element in the list to its uppercase equivalent using the toUpperCase method. Then, we collect the transformed elements into a new list using Collectors.toList().

However, it’s essential to note that each new element directly alters the corresponding original element in the list in place. This direct alteration demonstrates the transformation of the original list’s contents while preserving its integrity.

4.2. Numeric Scenario

Let’s apply the Java 8 Stream approach to a list of integers, multiplying each number by 3:

@Test
public void givenNumericalList_whenMultiplyByThreeWithJava8Stream_thenModifiedList() {
    numbers = numbers.stream().map(num -> num * 3).collect(Collectors.toList());
    assertIterableEquals(Arrays.asList(3, 6, 9), numbers);
}

In this test method, we utilize the map operation, each numeric element in the list undergoes a transformation where it is multiplied by 3. In addition, the resulting stream is collected into a new list through Collectors.toList().

5. Conclusion

In conclusion, we explored diverse methods, including ListIterator, enhanced for loop with a copy, and Java 8 Streams, for adding elements to a list during iteration in Java.

As usual, the accompanying source code can be found over on GitHub.

       

1. Overview

int result = 0b1100 & 0b0111;
assertEquals(0b0100, result);

int result = 0b1100 | 0b0111;
assertEquals(0b1111, result);

int result = 0b1100 ^ 0b0111;
assertEquals(0b1011, result);

int result = ~0b0101;
assertEquals(-0b0110, result);

int result = 0b0101 << 2;
assertEquals(0b10100, result);

int result = 0b0101 >> 1;
assertEquals(0b10, result);

int originalColor = 0xFF336699;
int alphaMask = 0xFF000000;
int redMask = 0x00FF0000;
int greenMask = 0x0000FF00;
int blueMask = 0x000000FF;

3.2. Extracting Color Components

int alpha = (originalColor & alphaMask) >>> 24;
int red = (originalColor & redMask) >>> 16;
int green = (originalColor & greenMask) >>> 8;
int blue = originalColor & blueMask;

red = Math.min(255, red + 50);
green = Math.min(255, green + 30);

int modifiedColor = (alpha << 24) | (red << 16) | (green << 8) | blue;

       

1. Overview

Querydsl and JPA Criteria are popular frameworks for constructing type-safe queries in Java. They both provide ways to express queries statically typed, making it easier to write efficient and maintainable code for interacting with databases. In this article, we’ll compare them from various perspectives.

2. Setup

First, we need to set up dependencies and configurations for our tests. In all the examples, we’ll use HyperSQL Database:

<dependency>
    <groupId>org.hsqldb</groupId>
    <artifactId>hsqldb</artifactId>
    <version>2.7.1</version>
</dependency>

We’ll use JPAMetaModelEntityProcessor and JPAAnnotationProcessor to generate metadata for our frameworks. For this purpose, we’ll add maven-processor-plugin with the following configuration:

<plugin>
    <groupId>org.bsc.maven</groupId>
    <artifactId>maven-processor-plugin</artifactId>
    <version>5.0</version>
    <executions>
        <execution>
            <id>process</id>
            <goals>
                <goal>process</goal>
            </goals>
            <phase>generate-sources</phase>
            <configuration>
                <processors>
                    <processor>org.hibernate.jpamodelgen.JPAMetaModelEntityProcessor</processor>
                    <processor>com.querydsl.apt.jpa.JPAAnnotationProcessor</processor>
                </processors>
            </configuration>
        </execution>
    </executions>
    <dependencies>
        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-jpamodelgen</artifactId>
            <version>6.2.0.Final</version>
        </dependency>
    </dependencies>
</plugin>

Then, let’s configure properties for our EntityManager:

<persistence-unit name="com.baeldung.querydsl.intro">
    <provider>org.hibernate.jpa.HibernatePersistenceProvider</provider>
    <properties>
        <property name="hibernate.connection.driver_class" value="org.hsqldb.jdbcDriver"/>
        <property name="hibernate.connection.url" value="jdbc:hsqldb:mem:test"/>
        <property name="hibernate.connection.username" value="sa"/>
        <property name="hibernate.connection.password" value=""/>
        <property name="hibernate.hbm2ddl.auto" value="update"/>
        <property name="hibernate.dialect" value="org.hibernate.dialect.HSQLDialect" />
    </properties>
</persistence-unit>

2.1. JPA Criteria

To use the EntityManager, we need to specify dependencies for any JPA provider. Let’s choose a Hibernate as the most popular one:

<dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-core</artifactId>
    <version>6.2.0.Final</version>
</dependency>

To support code generation functionality, we’ll add the Annotation Processor dependency:

<dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-jpamodelgen</artifactId>
    <version>6.2.0.Final</version>
</dependency>

2.2. Querydsl

Since we’re going to use it with the EntityManager, we still need to include the dependency from the previous section. Additionally, we’ll incorporate Querydsl dependency:

<dependency>
    <groupId>com.querydsl</groupId>
    <artifactId>querydsl-jpa</artifactId>
    <version>5.0.0</version>
</dependency>

To support code generation functionality, we’ll add APT based Source code generation dependency:

<dependency>
    <groupId>com.querydsl</groupId>
    <artifactId>querydsl-apt</artifactId>
    <classifier>jakarta</classifier>
    <version>5.0.0</version>
</dependency>

3. Simple Queries

Let’s start with the simple queries to one entity without any additional logic. We’ll use the next data model, the root entity will be a UserGroup:

@Entity
public class UserGroup {
    @Id
    @GeneratedValue
    private Long id;
    private String name;
    @ManyToMany(cascade = CascadeType.PERSIST)
    private Set<GroupUser> groupUsers = new HashSet<>();
    // getters and setters
}

In this entity, we’ll establish a many-to-many relationship with a GroupUser:

@Entity
public class GroupUser {
    @Id
    @GeneratedValue
    private Long id;
    private String login;
    @ManyToMany(mappedBy = "groupUsers", cascade = CascadeType.PERSIST)
    private Set<UserGroup> userGroups = new HashSet<>();
    @OneToMany(cascade = CascadeType.PERSIST, mappedBy = "groupUser")
    private Set<Task> tasks = new HashSet<>(0);
 
    // getters and setters
}

And, finally, we’ll add a Task entity that relates many-to-one to our User:

@Entity
public class Task {
    @Id
    @GeneratedValue
    private Long id;
    private String description;
    @ManyToOne
    private GroupUser groupUser;
    // getters and setters
}

3.1. JPA Criteria

Now, let’s select all the UserGroup items from the database:

@Test
void givenJpaCriteria_whenGetAllTheUserGroups_thenExpectedNumberOfItemsShouldBePresent() {
    CriteriaBuilder cb = em.getCriteriaBuilder();
    CriteriaQuery<UserGroup> cr = cb.createQuery(UserGroup.class);
    Root<UserGroup> root = cr.from(UserGroup.class);
    CriteriaQuery<UserGroup> select = cr.select(root);
    TypedQuery<UserGroup> query = em.createQuery(select);
    List<UserGroup> results = query.getResultList();
    Assertions.assertEquals(3, results.size());
}

We created an instance of CriteriaBuilder by calling the getCriteriaBuilder() method of EntityManager. Then, we created an instance of CriteriaQuery for our UserGroup model. After that, we obtained an instance of TypedQuery by calling the EntityManager createQuery() method. By calling the getResultList() method, we retrieved the list of entities from the database. As we can see, the expected number of items is present in the result collection.

3.2. Querydsl

Let’s prepare the JPAQueryFactory instance, which we’ll use to create our queries.

@BeforeEach
void setUp() {
    em = emf.createEntityManager();
    em.getTransaction().begin();
    queryFactory = new JPAQueryFactory(em);
}

Now, we’ll perform the same query as in the previous section using Querydsl:

@Test
void givenQueryDSL_whenGetAllTheUserGroups_thenExpectedNumberOfItemsShouldBePresent() {
    List<UserGroup> results = queryFactory.selectFrom(QUserGroup.userGroup).fetch();
    Assertions.assertEquals(3, results.size());
}

Using the selectFrom() method of the JPAQueryFactory starts building a query for our entity. Then, fetch() retrieves the values from the database into the persistence context. Finally, we’ve achieved the same result, but our query-building process is significantly shorter.

4. Filtering, Ordering and Grouping

Let’s delve into a more complex example. We’ll explore how our frameworks handle filtering, sorting, and data aggregation queries.

4.1. JPA Criteria

In this example, we’ll query all the UserGroup entities filtering them using names, which should be in one of two lists. The result we’ll sort by UserGroup name in descending order. Additionally, we’ll aggregate unique IDs per each UserGroup from the result:

@Test
void givenJpaCriteria_whenGetTheUserGroups_thenExpectedAggregatedDataShouldBePresent() {
    CriteriaBuilder cb = em.getCriteriaBuilder();
    CriteriaQuery<Object[]> cr = cb.createQuery(Object[].class);
    Root<UserGroup> root = cr.from(UserGroup.class);
    CriteriaQuery<Object[]> select = cr
      .multiselect(root.get(UserGroup_.name), cb.countDistinct(root.get(UserGroup_.id)))
      .where(cb.or(
        root.get(UserGroup_.name).in("Group 1", "Group 2"),
        root.get(UserGroup_.name).in("Group 4", "Group 5")
      ))
      .orderBy(cb.desc(root.get(UserGroup_.name)))
      .groupBy(root.get(UserGroup_.name));
    TypedQuery<Object[]> query = em.createQuery(select);
    List<Object[]> results = query.getResultList();
    assertEquals(2, results.size());
    assertEquals("Group 2", results.get(0)[0]);
    assertEquals(1L, results.get(0)[1]);
    assertEquals("Group 1", results.get(1)[0]);
    assertEquals(1L, results.get(1)[1]);
}

All the base methods here are the same as in the previous JPA Criteria section. Instead of selectFrom(), in this example, we use multiselect(), where we specify all the items that will be returned. We use the second parameter of this method for the aggregated amount of UserGroup IDs. In the where() method, we added filters we’ll apply to our query.

Then we call the orderBy() method, specifying the ordering field and type. Finally, in the groupBy() method, we specify a field that will be our key for aggregated data.

As we can see, a few UserGroup items were returned. They’re placed in the expected order and the result also contains aggregated data.

4.2. Querydsl

Now, let’s make the same query using Querydsl:

@Test
void givenQueryDSL_whenGetTheUserGroups_thenExpectedAggregatedDataShouldBePresent() {
    List<Tuple> results = queryFactory
      .select(userGroup.name, userGroup.id.countDistinct())
      .from(userGroup)
      .where(userGroup.name.in("Group 1", "Group 2")
        .or(userGroup.name.in("Group 4", "Group 5")))
      .orderBy(userGroup.name.desc())
      .groupBy(userGroup.name)
      .fetch();
    assertEquals(2, results.size());
    assertEquals("Group 2", results.get(0).get(userGroup.name));
    assertEquals(1L, results.get(0).get(userGroup.id.countDistinct()));
    assertEquals("Group 1", results.get(1).get(userGroup.name));
    assertEquals(1L, results.get(1).get(userGroup.id.countDistinct()));
}

Aiming for grouping functionality, we’ve replaced the selectFrom() method with two separate methods. In the select() method, we’ve specified the group field and aggregated function. In the from() method, we instruct the query builder on which entity should be applied. Similar to JPA Criteria, where(), orderBy(), and groupBy() are used to describe the filtering, ordering, and fields for grouping.

Finally, we achieved the same result with a slightly more compact syntax.

5. Complex Queries With JOINs

In this example, we’ll create complex queries that join all our entities. The result will contain a list of UserGroup entities with all their related entities.

Let’s prepare some data for our tests:

Stream.of("Group 1", "Group 2", "Group 3")
  .forEach(g -> {
      UserGroup userGroup = new UserGroup();
      userGroup.setName(g);
      em.persist(userGroup);
      IntStream.range(0, 10)
        .forEach(u -> {
            GroupUser groupUser = new GroupUser();
            groupUser.setLogin("User" + u);
            groupUser.getUserGroups().add(userGroup);
            em.persist(groupUser);
            userGroup.getGroupUsers().add(groupUser);
            IntStream.range(0, 10000)
              .forEach(t -> {
                  Task task = new Task();
                  task.setDescription(groupUser.getLogin() + " task #" + t);
                  task.setUser(groupUser);
                  em.persist(task);
              });
        });
      em.merge(userGroup);
  });

So, in our database, we’ll have three UserGroups, each containing ten GroupUsers. Each GroupUser will have ten thousand Tasks.

5.1. JPA Criteria

Now, let’s make a query using JPA CriteriaBuider:

@Test
void givenJpaCriteria_whenGetTheUserGroupsWithJoins_thenExpectedDataShouldBePresent() {
    CriteriaBuilder cb = em.getCriteriaBuilder();
    CriteriaQuery<UserGroup> query = cb.createQuery(UserGroup.class);
    query.from(UserGroup.class)
      .<UserGroup, GroupUser>join(GROUP_USERS, JoinType.LEFT)
      .join(tasks, JoinType.LEFT);
    List<UserGroup> result = em.createQuery(query).getResultList();
    assertUserGroups(result);
}

We use the join() method specifying the entity we want to join with and its type. After the execution, we retrieved a list of results. Let’s assert it using the following code:

private void assertUserGroups(List<UserGroup> userGroups) {
    assertEquals(3, userGroups.size());
    for (UserGroup group : userGroups) {
        assertEquals(10, group.getGroupUsers().size());
        for (GroupUser user : group.getGroupUsers()) {
            assertEquals(10000, user.getTasks().size());
        }
    }
}

As we can see, all the expected items were retrieved from the database.

5.2. Querydsl

Let’s achieve the same goal using Querydsl:

@Test
void givenQueryDSL_whenGetTheUserGroupsWithJoins_thenExpectedDataShouldBePresent() {
    List<UserGroup> result = queryFactory
      .selectFrom(userGroup)
      .leftJoin(userGroup.groupUsers, groupUser)
      .leftJoin(groupUser.tasks, task)
      .fetch();
    assertUserGroups(result);
}

Here, we use the leftJoin() method to add the connection to another entity. All the join types have separate methods. Both syntaxes are not very wordy. In the Querydsl implementation, our query is slightly more readable.

6. Modifying Data

Both frameworks support data modification. We can utilize it to update data based on complex and dynamic criteria. Let’s see how it works.

6.1. JPA Criteria

Let’s update UserGroup with a new name:

@Test
void givenJpaCriteria_whenModifyTheUserGroup_thenNameShouldBeUpdated() {
    CriteriaBuilder cb = em.getCriteriaBuilder();
    CriteriaUpdate<UserGroup> criteriaUpdate = cb.createCriteriaUpdate(UserGroup.class);
    Root<UserGroup> root = criteriaUpdate.from(UserGroup.class);
    criteriaUpdate.set(UserGroup_.name, "Group 1 Updated using Jpa Criteria");
    criteriaUpdate.where(cb.equal(root.get(UserGroup_.name), "Group 1"));
    em.createQuery(criteriaUpdate).executeUpdate();
    UserGroup foundGroup = em.find(UserGroup.class, 1L);
    assertEquals("Group 1 Updated using Jpa Criteria", foundGroup.getName());
}

To modify the data we use a CriteriaUpdate instance, which is used to create a Query. We set all the field names and values are going to be updated. Finally, we call the executeUpdate() method to run the update query. As we can see, we have a modified name field in the updated entity.

6.2. Querydsl

Now, let’s update UserGroup using Querydsl:

@Test
void givenQueryDSL_whenModifyTheUserGroup_thenNameShouldBeUpdated() {
    queryFactory.update(userGroup)
      .set(userGroup.name, "Group 1 Updated Using QueryDSL")
      .where(userGroup.name.eq("Group 1"))
      .execute();
    UserGroup foundGroup = em.find(UserGroup.class, 1L);
    assertEquals("Group 1 Updated Using QueryDSL", foundGroup.getName());
}

From queryFactory we create the update query by calling the update() method. Then, we set new values for the entity fields using the set() method. We’ve updated the name successfully. Similar to previous examples, Querydsl offers a slightly shorter and more declarative syntax.

7. Integration With Spring Data JPA

We can use Querydsl and JPA Criteria to implement dynamic filtering in our Spring Data JPA repositories. Let’s add Spring Data JPA starter dependency first:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-jpa</artifactId>
    <version>3.2.3</version>
</dependency>

7.1. JPA Criteria

Let’s create a Spring Data JPA repository for our UserGroup extending JpaSpecificationExecutor:

public interface UserGroupJpaSpecificationRepository 
  extends JpaRepository<UserGroup, Long>, JpaSpecificationExecutor<UserGroup> {
    default List<UserGroup> findAllWithNameInAnyList(List<String> names1, List<String> names2) {
        return findAll(specNameInAnyList(names1, names2));
    }
    default Specification<UserGroup> specNameInAnyList(List<String> names1, List<String> names2) {
        return (root, q, cb) -> cb.or(
          root.get(UserGroup_.name).in(names1),
          root.get(UserGroup_.name).in(names2)
        );
    }
}

In this repository, we’ve created a method that filters our results based on any of two name lists from the parameters. Let’s use it and see, how it works:

@Test
void givenJpaSpecificationRepository_whenGetTheUserGroups_thenExpectedDataShouldBePresent() {
    List<UserGroup> results = userGroupJpaSpecificationRepository.findAllWithNameInAnyList(
      List.of("Group 1", "Group 2"), List.of("Group 4", "Group 5"));
    assertEquals(2, results.size());
    assertEquals("Group 1", results.get(0).getName());
    assertEquals("Group 4", results.get(1).getName());
}

We can see that the result list contains exactly the expected groups.

7.2. Querydsl

The same functionality we can achieve using Querydsl Predicate. Let’s create another Spring Data JPA repository for the same entity:

public interface UserGroupQuerydslPredicateRepository 
  extends JpaRepository<UserGroup, Long>, QuerydslPredicateExecutor<UserGroup> {
    default List<UserGroup> findAllWithNameInAnyList(List<String> names1, List<String> names2) {
        return StreamSupport
          .stream(findAll(predicateInAnyList(names1, names2)).spliterator(), false)
          .collect(Collectors.toList());
    }
    default Predicate predicateInAnyList(List<String> names1, List<String> names2) {
        return new BooleanBuilder().and(QUserGroup.userGroup.name.in(names1))
          .or(QUserGroup.userGroup.name.in(names2));
    }
}

QuerydslPredicateExecutor provides only Iterable as the container for multiple results. If we want to use another type, we must handle the conversions ourselves. As we can see, the client code for this repository will be very similar to that for JPA specifications:

@Test
void givenQuerydslPredicateRepository_whenGetTheUserGroups_thenExpectedDataShouldBePresent() {
    List<UserGroup> results = userQuerydslPredicateRepository.findAllWithNameInAnyList(
      List.of("Group 1", "Group 2"), List.of("Group 4", "Group 5"));
    assertEquals(2, results.size());
    assertEquals("Group 1", results.get(0).getName());
    assertEquals("Group 4", results.get(1).getName());
}

8. Performance

Querydsl ultimately prepares the same criteria query but introduces additional conventions beforehand. Let’s measure how this process impacts the performance of our queries. To measure execution time, we can use our IDE functionality or create a timing extension.

We’ve executed all our test methods a few times and saved the median result into a list:

Method [givenJpaSpecificationRepository_whenGetTheUserGroups_thenExpectedDataShouldBePresent] took 128 ms.
Method [givenQuerydslPredicateRepository_whenGetTheUserGroups_thenExpectedDataShouldBePresent] took 27 ms.
Method [givenJpaCriteria_whenGetAllTheUserGroups_thenExpectedNumberOfItemsShouldBePresent] took 1 ms.
Method [givenQueryDSL_whenGetAllTheUserGroups_thenExpectedNumberOfItemsShouldBePresent] took 3 ms.
Method [givenJpaCriteria_whenModifyTheUserGroup_thenNameShouldBeUpdated] took 13 ms.
Method [givenQueryDSL_whenModifyTheUserGroup_thenNameShouldBeUpdated] took 161 ms.
Method [givenJpaCriteria_whenGetTheUserGroupsWithJoins_thenExpectedDataShouldBePresent] took 887 ms.
Method [givenQueryDSL_whenGetTheUserGroupsWithJoins_thenExpectedDataShouldBePresent] took 728 ms.
Method [givenJpaCriteria_whenGetTheUserGroups_thenExpectedAggregatedDataShouldBePresent] took 5 ms.
Method [givenQueryDSL_whenGetTheUserGroups_thenExpectedAggregatedDataShouldBePresent] took 88 ms.

As we can see, in most cases, we achieve similar execution times for both Querydsl and JPA Criteria. In the modification case, Querydsl uses JPQLSerializer and prepares a JPQL query string, which leads to additional overhead.

9. Conclusion

In this article, we’ve thoroughly compared JPA Criteria and Querydsl in various scenarios. In many cases, Querydsl emerged as the preferable option due to its slightly more user-friendly syntax. If a few more dependencies in the project are not an issue, we can consider it a good tool to improve our code readability. On the other hand, we can achieve all the functionality using JPA criteria.

As usual, the full source code can be found over on GitHub.

       

1. Overview

In today’s event-driven architectures, managing data streams effectively is essential. Apache Kafka is a popular choice for this, but integrating it into our applications has its challenges, despite helper frameworks such as Spring Kafka. One major challenge is implementing proper dynamic listener management, which provides flexibility and control that is crucial for adapting to our application’s changing workloads and maintenance.

In this tutorial, we’ll learn how to dynamically start and stop Kafka listeners in Spring Boot applications.

2. Prerequisites

First, let’s import the spring-kafka dependency into our project:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>3.1.2</version>
</dependency>

3. Configuring the Kafka Consumer

Producers are applications that publish (write) events to Kafka topics.

Throughout this tutorial, we’ll use unit tests to simulate producers sending events to Kafka topics. Consumers, who subscribe to topics and process the stream of events, are represented by a listener within our application. This listener is configured to process incoming messages from Kafka.

Let’s configure our Kafka consumer through the KafkaConsumerConfig class, which includes the Kafka broker’s address, the consumer group ID, and the deserializers for the key and value:

@Bean
public DefaultKafkaConsumerFactory<String, UserEvent> consumerFactory() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
    props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
    props.put(JsonDeserializer.TRUSTED_PACKAGES, "com.baeldung.spring.kafka.start.stop.consumer");
    return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(), new JsonDeserializer<>(UserEvent.class));
}

4. Configuring the Kafka Listener

In Spring Kafka, annotating a method with @KafkaListener creates a listener that consumes messages from a specified topic. To define it, let’s declare a UserEventListener class:

@KafkaListener(id = Constants.LISTENER_ID, topics = Constants.MULTI_PARTITION_TOPIC, groupId = "test-group",
  containerFactory = "kafkaListenerContainerFactory", autoStartup = "false")
public void processUserEvent(UserEvent userEvent) {
    logger.info("Received UserEvent: " + userEvent.getUserEventId());
    userEventStore.addUserEvent(userEvent);
}

The above listener waits for messages from the topic multi_partition_topic and processes them using the processUserEvent() method. We assign the groupId as test-group, ensuring that the consumer becomes part of a broader group, thus facilitating distributed processing across multiple instances.

We assign each listener a unique identifier using the id attribute. In this example, the assigned listener ID is listener-id-1.

The autoStartup property gives us control over whether the listener is started when the application is initialized. In our example, we’ll set it to false, which means the listener does not start automatically when the application boots up. This configuration provides us with the flexibility to initiate the listener manually.

This manual initiation can be triggered by various events, such as a new user registration, a specific condition within the application, such as reaching a certain data volume threshold, or an administrative action, such as manually starting the listener via a management interface. For example, if an online retail application detects a surge in traffic during a flash sale, it could automatically start additional listeners to handle the increased load, optimizing performance.

The UserEventStore acts as a temporary storage for events received by the listener:

@Component
public class UserEventStore {
    private final List<UserEvent> userEvents = new ArrayList<>();
    public void addUserEvent(UserEvent userEvent) {
        userEvents.add(userEvent);
    }
    public List<UserEvent> getUserEvents() {
        return userEvents;
    }
    public void clearUserEvents() {
        this.userEvents.clear();
    }
}

5. Dynamically Controlling the Listener

Let’s create a KafkaListenerControlService that dynamically starts and stops Kafka listeners using KafkaListenerEndpointRegistry:

@Service
public class KafkaListenerControlService {
    @Autowired
    private KafkaListenerEndpointRegistry registry;
    public void startListener(String listenerId) {
        MessageListenerContainer listenerContainer = registry.getListenerContainer(listenerId);
        if (listenerContainer != null && !listenerContainer.isRunning()) {
            listenerContainer.start();
        }
    }
    public void stopListener(String listenerId) {
        MessageListenerContainer listenerContainer = registry.getListenerContainer(listenerId);
        if (listenerContainer != null && listenerContainer.isRunning()) {
            listenerContainer.stop();
        }
    }
}

The KafkaListenerControlService can precisely manage individual listener instances based on their assigned ID. Both startListener() and stopListener() methods use listenerId as a parameter, allowing us to start and stop message consumption from the topics as required.

KafkaListenerEndpointRegistry acts as a central repository for all Kafka listener endpoints defined within a Spring application context. It monitors these listener containers, thus permitting programmatic control over their state, be it starting, stopping, or pausing. This capability is essential for applications that need to adjust their message processing activities in real-time, without necessitating a restart of the entire application.

6. Validate Dynamic Listener Controls

Next, let’s focus on testing the dynamic start and stop capabilities of Kafka listeners within our Spring Boot application. First, let’s start the listener:

kafkaListenerControlService.startListener(Constants.LISTENER_ID);

We then verify that the listener is activated by sending and then processing a test event:

UserEvent startUserEventTest = new UserEvent(UUID.randomUUID().toString()); 
producer.send(new ProducerRecord<>(Constants.MULTI_PARTITION_TOPIC, startUserEventTest)); 
await().untilAsserted(() -> assertEquals(1, this.userEventStore.getUserEvents().size())); 
this.userEventStore.clearUserEvents();

Now that the listener is active, we’ll send a batch of ten messages for processing. After sending four messages, we’ll stop the listener and then send the remaining messages to the Kafka topic:

for (long count = 1; count <= 10; count++) {
    UserEvent userEvent = new UserEvent(UUID.randomUUID().toString());
    Future<RecordMetadata> future = producer.send(new ProducerRecord<>(Constants.MULTI_PARTITION_TOPIC, userEvent));
    RecordMetadata metadata = future.get();
    if (count == 4) {
        await().untilAsserted(() -> assertEquals(4, this.userEventStore.getUserEvents().size()));
        this.kafkaListenerControlService.stopListener(Constants.LISTENER_ID);
        this.userEventStore.clearUserEvents();
    }
    logger.info("User Event ID: " + userEvent.getUserEventId() + ", Partition : " + metadata.partition());
}

Before starting the listener, we’ll verify that there are no messages in the event store:

assertEquals(0, this.userEventStore.getUserEvents().size());
kafkaListenerControlService.startListener(Constants.LISTENER_ID);
await().untilAsserted(() -> assertEquals(6, this.userEventStore.getUserEvents().size()));
kafkaListenerControlService.stopListener(Constants.LISTENER_ID);

Once the listener starts again, it processes the remaining six messages that we sent to the Kafka topic after the listener was stopped. This test showcases the Spring Boot application’s ability to dynamically manage Kafka listeners.

7. Use Cases

Dynamic listener management excels in scenarios requiring high adaptability. For example, during peak load times, we can dynamically start additional listeners to enhance throughput and reduce processing time. Conversely, during maintenance or low-traffic periods, we can stop listeners to conserve resources. This flexibility is also beneficial for deploying new features behind feature flags, allowing seamless on-the-fly adjustments without impacting the overall system.

Let’s consider a scenario where an e-commerce platform introduces a new recommendation engine designed to enhance the user experience by suggesting products based on browsing history and purchase patterns. To verify this feature’s effectiveness before its full-scale launch, we decide to deploy it behind a feature flag.

Activating this feature flag starts the Kafka listener. As end users interact with the platform, the recommendation engine, powered by Kafka listeners, processes the incoming stream of user activity data to generate personalized product recommendations.

When we deactivate the feature flag, we stop the Kafka listener, and the platform defaults to its existing recommendation engine. This ensures a seamless user experience, regardless of the testing phase of the new engine.

While the feature is active, we actively collect data, monitor performance metrics, and make adjustments to the recommendation engine. We repeat this feature testing across multiple iterations until we achieve the desired results.

Through this iterative process, dynamic listener management proves to be a valuable tool. It allows for the seamless introduction of new features

8. Conclusion

In this article, we’ve addressed Kafka integration with Spring Boot, focusing on dynamically managing Kafka listeners. This capability is crucial for managing fluctuating workloads and performing routine maintenance. Additionally, it enables feature toggling, scales services based on traffic patterns, and manages event-driven workflows with specific triggers.

As always, the source for the examples is available over on GitHub.

       

1. Introduction

When working with date and time data in Java applications, it’s often crucial to compare dates for various purposes, such as scheduling tasks, reminders, or reporting. One common scenario is the need to identify whether a given date corresponds to yesterday relative to the current date. In this tutorial, we’ll explore various approaches for determining whether a given date object falls on yesterday.

2. Using java.util.Calendar

One common approach is to use the java.util.Calendar classes to manipulate date and time information. To compare yesterday’s date, we instantiate a Calendar object via Calendar.getInstance(). Next, we set its time to the current date using calendar.setTime(new Date()), and then subtract one day using calendar.add(Calendar.DATE, -1). This yields yesterday’s date.

Here’s a code snippet demonstrating these concepts:

Date date = new Date(System.currentTimeMillis() - (1000 * 60 * 60 * 24));
Calendar expectedCalendar = Calendar.getInstance();
expectedCalendar.setTime(date);
Calendar actualCalendar = Calendar.getInstance();
actualCalendar.add(Calendar.DATE, -1);
boolean isEqualToYesterday = expectedCalendar.get(Calendar.YEAR) == actualCalendar.get(Calendar.YEAR) &&
  expectedCalendar.get(Calendar.MONTH) == actualCalendar.get(Calendar.MONTH) &&
  expectedCalendar.get(Calendar.DAY_OF_MONTH) == actualCalendar.get(Calendar.DAY_OF_MONTH);
assertTrue(isEqualToYesterday);

Directly comparing dates using this method may fail due to differences in milliseconds between the two dates. This is because the Date object captures both the date and time, including milliseconds. If the time component is not precisely the same, the comparison may yield incorrect results.

To mitigate this issue, one approach is to truncate the time component of both dates before performing the comparison. We extract the year, month, and day components from both the dates using their respective get(Calendar.YEAR), get(Calendar.MONTH), and get(Calendar.DAY_OF_MONTH) methods to solely determine whether the actualCalendar is equal to yesterday.

Although these classes are considered legacy APIs, they are still widely used, especially in environments where newer Java versions are not available.

3. Using java.util.Date Milliseconds

This approach leverages the fact that Date objects internally store milliseconds since the epoch (January 1, 1970, 00:00:00 UTC). It involves calculating yesterday’s midnight in milliseconds and comparing it with the getTime() value of the Date object.

We compute yesterday’s midnight in milliseconds using Instant.now().minus(1, ChronoUnit.DAYS).toEpochMilli(). This operation subtracts one day from the current instant and converts it to milliseconds since the epoch, effectively representing the timestamp for midnight of the previous day.

Subsequently, we compare this calculated value with the getTime() value of the given expectedDate. By verifying if expectedDate falls within the range from yesterday’s midnight timestamp to the next midnight (yesterdayMidnightMillis + 86_400_000 milliseconds), we ascertain whether yesterdayMidnightMillis corresponds to yesterday’s date.

Let’s demonstrate this approach with a quick example:

Date expectedDate = new Date(System.currentTimeMillis() - (1000 * 60 * 60 * 24));
long yesterdayMidnightMillis = Instant.now()
  .minus(1, ChronoUnit.DAYS)
  .toEpochMilli();
boolean isEqualToYesterday = expectedDate.getTime() >= yesterdayMidnightMillis && expectedDate.getTime() < yesterdayMidnightMillis + 86_400_000;
assertTrue(isEqualToYesterday);

This approach offers a simple and efficient method for comparing dates using milliseconds. It’s suitable for scenarios where direct comparison with legacy Date objects is required.

4. Using java.time.LocalDate 

Another approach to handling date manipulation is to use the LocalDate class along with its minusDays() method. This approach is part of the modern Date and Time API introduced in Java 8.

LocalDate offers a more straightforward and intuitive way to work with dates. It is designed to focus solely on the year, month, and day components, without a time component. This makes it ideal for date-only operations, such as recording historical dates or report date filtering.

To compare yesterday’s date using this method, we start by obtaining the current date using LocalDate.now(). Then, we simply call the minusDays(1) method on the LocalDate object to subtract one day, resulting in yesterday’s date:

Date date = new Date(System.currentTimeMillis() - (1000 * 60 * 60 * 24));
LocalDate expectedLocalDate = LocalDate.of(date.getYear() + 1900, date.getMonth() + 1, date.getDate());
LocalDate actualLocalDate = LocalDate.now().minusDays(1);
boolean isEqualToYesterday = expectedLocalDate.equals(actualLocalDate);
assertTrue(isEqualToYesterday);

In this example, we specify yesterday’s date by extracting the year, month, and day components from a Date object. However, note that the getYear() method returns the year since 1900, and getMonth() returns the month index starting from 0. Therefore, we add 1900 to getYear() and 1 to getMonth() to adjust them to the standard date representation.

Using LocalDate for date manipulation is recommended for Java versions 8 and above.

5. Using Joda-Time

Joda-Time is a popular date and time library for Java. We can use this library to check if a date object represents yesterday’s date. To compare yesterday’s date using Joda-Time, we utilize the DateTime class and its minusDays() method.

First, we obtain the current date and time using DateTime.now(). Then, we subtract one day from the current date by calling the minusDays(1) method, resulting in yesterday’s date:

Date date = new Date(System.currentTimeMillis() - (1000 * 60 * 60 * 24));
DateTime expectedDateTime = new DateTime(date).withTimeAtStartOfDay();
DateTime actualDateTime = DateTime.now().minusDays(1).withTimeAtStartOfDay();
boolean isEqualToYesterday = expectedDateTime.equals(actualDateTime);
assertTrue(isEqualToYesterday);

Similar to the Calendar class, Joda-Time’s DateTime class captures time along with the date, and thus, comparing them directly may lead to inaccuracies due to differences in milliseconds.

To resolve this, instead of breaking down the date value into year, month, and day components, we can leverage the withTimeAtStartOfDay() method provided by Joda-Time. This method sets the time component of a DateTime object to the start of the day, effectively resetting the time to midnight (00:00:00). By applying withTimeAtStartOfDay() to a DateTime object, we ensure that only the date component remains significant for comparison purposes.

6. Conclusion

In this article, we’ve explored a few approaches to determine whether a date object falls on yesterday’s date. For most modern Java applications using Java 8 or later, the LocalDate approach is generally recommended due to its clarity, immutability, and support for various date manipulations.

While Joda-Time is a mature and stable library, its future development and support may become less certain over time as more projects transition to the standard Java Date and Time API.

As always, the source code for the examples is available over on GitHub.

       

1. Overview

Caching data means our applications don’t have to access a slower storage layer, thereby improving their performance and responsiveness. We can implement caching using any in-memory implementation library like Caffeine.

Although doing this improves the performance of data retrieval, if the application is deployed to multiple replica sets, then the cache is not shared between instances. To overcome this problem, we can introduce a distributed cache layer that can be accessed by all instances.

In this article, we’ll learn how to implement the two-level caching mechanism in Spring. We’ll show how to implement both of these layers using Spring’s caching support, and how the distributed cache layer is called if the local cache layer incurs a cache miss.

2. Example Application in Spring Boot

Let’s imagine we need to build a simple application that calls a database to fetch some data.

2.1. Maven Dependency

First, let’s include the spring-boot-starter-web dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
    <version>3.1.5</version>
</dependency>

2.2. Implementing a Spring Service

We’ll implement a Spring service that fetches the data from a repository.

First, let’s model the Customer class:

public class Customer implements Serializable {
    private String id;
    private String name;
    private String email;
    // standard getters and setters
}

Then, let’s implement the CustomerService class and a getCustomer method:

@Service
public class CustomerService {
    
    private final CustomerRepository customerRepository;
    public Customer getCustomer(String id) {
        return customerRepository.getCustomerById(id);
    }
}

Finally, let’s define the CustomerRepository interface:

public interface CustomerRepository extends CrudRepository<Customer, String> {
}

Next, let’s implement the two levels of caching.

3. Implement the First Level of Cache

We’ll leverage Spring’s cache support and the Caffeine library to implement the first cache layer.

3.1. Caffeine Dependencies

Let’s include the spring-boot-starter-cache and caffeine dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-cache</artifactId>
    <version>3.1.5</version/
</dependency>
<dependency>
    <groupId>com.github.ben-manes.caffeine</groupId>
    <artifactId>caffeine</artifactId>
    <version>3.1.8</version>
</dependency>

3.2. Enable Caffeine Cache

To enable the Caffeine cache, we’ll need to add a few cache-related configurations.

First, let’s add the @EnableCaching annotation in the CacheConfig class and include a few Caffeine cache configs:

@Configuration
@EnableCaching
public class CacheConfig {
    @Bean
    public CaffeineCache caffeineCacheConfig() {
        return new CaffeineCache("customerCache", Caffeine.newBuilder()
          .expireAfterWrite(Duration.ofMinutes(1))
          .initialCapacity(1)
          .maximumSize(2000)
          .build());
    }
}

Next, let’s add the CaffeineCacheManager bean using the SimpleCacheManager class and set the cache config:

@Bean
public CacheManager caffeineCacheManager(CaffeineCache caffeineCache) {
    SimpleCacheManager manager = new SimpleCacheManager();
    manager.setCaches(Arrays.asList(caffeineCache));
    return manager;
}

3.3. Include the @Cacheable Annotation

To enable the above caching, we’ll need to add @Cacheable annotation in the getCustomer method:

@Cacheable(cacheNames = "customerCache", cacheManager = "caffeineCacheManager")
public Customer getCustomer(String id) {
}

As discussed earlier, this works well in a single instance deployment environment but is not very effective when the application runs with multiple replicas.

4. Implement the Second Level of Cache

We’ll implement the second level of caching using the Redis server. Of course, we can implement it with any other distributed cache like Memcached. This layer of cache will be accessible to all of our application’s replicas.

4.1. Redis Dependency

Let’s add the spring-boot-starter-redis dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
    <version>3.1.5</version>
</dependency>

4.2. Enable Redis Cache

We’ll need to add the Redis cache-related configuration to enable it in the application.

First, let’s configure the RedisCacheConfiguration bean with a few properties:

@Bean
public RedisCacheConfiguration cacheConfiguration() {
    return RedisCacheConfiguration.defaultCacheConfig()
      .entryTtl(Duration.ofMinutes(5))
      .disableCachingNullValues()
      .serializeValuesWith(RedisSerializationContext.SerializationPair.fromSerializer(new GenericJackson2JsonRedisSerializer()));
}

Then, let’s enable the CacheManager using the RedisCacheManager class:

@Bean
public CacheManager redisCacheManager(RedisConnectionFactory connectionFactory, RedisCacheConfiguration cacheConfiguration) {
    return RedisCacheManager.RedisCacheManagerBuilder
      .fromConnectionFactory(connectionFactory)
      .withCacheConfiguration("customerCache", cacheConfiguration)
      .build();
}

4.3. Include the @Caching and @Cacheable Annotations

We’ll include the second cache in the getCustomer method with the @Caching and @Cacheable annotations:

@Caching(cacheable = {
  @Cacheable(cacheNames = "customerCache", cacheManager = "caffeineCacheManager"),
  @Cacheable(cacheNames = "customerCache", cacheManager = "redisCacheManager")
})
public Customer getCustomer(String id) {
}

We should note that Spring will fetch the cache object from the first available cache. If both cache managers miss, it will run the actual method.

5. Implement the Integration Tests

To verify our setup, we’ll implement a few integration tests and validate both caches.

First, let’s create an integration test to verify both caches using an embedded Redis server:

@Test
void givenCustomerIsPresent_whenGetCustomerCalled_thenReturnCustomerAndCacheIt() {
    String CUSTOMER_ID = "100";
    Customer customer = new Customer(CUSTOMER_ID, "test", "test@mail.com");
    given(customerRepository.findById(CUSTOMER_ID))
      .willReturn(customer);
    
    Customer customerCacheMiss = customerService.getCustomer(CUSTOMER_ID);<code class="language-java">
    
    assertThat(customerCacheMiss).isEqualTo(customer);
    verify(customerRepository, times(1)).findById(CUSTOMER_ID);
    assertThat(caffeineCacheManager.getCache("customerCache").get(CUSTOMER_ID).get()).isEqualTo(customer);
    assertThat(redisCacheManager.getCache("customerCache").get(CUSTOMER_ID).get()).isEqualTo(customer);
}

We’ll run the above test case and find that this works fine.

Next, let’s imagine a scenario where the first level of cache data gets evicted due to expiry, and we try to get the same customer. Then, it should be a cache hit to the second cache level — Redis. Any further cache hit for the same customer should be to the first cache.

Let’s implement the above test scenario to check for both the caches after the local cache expiry:

@Test
void givenCustomerIsPresent_whenGetCustomerCalledTwiceAndFirstCacheExpired_thenReturnCustomerAndCacheIt() throws InterruptedException {
    String CUSTOMER_ID = "102";
    Customer customer = new Customer(CUSTOMER_ID, "test", "test@mail.com");
    given(customerRepository.findById(CUSTOMER_ID))
      .willReturn(customer);
    Customer customerCacheMiss = customerService.getCustomer(CUSTOMER_ID);
    TimeUnit.SECONDS.sleep(3);
    Customer customerCacheHit = customerService.getCustomer(CUSTOMER_ID);
    verify(customerRepository, times(1)).findById(CUSTOMER_ID);
    assertThat(customerCacheMiss).isEqualTo(customer);
    assertThat(customerCacheHit).isEqualTo(customer);
    assertThat(caffeineCacheManager.getCache("customerCache").get(CUSTOMER_ID).get()).isEqualTo(customer);
    assertThat(redisCacheManager.getCache("customerCache").get(CUSTOMER_ID).get()).isEqualTo(customer);
}

We now run the above test and see an unexpected assertion error with the Caffeine cache object:

org.opentest4j.AssertionFailedError: 
expected: Customer(id=102, name=test, email=test@mail.com)
but was: null
...
at com.baeldung.caching.twolevelcaching.CustomerServiceCachingIntegrationTest.
givenCustomerIsPresent_whenGetCustomerCalledTwiceAndFirstCacheExpired_thenReturnCustomerAndCacheIt(CustomerServiceCachingIntegrationTest.java:91)

From the above logs, it’s evident that the customer object is not in the Caffeine cache after the eviction, and even if we call the same method again, it’s not restored from the second cache. This is not an ideal situation for this use case, as every time the first level cache expires, it never gets updated until the second cache also expires. This puts additional load into the Redis cache.

We should note that Spring does not manage any data between multiple caches even if they are declared for the same method.

This tells us that we need to update the first-level cache whenever it’s accessed again.

6. Implement a Custom CacheInterceptor

To update the first cache, we’ll need to implement a custom cache interceptor to intercept whenever the cache is accessed.

We’ll add an interceptor to check if the current cache class is Redis type, and if the local cache does not exist, then we can update the cache value.

Let’s implement a custom CacheInterceptor by overriding the doGet method:

public class CustomerCacheInterceptor extends CacheInterceptor {
    private final CacheManager caffeineCacheManager;
    @Override
    protected Cache.ValueWrapper doGet(Cache cache, Object key) {
        Cache.ValueWrapper existingCacheValue = super.doGet(cache, key);
      
        if (existingCacheValue != null && cache.getClass() == RedisCache.class) {
            Cache caffeineCache = caffeineCacheManager.getCache(cache.getName());
            if (caffeineCache != null) {
                caffeineCache.putIfAbsent(key, existingCacheValue.get());
            }
        }
        return existingCacheValue;
    }
}

Also, we’ll need to register the CustomerCacheInterceptor bean to enable it:

@Bean
public CacheInterceptor cacheInterceptor(CacheManager caffeineCacheManager, CacheOperationSource cacheOperationSource) {
    CacheInterceptor interceptor = new CustomerCacheInterceptor(caffeineCacheManager);
    interceptor.setCacheOperationSources(cacheOperationSource);
    return interceptor;
}
@Bean
public CacheOperationSource cacheOperationSource() {
    return new AnnotationCacheOperationSource();
}

We should note that the custom interceptor will intercept the call whenever the Spring proxy method calls the get cache method internally.

We’ll re-run the integration tests, and see that the above test cases pass.

7. Conclusion

In this article, we’ve learned how to implement two levels of caching with Caffeine and Redis using Spring’s cache support. We’ve also seen how to update the first-level Caffeine cache with the custom cache interceptor implementation.

As always, the example code can be found over on GitHub.

       

1. Introduction

In Java, there are numerous ways to determine if a particular time lies within the two times without considering dates.

In this tutorial, we’ll look at several possible ways to achieve this.

2. Using the isAfter() and isBefore() Methods

Java 8 introduced the LocalTime class in the java.time package, providing convenient methods to work with time without considering the date:

LocalTime startTime = LocalTime.parse("09:00:00");
LocalTime endTime = LocalTime.parse("17:00:00");
LocalTime targetTime = LocalTime.parse("12:30:00");

Here, we initialize three LocalTime variables, startTime, endTime, and targetTime, with specific time values. These lines essentially set the starting time, ending time, and target time for comparison in the test methods.

LocalTime.parse(“09:00:00”) parses the string “09:00:00” into a LocalTime object representing 9:00 AM. Similarly, “17:00:00” represents 5:00 PM, and “12:30:00” represents 12:30 PM.

Let’s now test this approach:

@Test
void givenLocalTime_whenUsingIsAfterIsBefore_thenTimeIsBetween() {
    assertTrue(!targetTime.isBefore(startTime) && !targetTime.isAfter(endTime));
}

This method tests whether a given targetTime falls between a specified startTime and endTime using the isAfter() and isBefore() methods of the LocalTime class. We use the assetTrue() method to verify that the targetTime is neither before the startTime nor after the endTime.

3. Using the compareTo() Method

Another approach is to use the compareTo() method, which compares two LocalTime instances:

@Test
void givenLocalTime_whenUsingCompareTo_thenTimeIsBetween() {
    assertTrue(targetTime.compareTo(startTime) >= 0 && targetTime.compareTo(endTime) <= 0);
}

Here, the expression targetTime.compareTo(startTime) >= 0 && targetTime.compareTo(endTime) <= 0 combines two comparisons using logical AND (&&). It checks whether targetTime is greater than or equal to startTime and less than or equal to endTime.

This compound condition ensures that targetTime falls within the time range defined by startTime and endTime, inclusive. If both individual comparisons evaluate to true, the entire expression evaluates to true, indicating that the target time is between the start and end times, or coincides with them.

4. Using the after() and before() Methods

In this approach, we’ll use the legacy Date and Calendar objects to represent and compare times. Although this approach is considered less convenient than modern Java 8 LocalTime methods, it remains relevant for scenarios requiring compatibility with older codebases or systems:

@Test
void givenDate_whenUsingAfterBefore_thenTimeIsBetween() {
    Calendar startCalendar = Calendar.getInstance();
    startCalendar.set(Calendar.HOUR_OF_DAY, 9);
    startCalendar.set(Calendar.MINUTE, 0);
    Date startTime = startCalendar.getTime();
    Calendar endCalendar = Calendar.getInstance();
    endCalendar.set(Calendar.HOUR_OF_DAY, 17);
    endCalendar.set(Calendar.MINUTE, 0);
    Date endTime = endCalendar.getTime();
    Calendar targetCalendar = Calendar.getInstance();
    targetCalendar.set(Calendar.HOUR_OF_DAY, 12);
    targetCalendar.set(Calendar.MINUTE, 30);
    Date targetTime = targetCalendar.getTime();
    assertTrue(!targetTime.before(startTime) && !targetTime.after(endTime));
}

Firstly, we create Calendar instances for the startTime, endTime, and targetTime using the Calendar.getInstance() method, which returns a Calendar object initialized with the current date and time.

Once the Calendar objects are set up, we extract Date objects representing the start, end, and target times from their respective Calendar instances.

With the Date objects prepared, we compare the targetTime against the startTime and endTime. We achieve this by utilizing the Date class’s before() and after() methods.

5. Conclusion

In conclusion, Java offers multiple approaches, including modern LocalTime methods, legacy Date functionalities, and regular expressions, to accurately determine if a specific time falls within defined boundaries, catering to diverse development requirements.

As usual, the accompanying source code can be found over on GitHub.