Quantcast
Channel: Baeldung
Viewing all 4463 articles
Browse latest View live

Testing Kafka and Spring Boot

$
0
0

1. Overview

Apache Kafka is a powerful, distributed, fault-tolerant stream processing system. In a previous tutorial, we learned how to work with Spring and Kafka.

In this tutorial, we'll build on the previous one and learn how to write reliable, self-contained integration tests that don't rely on an external Kafka server running.

First, we'll start but looking at how to use and configure an embedded instance of Kafka. Then we'll see how we can make use of the popular framework Testcontainers from our tests.

2. Dependencies

Of course, we'll need to add the standard spring-kafka dependency to our pom.xml:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>2.6.3.RELEASE</version>
</dependency>

Then we'll need two more dependencies specifically for our tests. First, we'll add the spring-kafka-test artifact:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka-test</artifactId>
    <version>2.6.3.RELEASE</version>
    <scope>test</scope>
</dependency>

And finally, we'll add the Testcontainers Kafka dependency, which is also available over on Maven Central:

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>kafka</artifactId>
    <version>1.15.0</version>
    <scope>test</scope>
</dependency>

Now that we have all the necessary dependencies configured, we can write a simple Spring Boot application using Kafka.

3. A Simple Kafka Producer-Consumer Application

Throughout this tutorial, the focus of our tests will be a simple producer-consumer Spring Boot Kafka application.

Let's start by defining our application entry point:

@SpringBootApplication
@EnableAutoConfiguration
public class KafkaProducerConsumerApplication {
    public static void main(String[] args) {
        SpringApplication.run(KafkaProducerConsumerApplication.class, args);
    }
}

As we can see, this is a standard Spring Boot application. Where possible, we want to make use of default configuration values. With this in mind, we make use of the @EnableAutoConfiguration annotation to auto-config our application.

3.1. Producer Setup

Next, let's consider a producer bean that we'll use to send messages to a given Kafka topic:

@Component
public class KafkaProducer {
    private static final Logger LOGGER = LoggerFactory.getLogger(KafkaProducer.class);
    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;
    public void send(String topic, String payload) {
        LOGGER.info("sending payload='{}' to topic='{}'", payload, topic);
        kafkaTemplate.send(topic, payload);
    }
}

Our KafkaProducer bean defined above is merely a wrapper around the KafkaTemplate class. This class provides high-level thread-safe operations, such as sending data to the provided topic, which is exactly what we do in our send method.

3.2. Consumer Setup

Likewise, we'll now define a simple consumer bean which will listen to a Kafka topic and receives messages:

@Component
public class KafkaConsumer {
    private static final Logger LOGGER = LoggerFactory.getLogger(KafkaConsumer.class);
    private CountDownLatch latch = new CountDownLatch(1);
    private String payload = null;
    @KafkaListener(topics = "${test.topic}")
    public void receive(ConsumerRecord<?, ?> consumerRecord) {
        LOGGER.info("received payload='{}'", consumerRecord.toString());
        setPayload(consumerRecord.toString());
        latch.countDown();
    }
    public CountDownLatch getLatch() {
        return latch;
    }
    public String getPayload() {
        return payload;
    }
}

Our simple consumer uses the @KafkaListener annotation on the receive method to listen to messages on a given topic. We'll see later how we configure the test.topic from our tests.

Furthermore, the receive method stores the message content in our bean and decrements the count of the latch variable. This variable is a simple thread-safe counter field that we'll use later from our tests to ensure we successfully received a message.

Now that we have our simple Kafka application using Spring Boot implemented let's see how we can write integration tests.

4. A Word on Testing

In general, when writing clean integration tests, we shouldn't depend on external services that we might not be able to control or might suddenly stop working. This could have adverse effects on our test results.

Similarly, if we're dependent on an external service, in this case, a running Kafka broker, we likely won't be able to set it up, control it and tear it down in the way we want from our tests.

4.1. Application Properties

We're going to use a very light set of application configuration properties from our tests. We'll define these properties in our src/test/resources/application.yml file:

spring:
  kafka:
    consumer:
      auto-offset-reset: earliest
      group-id: baeldung
test:
  topic: embedded-test-topic

This is the minimum set of properties that we need when working with an embedded instance of Kafka or a local broker.

Most of these are self-explanatory, but the one we should highlight of particular importance is the consumer property auto-offset-reset: earliest. This property ensures that our consumer group gets the messages we send because the container might start after the sends have completed.

Additionally, we configure a topic property with the value embedded-test-topic, which is the topic we'll use from our tests.

5. Testing Using Embedded Kafka

In this section, we'll take a look at how to use an in-memory Kafka instance to run our tests against. This is also known as Embedded Kafka.

The dependency spring-kafka-test we added previously contains some useful utilities to assist with testing our application. Most notably, it contains the EmbeddedKafkaBroker class.

With that in mind, let's go ahead and write our first integration test:

@SpringBootTest
@DirtiesContext
@EmbeddedKafka(partitions = 1, brokerProperties = { "listeners=PLAINTEXT://localhost:9092", "port=9092" })
class EmbeddedKafkaIntegrationTest {
    @Autowired
    private KafkaConsumer consumer;
    @Autowired
    private KafkaProducer producer;
    @Value("${test.topic}")
    private String topic;
    @Test
    public void givenEmbeddedKafkaBroker_whenSendingtoSimpleProducer_thenMessageReceived() 
      throws Exception {
        producer.send(topic, "Sending with own simple KafkaProducer");
        consumer.getLatch().await(10000, TimeUnit.MILLISECONDS);
        
        assertThat(consumer.getLatch().getCount(), equalTo(0L));
        assertThat(consumer.getPayload(), containsString("embedded-test-topic"));
    }
}

Let's walk through the key parts of our test. First, we start by decorating our test class with two pretty standard Spring annotations:

  • The @SpringBootTest annotation will ensure that our test bootstraps the Spring application context
  • We also use the @DirtiesContext annotation, which will make sure this context is cleaned and reset between different tests

Here comes the crucial part, we use the @EmbeddedKafka annotation to inject an instance of an EmbeddedKafkaBroker into our tests. Moreover, there are several properties available we can use to configure the embedded Kafka node:

  • partitions – this is the number of partitions used per topic. To keep things nice and simple, we only want one to be used from our tests
  • brokerProperties – additional properties for the Kafka broker. Again we keep things simple and specify a plain text listener and a port number

Next, we auto-wire our consumer and producer classes and configure a topic to use the value from our application.properties.

For the final piece of the jigsaw, we simply send a message to our test topic and verify that the message has been received and contains the name of our test topic.

When we run our test, we'll see amongst the verbose Spring output:

...
12:45:35.099 [main] INFO  c.b.kafka.embedded.KafkaProducer -
  sending payload='Sending with our own simple KafkaProducer' to topic='embedded-test-topic'
...
12:45:35.103 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1]
  INFO  c.b.kafka.embedded.KafkaConsumer - received payload=
  'ConsumerRecord(topic = embedded-test-topic, partition = 0, leaderEpoch = 0, offset = 1,
  CreateTime = 1605267935099, serialized key size = -1, 
  serialized value size = 41, headers = RecordHeaders(headers = [], isReadOnly = false),
  key = null, value = Sending with our own simple KafkaProducer)'

This confirms that our test is working properly. Awesome! We now have a way to write self-contained, independent integration tests using an in-memory Kafka broker.

6. Testing Kafka With TestContainers

Sometimes we might see small differences between a real external service vs. an embedded in-memory instance of a service that has been specifically provided for testing purposes. Although unlikely, it could also be that the port used from our test might be occupied, causing a failure.

With that in mind, in this section, we'll see a variation on our previous approach to testing using the Testcontainers framework. We'll see how to instantiate and manage an external Apache Kafka broker hosted inside a Docker container from our integration test.

Let's define another integration test which will be quite similar to the one we saw in the previous section:

@RunWith(SpringRunner.class)
@Import(com.baeldung.kafka.testcontainers.KafkaTestContainersIntegrationTest.KafkaTestContainersConfiguration.class)
@SpringBootTest(classes = KafkaProducerConsumerApplication.class)
@DirtiesContext
public class KafkaTestContainersIntegrationTest {
    @ClassRule
    public static KafkaContainer kafka = 
      new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:5.4.3"));
    @Autowired
    private KafkaConsumer consumer;
    @Autowired
    private KafkaProducer producer;
    @Value("${test.topic}")
    private String topic;
    @Test
    public void givenKafkaDockerContainer_whenSendingtoSimpleProducer_thenMessageReceived() 
      throws Exception {
        producer.send(topic, "Sending with own controller");
        consumer.getLatch().await(10000, TimeUnit.MILLISECONDS);
        
        assertThat(consumer.getLatch().getCount(), equalTo(0L));
        assertThat(consumer.getPayload(), containsString("embedded-test-topic"));
    }
}

Let's take a look at the differences this time around. We're declaring the kafka field, which is a standard JUnit @ClassRule. This field is an instance of the KafkaContainer class that will prepare and manage the lifecycle of our container running Kafka.

To avoid port clashes, Testcontainers allocates a port number dynamically when our docker container starts. For this reason, we provide a custom consumer and producer factory configuration using the class KafkaTestContainersConfiguration:

@Bean
public Map<String, Object> consumerConfigs() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
    props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
    props.put(ConsumerConfig.GROUP_ID_CONFIG, "baeldung");
    // more standard configuration
    return props;
}
@Bean
public ProducerFactory<String, String> producerFactory() {
    Map<String, Object> configProps = new HashMap<>();
    configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka.getBootstrapServers());
    // more standard configuration
    return new DefaultKafkaProducerFactory<>(configProps);
}

We then reference this configuration via the @Import annotation at the beginning of our test.

The reason for this is that we need a way to inject the server address into our application, which as previously mentioned, is generated dynamically. We achieve this by calling the getBootstrapServers() method, which will return the bootstrap server location:

bootstrap.servers = [PLAINTEXT://localhost:32789]

Now when we run our test, we should see that Testcontainers does several things:

  • Checks our local Docker setup.
  • Pulls the confluentinc/cp-kafka:5.4.3 docker image if necessary
  • Starts a new container and waits for it to be ready
  • Finally, shuts down and deletes the container after our test finishes

Again, this is confirmed by inspecting the test output:

13:33:10.396 [main] INFO  🐳 [confluentinc/cp-kafka:5.4.3]
  - Creating container for image: confluentinc/cp-kafka:5.4.3
13:33:10.454 [main] INFO  🐳 [confluentinc/cp-kafka:5.4.3]
  - Starting container with ID: b22b752cee2e9e9e6ade38e46d0c6d881ad941d17223bda073afe4d2fe0559c3
13:33:10.785 [main] INFO  🐳 [confluentinc/cp-kafka:5.4.3]
  - Container confluentinc/cp-kafka:5.4.3 is starting: b22b752cee2e9e9e6ade38e46d0c6d881ad941d17223bda073afe4d2fe0559c3

Presto! A working integration test using a Kafka docker container.

7. Conclusion

In this article, we've learned about a couple of approaches for testing Kafka applications with Spring Boot. In the first approach, we saw how to configure and use a local in-memory Kafka broker.

Then we saw how to use Testcontainers to set up an external Kafka broker running inside a docker container from our tests.

As always, the full source code of the article is available over on GitHub.

The post Testing Kafka and Spring Boot first appeared on Baeldung.

        

Reusing Docker Layers with Spring Boot

$
0
0

1. Introduction

Docker is the de facto standard for creating self-contained applications. From version 2.3.0, Spring Boot includes several enhancements to help us create efficient Docker Images. Thus, it allows the decomposition of the application into different layers.

In other words, the source code resides in its own layer. Therefore, it can be independently rebuilt, improving efficiency and start-up time. In this tutorial, we'll see how to exploit the new capabilities of Spring Boot to reuse Docker layers.

2. Layered Jars in Docker

Docker containers consist of a base image and additional layers. Once the layers are built, they'll remain cached. Therefore, subsequent generations will be much faster:

Changes in the lower-level layers also rebuild the upper-level ones. Thus, the infrequently changing layers should remain at the bottom, and the frequently changing ones should be placed on top.

In the same way, Spring Boot allows mapping the content of the artifact into layers. Let's see the default mapping of layers:

As we can see, the application has its own layer. When modifying the source code, only the independent layer is rebuilt. The loader and the dependencies remain cached, reducing Docker image creation and startup time. Let's see how to do it with Spring Boot!

3. Creating Efficient Docker Images with Spring Boot

In the traditional way of building Docker images, Spring Boot uses the fat jar approach. As a result, a single artifact embeds all the dependencies and the application source code. So, any change in our source code forces the rebuilding of the entire layer.

3.1. Layers Configuration with Spring Boot

Spring Boot version 2.3.0 introduces two new features to improve the Docker image generation:

  • Buildpack support provides the Java runtime for the application, so it's now possible to skip the Dockerfile and build the Docker image automatically
  • Layered jars help us to get the most out of the Docker layer generation

In this tutorial, we'll extend the layered jar approach.

Initially, we'll set up the layered jar in Maven. When packaging the artifact, we'll generate the layers. Let's inspect the jar file:

jar tf target/spring-boot-docker-0.0.1-SNAPSHOT.jar

As we can see, a new layers.idx file in the BOOT-INF folder inside the fat jar is created. Certainly, it maps dependencies, resources, and application source code to independent layers:

BOOT-INF/layers.idx

Likewise, the content of the file breaks down the different layers stored:

- "dependencies":
  - "BOOT-INF/lib/"
- "spring-boot-loader":
  - "org/"
- "snapshot-dependencies":
- "application":
  - "BOOT-INF/classes/"
  - "BOOT-INF/classpath.idx"
  - "BOOT-INF/layers.idx"
  - "META-INF/"

3.2. Interacting with Layers

Let's list the layers inside the artifact:

java -Djarmode=layertools -jar target/docker-spring-boot-0.0.1.jar list

The result provides a simplistic view of the content of the layers.idx file:

dependencies
spring-boot-loader
snapshot-dependencies
application

We can also extract the layers into folders:

java -Djarmode=layertools -jar target/docker-spring-boot-0.0.1.jar extract

Then, we can reuse the folders inside the Dockerfile as we'll see in the next section:

$ ls
application/
snapshot-dependencies/
dependencies/
spring-boot-loader/

3.3. Dockerfile Configuration

To get the most out of the Docker capabilities, we need to add the layers to our image.

First, let's add the fat jar file to the base image:

FROM adoptopenjdk:11-jre-hotspot as builder
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} application.jar

Second, let's extract the layers of the artifact:

RUN java -Djarmode=layertools -jar application.jar extract

Finally, let's copy the extracted folders to add the corresponding Docker layers:

FROM adoptopenjdk:11-jre-hotspot
COPY --from=builder dependencies/ ./
COPY --from=builder snapshot-dependencies/ ./
COPY --from=builder spring-boot-loader/ ./
COPY --from=builder application/ ./
ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"]

With this configuration, when we change our source code, we'll only rebuild the application layer. The rest will remain cached.

4. Custom Layers

It seems everything is working like a charm. But if we look carefully, the dependency layer is not shared between our builds. That is to say, all of them come to a single layer, even the internal ones. Therefore, if we change the class of an internal library, we'll rebuild again all the dependency layers.

4.1. Custom Layers Configuration with Spring Boot

In Spring Boot, it's possible to tune custom layers through a separate configuration file:

<layers xmlns="http://www.springframework.org/schema/boot/layers"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/boot/layers
                     https://www.springframework.org/schema/boot/layers/layers-2.3.xsd">
    <application>
        <into layer="spring-boot-loader">
            <include>org/springframework/boot/loader/**</include>
        </into>
        <into layer="application" />
    </application>
    <dependencies>
        <into layer="snapshot-dependencies">
            <include>*:*:*SNAPSHOT</include>
        </into>
        <into layer="dependencies" />
    </dependencies>
    <layerOrder>
        <layer>dependencies</layer>
        <layer>spring-boot-loader</layer>
        <layer>snapshot-dependencies</layer>
        <layer>application</layer>
    </layerOrder>
</layers>

As we can see, we're mapping and ordering the dependencies and resources into layers. Furthermore, we can add as many custom layers as we want.

Let's name our file layers.xml. Then, in Maven, we can configure this file to customize the layers:

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <configuration>
        <layers>
            <enabled>true</enabled>
            <configuration>${project.basedir}/src/layers.xml</configuration>
        </layers>
    </configuration>
</plugin>

If we package the artifact, the result will be similar to the default behavior.

4.2. Adding New Layers

Let's create an internal dependency adding our application classes:

<into layer="internal-dependencies">
    <include>com.baeldung.docker:*:*</include>
</into>

In addition, we'll order the new layer:

<layerOrder>
    <layer>internal-dependencies</layer>
</layerOrder>

As a result, if we list the layers inside the fat jar, the new internal dependency appears:

dependencies
spring-boot-loader
internal-dependencies
snapshot-dependencies
application

4.3. Dockerfile Configuration

Once extracted, we can add the new internal layer to our Docker image:

COPY --from=builder internal-dependencies/ ./

So, if we generate the image, we'll see how Docker builds the internal dependency as a new layer:

$ mvn package
$ docker build -f src/main/docker/Dockerfile . --tag spring-docker-demo
....
Step 8/11 : COPY --from=builder internal-dependencies/ ./
 ---> 0e138e074118
.....

After that, we can check in the history the composition of layers in the Docker image:

$ docker history --format "{{.ID}} {{.CreatedBy}} {{.Size}}" spring-docker-demo
c0d77f6af917 /bin/sh -c #(nop)  ENTRYPOINT ["java" "org.s… 0B
762598a32eb7 /bin/sh -c #(nop) COPY dir:a87b8823d5125bcc4… 7.42kB
80a00930350f /bin/sh -c #(nop) COPY dir:3875f37b8a0ed7494… 0B
0e138e074118 /bin/sh -c #(nop) COPY dir:db6f791338cb4f209… 2.35kB
e079ad66e67b /bin/sh -c #(nop) COPY dir:92a8a991992e9a488… 235kB
77a9401bd813 /bin/sh -c #(nop) COPY dir:f0bcb2a510eef53a7… 16.4MB
2eb37d403188 /bin/sh -c #(nop)  ENV JAVA_HOME=/opt/java/o… 0B

As we can see, the layer now includes the internal dependencies of the project.

5. Conclusion

In this tutorial, we showed how to generate efficient Docker images. In short, we used the new Spring Boot features to create layered jars. For simple projects, we can use the default configuration. We also demonstrated a more advanced configuration to reuse the layers.

As always, the code is available over on GitHub.

The post Reusing Docker Layers with Spring Boot first appeared on Baeldung.

        

Creating a Generic Array in Java

$
0
0

1. Introduction

We may wish to use arrays as part of classes or functions that support generics. Due to the way Java handles generics, this can be difficult.

In this tutorial, we'll understand the challenges of using generics with arrays. Then, we'll create an example of a generic array.

We'll also look at where the Java API has solved a similar problem.

2. Considerations When Using Generic Arrays

An important difference between arrays and generics is how they enforce type checking. Specifically, arrays store and check type information at runtime. Generics, however, check for type errors at compile-time and don't have type information at runtime.

Java's syntax suggests we might be able to create a new generic array:

T[] elements = new T[size];

But, if we attempted this, we'd get a compile error.

To understand why, let's consider the following:

public <T> T[] getArray(int size) {
    T[] genericArray = new T[size]; // suppose this is allowed
    return genericArray;
}

As an unbound generic type T resolves to Object, our method at runtime will be:

public Object[] getArray(int size) {
    Object[] genericArray = new Object[size];
    return genericArray;
}

Then, if we call our method and store the result in a String array:

String[] myArray = getArray(5);

The code will compile fine but fail at runtime with a ClassCastException. This is because we've just assigned an Object[] to a String[] reference. Specifically, an implicit cast by the compiler would fail to convert Object[] to our required type String[].

Although we can't initialize generic arrays directly, it is still possible to achieve the equivalent operation if the precise type information is provided by the calling code.

3. Creating a Generic Array

For our example, let's consider a bounded stack data structure MyStack, where the capacity is fixed to a certain size. Also, as we'd like the stack to work with any type, a reasonable implementation choice would be a generic array.

First, let's create a field to store the elements of our stack, which is a generic array of type E:

private E[] elements;

Second, let's add a constructor:

public MyStack(Class<E> clazz, int capacity) {
    elements = (E[]) Array.newInstance(clazz, capacity);
}

Notice how we use java.lang.reflect.Array#newInstance to initialize our generic array, which requires two parameters. The first parameter specifies the type of object inside the new array. The second parameter specifies how much space to create for the array. As the result of Array#newInstance is of type Object, we need to cast it to E[] to create our generic array.

We should also note the convention of naming a type parameter clazz rather than class, which is a reserved word in Java.

4. Considering ArrayList

4.1. Using ArrayList in Place of an Array

It's often easier to use a generic ArrayList in place of a generic array. Let's see how we can change MyStack to use an ArrayList.

First, let's create a field to store our elements:

private List<E> elements;

Secondly, in our stack constructor, we can initialize the ArrayList with an initial capacity:

elements = new ArrayList<>(capacity);

It makes our class simpler, as we don't have to use reflection. Also, we aren't required to pass in a class literal when creating our stack. Finally, as we can set the initial capacity of an ArrayList, we can get the same benefits as an array.

Therefore, we only need to construct arrays of generics in rare situations or when we're interfacing with some external library that requires an array.

4.2. ArrayList Implementation

Interestingly, ArrayList itself is implemented using generic arrays. Let's peek inside ArrayList to see how.

First, let's see the list elements field:

transient Object[] elementData;

Notice ArrayList uses Object as the element type. As our generic type is not known until runtime, Object is used as the superclass of any type.

It's worth noting that nearly all the operations in ArrayList can use this generic array as they don't need to provide a strongly typed array to the outside world, except for one method – toArray!

5. Building an Array from a Collection

5.1. LinkedList Example

Let's look at using generic arrays in the Java Collections API, where we'll build a new array from a collection.

First, let's create a new LinkedList with a type argument String and add items to it:

List<String> items = new LinkedList();
items.add("first item");
items.add("second item");

Second, let's build an array of the items we've just added:

String[] itemsAsArray = items.toArray(new String[0]);

To build our array, the List.toArray method requires an input array. It uses this array purely to get the type information to create a return array of the right type.

In our example above, we've used new String[0] as our input array to build the resulting String array.

5.2. LinkedList.toArray Implementation

Let's take a peek inside LinkedList.toArray, to see how it's implemented in the Java JDK.

First, let's look at the method signature:

public <T> T[] toArray(T[] a)

Second, let's see how a new array is created when required:

a = (T[])java.lang.reflect.Array.newInstance(a.getClass().getComponentType(), size);

Notice how it makes use of Array#newInstance to build a new array, like in our stack example earlier. Also, notice how parameter a is used to provide a type to Array#newInstance. Finally, the result from Array#newInstance is cast to T[] create a generic array.

6. Conclusion

In this article, we first looked at differences between arrays and generics, followed by an example of creating a generic array. Then, we showed how using an ArrayList may be easier than using a generic array. Finally, we also looked at the use of a generic array in the Collections API.

As always, the example code is available over on GitHub.

The post Creating a Generic Array in Java first appeared on Baeldung.

        

Explanation of ClassCastException in Java

$
0
0

1. Overview

In this short tutorial, we'll focus on ClassCastException, a common Java exception.

ClassCastException is an unchecked exception that signals the code has attempted to cast a reference to a type of which it's not a subtype.

Let's look at some scenarios that lead to this exception being thrown and how we can avoid them.

2. Explicit Casting

For our next experiments, let's consider the following classes:

public interface Animal {
    String getName();
}
public class Mammal implements Animal {
    @Override
    public String getName() {
        return "Mammal";
    }
}
public class Amphibian implements Animal {
    @Override
    public String getName() {
        return "Amphibian";
    }
}
public class Frog extends Amphibian {
    @Override
    public String getName() {
        return super.getName() + ": Frog";
    }
}

2.1. Casting Classes

By far, the most common scenario for encountering a ClassCastException is explicitly casting to an incompatible type.

For example, let's try to cast a Frog to a Mammal:

Frog frog = new Frog();
Mammal mammal = (Mammal) frog;

We might expect a ClassCastException here, but in fact, we get a compilation error: “incompatible types: Frog cannot be converted to Mammal”. However, the situation changes when we use the common super-type:

Animal animal = new Frog();
Mammal mammal = (Mammal) animal;

Now, we get a ClassCastException from the second line:

Exception in thread "main" java.lang.ClassCastException: class Frog cannot be cast to class Mammal (Frog and Mammal are in unnamed module of loader 'app') 
at Main.main(Main.java:9)

A checked downcast to Mammal is incompatible from a Frog reference because Frog is not a subtype of Mammal. In this case, the compiler cannot help us, as the Animal variable may hold a reference of a compatible type.

It's interesting to note that the compilation error only occurs when we attempt to cast to an unequivocally incompatible class. The same is not true for interfaces because Java supports multiple interface inheritance, but only single inheritance for classes. Thus, the compiler can't determine if the reference type implements a specific interface or not. Let's exemplify:

Animal animal = new Frog();
Serializable serial = (Serializable) animal;

We get a ClassCastException on the second line instead of a compilation error:

Exception in thread "main" java.lang.ClassCastException: class Frog cannot be cast to class java.io.Serializable (Frog is in unnamed module of loader 'app'; java.io.Serializable is in module java.base of loader 'bootstrap') 
at Main.main(Main.java:11)

2.2. Casting Arrays

We've seen how classes handle casting, now let's look at arrays. Array casting works the same as class casting. However, we might get confused by autoboxing and type-promotion, or lack thereof.

Thus, let's see what happens for primitive arrays when we attempt the following cast:

Object primitives = new int[1];
Integer[] integers = (Integer[]) primitives;

The second line throws a ClassCastException as autoboxing doesn't work for arrays.

How about type promotion? Let's try the following:

Object primitives = new int[1];
long[] longs = (long[]) primitives;

We also get a ClassCastException because type promotion doesn't work for entire arrays.

2.3. Safe Casting

In the case of explicit casting, it is highly recommended to check the compatibility of the types before attempting to cast using instanceof.

Let's look at a safe cast example:

Mammal mammal;
if (animal instanceof Mammal) {
    mammal = (Mammal) animal;
} else {
    // handle exceptional case
}

3. Heap Pollution

As per the Java Specification: “Heap pollution can only occur if the program performed some operation involving a raw type that would give rise to a compile-time unchecked warning”.

For our experiment, let's consider the following generic class:

public static class Box<T> {
    private T content;
    public T getContent() {
        return content;
    }
    public void setContent(T content) {
        this.content = content;
    }
}

We will now attempt to pollute the heap as follows:

Box<Long> originalBox = new Box<>();
Box raw = originalBox;
raw.setContent(2.5);
Box<Long> bound = (Box<Long>) raw;
Long content = bound.getContent();

The last line will throw a ClassCastException as it cannot transform a Double reference to Long.

4. Generic Types

When using generics in Java, we must be wary of type erasure, which can lead to ClassCastException as well in some conditions.

Let's consider the following generic method:

public static <T> T convertInstanceOfObject(Object o) {
    try {
        return (T) o;
    } catch (ClassCastException e) {
        return null;
    }
}

And now let's call it:

String shouldBeNull = convertInstanceOfObject(123);

At first look, we can reasonably expect a null reference returned from the catch block. However, at runtime, due to type erasure, the parameter is cast to Object instead of String. Thus the compiler is faced with the task of assigning an Integer to String, which throws ClassCastException.

5. Conclusion

In this article, we have looked at a series of common scenarios for inappropriate casting.

Whether implicit or explicit, casting Java references to another type can lead to ClassCastException unless the target type is the same or a descendent of the actual type.

The code used in this article can be found over on GitHub.

The post Explanation of ClassCastException in Java first appeared on Baeldung.

        

Performance Difference Between save() and saveAll() in Spring Data

$
0
0

1. Overview

In this quick tutorial, we'll learn about the performance difference between save() and saveAll() methods in Spring Data.

2. Application

In order to test the performance, we'll need a Spring application with an entity and a repository.

Let's create a book entity:

@Entity
public class Book {
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private long id;
    private String title;
    private String author;
    // constructors, standard getters and setters
}

In addition, let's create a repository for it:

public interface BookRepository extends JpaRepository<Book, Long> {
}

3. Performance

To test the performance, we'll save 10,000 books using both methods.

First, we'll use the save() method:

for(int i = 0; i < bookCount; i++) {
    bookRepository.save(new Book("Book " + i, "Author " + i));
}

Then, we'll create a list of books and use the saveAll() method to save all of them at once:

List<Book> bookList = new ArrayList<>();
for (int i = 0; i < bookCount; i++) {
    bookList.add(new Book("Book " + i, "Author " + i));
}
bookRepository.saveAll(bookList);

In our tests, we noticed that the first method took around 2 seconds, and the second one took approximately 0.3 seconds.

Furthermore, when we enabled JPA Batch Inserts, we observed a decrease of up to 10% in the performance of the save() method, and an increase of up to 60% on the saveAll() method.

4. Differences

Looking into the implementation of the two methods, we can see that saveAll() iterates over each element and uses the save() method in each iteration. This implies that it shouldn't be such a big performance difference.

Looking more closely, we observe that both methods are annotated with @Transactional.

Furthermore, the default transaction propagation type is REQUIRED, which means that, if not provided, a new transaction is created each time the methods are called.

In our case, each time we call the save() method, a new transaction is created, whereas when we call saveAll(), only one transaction is created, and it's reused later by save().

This overhead translates into the performance difference that we noticed earlier.

Finally, the overhead is bigger when batching is enabled due to the fact that it's done at the transaction level.

5. Conclusion

In this article, we've learned about the performance difference between the save() and saveAll() methods in Spring Data.

Ultimately, choosing whether to use one method over another can have a big performance impact on the application.

As always, the code for these examples is available over on GitHub.

The post Performance Difference Between save() and saveAll() in Spring Data first appeared on Baeldung.

        

The Capacity of an ArrayList vs the Size of an Array in Java

$
0
0

1. Overview

Java allows us to create arrays of fixed size or use collection classes to do a similar job.

In this tutorial, we're going to look at the difference between the capacity of an ArrayList and the size of an Array.

We'll also look at examples of when we should initialize ArrayList with a capacity and the benefits and disadvantages in terms of memory usage.

2. Example

To understand the differences, let's first try both options.

2.1. Size of an Array

In java, it's mandatory to specify the size of an array while creating a new instance of it:

Integer[] array = new Integer[100]; 
System.out.println("Size of an array:" + array.length);

Here, we created an Integer array of size 100, which resulted in the below output

Size of an array:100

2.2. Capacity of an ArrayList

Now, let's create an ArrayList with an initial capacity of 100:

List<Integer> list = new ArrayList<>(100);
System.out.println("Size of the list is :" + list.size());
Size of the list is :0

As no elements have been added yet, the size is zero.

Now, let's add an element to the list and check the size of it:

list.add(10);
System.out.println("Size of the list is :" + list.size());
Size of the list is :1

3. Size in Arrays vs. ArrayList

Below are some major differences between the size of an array and the capacity of an ArrayList.

3.1. Modification of Size

Arrays are fixed size. Once we initialize the array with some int value as its size, it can't change. The size and capacity are equal to each other too.

ArrayList‘s size and capacity are not fixed. The logical size of the list changes based on the insertion and removal of elements in it. This is managed separately from its physical storage size. Also when the threshold of ArrayList capacity is reached, it increases its capacity to make room for more elements.

3.2. Memory Allocation

Array memory is allocated on creation. When we initialize an array, it allocates the memory according to the size and type of an array. It initializes all the elements with a null value for reference types and the default value for primitive types.

ArrayList changes the memory allocation as it grows. When we specify the capacity while initializing the ArrayList, it allocates enough memory to store objects up to that capacity. The logical size remains 0. When it is time to expand the capacity, a new, larger array is created, and the values are copied to it.

We should note that there's a special singleton 0-sized array for empty ArrayList objects, making them very cheap to create. It's also worth noting that ArrayList internally uses an array of Object references.

4. When to Initialize ArrayList with Capacity

We may expect to initialize the capacity of an ArrayList when we know its required size before we create it, but it's not usually necessary. However, there are a few reasons why this may be the best option.

4.1. Building a Large ArrayList

It is good to initialize a list with an initial capacity when we know that it will get large. This prevents some costly grow operations as we add elements.

Similarly, if the list is very large, the automatic grow operations may allocate more memory than necessary for the exact maximum size. This is because the amount to grow each time is calculated as a proportion of the size so far. So, with large lists, this could result in a waste of memory.

4.2. Building Small Multiple ArrayLists

If we have a lot of small collections, then the automatic capacity of an ArrayList may provide a large percentage of wasted memory. Let's say that ArrayList prefers a size of 10 with smaller numbers of elements, but we are only storing 2 or 3. That means 70% wasted memory, which might matter if we have a huge number of these lists.

Setting the capacity upfront can avoid this situation.

5. Avoiding Waste

We should note that ArrayList is a good solution for a flexible-sized container of objects that is to support random access. It consumes slightly more memory than an array but provides a richer set of operations.

In some use cases, especially around large collections of primitive values, the standard array may be faster and use less memory.

Similarly, for storing a variable number of elements that do not need to be accessed by index, LinkedList can be more performant. It does not come with any overhead of memory management.

6. Summary

In this short article, we saw the difference between the capacity of the ArrayList and the size of an array. We also looked at when we should initialize the ArrayList with capacity and its benefits with regards to memory usage and performance.

As always, the example code is available over on GitHub.

The post The Capacity of an ArrayList vs the Size of an Array in Java first appeared on Baeldung.

        

Java Weekly, Issue 360

$
0
0

1. Spring and Java

>> From Spring Boot to Quarkus [blog.frankel.ch]

The new kid on the block: a practical guide on how to migrate a typical Spring Boot app to Quarkus!

>> New language features since Java 8 to 15 [advancedweb.hu]

Java evolution for people in a hurry: an anthology of improvements and new features available in modern Java.

>> Getting Started with Spring Data Specifications [reflectoring.io]

Flexing data access with Specifications: a more maintainable approach for ad-hoc queries with a myriad of custom filters.

Also worth reading:

Webinars and presentations:

Time to upgrade:

2. Technical

>> On distributed systems and elliptic curve cryptography [martin.kleppmann.com]

Going all academic on distributed systems and EC cryptography: An academical and yet practical take on distributed systems and ECC.

Also worth reading:

3. Musings

>> The No-Excuses Guide to Innovation with APIs [blog.scottlogic.com]

Reinventing digital services by thinking and acting more strategically instead of pursuing one-off solutions.

Also worth reading:

4. Comics

And my favorite Dilberts of the week:

>> Information From Carl [dilbert.com]

>> Rewriting Shakespeare [dilbert.com]

>> Real Men Multitask [dilbert.com]

5. Pick of the Week

Datadog is back, with full visibility into an always growing number of solid, native integrations – both in and out of the Java ecosystem:

>> End-to-end visibility into performance and code efficiency with Datadog APM

Simply put, you can drill in starting with the browser all the way down to individual DB queries – with no sampling, which is a little bit crazy to think about 🙂

The post Java Weekly, Issue 360 first appeared on Baeldung.

        

What’s New in Java 15

$
0
0

1. Introduction

Java 15 reached general availability in September 2020 and is the next short-term release for the JDK platform. It builds on several features from earlier releases and also provides some new enhancements.

In this post, we'll look at some of the new features of Java 15, as well as other changes that are of interest to Java developers.

2. Records (JEP 384)

The record is a new type of class in Java that makes it easy to create immutable data objects.

Originally introduced in Java 14 as an early preview, Java 15 aims to refine a few aspects before becoming an official product feature.

Let's look at an example using current Java and how it could change with records.

2.1. Without Records

Prior to records, we would create an immutable data transfer object (DTO) as:

public class Person {
    private final String name;
    private final int age;
    public Person(String name, int age) {
        this.name = name;
        this.age = age;
    }
    public String getName() {
        return name;
    }
    public int getAge() {
        return age;
    }
}

Notice that there's a lot of code here to create an immutable object that really just holds state. All of our fields are explicitly defined using final, we have a single all-arguments constructor, and we have an accessor method for every field. In some cases, we might even declare the class itself as final to prevent any sub-classing.

In many cases, we would also go a step further and override the toString method to provide meaningful logging output. We would probably also want to override the equals and hashCode methods to avoid unexpected consequences when comparing two instances of these objects.

2.2. With Records

Using the new record class, we can define the same immutable data object in a much more compact way:

public record Person(String name, int age) {
}

A few things have happened here. First and foremost, the class definition has a new syntax that is specific for records. This header is where we provide the details about the fields inside the record.

Using this header, the compiler can infer the internal fields. This means we don't need to define specific member variables and accessors, as they're provided by default. We also don't have to provide a constructor.

Additionally, the compiler provides sensible implementations for the toString, equals, and hashCode methods.

While records eliminate a lot of boilerplate code, they do allow us to override some of the default behaviors. For example, we could define a canonical constructor that does some validation:

public record Person(String name, int age) {
    public Person {
        if(age < 0) {
            throw new IllegalArgumentException("Age cannot be negative");
        }
    }
}

It's worth mentioning that records do have some restrictions. Among other things, they are always final, they cannot be declared abstract, and they can't use native methods.

3. Sealed Classes (JEP 360)

Currently, Java provides no fine-grained control over inheritance. Access modifiers such as public, protected, private, as well as the default package-private, provide very coarse-grained control.

To that end, the goal of sealed classes is to allow individual classes to declare which types may be used as sub-types. This also applies to interfaces and determining which types can implement them.

Sealed classes involve two new keywords — sealed and permits:

public abstract sealed class Person
    permits Employee, Manager {
 
    //...
}

In this example, we've declared an abstract class named Person. We've also specified that the only classes that can extend it are Employee and Manager. Extending the sealed class is done just as it is today in Java, using the extends keyword:

public final class Employee extends Person {
}
public non-sealed class Manager extends Person {
}

It's important to note that any class that extends a sealed class must itself be declared sealed, non-sealed, or final. This ensures the class hierarchy remains finite and known by the compiler.

This finite and exhaustive hierarchy is one of the great benefits of using sealed classes. Let's see an example of this in action:

if (person instanceof Employee) {
    return ((Employee) person).getEmployeeId();
} 
else if (person instanceof Manager) {
    return ((Manager) person).getSupervisorId();
}

Without a sealed class, the compiler can't reasonably determine that all possible sub-classes are covered with our if-else statements. Without an else clause at the end, the compiler would likely issue a warning indicating our logic doesn't cover every case.

4. Hidden Classes (JEP 371)

A new feature being introduced in Java 15 is known as hidden classes. While most developers won't find a direct benefit from them, anyone who works with dynamic bytecode or JVM languages will likely find them useful.

The goal of hidden classes is to allow the runtime creation of classes that are not discoverable. This means they cannot be linked by other classes, nor can they be discovered via reflection. Classes such as these typically have a short lifecycle, and thus, hidden classes are designed to be efficient with both loading and unloading.

Note that current versions of Java do allow for the creation of anonymous classes similar to hidden classes. However, they rely on the Unsafe API. Hidden classes have no such dependency.

5. Pattern Matching Type Checks (JEP 375)

The pattern matching feature was previewed in Java 14, and Java 15 aims to continue its preview status with no new enhancements.

As a review, the goal of this feature is to remove a lot of boilerplate code that typically comes with the instanceof operator:

if (person instanceof Employee) {
    Employee employee = (Employee) person;
    Date hireDate = employee.getHireDate();
    //...
}

This is a very common pattern in Java. Whenever we check if a variable is a certain type, we almost always follow it with a cast to that type.

The pattern matching feature simplifies this by introducing a new binding variable:

if (person instanceof Employee employee) {
    Date hireDate = employee.getHireDate();
    //...
}

Notice how we provide a new variable name, employee, as part of the type check. If the type check is true, then the JVM automatically casts the variable for us and assigns the result to the new binding variable.

We can also combine the new binding variable with conditional statements:

if (person instanceof Employee employee && employee.getYearsOfService() > 5) {
    //...
}

In future Java versions, the goal is to expand pattern matching to other language features such as switch statements.

6. Foreign Memory API (JEP 383)

Foreign memory access is already an incubating feature of Java 14. In Java 15, the goal is to continue its incubation status while adding several new features:

  • A new VarHandle API, to customize memory access var handles
  • Support for parallel processing of a memory segment using the Spliterator interface
  • Enhanced support for mapped memory segments
  • Ability to manipulate and dereference addresses coming from things like native calls

Foreign memory generally refers to memory that lives outside the managed JVM heap. Because of this, it's not subject to garbage collection and can typically handle incredibly large memory segments.

While these new APIs likely won't impact most developers directly, they will provide a lot of value to third-party libraries that deal with foreign memory. This includes distributed caches, denormalized document stores, large arbitrary byte buffers, memory-mapped files, and more.

7. Garbage Collectors

In Java 15, both ZGC (JEP 377) and Shenandoah (JEP 379) will be no longer be experimental. Both will be supported configurations that teams can opt to use, while the G1 collector will remain the default.

Both were previously available using experimental feature flags. This approach allows developers to test the new garbage collectors and submit feedback without downloading a separate JDK or add-on.

One note on Shenandoah: it isn't available from all vendor JDKs — most notably, Oracle JDK doesn't include it.

8. Other Changes

There are several other noteworthy changes in Java 15.

After multiple rounds of previews in Java 13 and 14, text blocks will be a fully supported product feature in Java 15.

Helpful null pointer exceptions, originally delivered in Java 14 under JEP 358, are now enabled by default.

The legacy DatagramSocket API has been rewritten. This is a follow-on to a rewrite in Java 14 of the Socket API. While it won't impact most developers, it is interesting as it's a prerequisite for Project Loom.

Also of note, Java 15 includes cryptographic support for Edwards-Curve Digital Signature Algorithm. EdDSA is a modern elliptic curve signature scheme that has several advantages over the existing signature schemes in the JDK.

Finally, several things have been deprecated in Java 15. Biased locking, Solaris/SPARC ports, and RMI Activation are all removed or scheduled for removal in a future release.

Of note, The Nashorn JavaScript engine, originally introduced in Java 8, is now removed. With the introduction of GraalVM and other VM technologies recently, it's clear Nashorn no longer has a place in the JDK ecosystem.

9. Conclusion

Java 15 builds on several features of past releases, including records, text blocks, new garbage collection algorithms, and more. It also adds new preview features, including sealed classes and hidden classes.

As Java 15 is not a long-term-support release, we can expect support for it to end in March 2021. At that time, we can look forward to Java 16, followed soon after with a new long-term-support version in Java 17.

The post What's New in Java 15 first appeared on Baeldung.

        

How to Access an Iteration Counter in a For Each Loop

$
0
0

1. Overview

While iterating over data in Java, we may wish to access both the current item and its position in the data source.

This is very easy to achieve in a classic for loop, where the position is usually the focus of the loop's calculations, but it requires a little more work when we use constructs like for each loops or streams.

In this short tutorial, we'll look at a few ways that for each operation can include a counter.

2. Implementing a Counter

Let's start with a simple example. We'll take an ordered list of movies and output them with their ranking.

List<String> IMDB_TOP_MOVIES = Arrays.asList("The Shawshank Redemption",
  "The Godfather", "The Godfather II", "The Dark Knight");

2.1. for Loop

A for loop uses a counter to reference the current item, so it's an easy way to operate over both the data and its index in the list:

List rankings = new ArrayList<>();
for (int i = 0; i < movies.size(); i++) {
    String ranking = (i + 1) + ": " + movies.get(i);
    rankings.add(ranking);
}

As this List is probably an ArrayList, the get operation is efficient, and the above code is a simple solution to our problem.

assertThat(getRankingsWithForLoop(IMDB_TOP_MOVIES))
  .containsExactly("1: The Shawshank Redemption",
      "2: The Godfather", "3: The Godfather II", "4: The Dark Knight");

However, not all data sources in Java can be iterated over this way. Sometimes get is a time-intensive operation, or we can only process the next element of a data source using Stream or Iterable.

2.2. for Each loop

We'll continue using our list of movies, but let's pretend that we can only iterate over it using Java's for each construct:

for (String movie : IMDB_TOP_MOVIES) {
   // use movie value
}

Here we need to use a separate variable to track the current index. We can construct that outside of the loop, and increment it inside:

int i = 0;
for (String movie : movies) {
    String ranking = (i + 1) + ": " + movie;
    rankings.add(ranking);
    i++;
}

We should note that we have to increment the counter after it has been used within the loop.

3. A Functional for Each

Writing the counter extension every time we need it might result in code duplication and may risk accidental bugs concerning when to update the counter variable. We can, therefore, generalize the above using Java's functional interfaces.

First, we should think of the behavior inside the loop as a consumer of both the item in the collection and also the index. This can be modeled using BiConsumer, which defines an accept function that takes two parameters

@FunctionalInterface
public interface BiConsumer<T, U> {
   void accept(T t, U u);
}

As the inside of our loop is something that uses two values, we could write a general looping operation. It could take the Iterable of the source data, over which the for each loop will run, and the BiConsumer for the operation to perform on each item and its index. We can make this generic with the type parameter T:

static <T> void forEachWithCounter(Iterable<T> source, BiConsumer<Integer, T> consumer) {
    int i = 0;
    for (T item : source) {
        consumer.accept(i, item);
        i++;
    }
}

We can use this with our movie rankings example by providing the implementation for the BiConsumer as a lambda:

List rankings = new ArrayList<>();
forEachWithCounter(movies, (i, movie) -> {
    String ranking = (i + 1) + ": " + movies.get(i);
    rankings.add(ranking);
});

4. Adding a Counter to forEach with Stream

The Java Stream API allows us to express how our data passes through filters and transformations. It also provides a forEach function. Let's try to convert that into an operation that includes the counter.

The Stream forEach function takes a Consumer to process the next item. We could, however, create that Consumer to keep track of the counter and pass the item onto a BiConsumer:

public static <T> Consumer<T> withCounter(BiConsumer<Integer, T> consumer) {
    AtomicInteger counter = new AtomicInteger(0);
    return item -> consumer.accept(counter.getAndIncrement(), item);
}

This function returns a new lambda. That lambda uses the AtomicInteger object to keep track of the counter during iteration. The getAndIncrement function is called every time there's a new item.

The lambda created by this function delegates to the BiConsumer passed in so that the algorithm can process both the item and its index.

Let's see this in use by our movie ranking example against a Stream called movies:

List rankings = new ArrayList<>();
movies.forEach(withCounter((i, movie) -> {
    String ranking = (i + 1) + ": " + movie;
    rankings.add(ranking);
}));

Inside the forEach is a call to the withCounter function to create an object which both tracks the count and acts as the Consumer that the forEach operation passes its values to.

5. Conclusion

In this short article, we've looked at three ways to attach a counter to a Java for each operation.

We saw how to track the index of the current item on each implementation of the for loop. We then looked at how to generalize this pattern and how to add it to streaming operations.

As always the example code for this article is available over on GitHub.

The post How to Access an Iteration Counter in a For Each Loop first appeared on Baeldung.

        

Java Weekly, Issue 361

$
0
0

1. Spring and Java

>> Announcing GraalVM 20.3 [medium.com]

Improved container awareness, G1 GC for native images, and many more performance improvements, all in a new GraalVM version!

>> Vanilla Java: Using Java SE as a Framework [renato.athaydes.com]

A minimal server framework on top of Java SE: hot-reloading, annotation processors and code generation, native images, and JUnit integration.

>> What We Know About Java 16 and 17 So Far [infoq.com]

Speculating over the Loom and Valhalla release: delaying the Java 17 LTS release or delivering small chunks of big features?

Also worth reading:

Webinars and presentations:

Time to upgrade:

2. Technical

>> Integrating PIT Mutation Testing and GitHub Pages with GitHub Actions [4comprehension.com]

Publishing PIT mutation test reports to GitHub pages: a practical guide to integrating GitHub actions with PIT.

Also worth reading:

3. Musings

>> Humility [benjiweber.co.uk]

In praise of Humility: enabling compassion, keeping things simple, valuing feedback, and courage!

Also worth reading:

4. Comics

And my favorite Dilberts of the week:

>> Dilbert Not On Mute [dilbert.com]

>> Working At Home Benefits [dilbert.com]

>> Online Class Muted [dilbert.com]

5. Pick of the Week

Not exactly a launch but a couple of weeks ago, I introduced our longer-term COVID pricing for 2020:

>> All Courses on Baeldung

Here's the plan – I'm going to keep the courses at 25% off their standard price in November and then 20% off in December.

Hope that helps.

The post Java Weekly, Issue 361 first appeared on Baeldung.

        

Guide to the System Stubs Library

$
0
0

1. Overview

It can be hard to test our software when it depends on system resources like environment variables, system properties, or uses process-level operations like System.exit.

Java doesn't provide a direct method for setting environment variables, and we run the risk of the values set in one test affecting the execution of another. Similarly, we may find ourselves avoiding writing JUnit tests for code that might perform a System.exit as there's a chance it would abort the tests.

The System Rules and System Lambda Libraries were early solutions to these problems. In this tutorial, we'll look at a new fork of System Lambda called System Stubs, which provides a JUnit 5 alternative.

2. Why System Stubs?

2.1. System Lambda is Not a JUnit Plugin

The original System Rules library was only usable with JUnit 4. It could still be used with JUnit Vintage under JUnit 5, but that required the continued creation of JUnit 4 tests. The creators of the library produced a test framework agnostic version called System Lambda, which was intended for use inside each test method:

@Test
void aSingleSystemLambda() throws Exception {
    restoreSystemProperties(() -> {
        System.setProperty("log_dir", "test/resources");
        assertEquals("test/resources", System.getProperty("log_dir"));
    });
    // more test code here
}

The test code is expressed as a lambda, passed to a method that sets up the necessary stubbing. The cleanup happens just before control is returned to the rest of the test method.

Though this works well in some cases, the approach has a few disadvantages.

2.2. Avoiding Extra Code

The benefit of the System Lambda approach is that there are some common recipes inside its factory class for performing specific types of tests. However, this leads to some code bloat when we want to use it across many test cases.

Firstly, even if the test code itself doesn't throw a checked exception, the wrapper method does, so all methods gain a throws Exception. Secondly, setting up the same rule across multiple tests requires code duplication. Each test needs to perform the same configuration independently.

However, the most cumbersome aspect of this approach comes when we try to set up more than one tool at a time. Let's say we want to set some environment variables and system properties. We end up needing two levels of nesting before our test code starts:

@Test
void multipleSystemLambdas() throws Exception {
    restoreSystemProperties(() -> {
        withEnvironmentVariable("URL", "https://www.baeldung.com")
            .execute(() -> {
                System.setProperty("log_dir", "test/resources");
                assertEquals("test/resources", System.getProperty("log_dir"));
                assertEquals("https://www.baeldung.com", System.getenv("URL"));
            });
    });
}

This is where a JUnit plugin or extension can help us cut down the amount of code we need in our tests.

2.3. Using Less Boilerplate

We should expect to be able to write our tests with a minimum of boilerplate:

@SystemStub
private EnvironmentVariables environmentVariables = ...;
@SystemStub
private SystemProperties restoreSystemProperties;
@Test
void multipleSystemStubs() {
    System.setProperty("log_dir", "test/resources");
    assertEquals("test/resources", System.getProperty("log_dir"));
    assertEquals("https://www.baeldung.com", System.getenv("ADDRESS"));
}

This approach is provided by the SystemStubs JUnit 5 extension and allows our tests to be composed with less code.

2.4. Test Lifecycle Hooks

When the only available tool is the execute-around pattern, it's impossible to hook in the stubbing behavior to all parts of the test lifecycle. This is particularly challenging when trying to combine it with other JUnit extensions, like @SpringBootTest.

If we wanted to set up some environment variables around a Spring Boot test, then there is no way we could reasonably embed that whole test eco-system inside a single test method. We would need a way to activate the test set-up around a test suite.

This was never going to be possible with the methodology employed by System Lambda and was one of the main reasons to create System Stubs.

2.5. Encourage Dynamic Properties

Other frameworks for setting system properties, such as JUnit Pioneer, emphasize configurations known at compile time. In modern tests, where we may be using Testcontainers or Wiremock, we need to set up our system properties based on random runtime settings after those tools startup. This works best with a test library that can be used across the whole test lifecycle.

2.6. More Configurability

It's beneficial having ready-made test recipes, like catchSystemExit, which wrap around test code to do a single job. However, this relies on the test library developers to provide each variation of configuration option we might need.

Configuration by composition is more flexible and is a large part of the new System Stubs implementation.

However, System Stubs supports the original test constructs from System Lambda for backward compatibility. Additionally, it provides a new JUnit 5 extension, a set of JUnit 4 rules, and many more configuration options. Though based on the original code, it has been heavily refactored and modularised to provide a richer set of features.

Let's learn more about it.

3. Getting Started

3.1. Dependencies

The JUnit 5 extension requires a reasonably up to date version of JUnit 5:

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>5.6.2</version>
    <scope>test</scope>
</dependency>

Let's add all the System Stubs library dependencies to our pom.xml:

<!-- for testing with only lambda pattern -->
<dependency>
    <groupId>uk.org.webcompere</groupId>
    <artifactId>system-stubs-core</artifactId>
    <version>1.1.0</version>
    <scope>test</scope>
</dependency>
<!-- for JUnit 4 -->
<dependency>
    <groupId>uk.org.webcompere</groupId>
    <artifactId>system-stubs-junit4</artifactId>
    <version>1.1.0</version>
    <scope>test</scope>
</dependency>
<!-- for JUnit 5 -->
<dependency>
    <groupId>uk.org.webcompere</groupId>
    <artifactId>system-stubs-jupiter</artifactId>
    <version>1.1.0</version>
    <scope>test</scope>
</dependency>

We should note that we only need to import as many of these as we need for the test framework we're using. Indeed, both the latter two transitively include the core dependency.

Now let's write our first test.

3.2. JUnit 4 Environment Variables

We can control environment variables by declaring a JUnit 4 @Rule annotated field in our test class of type EnvironmentVariablesRule. This will be activated by JUnit 4 when our tests run and will allow us to set environment variables inside the test:

@Rule
public EnvironmentVariablesRule environmentVariablesRule = new EnvironmentVariablesRule();
@Test
public void givenEnvironmentCanBeModified_whenSetEnvironment_thenItIsSet() {
    environmentVariablesRule.set("ENV", "value1");
    assertThat(System.getenv("ENV")).isEqualTo("value1");
}

In practice, we may prefer to set the environment variable values in a @Before method so that the set-up can be shared across all tests:

@Before
public void before() {
    environmentVariablesRule.set("ENV", "value1")
      .set("ENV2", "value2");
}

Here we should note using the fluent set method, which makes setting multiple values easy through method chaining.

We can also use the constructor of the EnvironmentVariablesRule object to provide values on construction:

@Rule
public EnvironmentVariablesRule environmentVariablesRule =
  new EnvironmentVariablesRule("ENV", "value1",
    "ENV2", "value2");

There are several overloads of the constructor, allowing variables to be provided in different forms. The one in the above example allows any number of name-value pairs to be provided using varargs.

Each of the System Stubs JUnit 4 rules is a subclass of one of the core stubbing objects. They can also be used across the lifecycle of a whole test class with the @ClassRule annotation on a static field, which will cause them to be activated before the first test, and then cleaned up just after the last.

3.3. JUnit 5 Environment Variables

Before we use System Stubs objects inside a JUnit 5 test, we must add the extension to our test class:

@ExtendWith(SystemStubsExtension.class)
class EnvironmentVariablesJUnit5 {
    // tests
}

Then we can create a field in the test class for JUnit 5 to manage for us. We annotate this with @SystemStub so that the extension knows to activate it:

@SystemStub
private EnvironmentVariables environmentVariables;

The extension will only manage objects marked with @SystemStub, which allows us to use other System Stubs objects in the test manually if we prefer.

Here, we haven't provided any construction of the stub object. The extension constructs one for us, in the same way as the Mockito extension constructs mocks.

We can now use the object to help us set environment variables inside one of our tests:

@Test
void givenEnvironmentCanBeModified_whenSetEnvironment_thenItIsSet() {
    environmentVariables.set("ENV", "value1");
    assertThat(System.getenv("ENV")).isEqualTo("value1");
}

If we wanted to provide the environment variables that apply to all tests from outside of the test method, we can do that inside a @BeforeEach method or can use the constructor of EnvironmentVariables to set our values:

@SystemStub
private EnvironmentVariables environmentVariables =
  new EnvironmentVariables("ENV", "value1");

As with EnvironmentVariablesRule there are several overloads of the constructor, allowing us many ways to set the desired variables. We can also use the set method fluently to set values if we prefer:

@SystemStub
private EnvironmentVariables environmentVariables =
  new EnvironmentVariables()
    .set("ENV", "value1")
    .set("ENV2", "value2");

We can also make our fields static for them to be managed as part of the @BeforeAll/@AfterAll lifecycle.

3.4. JUnit 5 Parameter Injection

While placing the stub objects in fields is useful when using them for all of our tests, we may prefer to use them only for selected ones. This can be achieved by JUnit 5 parameter injection:

@Test
void givenEnvironmentCanBeModified(EnvironmentVariables environmentVariables) {
    environmentVariables.set("ENV", "value1");
    assertThat(System.getenv("ENV")).isEqualTo("value1");
}

In this case, the EnvironmentVariables object was constructed for us with its default constructor, allowing us to use it within a single test. The object has also been activated so that it is operating on the runtime environment. It will be tidied up when the test is finished.

All of the System Stubs objects have a default constructor and the ability to be reconfigured while running. We can inject as many as we need into our tests.

3.5. Execute-Around Environment Variables

The original System Lambda facade methods for creating stubs are also available via the SystemStubs class. Internally they are implemented by creating instances of the stubbing objects. Sometimes the object returned from the recipe is a stub object for further configuration and use:

withEnvironmentVariable("ENV3", "val")
    .execute(() -> {
        assertThat(System.getenv("ENV3")).isEqualTo("val");
    });

Behind the scenes, withEnvironmentVariable is doing the equivalent of:

return new EnvironmentVariables().set("ENV3", "val");

The execute method is common to all SystemStub objects. It sets up the stubbing defined by the object, then executes the lambda passed in. Afterward, it tidies up and returns control to the surrounding test.

If the test code returns a value, then that value can be returned by execute:

String extracted = new EnvironmentVariables("PROXY", "none")
  .execute(() -> System.getenv("PROXY"));
assertThat(extracted).isEqualTo("none");

This can be useful when the code we are testing needs to have access to environment settings to construct something. It's commonly used when testing things like AWS Lambda handlers, which are often configured through environment variables.

The advantage of this pattern for occasional tests is that we have to set up the stubbing explicitly, only where needed. Therefore it can be more precise and visible. However, it does not allow us to share the setup between tests and can be more long-winded.

3.6. Multiple System Stubs

We have already seen how the JUnit 4 and JUnit 5 plugins construct and activate stubbing objects for us. If there are multiple stubs, they are set up and torn down appropriately by the framework code.

However, when we construct stubbing objects for the execute-around pattern, we need our test code to run inside them all.

This can be achieved using the with/execute methods. These work by creating a composite from multiple stubbing objects used with a single execute:

with(new EnvironmentVariables("FOO", "bar"), new SystemProperties("prop", "val"))
  .execute(() -> {
      assertThat(System.getenv("FOO")).isEqualTo("bar");
      assertThat(System.getProperty("prop")).isEqualTo("val");
  });

Now we have seen the general form of using the System Stubs objects, both with and without JUnit framework support, let's look at the rest of the library's capabilities.

4. System Properties

We can call System.setProperty at any time in Java. However, this runs the risk of leaking the settings out of one test into another. The primary aim of SystemProperties stubbing is to restore the system properties to their original settings after the test is complete. However, it's also useful for common setup code to define which system properties should be used before the test starts.

4.1. JUnit 4 System Properties

By adding the rule to the JUnit 4 test class, we can insulate each test from any System.setProperty calls made in other test methods. We can also provide some upfront properties via the constructor:

@Rule
public SystemPropertiesRule systemProperties =
  new SystemPropertiesRule("db.connection", "false");

With this object, we can also set some additional properties in the JUnit @Before method:

@Before
public void before() {
    systemProperties.set("before.prop", "before");
}

We can also use the set method in the body of a test or use System.setProperty if we wish. We must only use set in creating the SystemPropertiesRule, or in the @Before method, as it stores the setting in the rule, ready for applying later.

4.2. JUnit 5 System Properties

We have two main use cases for using the SystemProperties object. We may wish to reset the system properties after each test case, or we may wish to prepare some common system properties in a central place for each test case to use.

Restoring system properties requires us to add both the JUnit 5 extension and a SystemProperties field to our test class:

@ExtendWith(SystemStubsExtension.class)
class RestoreSystemProperties {
    @SystemStub
    private SystemProperties systemProperties;
}

Now, each test will have any system properties it changes cleaned up afterward.

We can also do this for selected tests by parameter injection:

@Test
void willRestorePropertiesAfter(SystemProperties systemProperties) {
}

If we want the test to have properties set in it, then we can either assign those properties in the construction of our SystemProperties object or use a @BeforeEach method:

@ExtendWith(SystemStubsExtension.class)
class SetSomeSystemProperties {
    @SystemStub
    private SystemProperties systemProperties;
    @BeforeEach
    void before() {
        systemProperties.set("beforeProperty", "before");
    }
}

Again, let's note that the JUnit 5 test needs to be annotated with @ExtendWith(SystemStubsExtension.class). The extension will create the System Stubs object if we do not provide a new statement in the initializer list.

4.3. System Properties with Execute Around

The SystemStubs class provides a restoreSystemProperties method to allow us to run test code with properties restored:

restoreSystemProperties(() -> {
    // test code
    System.setProperty("unrestored", "true");
});
assertThat(System.getProperty("unrestored")).isNull();

This takes a lambda that returns nothing. If we wish to use a common set-up function to create properties, get a return value from the test method, or combine SystemProperties with other stubs via with/execute, then we can create the object explicitly:

String result = new SystemProperties()
  .execute(() -> {
      System.setProperty("unrestored", "true");
      return "it works";
  });
assertThat(result).isEqualTo("it works");
assertThat(System.getProperty("unrestored")).isNull();

4.4. Properties in Files

Both the SystemProperties and EnvironmentVariables objects can be constructed from a Map. This allows Java's Properties object to be provided as the source of either system properties or environment variables.

There are helper methods inside the PropertySource class for loading up Java properties from files or resources. These properties files are name/value pairs:

name=baeldung
version=1.0

We can load from the resource test.properties by using the fromResource function:

SystemProperties systemProperties =
  new SystemProperties(PropertySource.fromResource("test.properties"));

There are similar convenience methods in PropertySource for other sources, such as fromFile or fromInputStream.

5. System Out and System Err

When our application writes to System.out, it can be hard to test. This is sometimes solved by using an interface as the target of output and mocking that at test time:

interface LogOutput {
   void write(String line);
}
class Component {
    private LogOutput log;
    public void method() {
        log.write("Some output");
    }
}

Techniques like this work well with Mockito mocks but are not necessary if we can just trap System.out itself.

5.1. JUnit 4 SystemOutRule and SystemErrRule

To trap output to System.out in a JUnit 4 test, we add the SystemOutRule:

@Rule
public SystemOutRule systemOutRule = new SystemOutRule();

After that, any output to System.out can be read within the test:

System.out.println("line1");
System.out.println("line2");
assertThat(systemOutRule.getLines())
  .containsExactly("line1", "line2");

We have a choice of formats for the text. The above example uses the Stream<String> provided by getLines. We may also choose to get the whole block of text:

assertThat(systemOutRule.getText())
  .startsWith("line1");

However, we should note that this text will have newline characters that vary between platforms. We can replace newlines with \n on every platform by using the normalized form:

assertThat(systemOutRule.getLinesNormalized())
  .isEqualTo("line1\nline2\n");

The SystemErrRule works in the same way for System.err as its System.out counterpart:

@Rule
public SystemErrRule systemErrRule = new SystemErrRule();
@Test
public void whenCodeWritesToSystemErr_itCanBeRead() {
    System.err.println("line1");
    System.err.println("line2");
    assertThat(systemErrRule.getLines())
      .containsExactly("line1", "line2");
}

There is also a SystemErrAndOutRule class, which taps both System.out and System.err simultaneously into a single buffer.

5.2. JUnit 5 Example

As with the other System Stubs objects, we only need to declare a field or parameter of type SystemOut or SystemErr. This will provide us with a capture of the output:

@SystemStub
private SystemOut systemOut;
@SystemStub
private SystemErr systemErr;
@Test
void whenWriteToOutput_thenItCanBeAsserted() {
    System.out.println("to out");
    System.err.println("to err");
    assertThat(systemOut.getLines()).containsExactly("to out");
    assertThat(systemErr.getLines()).containsExactly("to err");
}

We can also use the SystemErrAndOut class to direct both sets of output into the same buffer.

5.3. Execute-Around Example

The SystemStubs facade provides some functions for tapping the output and returning it as a String:

@Test
void givenTapOutput_thenGetOutput() throws Exception {
    String output = tapSystemOutNormalized(() -> {
        System.out.println("a");
        System.out.println("b");
    });
    assertThat(output).isEqualTo("a\nb\n");
}

We should note that these methods do not provide as rich an interface as the raw objects themselves. The capture of output can't easily be combined with other stubbing, such as setting environment variables.

However, the SystemOut, SystemErr, and SystemErrAndOut objects can be used directly. For example, we could combine them with some SystemProperties:

SystemOut systemOut = new SystemOut();
SystemProperties systemProperties = new SystemProperties("a", "!");
with(systemOut, systemProperties)
  .execute(()  -> {
    System.out.println("a: " + System.getProperty("a"));
});
assertThat(systemOut.getLines()).containsExactly("a: !");

5.4. Muting

Sometimes our aim is not to capture output but to keep it from cluttering our test run logs. We can achieve this using the muteSystemOut or muteSystemErr functions:

muteSystemOut(() -> {
    System.out.println("nothing is output");
});

We can achieve the same thing across all tests via the JUnit 4 SystemOutRule:

@Rule
public SystemOutRule systemOutRule = new SystemOutRule(new NoopStream());

In JUnit 5, we can use the same technique:

@SystemStub
private SystemOut systemOut = new SystemOut(new NoopStream());

5.5. Customization

As we have seen, there are several variations for intercepting output. They all share a common base class in the library. For convenience, several helper methods and types, like SystemErrAndOut, help do common things. However, the library itself is easily customized.

We could provide our own target for capturing the output as an implementation of Output. We've already seen the Output class TapStream in use in the first examples. NoopStream is used for muting. We also have DisallowWriteStream that throws an error if something writes to it:

// throws an exception:
new SystemOut(new DisallowWriteStream())
  .execute(() -> System.out.println("boo"));

6. Mocking System In

We may have an application that reads input on stdin. Testing this could involve extracting the algorithm into a function that reads from any InputStream and then feeding it with a pre-preprepared input stream. In general, modular code is better, so this is a good pattern.

However, if we only test the core functions, we lose test coverage on the code which provides System.in as the source.

In any case, it can be inconvenient to construct our own streams. Luckily, System Stubs has solutions for all of these.

6.1. Test Input Streams

System Stubs provides a family of AltInputStream classes as alternative inputs for any code that reads from an InputStream:

LinesAltStream testInput = new LinesAltStream("line1", "line2");
Scanner scanner = new Scanner(testInput);
assertThat(scanner.nextLine()).isEqualTo("line1");

In this example, we've used an array of strings to construct LinesAltStream, but we could have supplied the input from a Stream<String>, allowing this to be used with any source of text data without necessarily loading it all into memory at once.

6.2. JUnit 4 Example

We can provide lines for input in a JUnit 4 test using the SystemInRule:

@Rule
public SystemInRule systemInRule =
  new SystemInRule("line1", "line2", "line3");

Then, the test code can read this input from System.in:

@Test
public void givenInput_canReadFirstLine() {
    assertThat(new Scanner(System.in).nextLine())
      .isEqualTo("line1");
}

6.3. JUnit 5 Example

For JUnit 5 tests, we create a SystemIn field:

@SystemStub
private SystemIn systemIn = new SystemIn("line1", "line2", "line3");

Then our tests will run with System.in providing these lines as input.

6.4. Execute-Around Example

The SystemStubs facade provides withTextFromSystemIn as a factory method that creates a SystemIn object for use with its execute method:

withTextFromSystemIn("line1", "line2", "line3")
  .execute(() -> {
      assertThat(new Scanner(System.in).nextLine())
        .isEqualTo("line1");
  });

6.5. Customization

More features can be added to the SystemIn object either on construction or while it is running within a test.

We can call andExceptionThrownOnInputEnd, which causes reading from System.in to throw an exception when it runs out of text. This can simulate an interrupted read from a file.

We can also set the input stream to come from any InputStream, like FileInputStream, by using setInputStream. We also have LinesAltStream and TextAltStream, which operate on input text.

7. Mocking System.Exit

As mentioned previously, if our code can call System.exit, it can make for dangerous and hard to debug test faults. One of our aims in stubbing System.exit is to make an accidental call into a traceable error. Another motivation is to test intentional exits from the software.

7.1. JUnit 4 Example

Let's add the SystemExitRule to a test class as a safety measure to prevent any System.exit from stopping the JVM:

@Rule
public SystemExitRule systemExitRule = new SystemExitRule();

However, we may also wish to see if the right exit code was used. For that, we need to assert that the code throws the AbortExecutionException, which is the System Stubs signal that System.exit was called.

@Test
public void whenExit_thenExitCodeIsAvailable() {
    assertThatThrownBy(() -> {
        System.exit(123);
    }).isInstanceOf(AbortExecutionException.class);
    assertThat(systemExitRule.getExitCode()).isEqualTo(123);
}

In this example, we've used assertThatThrownBy from AssertJ to catch and check the exception signaling exit occurred. Then we looked at getExitCode from the SystemExitRule to assert the exit code.

7.2. JUnit 5 Example

For JUnit 5 tests, we declare the @SystemStub field:

@SystemStub
private SystemExit systemExit;

Then we use the SystemExit class in the same way as SystemExitRule in JUnit 4. Given that the SystemExitRule class is a subclass of SystemExit, they have the same interface.

7.3. Execute-Around Example

The SystemStubs class provides catchSystemExit, which internally uses SystemExit‘s execute function:

int exitCode = catchSystemExit(() -> {
    System.exit(123);
});
assertThat(exitCode).isEqualTo(123);

Compared with the JUnit plugin examples, this code does not throw an exception to indicate a system exit. Instead, it catches the error and records the exit code. With the facade method, it returns the exit code.

When we use the execute method directly, exit is caught, and the exit code is set inside the SystemExit object. We can then call getExitCode to get the exit code, or null if there was none.

8. Custom Test Resources in JUnit 5

JUnit 4 already provides a simple structure for creating test rules like the ones used in System Stubs. If we want to make a new test rule for some resource, with a setup and teardown, we can subclass ExternalResource and provide overrides of the before and after methods.

JUnit 5 has a more complex pattern for resource management. For simple use cases, it's possible to use the System Stubs library as a starting point. The SystemStubsExtension operates on anything that satisfies the TestResource interface.

8.1. Creating a TestResource

We can create a subclass of TestResource and then use our custom objects in the same way we use System Stubs ones. We should note that we need to provide a default constructor if we want to use the automatic creation of fields and parameters.

Let's say we wanted to open a connection to a database for some tests and close it afterward:

public class FakeDatabaseTestResource implements TestResource {
    // let's pretend this is a database connection
    private String databaseConnection = "closed";
    @Override
    public void setup() throws Exception {
        databaseConnection = "open";
    }
    @Override
    public void teardown() throws Exception {
        databaseConnection = "closed";
    }
    public String getDatabaseConnection() {
        return databaseConnection;
    }
}

We're using the databaseConnection string as an illustration of a resource like a database connection. We modify the state of the resource in the setup and teardown methods.

8.2. Execute-Around is Built-In

Now let's try using this with the execute-around pattern:

FakeDatabaseTestResource fake = new FakeDatabaseTestResource();
assertThat(fake.getDatabaseConnection()).isEqualTo("closed");
fake.execute(() -> {
    assertThat(fake.getDatabaseConnection()).isEqualTo("open");
});

As we can see, the TestResource interface gave it the execute-around capabilities of the other objects.

8.3. Custom TestResource in JUnit 5 Test

We can also use this inside a JUnit 5 test:

@ExtendWith(SystemStubsExtension.class)
class FakeDatabaseJUnit5UnitTest {
    @Test
    void useFakeDatabase(FakeDatabaseTestResource fakeDatabase) {
        assertThat(fakeDatabase.getDatabaseConnection()).isEqualTo("open");
    }
}

So, it is easy to create additional test objects that follow the System Stubs design.

9. Environment and Property Overrides for JUnit 5 Spring Tests

Setting environment variables for Spring tests can be difficult. We might compose a custom rule for integration testing to set some system properties for Spring to pick up.

We may also use an ApplicationContextInitializer class to plug into our Spring Context, providing extra properties for the test.

As many Spring applications are controlled by system property or environment variable overrides, it may be easier to use System Stubs to set these in an outer test, with the Spring test running as an inner class.

There's a full example provided in the System Stubs documentation. We start by creating an outer class:

@ExtendWith(SystemStubsExtension.class)
public class SpringAppWithDynamicPropertiesTest {
    // sets the environment before Spring even starts
    @SystemStub
    private static EnvironmentVariables environmentVariables;
}

In this instance, the @SystemStub field is static and is initialized in the @BeforeAll method:

@BeforeAll
static void beforeAll() {
     String baseUrl = ...;
     environmentVariables.set("SERVER_URL", baseUrl);
}

This point in the test lifecycle allows some global resources to be created and applied to the running environment before the Spring test runs.

Then, we can put the Spring test into a @Nested class. This causes it to be run only when the parent class is set up:

@Nested
@SpringBootTest(classes = {RestApi.class, App.class},
    webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
class InnerSpringTest {
    @LocalServerPort
    private int serverPort;
    // Test methods
}

The Spring context is created against the state of the environment set by the @SystemStub objects in the outer class.

This technique also allows us to control the configuration of any other libraries that depend on the state of system properties or environment variables that may be running behind Spring Beans.

This can allow us to hook into the test lifecycle to modify things like proxy settings or HTTP connection pool parameters before a Spring test runs.

10. Conclusion

In this article, we've looked at the importance of being able to mock system resources and how System Stubs allows for complex configurations of stubbing with a minimum of code repetition through its JUnit 4 and JUnit 5 plugins.

We saw how to provide and isolate environment variables and system properties in our tests. Then we looked at capturing the output and controlling the input on the standard streams. We also looked at capturing and asserting calls to System.exit.

Finally, we looked at how to create custom test resources and how to use System Stubs with Spring.

As always, the full source code of the examples is available over on GitHub.

The post Guide to the System Stubs Library first appeared on Baeldung.

        

Digital Certificate: How to Import .cer File into Truststore File

$
0
0

1. Overview

The SSL protocol is usually the preferred choice whenever applications need to communicate with clients over the network. Together with encryption of data, SSL makes it mandatory for an application, like a browser, to exchange asymmetric keys during the handshake in order to establish a secure connection.

Generally, applications share the asymmetric keys in X.509 certificates format. Therefore, before SSL handshaking, clients must import such certificates into their truststore files. 

In this article, we'll discuss a few tools that we can use to import certificates in .cer format into the client's truststore.

2. The keytool Command

The JDK distribution provides a keytool utility that we can use to manage Java keystores (JKS). The most important purpose of this command is to generate self-signed X.509 certificates for testing SSL communication between a client and a server.

We can also import self-signed or CA-signed certificates into a JKS file and use it as a truststore:

keytool -importcert -alias trustme -file baeldung.cer -keystore cacerts
Enter keystore password:
Trust this certificate? [no]:  yes
Certificate was added to keystore

Here, we've imported a self-signed baeldung.cer certificate using the keytool command. We can import this certificate into any Java keystore. For example, the one shown here is adding the certificate in the cacerts keystore in the JDK.

If we now list the certificates in the keystore, we'll see an alias trustme:

keytool -list -keystore cacerts
trustme, Oct 31, 2020, trustedCertEntry,
Certificate fingerprint (SHA1): 04:40:6C:B0:06:65:EE:80:9A:90:A5:E9:DA:19:05:4A:AA:F2:CF:A4

3. The openssl Command

Until now, we've only discussed importing the certificates into a JKS file. Such keystores can only be used with Java applications. If we have to implement an SSL library in other languages or use the same certificate across multiple language platforms, we're more likely to use PKCS12 keystores.

To import a certificate into a PKCS12 keystore, we can also use openssl :

openssl pkcs12 -export -in baeldung.cer -inkey baeldung.key -out baeldung.keystore -name trustme

This command will import a certificate named baeldung.cer into a keystore baeldung.keystore with an alias trustme. 

We can see the imported certificate in the keystore:

openssl pkcs12 -info -in baeldung.keystore
Enter Import Password:
MAC: sha1, Iteration 2048
MAC length: 20, salt length: 8
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
Bag Attributes
    friendlyName: trustme
    localKeyID: F4 36 4E 19 E4 E4 E7 65 74 56 FB 50 40 02 68 8B EC F0 4D B3
subject=C = IN, ST = DE, L = DC, O = BA, OU = AU, CN = baeldung.com
issuer=C = IN, ST = DE, L = DC, O = BA, OU = AU, CN = baeldung.com
-----BEGIN CERTIFICATE-----
MIIFkTCCA3mgAwIBAgIUL/OjGExnppeZkiNNh0i2+TPHaCQwDQYJKoZIhvcNAQEL
BQAwWDELMAkGA1UEBhMCSU4xCzAJBgNVBAgMAkRFMQswCQYDVQQHDAJEQzELMAkG
A1UECgwCQkExCzAJBgNVBAsMAkFVMRUwEwYDVQQDDAxiYWVsZHVuZy5jb20wHhcN
MjAxMTAzMTIwMjI5WhcNMjExMTAzMTIwMjI5WjBYMQswCQYDVQQGEwJJTjELMAkG
A1UECAwCREUxCzAJBgNVBAcMAkRDMQswCQYDVQQKDAJCQTELMAkGA1UECwwCQVUx
FTATBgNVBAMMDGJhZWxkdW5nLmNvbTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCC
AgoCggIBAK/XF/xmqQRJlTx2Vtq70x1KFwkHJEcZOyFbQP7O9RgicvMTAnbZtKpS
BSVjwroklIr4OVK4wmwdaTnlIm22CsFrbn+iBVL00tVs+sBYEcgO5nphVWGFbvHl
Q3PO4vTedSyH1qIyYrrhAn8wYvzdmr2g6tRwBX8K5H948Zb32Xbp5r9aR5M2i8Qz
fc0QasJUM5b71TNt8Qcsru3pFKj5hUMBTNrGCQrr6vrADTcG0YHuVSMeJId7f67h
l0vEY0BmRPnWNwGe+Sg/jqOWH9WWvkk/umkEQNWCQZaXZNZZ8jl5WMKFnmA7sPQ+
UjZPabNOTxhz6fJv5nJu7aMS/6tUWO0SdQ+ctO3HgR42wtBPoEOOuFMP6OqHI4hf
CXFTYg6aLwxFJP7LngfRvETgzVlsb9L/m++JBeoWRqpWaQUEgxDYJGFGA5dwQJaf
f24d042i44X0WqBBoWLjSQd/JFVH5MF17waiYpxFBOgpz3XEM/1j+juJPVut2k96
3ecgR54iKILbibizPUojn7t3AFT1Ug8exdefHdf+QsL8/L5+8/xOYkp/pnglQJJl
W0Lq4Sh9LWiux9XVdY6n2UYf/crgLSHatVkPa26cysdXhiiEPn4yYr2AdYVf0Xr5
W5PULufdi0HW2Eja/TfeXoBQtkdidqP8SMW+DwqafX80s37bZZBvAgMBAAGjUzBR
MB0GA1UdDgQWBBQPHIpCFhAy3kGAbzHpXMjXCMVQRzAfBgNVHSMEGDAWgBQPHIpC
FhAy3kGAbzHpXMjXCMVQRzAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUA
A4ICAQBzOK52I7lDX+7CHy6cQ8PnLZjAD4S5qC1P9dMy50E9N6Tmw2TiPbWl9CnU
7a/kVO6xDJDmNMnqRbHmlZclJaTFv6naXSX27PdIWjhwAfLjNa+FO9JNwMgiP25I
ISVjyrA3HwbhFnMs5FyBW9hbxfQ+X2Q2ooa+J3TKU7FImuDRKF3Sdb63+/j0go8S
5/TsoYpQxg86xbWf6IYGYwegd2SPSWUZ0HQSobZ7fRA7Y0EyPKgyqsBbmDtJ+X1g
P8Kep4N1oocc7ZkkX4pNfXTgXib9fUkKMAfRJz8w62z8I1OM61bciW7V2VSp/Y5p
iTihyuwO0aHG+YTnsr3qFrSFQLQUjCeBvx+euQelsGm8W9xM9YfASXiaEwCmb9PO
i/umD70J1e0HFDay9FW6mMoCCEBTZIF9ARqzhHgg9fi9iH2ctrsxadFAlOTFp5+/
p+nxrencfvc4CP6aHoqkE45HpMBoNDAxRMVd/FRzIG2as0q5At873MNFXP6WxmQV
4KGIhteNLyrXk82yHdHfm1EENTI7OEst/Fc8O3fFy9wn0OvoHIuCv1FVCq4Gnjmq
vNnBnGldrYruCNvj5KT6U14FFdK/5Zng0nSky4oMTs49zt392bdYi31SHWeATdw/
PscFRdig2stoI8ku6R+K7XvmseGzPmUW4K2IWU0zdRP2a4YyvA==
-----END CERTIFICATE-----
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048
Bag Attributes
    friendlyName: trustme
    localKeyID: F4 36 4E 19 E4 E4 E7 65 74 56 FB 50 40 02 68 8B EC F0 4D B3
Key Attributes: <No Attributes>

So, we've successfully imported our certificate into the PKCS12 keystore. As a result, this keystore can now be used as a truststore file in SSL client applications like HTTP client libraries. Likewise, this file can also be used as a keystore in SSL server applications like Tomcat.

4. Conclusion

In this article, we discussed two popular SSL tools for managing digital certificates — OpenSSL and Java Keytool. We further used the keytool and openssl commands to import a certificate in .cer format into JKS and PKCS12 files, respectively.

The post Digital Certificate: How to Import .cer File into Truststore File first appeared on Baeldung.

        

Security Context Basics: User, Subject and Principal

$
0
0

1. Overview

Security is a fundamental part of any Java application. Also, we can find many security frameworks that can handle security concerns. Additionally, we use a few terms commonly like the subject, principal, and user in these frameworks.

In this tutorial, we're going to explain these basic concepts of security frameworks. Also, we'll show their relationships and differences.

2. Subject

In a security context, the subject represents the source of a request. The subject is an entity that obtains information about resources or modifies resources. Additionally, a subject can also be a user, a program, a process, a file, a computer, a database, etc.

For example, a person needs to authorize access to resources and applications to authenticate the request source. In this case, this person is a subject.

Let's take a look at our example that implemented base on the JAAS framework:

Subject subject = loginContext.getSubject();
PrivilegedAction privilegedAction = new ResourceAction();
Subject.doAsPrivileged(subject, privilegedAction, null);

3. Principal

After successful authentication, we have a populated subject with many associated identities, such as roles, social security number(SSN), etc. In other words, these identifiers are principals, and the subject represents them.

For instance, a person may have an account number principal (“87654-3210”) and other unique identifiers, distinguishing it from other subjects.

Let's see how to create an UserPrincipal after a successful login and add it to a Subject:

@Override
public boolean commit() throws LoginException {
    if (!loginSucceeded) {
        return false;
    }
    userPrincipal = new UserPrincipal(username);
    subject.getPrincipals().add(userPrincipal);
    return true;
}

4. User

Typically, a user represents a person who accesses resources to perform some action or accomplish a work task.

Also, we can use a user as a principal, and on the other hand, a principal is an identity assigned to a user. UserPrincipal is an excellent example of a user in the JAAS framework discussed in the previous section.

5. Difference Between Subject, Principal, and User

As we saw in the above sections, we can represent different aspects of the same user's identity by using principals. They are subsets of subjects, and users are subsets of principals that are referring to the end-user or interactive operators.

6. Conclusion

In this tutorial, we discussed the definition of the subject, principal, and user that they are common in most of the security frameworks. Also, we showed the difference between them.

The implementation of all these examples and code snippets can be found in the GitHub project.

The post Security Context Basics: User, Subject and Principal first appeared on Baeldung.

        

Thymeleaf Variables

$
0
0

1. Introduction

In this tutorial, we're going to take a look at variables in Thymeleaf. We'll create a Spring Boot example that will fetch a list of Baeldung articles and display them in a Thymeleaf HTML template.

2. Maven Dependencies

To work with Thymeleaf, we'll need to add the spring-boot-starter-thymeleaf and spring-boot-starter-web dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-thymeleaf</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

3. Web Controller

First, we'll create a web controller with a GET endpoint that returns a page with a list of Baeldung articles.

The method annotated with @GetMapping will take a single parameter – the Model. It holds all the global variables that can be further used inside the Thymeleaf template. In our case, the model will have just one parameter – the list of articles.

The Article class will consist of two String fields, name and url:

public class Article {
    private String name;
    private String url;
    // constructor, getters and setters
}

The return value of our controller's method should be the name of the desired Thymeleaf template. This name should correspond to the HTML file located in the src/resource/template directory. In our case, it'll be src/resource/template/articles-list.html.

Let's take a quick look at our Spring controller:

@Controller
@RequestMapping("/api/articles")
public class ArticlesController {
    @GetMapping
    public String allArticles(Model model) {
        model.addAttribute("articles", fetchArticles());
        return "articles-list";
    }
    private List<Article> fetchArticles() {
        return Arrays.asList(
          new Article(
            "Introduction to Using Thymeleaf in Spring",
            "https://www.baeldung.com/thymeleaf-in-spring-mvc"
          ),
          // a few other articles
        );
    }
}

After running the application, the articles page will be available at http://localhost:8080/articles.

4. Thymeleaf Template

Now, let's move into the Thymeleaf HTML template. It should have the standard HTML document structure with just the additional Thymeleaf namespace definition:

<html xmlns:th="http://www.thymeleaf.org">

We'll use this as a template in further examples, where we'll be replacing just the content of the <main> tag:

<!DOCTYPE HTML>
<html xmlns:th="http://www.thymeleaf.org">
<head>
    <title>Thymeleaf Variables</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
    <main>
        ...
    </main>
</body>
</html>

5. Define Variables

There are two ways we can define a variable in a Thymeleaf template. The first option is to take a single element while iterating over an array:

<div th:each="article : ${articles}">
    <a th:text="${article.name}" th:href="https://feeds.feedblitz.com/~/t/0/_/baeldung/~${article.url}"></a>
</div>

As a result, we'll get a <div> with several <a> elements corresponding to the number of articles in the articles variable.

Another way is to define a new variable based on another one. For example, we can take the first element of the articles array:

<div th:with="firstArticle=${articles[0]}">
    <a th:text="${firstArticle.name}" th:href="https://feeds.feedblitz.com/~/t/0/_/baeldung/~${firstArticle.url}"></a>
</div>

Or we can create a new variable that holds just the article's name:

<div th:each="article : ${articles}", th:with="articleName=${article.name}">
    <a th:text="${articleName}" th:href="https://feeds.feedblitz.com/~/t/0/_/baeldung/~${article.url}"></a>
</div>

In the above example, the ${article.name} and ${articleName} fragments are replaceable.

It's also possible to define multiple variables. For example, we can create two separate variables to hold the article name and URL:

<div th:each="article : ${articles}" th:with="articleName=${article.name}, articleUrl=${article.url}">
    <a th:text="${articleName}" th:href="https://feeds.feedblitz.com/~/t/0/_/baeldung/~${articleUrl}"></a>
</div>

6. Variables Scope

Variables passed to the Model in a controller have a global scope. This means they can be used in every place of our HTML templates.

On the other hand, variables defined in the HTML template have a local scope. They can be used only within the range of the element that they were defined in.

For example, the below code is correct as the <a> element is within the firstDiv:

<div id="firstDiv" th:with="firstArticle=${articles[0]}">
    <a th:text="${firstArticle.name}" th:href="https://feeds.feedblitz.com/~/t/0/_/baeldung/~${firstArticle.url}"></a>
</div>

On the other hand, when we try to use the firstArticle in another div:

<div id="firstDiv" th:with="firstArticle=${articles[0]}">
    <a th:text="${firstArticle.name}" th:href="https://feeds.feedblitz.com/~/t/0/_/baeldung/~${firstArticle.url}"></a>
</div>
<div id="secondDiv">
    <h2 th:text="${firstArticle.name}"></h2>
</div>

We'll get an exception during compile-time saying that the firstArticle is null:

org.springframework.expression.spel.SpelEvaluationException: EL1007E: Property or field 'name' cannot be found on null

This is because the <h2> element is trying to use a variable defined in the firstDiv, which is out of scope.

If we still need to use the firstArticle variable inside the secondDiv, we would need to define it again in secondDiv or wrap these two div tags in a common element and define the firstArticle in it.

7. Changing a Variable's Value

It's also possible to overwrite a variable's value in a given scope:

<div id="mainDiv" th:with="articles = ${ { articles[0], articles[1] } }">
    <div th:each="article : ${articles}">
        <a th:text="${article.name}" th:href="https://feeds.feedblitz.com/~/t/0/_/baeldung/~${article.url}"></a>
    </div>
</div>

In the above example, we redefined the articles variable to have just two first elements.

Note that outside of the mainDiv, the articles variable will still have its original value passed in the controller.

8. Conclusion

In this tutorial, we've learned how to define and use variables in Thymeleaf. As always, all the source code is available over on GitHub.

The post Thymeleaf Variables first appeared on Baeldung.

        

NoSuchFieldError in Java

$
0
0

1. Overview

In this article, we'll demonstrate the reason behind NoSuchFieldError and discover how to resolve it.

2. NoSuchFieldError

As the name suggests, NoSuchFieldError occurs when a specified field doesn't exist. NoSuchFieldError extends the IncompatibleClassChangeError class and is thrown when the application tries to access or modify a field of an object or a static field of a class but the object or class no longer has that field.

IncompatibleClassChangeError class extends the LinkageError class and occurs when we perform incompatible class definition changes. Finally, LinkageError extends Error and shows that a class has some dependency on another incompatibly changed class.

Let's see this error in action with the help of an example. As a first step, let's create a Dependency class:

public class Dependency {
    public static String message = "Hello Baeldung!!";
}

Then we'll create a FieldErrorExample class that refers to a field of our Dependency class:

public class FieldErrorExample {
    public static String getDependentMessage() {
        return Dependency.message;
    }
}

Let's also add code to check whether we're getting a message from the Dependency class:

public static void fetchAndPrint() {
    System.out.println(getDependentMessage());
}

Now, we can compile these files using the javac command, and upon execution of the FieldErrorExample class using the java command, it will print the specified message.

However, if we comment out, remove, or change the attribute name in the Dependency class and recompile it, then we'll run into our error.

For example, let's change the attribute name in our Dependency class:

public class Dependency {
    public static String msg = "Hello Baeldung!!";
}

Now, if we recompile only our Dependency class, and then execute FieldErrorExample again, we'll encounter the NoSuchFieldError:

Exception in thread "main" java.lang.NoSuchFieldError: message

The above error occurred because the FieldErrorExample class still refers to the static field message of the Dependency class, but it no longer exists — we've made an incompatible change to the Dependency class.

3. Resolving the Error

To avoid this error, we need to clean and compile the existing files. We can do this using the javac command or with Maven by running mvn clean install. By performing this step, we'll have all the latest compiled files, and we'll avoid running into the error.

If the error persists, then the problem might be multiple JAR files: one while compiling, and another while running. This often happens when the application depends on external JARs. Here, we should validate the order of the JARs in the build path to identify the inconsistent JAR.

If we have to investigate further, it's helpful to run the application with -verbose: class option to check the loaded classes. This can help us identify the outdated class.

Sometimes a third-party JAR might be internally referring to another version, which results in NoSuchFieldError. If this happens, we can use mvn dependency:tree -Dverbose. This generates the maven dependency tree and helps us in identifying the inconsistent JAR.

4. Conclusion

In this short tutorial, we have shown why NoSuchFieldError occurs and looked at how we can resolve it.

As always, the code is available over on GitHub.

The post NoSuchFieldError in Java first appeared on Baeldung.

        

Java AES Encryption and Decryption

$
0
0

1. Overview

The symmetric-key block cipher plays an important role in data encryption. It means that the same key is used for both encryption and decryption. The Advanced Encryption Standard (AES) is a widely used symmetric-key encryption algorithm.

In this tutorial, we’ll see how to implement AES encryption and decryption using the Java Cryptography Architecture (JCA) within the JDK.

2. AES Algorithm

The AES algorithm is an iterative, symmetric-key block cipher that supports cryptographic keys (secret keys) of 128, 192, and 256 bits to encrypt and decrypt data in blocks of 128 bits. The below figure shows the high-level AES algorithm:

High Level AES Algorithm

If the data to be encrypted does not meet the block size of 128 bits requirement, it must be padded. Padding is a process of filling up the last block to 128 bits.

3. AES Variations

The AES algorithm has six modes of operation:

  1. ECB (Electronic Code Book)
  2. CBC (Cipher Block Chaining)
  3. CFB (Cipher FeedBack)
  4. OFB (Output FeedBack)
  5. CTR (Counter)
  6. GCM (Galois/Counter Mode)

The mode of operation may be applied in order to strengthen the effect of the encryption algorithm. Moreover, the mode of operation may convert the block cipher into a stream cipher. Each mode has its strength and weakness. Let’s have a quick review.

3.1. ECB

This mode of operation is the simplest of all. The plaintext is divided into blocks with a size of 128 bits. Then each block will be encrypted with the same key and algorithm. Therefore, it produces the same result for the same block. This is the main weakness of this mode and it is not recommended for encryption. It requires padding data.

3.2. CBC

In order to overcome the ECB weakness, CBC mode uses an Initialization Vector (IV) to augment the encryption. First, CBC uses the plaintext block xor with the IV. Then it encrypts the result to the ciphertext block. In the next block, it uses the encryption result to xor with the plaintext block until the last block.

In this mode, encryption can not be parallelized, but decryption can be parallelized. It also requires padding data.

3.3. CFB

This mode can be used as a stream cipher. First, it encrypts the IV, then it will xor with the plaintext block to get ciphertext. Then CFB encrypts the encryption result to xor the plaintext. It needs an IV.

In this mode, decryption can be parallelized but encryption can not be parallelized.

3.4. OFB

This mode can be used as a stream cipher. First, it encrypts the IV. Then it uses the encryption results to xor the plaintext to get ciphertext.

It doesn’t require padding data and will not be affected by the noisy block.

3.5. CTR

This mode uses the value of a counter as an IV. It's very similar to OFB, but it uses the counter to be encrypted every time instead of the IV.

This mode has two strengths, including encryption/decryption parallelization, and noise in one block does not affect other blocks.

3.6. GCM

This mode is an extension of the CTR mode. The GCM has received significant attention and is recommended by NIST. The GCM mode outputs ciphertext and an authentication tag. The main advantage of this mode, compared to other operation modes of the algorithm, is its efficiency.

In this tutorial, we'll use the AES/CBC/PKCS5Padding algorithm because it is widely used in many projects.

3.7. Size of Data After Encryption

As mentioned earlier, the AES has a block size of 128 bits or 16 bytes. The AES does not change the size, and the ciphertext size is equal to the cleartext size. Also, in ECB and CBC modes, we should use a padding algorithm likes PKCS 5. So, the size of data after encryption is:

ciphertext_size (bytes) = cleartext_size + (16 - (cleartext_size % 16))

For storing IV with ciphertext, we need to add 16 more bytes.

4. AES Parameters

In the AES algorithm, we need three parameters: input data, secret key, and IV. IV is not used in ECB mode.

4.1. Input Data

The input data to the AES can be string, file, object, and password-based.

4.2. Secret Key

There are two ways for generating a secret key in the AES: generating from a random number or deriving from a given password.

In the first approach, the secret key should be generated from a Cryptographically Secure (Pseudo-)Random Number Generator like the SecureRandom class.

For generating a secret key, we can use the KeyGenerator class. Let’s define a method for generating the AES key with the size of n (128, 192, and 256) bits:

public static SecretKey generateKey(int n) throws NoSuchAlgorithmException {
    KeyGenerator keyGenerator = KeyGenerator.getInstance("AES");
    keyGenerator.init(n);
    SecretKey key = keyGenerator.generateKey();
    return key;
}

In the second approach, the AES secret key can be derived from a given password using a password-based key derivation function like PBKDF2. We also need a salt value for turning a password into a secret key. The salt is also a random value.

We can use the SecretKeyFactory class with the PBKDF2WithHmacSHA256 algorithm for generating a key from a given password.

Let’s define a method for generating the AES key from a given password with 65,536 iterations and a key length of 256 bits:

public static SecretKey getKeyFromPassword(String password, String salt)
    throws NoSuchAlgorithmException, InvalidKeySpecException {
    
    SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256");
    KeySpec spec = new PBEKeySpec(password.toCharArray(), salt.getBytes(), 65536, 256);
    SecretKey secret = new SecretKeySpec(factory.generateSecret(spec)
        .getEncoded(), "AES");
    return secret;
}

4.3. Initialization Vector (IV)

IV is a pseudo-random value and has the same size as the block that is encrypted. We can use the SecureRandom class to generate a random IV.

Let’s define a method for generating an IV:

public static IvParameterSpec generateIv() {
    byte[] iv = new byte[16];
    new SecureRandom().nextBytes(iv);
    return new IvParameterSpec(iv);
}

5. Encryption and Decryption

5.1. String

To implement input string encryption, we first need to generate the secret key and IV according to the previous section. As the next step, we create an instance from the Cipher class by using the getInstance() method.

Additionally, we configure a cipher instance using the init() method with a secret key, IV, and encryption mode. Finally, we encrypt the input string by invoking the doFinal() method. This method gets bytes of input and returns ciphertext in bytes:

public static String encrypt(String algorithm, String input, SecretKey key,
    IvParameterSpec iv) throws NoSuchPaddingException, NoSuchAlgorithmException,
    InvalidAlgorithmParameterException, InvalidKeyException,
    BadPaddingException, IllegalBlockSizeException {
    
    Cipher cipher = Cipher.getInstance(algorithm);
    cipher.init(Cipher.ENCRYPT_MODE, key, iv);
    byte[] cipherText = cipher.doFinal(input.getBytes());
    return Base64.getEncoder()
        .encodeToString(cipherText);
}

For decrypting an input string, we can initialize our cipher using the DECRYPT_MODE to decrypt the content:

public static String decrypt(String algorithm, String cipherText, SecretKey key,
    IvParameterSpec iv) throws NoSuchPaddingException, NoSuchAlgorithmException,
    InvalidAlgorithmParameterException, InvalidKeyException,
    BadPaddingException, IllegalBlockSizeException {
    
    Cipher cipher = Cipher.getInstance(algorithm);
    cipher.init(Cipher.DECRYPT_MODE, key, iv);
    byte[] plainText = cipher.doFinal(Base64.getDecoder()
        .decode(cipherText));
    return new String(plainText);
}

Let's write a test method for encrypting and decrypting a string input:

@Test
void givenString_whenEncrypt_thenSuccess()
    throws NoSuchAlgorithmException, IllegalBlockSizeException, InvalidKeyException,
    BadPaddingException, InvalidAlgorithmParameterException, NoSuchPaddingException { 
    
    String input = "baeldung";
    SecretKey key = AESUtil.generateKey(128);
    IvParameterSpec ivParameterSpec = AESUtil.generateIv();
    String algorithm = "AES/CBC/PKCS5Padding";
    String cipherText = AESUtil.encrypt(algorithm, input, key, ivParameterSpec);
    String plainText = AESUtil.decrypt(algorithm, cipherText, key, ivParameterSpec);
    Assertions.assertEquals(input, plainText);
}

5.2. File

Now let's encrypt a file using the AES algorithm. The steps are the same, but we need some IO classes to work with the files. Let's encrypt a text file:

public static void encryptFile(String algorithm, SecretKey key, IvParameterSpec iv,
    File inputFile, File outputFile) throws IOException, NoSuchPaddingException,
    NoSuchAlgorithmException, InvalidAlgorithmParameterException, InvalidKeyException,
    BadPaddingException, IllegalBlockSizeException {
    
    Cipher cipher = Cipher.getInstance(algorithm);
    cipher.init(Cipher.ENCRYPT_MODE, key, iv);
    FileInputStream inputStream = new FileInputStream(inputFile);
    FileOutputStream outputStream = new FileOutputStream(outputFile);
    byte[] buffer = new byte[64];
    int bytesRead;
    while ((bytesRead = inputStream.read(buffer)) != -1) {
        byte[] output = cipher.update(buffer, 0, bytesRead);
        if (output != null) {
            outputStream.write(output);
        }
    }
    byte[] outputBytes = cipher.doFinal();
    if (outputBytes != null) {
        outputStream.write(outputBytes);
    }
    inputStream.close();
    outputStream.close();
}

Please note that trying to read the entire file – particularly if it is large – into memory is not recommended. Instead, we encrypt a buffer at a time.

For decrypting a file, we use similar steps and initialize our cipher using DECRYPT_MODE as we saw before.

Again, let's define a test method for encrypting and decrypting a text file. In this method, we read the baeldung.txt file from the test resource directory, encrypt it into a file called baeldung.encrypted, and then decrypt the file into a new file:

@Test
void givenFile_whenEncrypt_thenSuccess() 
    throws NoSuchAlgorithmException, IOException, IllegalBlockSizeException, 
    InvalidKeyException, BadPaddingException, InvalidAlgorithmParameterException, 
    NoSuchPaddingException {
    
    SecretKey key = AESUtil.generateKey(128);
    String algorithm = "AES/CBC/PKCS5Padding";
    IvParameterSpec ivParameterSpec = AESUtil.generateIv();
    Resource resource = new ClassPathResource("inputFile/baeldung.txt");
    File inputFile = resource.getFile();
    File encryptedFile = new File("classpath:baeldung.encrypted");
    File decryptedFile = new File("document.decrypted");
    AESUtil.encryptFile(algorithm, key, ivParameterSpec, inputFile, encryptedFile);
    AESUtil.decryptFile(
      algorithm, key, ivParameterSpec, encryptedFile, decryptedFile);
    assertThat(inputFile).hasSameTextualContentAs(decryptedFile);
}

5.3. Password-Based

We can do the AES encryption and decryption using the secret key that is derived from a given password.

For generating a secret key, we use the getKeyFromPassword() method. The encryption and decryption steps are the same as those shown in the string input section. We can then use the instantiated cipher and the provided secret key to perform the encryption.

Let's write a test method:

@Test
void givenPassword_whenEncrypt_thenSuccess() 
    throws InvalidKeySpecException, NoSuchAlgorithmException, 
    IllegalBlockSizeException, InvalidKeyException, BadPaddingException, 
    InvalidAlgorithmParameterException, NoSuchPaddingException {
    
    String plainText = "www.baeldung.com";
    String password = "baeldung";
    String salt = "12345678";
    IvParameterSpec ivParameterSpec = AESUtil.generateIv();
    SecretKey key = AESUtil.getKeyFromPassword(password,salt);
    String cipherText = AESUtil.encryptPasswordBased(plainText, key, ivParameterSpec);
    String decryptedCipherText = AESUtil.decryptPasswordBased(
      cipherText, key, ivParameterSpec);
    Assertions.assertEquals(plainText, decryptedCipherText);
}

5.4. Object

For encrypting a Java object, we need to use the SealedObject class. The object should be Serializable. Let's begin by defining a Student class:

public class Student implements Serializable {
    private String name;
    private int age;
    // standard setters and getters
}

Next, let's encrypt the Student object :

public static SealedObject encryptObject(String algorithm, Serializable object,
    SecretKey key, IvParameterSpec iv) throws NoSuchPaddingException,
    NoSuchAlgorithmException, InvalidAlgorithmParameterException, 
    InvalidKeyException, IOException, IllegalBlockSizeException {
    
    Cipher cipher = Cipher.getInstance(algorithm);
    cipher.init(Cipher.ENCRYPT_MODE, key, iv);
    SealedObject sealedObject = new SealedObject(object, cipher);
    return sealedObject;
}

The encrypted object can later be decrypted using the correct cipher:

public static Serializable decryptObject(String algorithm, SealedObject sealedObject,
    SecretKey key, IvParameterSpec iv) throws NoSuchPaddingException,
    NoSuchAlgorithmException, InvalidAlgorithmParameterException, InvalidKeyException,
    ClassNotFoundException, BadPaddingException, IllegalBlockSizeException,
    IOException {
    
    Cipher cipher = Cipher.getInstance(algorithm);
    cipher.init(Cipher.DECRYPT_MODE, key, iv);
    Serializable unsealObject = (Serializable) sealedObject.getObject(cipher);
    return unsealObject;
}

Let's write a test case:

@Test
void givenObject_whenEncrypt_thenSuccess() 
    throws NoSuchAlgorithmException, IllegalBlockSizeException, InvalidKeyException,
    InvalidAlgorithmParameterException, NoSuchPaddingException, IOException, 
    BadPaddingException, ClassNotFoundException {
    
    Student student = new Student("Baeldung", 20);
    SecretKey key = AESUtil.generateKey(128);
    IvParameterSpec ivParameterSpec = AESUtil.generateIv();
    String algorithm = "AES/CBC/PKCS5Padding";
    SealedObject sealedObject = AESUtil.encryptObject(
      algorithm, student, key, ivParameterSpec);
    Student object = (Student) AESUtil.decryptObject(
      algorithm, sealedObject, key, ivParameterSpec);
    assertThat(student).isEqualToComparingFieldByField(object);
}

6. Conclusion

In summary, we've learned how to encrypt and decrypt input data like strings, files, objects, and password-based data, using the AES algorithm in Java. Additionally, we've discussed the AES variations and the size of data after encryption.

As always, the full source code of the article is available over on GitHub.

The post Java AES Encryption and Decryption first appeared on Baeldung.

        

Introduction to Servlets and Servlet Containers

$
0
0

1. Overview

In this tutorial, we'll understand conceptually what servlets and servlet containers are and how they work.

We'll also see them in the context of a request, response, session objects, shared variables, and multithreading.

2. What are Servlets and Their Containers

Servlets are a component of the JEE framework used for web development. They are basically Java programs that run inside the boundaries of a container. On the whole, they are responsible for accepting a request, processing it, and sending a response back. Introduction to Java servlets provides a good basic understanding of the subject.

To use them, servlets need to be registered first so that a container, either JEE or Spring-based, can pick them up at start-up. In the beginning, the container instantiates a servlet by calling its init() method.

Once its initialization is complete, the servlet is ready to accept incoming requests. Subsequently, the container directs these requests for processing in the servlet's service() method. After that, it further delegates the request to the appropriate method such as doGet() or doPost() based on the HTTP request type.

With destroy(), the container tears the servlet down, and it can no longer accept incoming requests. We call this cycle of init-service-destroy as the lifecycle of a servlet.

Now let's look at this from the point of view of a container, such as Apache Tomcat or Jetty. At start-up, it creates an object of ServletContext. The job of the ServletContext is to function as the server or container's memory and remember all the servlets, filters, and listeners associated with the web application, as described in its web.xml or equivalent annotations. Until we stop or terminate the container, ServletContext stays with it.

However, the servlet's load-on-startup parameter plays an important role here. If this parameter has a value greater than zero, only then the server initializes it at start-up. If this parameter is not specified, then the servlet's init() is called when a request hits it for the very first time.

3. Request, Response, and Session

In the previous section, we talked about sending requests and receiving responses, which basically is the cornerstone of any client-server application. Now, let's look at them in detail with respect to servlets.

In this case, a request would be represented by HttpServletRequest and response with HttpServletResponse.

Whenever a client such as a browser, or a curl command, sends in a request, the container creates a new HttpServletRequest and HttpServletResponse object. It then passes on these new objects to the servlet's service method. Based on the HttpServletRequest‘s method attribute, this method determines which of the doXXX methods should be called.

Apart from the information about the method, the request object also carries other information such as headers, parameters, and body. Similarly, the HttpServletResponse object also carries headers, parameters, and body – we can set them up in our servlet's doXXX method.

These objects are short-lived. When the client gets the response back, the server marks the request and response objects for garbage collection.

How would we then maintain a state between subsequent client requests or connections? HttpSession is the answer to this riddle.

This basically binds objects to a user session, so that information pertaining to a particular user can be persisted across multiple requests. This is generally achieved using the concept of cookies, using JSESSIONID as a unique identifier for a given session. We can specify the timeout for the session in web.xml:

<session-config>
    <session-timeout>10</session-timeout>
</session-config>

This means if our session has been idle for 10 minutes, the server will discard it. Any subsequent request would create a new session.

4. How do Servlets Share Data

There're various ways in which servlets can share data, based on the required scope.

As we saw in the earlier sections, different objects have different lifetimes. HttpServletRequest and HttpServletResponse objects only live between one servlet call. HttpSession lives as long as it's active and hasn't timed out.

ServletContext‘s lifespan is the longest. It's born with the web application and gets destroyed only when the application itself shuts down. Since servlet, filter, and listener instances are tied to the context, they also live as long as the web application is up and running.

Consequently, if our requirement is to share data between all servlets, let's say if we want to count the number of visitors to our site, then we should put the variable in the ServletContext. If we need to share data within a session, then we'd save it in the session scope. A user's name would be an example in this case.

Lastly, there's the request scope pertaining to data for a single request, such as the request payload.

5. Handling Multithreading

Multiple HttpServletRequest objects share servlets among each other such that each request operates with its own thread of the servlet instance.

What that effectively means in terms of thread-safety is that we should not assign a request or session scoped data as an instance variable of the servlet.

For example, let's consider this snippet:

public class ExampleThree extends HttpServlet {
    
    private String instanceMessage;
    @Override
    protected void doGet(HttpServletRequest request, HttpServletResponse response) 
      throws ServletException, IOException {
        String message = request.getParameter("message");
        instanceMessage = request.getParameter("message");
        request.setAttribute("text", message);
        request.setAttribute("unsafeText", instanceMessage);
        request.getRequestDispatcher("/jsp/ExampleThree.jsp").forward(request, response);
    }
}

In this case, all requests in the session share instanceMessage, whereas message is unique to a given request object. Consequently, in the case of concurrent requests, the data in instanceMessage could be inconsistent.

6. Conclusion

In this tutorial, we looked at some concepts around servlets, their containers, and a few essential objects they revolve around. We also saw how servlets share data and how multi-threading affects them.

As always, source code is available over on GitHub.

The post Introduction to Servlets and Servlet Containers first appeared on Baeldung.

        

Running Spring Boot with PostgreSQL in Docker Compose

$
0
0

1. Introduction

In this tutorial, we want to run a Spring Boot application with the popular open-source database PostgreSQL. In a previous article, we looked at Docker Compose to handle multiple containers at once. So instead of installing PostgreSQL as a separate application, we'll use Docker Compose to run Spring Boot and PostgreSQL.

2. Creating the Spring Boot Project

Let’s go to the Spring Initializer and create our Spring Boot project. We’ll add the PostgreSQL Driver and Spring Data JPA modules. After we download the resulting ZIP file and extract it to a folder, we can run our new application:

./mvnw spring-boot:run

The application fails because it can't connect to the database:

***************************
APPLICATION FAILED TO START
***************************
Description:
Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.
Reason: Failed to determine a suitable driver class

3. Dockerfile

Before we can start PostgreSQL with Docker Compose, we need to turn our Spring Boot application into a Docker image. The first step is to package the application as a JAR file:

./mvnw clean package -DskipTests

Here, we first clean-up our previous builds before packaging the application. In addition, we skip the tests because they fail without PostgreSQL.

We now have an application JAR file in the target directory. That file has the project name and version number in its name and ends with -SNAPSHOT.jar. So its name could be docker-spring-boot-postgres-0.0.1-SNAPSHOT.jar.

Let's make the new src/main/docker directory. After that, we copy the application JAR file there:

cp target/docker-spring-boot-postgres-0.0.1-SNAPSHOT.jar src/main/docker

Finally, we create this Dockerfile in that same directory:

FROM adoptopenjdk:11-jre-hotspot
ARG JAR_FILE=*.jar
COPY ${JAR_FILE} application.jar
ENTRYPOINT ["java", "-jar", "application.jar"]

This file describes how Docker should run our Spring Boot application. It uses Java 11 from AdoptOpenJDK and copies the application JAR file to application.jar. It then runs that JAR file to start our Spring Boot application.

4. Docker Compose File

Now let's write our Docker Compose file, docker-compose.yml, and save it in src/main/docker:

version: '2'
services:
  app:
    image: 'docker-spring-boot-postgres:latest'
    build:
      context: .
    container_name: app
    depends_on:
      - db
    environment:
      - SPRING_DATASOURCE_URL=jdbc:postgresql://db:5432/compose-postgres
      - SPRING_DATASOURCE_USERNAME=compose-postgres
      - SPRING_DATASOURCE_PASSWORD=compose-postgres
      - SPRING_JPA_HIBERNATE_DDL_AUTO=update
          
  db:
    image: 'postgres:13.1-alpine'
    container_name: db
    environment:
      - POSTGRES_USER=compose-postgres
      - POSTGRES_PASSWORD=compose-postgres

Our application's name is app. It's the first of two services (lines 4-15):

  • The Spring Boot Docker image has the name docker-spring-boot-postgres:latest (line 5). Docker builds that image from the Dockerfile in the current directory (lines 6-7)
  • The container name is app (line 8). It depends on the db service (line 10). That's why it starts after the db container
  • Our application uses the db PostgreSQL container as the data source (line 12). The database name, the user name, and the password are all compose-postgres (lines 12-14)
  • Hibernate will automatically create or update any database tables needed (line 15)

The PostgreSQL database has the name db and is the second service (lines 17-22):

  • We use PostgreSQL 13.1 (line 18)
  • The container name is db (line 19)
  • The user name and password are both compose-postgres (lines 21-22)

5. Running with Docker Compose

Let's run our Spring Boot application and PostgreSQL with Docker Compose:

docker-compose up

Firstly, this will build the Docker Image for our Spring Boot application. Next, it will start a PostgreSQL container. Finally, it will launch our application Docker image. This time, our application runs fine:

Starting DemoApplication v0.0.1-SNAPSHOT using Java 11.0.9 on f94e79a2c9fc with PID 1 (/application.jar started by root in /)
[...]
Finished Spring Data repository scanning in 28 ms. Found 0 JPA repository interfaces.
[...]
Started DemoApplication in 4.751 seconds (JVM running for 6.512)

As we can see, Spring Data found no repository interface. That is correct – we didn't create one yet!

If we want to stop all containers, we need to press [Ctrl-C] first. Then we can stop Docker Compose:

docker-compose down

6. Creating a Customer Entity and Repository

To use the PostgreSQL database in our application, we'll create a simple customer entity:

@Entity
@Table(name = "customer")
public class Customer {
    @Id
    @GeneratedValue
    private long id;
    
    @Column(name = "first_name", nullable = false)
    private String firstName;
    
    @Column(name = "last_name", nullable = false)
    private String lastName;

The Customer has a generated id attribute and two mandatory attributes: firstName and lastName.

Now, we can write the repository interface for this entity:

public interface CustomerRepository extends JpaRepository<Customer, Long> { }

By simply extending JpaRepository, we inherit methods for creating and querying our Customer entity.

Finally, we'll use these methods in our application:

@SpringBootApplication
public class DemoApplication {
    @Autowired 
    private CustomerRepository repository; 
  
    @EventListener(ApplicationReadyEvent.class)
    public void runAfterStartup() {
        List allCustomers = this.repository.findAll(); 
        logger.info("Number of customers: " + allCustomers.size());
 
        Customer newCustomer = new Customer(); 
        newCustomer.setFirstName("John"); 
        newCustomer.setLastName("Doe"); 
        logger.info("Saving new customer..."); 
        this.repository.save(newCustomer); 
 
        allCustomers = this.repository.findAll(); 
        logger.info("Number of customers: " + allCustomers.size());
    }
}
  • We access our Customer repository through dependency injection
  • We query the number of existing customers with the repository — this will be zero
  • Then we create and save a customer
  • When we then query the existing customers again, we expect to find the one we just created

7. Running with Docker Compose Again

To run the updated Spring Boot application, we need to rebuild it first. Therefore, we execute these commands once more in the project root directory:

./mvnw clean package -DskipTests
cp target/docker-spring-boot-postgres-0.0.1-SNAPSHOT.jar src/main/docker

How do we rebuild our Docker image with this updated application JAR file? The best way is to remove the existing Docker image whose name we specified in the docker-compose.yml. This forces Docker to build the image again the next time we start our Docker Compose file:

cd src/main/docker
docker-compose down
docker rmi docker-spring-boot-postgres:latest
docker-compose up

So after stopping our containers, we delete the application Docker image. We then start our Docker Compose file again, which rebuilds the application image.

Here's the application output:

Finished Spring Data repository scanning in 180 ms. Found 1 JPA repository interfaces.
[...]
Number of customers: 0
Saving new customer...
Number of customers: 1

Spring Boot finds our empty customer repository. Therefore, we start with no customer but then successfully create one.

8. Conclusion

In this short tutorial, we started by creating a Spring Boot application for PostgreSQL. Next, we wrote a Docker Compose file to run our application container with a PostgreSQL container.

Finally, we created a customer entity and repository, which allowed us to save a customer to PostgreSQL.

As usual, the source code for this tutorial can be found over on GitHub.

The post Running Spring Boot with PostgreSQL in Docker Compose first appeared on Baeldung.

        

Java Weekly, Issue 362

$
0
0

1. Spring and Java

>> Metaspace in OpenJDK 16 [lkorinth.github.io]

More efficient Metaspace in JDK 16: reduced fragmentation and uncommitting the unused memory.

>> JEP proposed to target JDK 16: Warnings for Value-Based Classes [openjdk.java.net]

One step closer to value types: considering primitive wrappers as value types and deprecating their constructors!

>> Cloud Native Buildpacks with Paketo.io & layered jars for Spring Boot [blog.codecentric.de]

Goodbye Dockerfiles: transforming source codes to Docker images without Dockerfiles with the added benefits of layered Jars.

Also worth reading:

Webinars and presentations:

Time to upgrade:

2. Technical

>> HATEOAS without links [mscharhag.com]

Discovering the next steps on the client-side: decoupling client and servers of a REST API using HATEOAS.

Also worth reading:

3. Musings

>> Remote work challenges as a team lead [arnoldgalovics.com]

On being an effective remote leader: embracing async communications, better meetings, and team building.

Also worth reading:

4. Comics

And my favorite Dilberts of the week:

>> Boss Hires Stalker [dilbert.com]

>> Zoom Happy Hour [dilbert.com]

5. Pick of the Week

>> How to Get Lucky: Focus On The Fat Tails [taylorpearson.me]

The post Java Weekly, Issue 362 first appeared on Baeldung.

        

Light-Weight Concurrency in Java and Kotlin

$
0
0

1. Introduction

In this tutorial, we'll explore the basic concepts of concurrency and how different programming languages address them, particularly Java and Kotlin.

We'll focus primarily on the light-weight concurrency models and compare coroutines in Kotlin with the upcoming proposals in Java as part of Project Loom.

2. Basics of Concurrency

Concurrency is the ability to decompose a program into components that are order-independent or partially ordered. The objective here is to have multiple independent processes working together without affecting the outcome.

Within the operating system kernel, we refer to an instance of a program as a process. The kernel isolates processes by assigning them different address spaces for security and fault tolerance. Since each process has its own address space, open file handles, etc., they are quite expensive to create.

Moreover, since processes cannot access each other's memory, inter-process communication becomes non-trivial.

This is where kernel-level threads bring relief for concurrent programming:

Threads are separate lines of execution within a process. A process can typically have multiple threads. While threads share the same file handles and address spaces, they maintain their own programming stacks. This makes inter-thread communication much easier.

The operating system kernel supports and manages the kernel-level threads directly. The kernel does provide system calls to create and manage these threads from outside. However, the kernel has full control of these threads, including their scheduling. This makes kernel-level threads slow and inefficient, resulting in costly thread operations.

On the other hand, we also have user-level threads that are supported in the user-space, part of the system memory allocated to the running applications:

There are various models that map user-level threads to kernel-level threads like one-to-one or many-to-one. But a runtime system like a virtual machine directly manages user-level threads.

The kernel isn't aware of user-level threads. Hence thread operations on user-level threads are much faster. Of course, this requires coordination between the user-level thread scheduler and the kernel.

3. Concurrency in Programming Languages

We discussed broadly the concurrency primitives that the operating systems provide us. But, how do concurrency abstractions available in different programming languages make use of them? While a detailed analysis is beyond this tutorial's scope, we'll discuss some of the popular patterns here.

Most modern programming languages support concurrent programming and provide one or more primitives to work with. For instance, Java has the first-class support for concurrency through an abstraction called the Thread class. This provides a system-independent definition for a thread in Java. However, under-the-hood, Java maps every thread to the kernel level thread through system calls.

As we've already seen, while kernel threads are easier to program with, they are quite bulky and inefficient. The alternative, in fact, is to use the user-level threads. Many programming languages support the concept of light-weight threads natively, while there are several external libraries to enable this as well.

The fundamental approach is to handle the scheduling of these light-weight threads within the execution environment. The scheduling here is cooperative rather than preemptive, which makes it much more efficient. Also, as we manage these threads in the user-space, we can multiplex them on just a few kernel threads, reducing the kernel threads' overall cost.

Different programming languages have different names for them. For instance, we have coroutines in Kotlin, goroutines in Golang, processes in Erlang, and threads in Haskell, to name a few. Although there's no native support for them in Java, this is in the active proposal under Project Loom. We'll examine some of them later in this tutorial.

4. Additional Approaches Towards Concurrency

The concurrency models discussed so far have a commonality that we can mostly reason about the program's flow in a synchronous manner. Even though they provide asynchronicity, fundamental primitives like threads, or coroutines, abstract it mostly.

However, with more explicit asynchronous programming, we break this abstraction and allow parts of the program to run arbitrarily.

For instance, reactive programming sees concurrency with a completely different perspective. It transforms the program flow as a sequence of events that occur asynchronously. Hence, the program code becomes functions that listen to these asynchronous events, process them, and, if necessary, publish new events.

We often depict these graphically as marble diagrams:

More importantly, the thread on which we publish or subscribe to these events is actually not significant in reactive programming. The reactor core process uses a limited number of threads, typically matching the available CPU cores. It executes a function on a free thread from the pool and releases it back.

So, if we can avoid using any blocking code, it can result in a program that executes much more efficiently, even on a single thread. Also, it addresses some of the pain points like call-back hell typically associated with other asynchronous programming styles.

However, it increases the level of difficulty in reading and writing the program, making it difficult to test and maintain.

5. A Case for Structured Concurrency

A typical concurrent application with more than one execution path is difficult to reason about. Part of the problem is that it lacks abstraction. For instance, if we call a function in such an application, we can't guarantee that processing has terminated when the faction terminates. This is because the function may have spawned multiple concurrent execution paths, of which we're completely unaware.

A sequential flow of the program is much easier to read and write. Of course, to support concurrency, this flow needs to branch out. But, it's much simpler to comprehend if all the branches terminate back into the main flow:

So, maintaining the abstraction, we don't really care how the function internally decomposes the program. It's all fine, so far, as all lines of execution terminate with the function. Alternatively, the scopes of concurrent executions are cleanly nested. This is the fundamental premise of structured concurrency. It emphasizes that if control splits into concurrent tasks, they must join up again.

If we see some of the asynchronous programming models like reactive programming, we'll understand that it's difficult to achieve structured concurrency. In fact, concurrent programming has mostly involved arbitrary jumps, even with simpler primitives like threads.

However, we can achieve structured concurrency in Kotlin with a solution like coroutines.

We'll see how, later in this tutorial.

6. Kotlin: How Do They Do It?

Now we've gathered enough background to examine how Kotlin solves the problem of concurrency while keeping most of the issues at bay. Kotlin is an open-source programming language that was started by JetBrains back in 2010. Kotlin targets the JVM along with other platforms like JavaScript, and even Native. Hence, it can produce Java-compatible bytecode.

Kotlin provides the support for light-weight threads in the form of coroutines, which are implemented as a rich librarykotlinx.coroutines. Interestingly, the JVM does not have native support for a light-weight concurrency construct like coroutine — well, at least yet! Nonetheless, Kotlin introduced coroutines as an experimental language feature quite early, and they became official in version 1.3.

We'll see how Kotlin implements coroutines and how we can use them to write concurrent applications with the benefits of structured concurrency.

6.1. What Exactly Is a Coroutine?

Generally speaking, coroutines are parts of a computer program or generalized subroutines that can suspend and resume their execution at any point. It first appeared as a method in assembly languages way back in the 1950s. Coroutines can have several interesting applications.

When we use them for concurrency, they appear to be similar to kernel threads. However, there are subtle differences. For instance, a scheduler manages kernel threads preemptively, while coroutines voluntarily yield control, resulting in cooperative multitasking.

Let's see a general construction of coroutines:

coroutine
    loop
        while some_condition
            some_action
        yield

Here, as we can see, we have a coroutine that performs some action in a loop but cooperatively yields the control on every step instead of blocking. This can help to utilize underlying kernel threads much more efficiently. This is exactly what other asynchronous programming styles like reactive programming do, but without the complexities.

While a coroutine can choose to yield to a specific coroutine, there can also be a controller that schedules multiple coroutines. More interestingly, we can multiplex thousands of coroutines on just a single underlying kernel thread.

But consequently, coroutines don't necessarily provide parallelism, even on a multi-core system.

6.2. Kotlin Coroutine in Action

Kotlin provides many coroutine builders to create a coroutine, like launch, async, and runBlocking. Further, coroutines in Kotlin are always bound to a coroutine scope. The coroutine scope contains the coroutine context and sets the new coroutine scope that is launched by a coroutine builder.

We'll see shortly how to launch a coroutine, but let's first understand suspending functions. Kotlin provides a special keyword called suspend to mark such functions. This allows the compiler to sprinkle some magic into these functions, which we'll see later.

Let's create a suspending function:

suspend fun doSomething() {
    // heavy computation here
}

Barring the use of this keyword, we can see that these are just regular functions. However, there is an important limitation: Suspending functions can only be invoked from within a coroutine or from another suspending function.

So, let's use one of the coroutine builders to launch a coroutine and call our simple but suspending function:

GlobalScope.launch {
    doSomething() // does some heavy computation in the background
    ... do other stuff
}

Here, we're starting a new coroutine with the launch coroutine builder.

6.3. Structured Concurrency with Coroutines

Now, we should avoid launching a coroutine bound to the GlobalScope, unless intended for the right reasons. This is because such coroutines operate on the whole application lifecycle and, more importantly, deviate from the principles of structured concurrency.

To adhere to structured concurrency, we should rather create an application-specific CoroutineScope and use coroutine builder on its instance:

var job = Job()
val coroutineScope = CoroutineScope(Dispatchers.Main + job)
coroutineScope.launch { 
    doSomething() // does some heavy computation in the background 
    ... do other stuff 
}

To create an instance of CoroutineScope, we have to define a Dispatcher, which controls which thread runs a coroutine. The Job here is responsible for the coroutine's lifecycle, cancellation, and parent-child relations.

All the coroutines launched using this CoroutineScope can be simply canceled by canceling this parent Job. This prevents coroutines from leaking unintentionally. This also avoids having side-effects of launching coroutines from a suspending function. Hence, we achieve structured concurrency, which we've discussed before.

6.4. Looking Under the Hood

So, the question now is: How does Kotlin implement coroutines? Broadly speaking, coroutines are implemented in Kotlin as a finite state machine with suspension points and continuations. For those of us uninitiated in this area, this may not make any sense! But we'll try to describe them briefly.

Let's first understand some of the terms we've just introduced. A suspension point is a point in the suspending function at which we want to suspend our execution and resume later. At the same time, a continuation is actually the encapsulation of the state of a function at a suspension point. Basically, a continuation captures the rest of the execution after the suspension point.

Now Kotlin, upon compilation, transforms all suspending functions to add a parameter, which is the continuation object. The compiler will transform the signature of our suspending function from the previous section:

fun doSomething(continuation: Continuation): Any?

This programming style is typical of functional programming and is known as Continuation Passing Style (CPS). Here, the control is passed explicitly in the form of a continuation. This is somewhat similar to the asynchronous programming style where we pass a callback function to get notified. However, with coroutines in Kotlin, the compiler implicitly handles the continuations.

The Kotlin compiler identifies all possible suspension points in a suspending function and creates states with labels for everything delimited by the suspension points. The resulting continuation is nothing but a huge switch statement with these states as labels.

Hence, we can think of continuation as packing this as a finite state machine.

7. Java: What Is the Proposal?

Java has had first-class support for concurrency since the early days of its inception. However, Java does not have native support for what we know as light-weight threads. Although there have been several attempts to build such support outside the core Java, none of them could find enough success.

For the last couple of years, OpenJDK has been working on Project Loom to bridge this gap.

7.1. A Brief History of Concurrency in Java

Since JDK 1.0, the class Thread has provided a core abstraction for concurrency in Java. It was intended to run on all platforms alike, to match the promise “write once, run anywhere”. Unfortunately, some of the target platforms didn't have native support for threads back then. Hence, Java had to implement something called green threads to deliver that promise.

Basically, green threads are the implementation of threads that are managed in the user-space and scheduled by the virtual machine. We've already seen such threads' general definition and discussed how coroutines in Kotlin or goroutines in Golang and similar concepts. Although green threads may vary in terms of the implementation, the basic idea was actually quite similar.

In the initial days, Java struggled to refine the implementation of green threads. It was difficult to scale green threads over multiple processors and hence benefit from parallelism on multi-core systems. To get around this problem and simplify the concurrent programming model, Java decided to abandon green threads in version 1.3.

So, Java decided to map every thread to a separate native kernel thread. Essentially the JVM threads became a thin wrapper around the operating system threads. This simplified the programming model, and Java could leverage the benefits of parallelism with preemptive scheduling of threads by the kernel across multiple cores.

7.2. Problems with the Java Concurrency Model

The concurrency model in Java was actually quite easy to use and has been improved substantially with the introduction of ExecutorService and CompletableFuture. This also worked well for a large period of time. However, the problem is how concurrent applications that were written with this model have to face an unprecedented scale today.

For instance, typical servlet containers are written in the thread-per-request model. But, it's impossible to create as many threads on a system as the number of concurrent requests we expect them to handle. This calls for alternate programming models like event-loop or reactive programming that are inherently non-blocking, but they have their own share of issues.

7.3. Proposals of Project Loom

By now, it should not be difficult for us to guess that perhaps it's time for Java to bring back the support for light-weight threads. This is actually the motivation behind Project Loom. The purpose of this project is to explore and incubate a light-weight concurrency model on the Java platform. The idea is to build support for light-weight threads on top of the JVM threads and fundamentally decouple the JVM threads from the native kernel threads.

The current proposal is to introduce support for some core concurrency related constructs right at the level of JVM. These include virtual threads (previously called fibers), delimited continuation, and tail-call elimination. The current construct of the thread is basically a composition of continuation and scheduler. The idea is to separate these concerns and support virtual threads on top of these building blocks.

As the current JVM thread is just a wrapper over the underlying kernel thread, it relies on the kernel to provide the implementation for both continuation and scheduler. However, by exposing continuation as a construct within the Java platform, it's possible to combine it with a global scheduler. This gives rise to virtual threads as light-weight threads managed entirely within the JVM.

Of course, the idea behind Project Loom is not just to provide a construct like the virtual thread in Java but also to address some of the other issues that arise due to them. For instance, a flexible mechanism to pass data among a large number of virtual threads. A more intuitive way to organize and supervise so many virtual threads, a concept close to structured concurrency. Or managing context-data for so many virtual threads, similar to what we have as thread-local for current threads.

7.4. Understanding Continuations

Let's understand what we actually mean by delimited continuations in the scope of Project Loom. Actually, the basic idea behind a delimited continuation is nothing different from a coroutine that we've already discussed before. Hence, we can see a delimited continuation as a sequential code that can suspend its execution at any point and resume again from the same point.

In Java, however, the proposal is to expose continuations as a public API. The proposed API may look like the following:

class _Continuation {
    public _Continuation(_Scope scope, Runnable target) 
    public boolean run()
    public static _Continuation suspend(_Scope scope, Consumer<_Continuation> ccc)
    public ? getStackTrace()
}

Please note that continuation is a general construct and has nothing specific to virtual threads. Although virtual threads require continuations for implementation, there are other possible uses of continuations as well. For instance, we can use it to implement a generator, which is an iterator that yields after producing a single value.

7.5. Implementation of Virtual Threads

The focus of Project Loom is to provide support for virtual threads as a basic construct. Virtual threads are the higher-level construct that is proposed to provide the capabilities of user-mode threads in Java. Basically, virtual threads should allow us to run an arbitrary code concurrently with the ability to suspend and resume execution.

As we can already guess, continuations will be used to create higher-level constructs like virtual threads. The idea is that the virtual thread will hold a private instance of the continuation class along with other necessary parts:

class _VirtualThread {
    private final _Continuation continuation;
    private final Executor scheduler;
    private volatile State state;
    private final Runnable task;
​
    private enum State { NEW, LEASED, RUNNABLE, PAUSED, DONE; }
  
    public _VirtualThread(Runnable target, Executor scheduler) {
        .....
    }
  
    public void start() {
        .....
    }
  
    public static void park() {
        _Continuation.suspend(_FIBER_SCOPE, null);
    }
  
    public void unpark() {
        .....
    }
}

Above is a simple representation of how we can compose a virtual thread with low-level primitive, like continuations. Also, note that schedulers are an essential part of implementing the virtual thread. However, the initial default global scheduler for virtual threads will be the ForkJoinPool that already exists in Java and implements a work-stealing algorithm.

More importantly, the proposal is to keep the API of virtual threads very close to that of the current heavy-weight threads. The heavy-weight thread as it exists today will continue to exist. So the conformity of the API that heavy-weight or the new light-weight threads support will lead to a better user experience.

7.6. A Sneak Peek into the Current State

Project Loom has been in progress for a couple of years now, and some parts of the proposal may be available as part of Java 16 in 2021. However, early-access builds are available for some time to experiment with the new features and provide feedback.

So, first, let's see how working with heavy-weight threads, or the threads as we know them currently, will change:

Runnable printThread = () -> System.out.println(Thread.currentThread());
ThreadFactory kernelThreadFactory = Thread.builder().factory();
Thread kernelThread = kernelThreadFactory.newThread(printThread);
kernelThread.start();

As we can see, we have a new interface called Thread.Builder, which is a mutable builder for Thread or ThreadFactory. This is to facilitate creating a kernel thread, as we're doing here, or a virtual thread. Everything else is quite similar to what exists today.

So, let's see how to create and use a virtual thread instead:

Runnable printThread = () -> System.out.println(Thread.currentThread());
ThreadFactory virtualThreadFactory = Thread.builder().virtual().factory();
Thread virtualThread = virtualThreadFactory.newThread(printThread);
virtualThread.start();

Apart from the fact that there's a different thread factory to create virtual threads, there is actually no difference! This is because the current implementation of virtual threads does not introduce a new class but just a new implementation of the Thread class.

Apart from the fact that this new implementation of Thread differs in scheduling, there are other aspects that will not work the same for them. For instance, the behavior and implications of some of the existing constructs like ThreadGroup and ThreadLocal will be different for the virtual threads.

8. How Are Java Virtual Threads Different from Kotlin Coroutines?

We've discussed in detail the support for the light-weight concurrency model that Kotlin has in terms of coroutines and the model Java is proposing to bring as virtual threads.

The obvious question is, how do they compare against each other, and is it possible to benefit from both of them when they target the same JVM. In this section, we'll explore some of the important aspects like continuations and scheduling.

8.1. Stackful vs. Stackless Continuations

Since continuations form the basis of any form of user-mode thread implementation, let's begin by examining their implementation in Kotlin and how they are different from the proposal in Java. Broadly speaking about design choice, Kotlin coroutines are stackless, whereas continuations in Java are proposed to be stackful.

As the name suggests, stackful continuations or coroutines maintain their own function call stack. A stack here is a contiguous block of memory that is needed to store the local variables and the function arguments. On the contrary, stackless coroutines do not maintain their stack and rely on the caller. This makes them strongly connected to the caller.

As an immediate fallout, stackless coroutines can suspend themselves only from the top-level function. So, all functions called from the coroutine must finish before suspending the coroutine. In comparison, a stackful continuation or coroutine can suspend at any nested depth of the call stack. So, stackful coroutines are more powerful and general-purpose than stackless coroutines.

However, since stackless coroutines have a lower memory footprint than stackful coroutines, they prove to be more efficient. This is because context switching between stackless coroutines comes out to be less expensive. Moreover, the compiler locally handles the code transformations for stackless coroutines with very little support from the runtime.

8.2. Preemptive vs. Cooperative Scheduling

Apart from continuations, another important part of the implementation of a light-weight thread is scheduling. We've seen how the operating system scheduler schedules the kernel threads preemptively. This is, in fact, one of the reasons why kernel threads prove to be inefficient. So, typically, the approach for scheduling light-weight threads is more structured than arbitrary.

As we've seen earlier, the scheduling in Kotlin coroutines is cooperative where coroutines voluntarily yield the control at logical points. For instance, we can decide to wrap a computationally heavy or blocking operation in a suspending function. When we call such functions from a coroutine or another suspending function, these become natural suspension points.

However, the current proposal in Java is to keep the scheduling preemptive rather than cooperative. Hence, it's not possible to define suspension points in Java virtual threads. So, does that mean it will carry the burden of kernel scheduler? Not really. Note that the kernel threads are preempted arbitrarily, based on the notion of time-slice.

However, the proposal for virtual thread scheduler in Java is to preempt them when they block on I/O or synchronization.

Regardless of how they're scheduled, light-weight threads are finally executed on the underlying kernel threads. In the case of Kotlin coroutines, coroutine context includes a coroutine dispatcher. The coroutine dispatcher decides which kernel thread the coroutine uses for its execution.

On the other hand, the Java virtual thread scheduler maintains a pool of kernel threads as workers and mounts a runnable virtual thread on one of the available workers.

9. Conclusion

In this tutorial, we understood the basic concepts of concurrency and how light-weight concurrency differs from heavy-weight concurrency. We also touched upon how concurrency is generally approached in programming languages and what we mean by structured concurrency.

Further, we understood how light-weight concurrency is supported in Kotlin as coroutines and how Java is proposing to introduce virtual threads in that regard. We discussed these constructs in some detail and then touched upon how their implementations differ from each other.

The post Light-Weight Concurrency in Java and Kotlin first appeared on Baeldung.

        
Viewing all 4463 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>