Difference Between Maven Surefire and Failsafe Plugins

1. Overview

In typical Test Driven Development, we aim to write lots of low-level unit tests that are fast to run and set up in isolation. Additionally, there are also few high-level integration tests that are dependent on external systems, for example, setting up a server or databases. Unsurprisingly, these are typically both resource and time-consuming.

Hence, these tests mostly require some pre-integration setup and post-integration cleanup for the graceful termination. Therefore, it's desirable to distinguish between the two types of tests and be able to run them separately during the build process.

In this tutorial, we'll compare the Surefire and Failsafe plugins most commonly used for running various types of tests in a typical Apache Maven build.

2. Surefire Plugin

The Surefire Plugin belongs to a set of Maven core plugins and runs the unit tests of the application.

The project POM includes this plugin by default, but we can also configure it explicitly:

<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M5</version>
                ....
            </plugin>
         </plugins>
    </pluginManagement>
</build>

The plugin binds to the test phase of the default lifecycle. Therefore, let's execute it with the command:

mvn clean test

This runs all the unit tests in our project. Since the Surefire plugin binds with the test phase, in case of any test failures, the build fails, and no further phases execute during the build process.

Alternatively, we can modify the plugin configuration to run integration tests, as well as the unit tests. However, this may not be desirable behavior for integration tests which could require some environment setup before, as well as some clean-up after test execution.

Maven provides another plugin precisely for this purpose.

3. Failsafe Plugin

The Failsafe Plugin is designed to run the integration tests in the project.

3.1. Configuration

First, let's configure this in the project POM:

<plugin>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>3.0.0-M5</version>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
            ....
        </execution>
    </executions>
</plugin>

Here, the plugin's goals bind to the integration-test and verify phases of the build cycle in order to execute the integration tests.

Now, let's execute the verify phase from the command line:

mvn clean verify

This runs all the integration tests, but if any tests fail during the integration-test phase, the plugin does not fail the build immediately.

Instead, Maven still executes the post-integration-test phase. Therefore we can still perform any cleanup and environment tear-down as part of the post-integration-test phase. The subsequent verify phase of the build process reports any test failures.

3.2. Example

In our example, we'll configure a Jetty server to start prior to running the integration tests and stop after the test execution.

First, let's add the Jetty Plugin to our POM:

<plugin>
    <groupId>org.eclipse.jetty</groupId>
    <artifactId>jetty-maven-plugin</artifactId>
    <version>9.4.11.v20180605</version>
    ....
    <executions>
        <execution>
            <id>start-jetty</id>
            <phase>pre-integration-test</phase>
            <goals>
                <goal>start</goal>
            </goals>
        </execution>
        <execution>
            <id>stop-jetty</id>
            <phase>post-integration-test</phase>
            <goals>
                <goal>stop</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Here we've added the configuration to start and stop the Jetty server during the pre-integration-test and post-integration-test phases respectively.

Now, let's execute our integration tests once again and see the console output:

....
[INFO] <<< jetty-maven-plugin:9.4.11.v20180605:start (start-jetty) 
  < validate @ maven-integration-test <<<
[INFO] --- jetty-maven-plugin:9.4.11.v20180605:start (start-jetty)
  @ maven-integration-test ---
[INFO] Started ServerConnector@4b9dc62f{HTTP/1.1,[http/1.1]}{0.0.0.0:8999}
[INFO] Started @6794ms
[INFO] Started Jetty Server
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M5:integration-test (default)
  @ maven-integration-test ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.baeldung.maven.it.FailsafeBuildPhaseIntegrationTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.024 s
  <<< FAILURE! - in com.baeldung.maven.it.FailsafeBuildPhaseIntegrationTest
[ERROR] com.baeldung.maven.it.FailsafeBuildPhaseIntegrationTest.whenTestExecutes_thenPreAndPostIntegrationBuildPhasesAreExecuted
  Time elapsed: 0.012 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
	at com.baeldung.maven.it.FailsafeBuildPhaseIntegrationTest
          .whenTestExecutes_thenPreAndPostIntegrationBuildPhasesAreExecuted(FailsafeBuildPhaseIntegrationTest.java:11)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   FailsafeBuildPhaseIntegrationTest.whenTestExecutes_thenPreAndPostIntegrationBuildPhasesAreExecuted:11
  expected: <true> but was: <false>
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] --- jetty-maven-plugin:9.4.11.v20180605:stop (stop-jetty)
  @ maven-integration-test ---
[INFO]
[INFO] --- maven-failsafe-plugin:3.0.0-M5:verify (default)
  @ maven-integration-test ---
[INFO] Stopped ServerConnector@4b9dc62f{HTTP/1.1,[http/1.1]}{0.0.0.0:8999}
[INFO] node0 Stopped scavenging
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
....

Here, as per our configuration, the Jetty server starts prior to the integration test execution. For demonstration, we have a failing integration test, but this does not fail the build immediately. The post-integration-test phase executes after the test execution, and the server stops before build failure.

In contrast, if we use Surefire Plugin to run these integration tests, the build would have stopped at the integration-test phase without performing any required cleanup.

An additional benefit of using different plugins for different types of tests is the separation between the various configurations. This improves the maintainability of the project build.

4. Conclusion

In this article, we compared Surefire and Failsafe plugins for separating and running different types of tests. We also looked at an example and saw how Failsafe Plugin provides additional functionality for running tests that require further setup and cleanup.

As always, the code is available over on GitHub.