A Quick JUnit vs TestNG Comparison

1. Overview

JUnit and TestNG are undoubtedly the two most popular unit-testing frameworks in the Java ecosystem. While TestNG itself is inspired by JUnit, it provides its own distinctive features and unlike JUnit, it works for functional and higher levels of testing.

In this article, we will discuss and compare these frameworks by covering their features and common use cases.

2. Test Setup

While writing test cases, often we need to execute some configuration or initialization instructions before test executions, and also some cleanup after completion of tests. Let’s evaluate these in both frameworks.

JUnit offers initialization and cleanup at two levels, before and after each method and class. @Before and @After annotations at method level and @BeforeClass and @AfterClass at the class level:

public class SummationServiceTest {

    private static List<Integer> numbers;

    @BeforeClass
    public static void initialize() {
        numbers = new ArrayList<>();
    }

    @AfterClass
    public static void tearDown() {
        numbers = null;
    }

    @Before
    public void runBeforeEachTest() {
        numbers.add(1);
        numbers.add(2);
        numbers.add(3);
    }

    @After
    public void runAfterEachTest() {
        numbers.clear();
    }

    @Test
    public void givenNumbers_sumEquals_thenCorrect() {
        int sum = numbers.stream().reduce(0, Integer::sum);
        assertEquals(6, sum);
    }
}

Similar to JUnit, TestNG also provides initialization and cleanup at the method and class level. While @BeforeClass and @AfterClass remain the same at the class level, the method level annotations are @BeforeMethod and @AfterMethod:

@BeforeClass
public void initialize() {
    numbers = new ArrayList<>();
}

@AfterClass
public void tearDown() {
    numbers = null;
}

@BeforeMethod
public void runBeforeEachTest() {
    numbers.add(1);
    numbers.add(2);
    numbers.add(3);
}

@AfterMethod
public void runAfterEachTest() {
    numbers.clear();
}

TestNG also offers, @BeforeSuite, @AfterSuite, @BeforeGroup and @AfterGroup annotations, for configurations at suite and group levels:

@BeforeGroups("positive_tests")
public void runBeforeEachGroup() {
    numbers.add(1);
    numbers.add(2);
    numbers.add(3);
}

@AfterGroups("negative_tests")
public void runAfterEachGroup() {
    numbers.clear(); 
}

Also, we can use the @BeforeTest and @AfterTest if we need any configuration before or after test cases included in the <test> tag in TestNG XML configuration file:

<test name="test setup">
    <classes>
        <class name="SummationServiceTest">
            <methods>
                <include name="givenNumbers_sumEquals_thenCorrect" />
            </methods>
        </class>
    </classes>
</test>

Note that, the declaration of @BeforeClass and @AfterClass method has to be static in JUnit whereas, there is more flexibility in TestNG in the method declaration, it does not have these constraints.

3. Ignoring Tests

Both frameworks support ignoring test cases, though they do it quite differently. JUnit offers @Ignore annotation:

@Ignore
@Test
public void givenNumbers_sumEquals_thenCorrect() {
    int sum = numbers.stream().reduce(0, Integer::sum);
    Assert.assertEquals(6, sum);
}

while TestNG uses @Test with a parameter “enabled” with a boolean value true or false:

@Test(enabled=false)
public void givenNumbers_sumEquals_thenCorrect() {
    int sum = numbers.stream.reduce(0, Integer::sum);
    Assert.assertEquals(6, sum);
}

4. Running Tests Together

Running tests together as a collection is possible in both JUnit 4 and TestNG, but they do it in different ways.

The @RunWith and @Suite annotations are used to group test cases and run them as a suite. A suite is a collection of test cases that can be grouped together and run as a single test.

All the declarations are defined inside a single class, let’s look at the example:

@RunWith(Suite.class)
@Suite.SuiteClasses({ RegistrationTest.class, SignInTest.class })
public class SuiteTest {

}

In TestNG grouping tests in groups are done using XML file:

<suite name="suite">
    <test name="test suite">
        <classes>
            <class name="com.baeldung.RegistrationTest" />
            <class name="com.baeldung.SignInTest" />
        </classes>
    </test>
</suite>

This indicates RegistrationTest and SignInTest will run together.

Apart from grouping classes, TestNG can group methods as well. Methods are grouped with @Test(groups=”groupName”) annotation:

@Test(groups = "regression")
public void givenNegativeNumber_sumLessthanZero_thenCorrect() {
    int sum = numbers.stream().reduce(0, Integer::sum);
    Assert.assertTrue(sum < 0);
}

Let’s use an XML to execute the groups:

<test name="test groups">
    <groups>
        <run>
            <include name="regression" />
        </run>
    </groups>
    <classes>
        <class 
          name="com.baeldung.SummationServiceTest" />
    </classes>
</test>

This will execute the test method tagged with group regression.

5. Testing Exceptions

The feature for testing for exceptions using annotations is available in both JUnit 4:

@Test(expected = ArithmeticException.class) 
public void givenNumber_whenThrowsException_thenCorrect() { 
    int i = 1 / 0;
}

and TestNG:

@Test(expectedExceptions = ArithmeticException.class) 
public void givenNumber_whenThrowsException_thenCorrect() { 
    int i = 1 / 0;
}

This feature implies what exception is thrown from a piece of code, that is being unit tested.

6. Parameterized Tests

Parameterized unit tests are used for testing the same code under several conditions. With the help of parameterized unit tests, we can set up a test method that obtains data from some data source. The main idea is to make the unit test method reusable and to test with a different set of inputs.

In JUnit, the test class has to be annotated with @RunWith to make it a parameterized class and @Parameter to use the denote the parameter values for unit test. @Parameters will have to return a List[], and these parameters will be passed to the constructor of the test class as argument:

@RunWith(value = Parameterized.class)
public class ParametrizedTests {

    private int value;
    private boolean isEven;

    public ParametrizedTests(int value, boolean isEven) {
        this.value = value;
        this.isEven = isEven;
    }

    @Parameters
    public static Collection<Object[]> data() {
        Object[][] data = new Object[][]{{1, false}, {2, true}, {4, true}};
        return Arrays.asList(data);
    }

    @Test
    public void givenParametrizedNumber_ifEvenCheckOK_thenCorrect() {
        Assert.assertEquals(isEven, value % 2 == 0);
    }
}

Note that the declaration of the parameter has to be static and it has to be passed to the constructor in order to initialize the class member as value for testing. In addition to that, the return type of parameter class must be a List, and the values are limited to String or primitive data types.

In TestNG, we can parametrize tests using @Parameter or @DataProvider annotation. While using the XML file annotate the test method with @Parameter:

@Test
@Parameters({"value", "isEven"})
public void 
  givenNumberFromXML_ifEvenCheckOK_thenCorrect(int value, boolean isEven) {
    Assert.assertEquals(isEven, value % 2 == 0);
}

and provide the data in the XML file:

<suite name="My test suite">
    <test name="numbersXML">
        <parameter name="value" value="1"/>
        <parameter name="isEven" value="false"/>
        <classes>
            <class name="baeldung.com.ParametrizedTests"/>
        </classes>
    </test>
</suite>

While using data in the XML file is simple and useful, in some cases, you might need to provide more complex data.

@DataProvider annotation is used to handle these scenarios, which can be used to map complex parameter types for testing methods.

@DataProvider for primitive data types:

@DataProvider(name = "numbers")
public static Object[][] evenNumbers() {
    return new Object[][]{{1, false}, {2, true}, {4, true}};
}

@Test(dataProvider = "numbers")
public void givenNumberFromDataProvider_ifEvenCheckOK_thenCorrect
  (Integer number, boolean expected) {
    Assert.assertEquals(expected, number % 2 == 0);
}

@DataProviderfor objects:

@Test(dataProvider = "numbersObject")
public void givenNumberObjectFromDataProvider_ifEvenCheckOK_thenCorrect
  (EvenNumber number) {
    Assert.assertEquals(number.isEven(), number.getValue() % 2 == 0);
}

@DataProvider(name = "numbersObject")
public Object[][] parameterProvider() {
    return new Object[][]{{new EvenNumber(1, false)},
      {new EvenNumber(2, true)}, {new EvenNumber(4, true)}};
}

In the same way, any specific objects that are to be tested can be created and returned using data provider. It’s useful when integrating with frameworks like Spring.

7. Test Timeout

Timed out tests means, a test case should fail if the execution is not completed within certain specified time period. Both JUnit and TestNG support timed out tests, in the same way, annotating with @Test(timeout=1000):

@Test(timeOut = 1000)
public void givenExecution_takeMoreTime_thenFail() {
    while (true);
}

8. Dependent Tests

TestNG supports dependency testing. This means in a set of test methods, if the initial test fails, then all subsequent dependent tests will be skipped, not marked as failed as in the case for JUnit.

Let’s have a look at a scenario, where we need to validate email, and if it’s successful, will proceed to log in:

@Test
public void givenEmail_ifValid_thenTrue() {
    boolean valid = email.contains("@");
    Assert.assertEquals(valid, true);
}

@Test(dependsOnMethods = {"givenEmail_ifValid_thenTrue"})
public void givenValidEmail_whenLoggedIn_thenTrue() {
    LOGGER.info("Email {} valid >> logging in", email);
}

9. Conclusion

Both JUnit and TestNG are popular tools for testing in the Java ecosystem.

In this article we had a quick look at the various needs and ways of writing tests with each of these two test frameworks.

The implementation of all the code snippets can be found in TestNG and  core-java Github project.