Custom Reporting with TestNG

1. Overview

In this article, we will discuss generating custom logging and reports using TestNG.

TestNG provides its own reporting feature – generating reports in either HTML/XML formats. If tests are run using the maven-surefire-plugin, the report will take the default form defined by the plugin. Apart from built-in reporting, it provides a mechanism for easy customization of logged information and generated reports.

If you’d like to start with the TestNG basics, check out this article.

2. Custom Logging

Before we implement custom logging, let’s look into the default logs by executing mvn test command:

Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, 
  Time elapsed: 1.21 sec <<< FAILURE! 
- in TestSuite
whenCalledFromSuite_thanOK(baeldung.com.RegistrationTest)  
Time elapsed: 0.01 sec  <<< FAILURE!
java.lang.AssertionError: Test Failed due to some reason
    at baeldung.com.RegistrationTest.whenCalledFromSuite_thanOK(
      RegistrationTest.java:15)


Results :

Failed tests:
  RegistrationTest.whenCalledFromSuite_thanOK:15 
    Test Failed due to some reason

Tests run: 11, Failures: 1, Errors: 0, Skipped: 0

[ERROR] There are test failures.

These logs do not give us any information about the order of execution, or about when a particular test was started/completed etc.

If we want to know the result of each run along with some custom data, we can implement our own logs and reports. TestNG provides a way of implementing custom reports and logging.

Simply put, we can either implement the org.testng.ITestListener interface for logging or the org.testng.IReporter interface for reporting. These implemented classes get notified for events like start, end, failure etc of tests and suites.

Let’s go ahead and implement some simple custom logging:

public class CustomisedListener implements ITestListener {
    
    // ...
    @Override
    public void onFinish(ITestContext testContext) {
        LOGGER.info("PASSED TEST CASES");
        testContext.getPassedTests().getAllResults()
          .forEach(result -> {LOGGER.info(result.getName());});
        
        LOGGER.info("FAILED TEST CASES");
        testContext.getFailedTests().getAllResults()
          .forEach(result -> {LOGGER.info(result.getName());});
        
        LOGGER.info(
          "Test completed on: " + testContext.getEndDate().toString());
    }  
 
    //...
}

Notice how we’ve overridden the onFinish() method, which will be invoked when all test executions are complete and all configurations are done. Similarly, we can override other methods – such as onTestStart(), onTestFailure() etc (and find details about these other methods here).

Now let’s include this listener in the XML configuration:

<suite name="My test suite">
    <listeners>
        <listener class-name="com.baeldung.reports.CustomisedListener" />
    </listeners>
    <test name="numbersXML">
        <parameter name="value" value="1" />
        <parameter name="isEven" value="false" />
        <classes>
            <class name="baeldung.com.ParametrizedTests" />
        </classes>
    </test>
</suite>

Once executed, the listener will be invoked on every event and will log information as we implemented. This might be useful for debugging our test execution.

The output logs:

...
INFO CUSTOM_LOGS - Started testing on: Sat Apr 22 14:39:43 IST 2017
INFO CUSTOM_LOGS - Testing: 
  givenNumberFromDataProvider_ifEvenCheckOK_thenCorrect
INFO CUSTOM_LOGS - Tested: 
  givenNumberFromDataProvider_ifEvenCheckOK_thenCorrect Time taken:6 ms
INFO CUSTOM_LOGS - Testing: 
  givenNumberObjectFromDataProvider_ifEvenCheckOK_thenCorrect
INFO CUSTOM_LOGS - Failed : 
  givenNumberObjectFromDataProvider_ifEvenCheckOK_thenCorrect
INFO CUSTOM_LOGS - PASSED TEST CASES
INFO CUSTOM_LOGS - givenNumberFromDataProvider_ifEvenCheckOK_thenCorrect
INFO CUSTOM_LOGS - FAILED TEST CASES
INFO CUSTOM_LOGS - 
  givenNumberObjectFromDataProvider_ifEvenCheckOK_thenCorrect
INFO CUSTOM_LOGS - Test completed on: Sat Apr 22 14:39:43 IST 2017
...

The custom logs give us missing information in the default logs.

3. Custom Reports

When we run tests using the plugin, it generates reports in HTML/XML formats in target/surefire-reports directory:

If we want to run a particular test suite using a TestNG XML file, we need to list it in the surefire-plugin configuration tag:

<configuration>
    <suiteXmlFiles>
        <suiteXmlFile>
            src\test\resources\parametrized_testng.xml
        </suiteXmlFile>
    </suiteXmlFiles>
</configuration>

After custom logging, now let’s try to create some custom reports where we implement the org.testng.IReporter interface and override the generateReport() method:

public void generateReport(
  List<XmlSuite> xmlSuites, 
  List<ISuite> suites, String outputDirectory) {
 
    String reportTemplate = initReportTemplate();

    String body = suites
      .stream()
      .flatMap(suiteToResults())
      .collect(Collectors.joining());

    String report
      = reportTemplate.replaceFirst("</tbody>", String.format("%s</tbody>", body));
    saveReportTemplate(outputDirectory, report);
}

The overridden method takes three arguments:

• xmlSuite – contains a list of all suites mentioned in the XML file
• suites – a list object, holding all information about the test execution
• outputDirectory – the directory path where reports are generated

We used the initReportTemplate() method to load an HTML template, suiteToResults() function which invokes the resultsToRow() function to handle the internals of generating the report:

private Function<ISuite, Stream<? extends String>> suiteToResults() {
    return suite -> suite.getResults().entrySet()
      .stream()
      .flatMap(resultsToRows(suite));
}

private Function<Map.Entry<String, ISuiteResult>, 
  Stream<? extends String>> resultsToRows(ISuite suite) {
    return e -> {
        ITestContext testContext = e.getValue().getTestContext();

        Set<ITestResult> failedTests 
          = testContext.getFailedTests().getAllResults();
        Set<ITestResult> passedTests 
          = testContext.getPassedTests().getAllResults();
        Set<ITestResult> skippedTests 
          = testContext.getSkippedTests().getAllResults();

        String suiteName = suite.getName();

        return Stream
          .of(failedTests, passedTests, skippedTests)
          .flatMap(results ->
            generateReportRows(e.getKey(), suiteName, results).stream());
    };
}

and saveReportTemplate() for saving the complete result.

Include the reporter in the XML configuration file:

<suite name="suite">
    <listeners>
        <listener class-name="com.baeldung.reports.CustomisedReports" />
    </listeners>
    <test name="test suite">
        <classes>
	    <class name="baeldung.com.RegistrationTest" />
            <class name="baeldung.com.SignInTest" />
        </classes>
    </test>
</suite>

Here’s the output of our reports:

In comparison to the default sure-fire HTML report, this report gives a clear and crisp picture of the result, in a single table. Which is more convenient and easy to read.

4. Conclusion

In this quick tutorial, we learned about how to generate test reports with Surefire Maven plugin. We also looked into customizing the logs and generating customized reports with TestNG. For more details about TestNG like how to write test cases, suites, etc. refer to our introductory

For more details about TestNG, such as how to write test cases, suites, definitely start with our introductory article.

As always the implementation of the snippets can be found over on GitHub.