Configuring Skip Logic in Spring Batch

1. Introduction

By default, any errors encountered during a Spring Batch job processing will make a corresponding step fail. However, there are many situations where we'd rather like to skip the currently processed item for certain exceptions.

In this tutorial, we'll explore two approaches to configure skip logic in the Spring Batch framework.

2. Our Use Case

For the purpose of examples, we'll reuse a simple, chunk-oriented job presented already in our Spring Batch introductory article.

This job converts some financial data from a CSV to XML format.

2.1. Input Data

First, let's add a few rows to the original CSV file:

username, user_id, transaction_date, transaction_amount
devendra, 1234, 31/10/2015, 10000
john, 2134, 3/12/2015, 12321
robin, 2134, 2/02/2015, 23411
, 2536, 3/10/2019, 100
mike, 9876, 5/11/2018, -500
, 3425, 10/10/2017, 9999

As we can see, the last three rows contain some invalid data – rows 5 and 7 are missing the username field, and the transaction amount in row 6 is negative.

In the later sections, we'll configure our batch job to skip these corrupted records.

3. Configuring Skip Limit and Skippable Exceptions

3.1. Using skip and skipLimit

Let's now discuss the first of two ways to configure our job to skip items in case of a failure — the skip and skipLimit methods:

@Bean
public Step skippingStep(
  ItemProcessor<Transaction, Transaction> processor,
  ItemWriter<Transaction> writer) throws ParseException {
    return stepBuilderFactory
      .get("skippingStep")
      .<Transaction, Transaction>chunk(10)
      .reader(itemReader(invalidInputCsv))
      .processor(processor)
      .writer(writer)
      .faultTolerant()
      .skipLimit(2)
      .skip(MissingUsernameException.class)
      .skip(NegativeAmountException.class)
      .build();
}

First of all, to enable skip functionality, we need to include a call to faultTolerant() during the step-building process.

Within skip() and skipLimit(), we define the exceptions we want to skip and the maximum number of skipped items.

In the above example, if either a MissingUsernameException or NegativeAmountException is thrown during read, process, or write phase, then the currently processed item will be omitted and counted against the total limit of two.

Consequently, if any exception is thrown a third time, then the whole step will fail.

3.1. Using noSkip

In the previous example, any other exception besides MissingUsernameException and NegativeAmountException makes our step fail.

In some situations, however, it may be more appropriate to identify exceptions that should make our step fail and skip on any other.

Let's see how we can configure this using skip, skipLimit, and noSkip:

@Bean
public Step skippingStep(
  ItemProcessor<Transaction, Transaction> processor,
  ItemWriter<Transaction> writer) throws ParseException {
    return stepBuilderFactory
      .get("skippingStep")
      .<Transaction, Transaction>chunk(10)
      .reader(itemReader(invalidInputCsv))
      .processor(processor)
      .writer(writer)
      .faultTolerant()
      .skipLimit(2)
      .skip(Exception.class)
      .noSkip(SAXException.class)
      .build();
}

With the above configuration, we instruct Spring Batch framework to skip on any Exception (within a configured limit) except SAXException. This means SAXException always causes a step failure.

The order of the skip() and noSkip() calls doesn't matter.

4. Using Custom SkipPolicy

Sometimes we may need a more sophisticated skip-checking mechanism. For that purpose, Spring Batch framework provides the SkipPolicy interface.

We can then provide our own implementation of skip logic and plug it into our step definition.

Keeping the preceding example in mind, imagine we still want to define a skip limit of two items and make only MissingUsernameException and NegativeAmountException skippable.

However, an additional constraint is that we can skip NegativeAmountException, but only if the amount doesn't exceed a defined limit.

Let's implement our custom SkipPolicy:

public class CustomSkipPolicy implements SkipPolicy {

    private static final int MAX_SKIP_COUNT = 2;
    private static final int INVALID_TX_AMOUNT_LIMIT = -1000;

    @Override
    public boolean shouldSkip(Throwable throwable, int skipCount) 
      throws SkipLimitExceededException {

        if (throwable instanceof MissingUsernameException && skipCount < MAX_SKIP_COUNT) {
            return true;
        }

        if (throwable instanceof NegativeAmountException && skipCount < MAX_SKIP_COUNT ) {
            NegativeAmountException ex = (NegativeAmountException) throwable;
            if(ex.getAmount() < INVALID_TX_AMOUNT_LIMIT) {
                return false;
            } else {
                return true;
            }
        }

        return false;
    }
}

Now, we can use our custom policy in a step definition:

    @Bean
    public Step skippingStep(
      ItemProcessor<Transaction, Transaction> processor,
      ItemWriter<Transaction> writer) throws ParseException {
        return stepBuilderFactory
          .get("skippingStep")
          .<Transaction, Transaction>chunk(10)
          .reader(itemReader(invalidInputCsv))
          .processor(processor)
          .writer(writer)
          .faultTolerant()
          .skipPolicy(new CustomSkipPolicy())
          .build();
    }

And, similarly to our previous example, we still need to use faultTolerant() to enable skip functionality.

This time, however, we do not call skip() or noSkip(). Instead, we use the skipPolicy() method to provide our own implementation of the SkipPolicy interface.

As we can see, this approach gives us more flexibility, so it can be a good choice in certain use cases.

5. Conclusion

In this tutorial, we presented two ways to make a Spring Batch job fault-tolerant.

Even though using a skipLimit() together with skip() and noSkip() methods seems to be more popular, we may find implementing a custom SkipPolicy to be more convenient in some situations.

As usual, all the code examples are available over on GitHub.