1. Introduction
In this tutorial, we're going to compare the performance of traditional JDK collections with Eclipse Collections. We'll create different scenarios and explore the results.
2. Configuration
First, note that for this article, we'll use the default configuration to run the tests. No flags or other parameters will be set on our benchmark.
We'll use the following hardware and libraries:
- JDK 11.0.3, Java HotSpot(TM) 64-Bit Server VM, 11.0.3+12-LTS.
- MacPro 2.6GHz 6-core i7 with 16GB DDR4.
- Eclipse Collections 10.0.0 (latest available at the time of writing)
- We'll leverage JMH (Java Microbenchmark Harness) to run our benchmarks
- JMH Visualizer to generate charts from JMH results
The easiest way to create our project is via the command-line:
mvn archetype:generate \ -DinteractiveMode=false \ -DarchetypeGroupId=org.openjdk.jmh \ -DarchetypeArtifactId=jmh-java-benchmark-archetype \ -DgroupId=com.baeldung \ -DartifactId=benchmark \ -Dversion=1.0
After that, we can open the project using our favorite IDE and edit the pom.xml to add the Eclipse Collections dependencies:
<dependency>
<groupId>org.eclipse.collections</groupId>
<artifactId>eclipse-collections</artifactId>
<version>10.0.0</version>
</dependency>
<dependency>
<groupId>org.eclipse.collections</groupId>
<artifactId>eclipse-collections-api</artifactId>
<version>10.0.0</version>
</dependency>
3. First Benchmark
Our first benchmark is simple. We want to calculate the sum of a previously created List of Integers.
We'll test six different combinations while running them in serial and parallel:
private List<Integer> jdkIntList;
private MutableList<Integer> ecMutableList;
private ExecutorService executor;
private IntList ecIntList;
@Setup
public void setup() {
PrimitiveIterator.OfInt iterator = new Random(1L).ints(-10000, 10000).iterator();
ecMutableList = FastList.newWithNValues(1_000_000, iterator::nextInt);
jdkIntList = new ArrayList<>(1_000_000);
jdkIntList.addAll(ecMutableList);
ecIntList = ecMutableList.collectInt(i -> i, new IntArrayList(1_000_000));
executor = Executors.newWorkStealingPool();
}
@Benchmark
public long jdkList() {
return jdkIntList.stream().mapToLong(i -> i).sum();
}
@Benchmark
public long ecMutableList() {
return ecMutableList.sumOfInt(i -> i);
}
@Benchmark
public long jdkListParallel() {
return jdkIntList.parallelStream().mapToLong(i -> i).sum();
}
@Benchmark
public long ecMutableListParallel() {
return ecMutableList.asParallel(executor, 100_000).sumOfInt(i -> i);
}
@Benchmark
public long ecPrimitive() {
return this.ecIntList.sum();
}
@Benchmark
public long ecPrimitiveParallel() {
return this.ecIntList.primitiveParallelStream().sum();
}
To run our first benchmark we need to execute:
mvn clean install java -jar target/benchmarks.jar IntegerListSum -rf json
This will trigger the benchmark at our IntegerListSum class and save the result to a JSON file.
We'll measure the throughput or number of operations per second in our tests, so the higher the better:
Benchmark Mode Cnt Score Error Units IntegerListSum.ecMutableList thrpt 10 573.016 ± 35.865 ops/s IntegerListSum.ecMutableListParallel thrpt 10 1251.353 ± 705.196 ops/s IntegerListSum.ecPrimitive thrpt 10 4067.901 ± 258.574 ops/s IntegerListSum.ecPrimitiveParallel thrpt 10 8827.092 ± 11143.823 ops/s IntegerListSum.jdkList thrpt 10 568.696 ± 7.951 ops/s IntegerListSum.jdkListParallel thrpt 10 918.512 ± 27.487 ops/s
Accordingly to our tests, Eclipse Collections's parallel list of primitives had the highest throughput of all. Also, it was the most efficient with a performance almost 10x faster than the Java JDK running also in parallel.
Of course, a portion of that can be explained by the fact that when working with primitive lists, we don't have the cost associated with boxing and unboxing.
We can use JMH Visualizer to analyze our results. The chart below shows a better visualization:
4. Filtering
Next, we'll modify our list to get all elements that are multiple of 5. We'll reuse a big portion of our previous benchmark and a filter function:
private List<Integer> jdkIntList;
private MutableList<Integer> ecMutableList;
private IntList ecIntList;
private ExecutorService executor;
@Setup
public void setup() {
PrimitiveIterator.OfInt iterator = new Random(1L).ints(-10000, 10000).iterator();
ecMutableList = FastList.newWithNValues(1_000_000, iterator::nextInt);
jdkIntList = new ArrayList<>(1_000_000);
jdkIntList.addAll(ecMutableList);
ecIntList = ecMutableList.collectInt(i -> i, new IntArrayList(1_000_000));
executor = Executors.newWorkStealingPool();
}
@Benchmark
public List<Integer> jdkList() {
return jdkIntList.stream().filter(i -> i % 5 == 0).collect(Collectors.toList());
}
@Benchmark
public MutableList<Integer> ecMutableList() {
return ecMutableList.select(i -> i % 5 == 0);
}
@Benchmark
public List<Integer> jdkListParallel() {
return jdkIntList.parallelStream().filter(i -> i % 5 == 0).collect(Collectors.toList());
}
@Benchmark
public MutableList<Integer> ecMutableListParallel() {
return ecMutableList.asParallel(executor, 100_000).select(i -> i % 5 == 0).toList();
}
@Benchmark
public IntList ecPrimitive() {
return this.ecIntList.select(i -> i % 5 == 0);
}
@Benchmark
public IntList ecPrimitiveParallel() {
return this.ecIntList.primitiveParallelStream()
.filter(i -> i % 5 == 0)
.collect(IntLists.mutable::empty, MutableIntList::add, MutableIntList::addAll);
}
We'll execute the test just like before:
mvn clean install java -jar target/benchmarks.jar IntegerListFilter -rf json
And the results:
Benchmark Mode Cnt Score Error Units IntegerListFilter.ecMutableList thrpt 10 145.733 ± 7.000 ops/s IntegerListFilter.ecMutableListParallel thrpt 10 603.191 ± 24.799 ops/s IntegerListFilter.ecPrimitive thrpt 10 232.873 ± 8.032 ops/s IntegerListFilter.ecPrimitiveParallel thrpt 10 1029.481 ± 50.570 ops/s IntegerListFilter.jdkList thrpt 10 155.284 ± 4.562 ops/s IntegerListFilter.jdkListParallel thrpt 10 445.737 ± 23.685 ops/s
As we can see, the Eclipse Collections Primitive was the winner again. With a throughput more than 2x faster than the JDK parallel list.
Note that for filtering, the effect of parallel processing is more visible. Summing is a cheap operation for the CPU and we won't see the same differences between serial and parallel.
Also, the performance boost that Eclipse Collections primitive lists got earlier begins to evaporate as the work done on each element begins to outweigh the cost of boxing and unboxing.
To finalize, we could see that operations on primitives are faster than objects:
5. Conclusion
In this article, we created a couple of benchmarks to compare Java Collections with Eclipse Collections. We've leveraged JMH to try to minimize the environment bias.
As always, the source code is available over on GitHub.