Micro-benchmarking Framework for Jenkins Plugins
-
I have been working on improving the performance of the Role Strategy Plugin as a part of my Google Summer of Code project. Since there was no existing way to measure performance and do benchmarks on Jenkins Plugins, my work for the first phase of the project was to create a framework for running benchmarks in Jenkins plugins with a Jenkins instance available. To make our job a bit easier, we chose Java Microbenchmark Harness for running these benchmarks. This allows us to reliably measure performance of our time-critical functions and will help make Jenkins perform faster for everyone.
The micro-benchmarking framework was recently released in the Jenkins Unit Test Harness 2.50. The blog post below shows how to run benchmarks in your plugins.
Introduction
The framework runs works by starting a temporary Jenkins instance for each fork of the JMH benchmark, just like
JenkinsRule
from Jenkins Test Harness. Benchmarks are run directly from your JUnit Tests which allows you to fail builds on the fly and easily run benchmarks from your IDE, just like unit tests. You can easily configure your benchmarks by either using your Java methods, or by using Jenkins Configuration-as-Code plugin and passing the path to your YAML file.To run benchmarks from your plugins, you need to do the following:
-
bump up the minimum required Jenkins version to 2.60.3 or above
-
bump Plugin-POM to a version ≥ 3.46 or manually upgrade to Jenkins Test Harness ≥ 2.51.
Now, to run the benchmarks, you need to have a benchmark runner that contains a
@Test
so it can run like a JUnit test. From inside a test method, you can use theOptionsBuilder
provided by JMH to configure your benchmarks. For example:public class BenchmarkRunner { @Test public void runJmhBenchmarks() throws Exception { ChainedOptionsBuilder options = new OptionsBuilder() .mode(Mode.AverageTime) .forks(2) .result("jmh-report.json"); // Automatically detect benchmark classes annotated with @JmhBenchmark new BenchmarkFinder(getClass()).findBenchmarks(options); new Runner(options.build()).run(); } }
Sample benchmarks
Now, you can write your first benchmark:
Without any special setup
@JmhBenchmark public class JmhStateBenchmark { public static class MyState extends JmhBenchmarkState { } @Benchmark public void benchmark(MyState state) { // benchmark code goes here state.getJenkins().setSystemMessage("Hello world"); } }
Using Configuration as Code
To use configuration as code, apart from the dependencies above you also need to add the following to your
pom.xml
:<dependency> <groupId>io.jenkins</groupId> <artifactId>configuration-as-code</artifactId> <version>1.21</version> <optional>true</optional> </dependency> <dependency> <groupId>io.jenkins</groupId> <artifactId>configuration-as-code</artifactId> <version>1.21</version> <classifier>tests</classifier> <scope>test</scope> </dependency>
Now configuring a benchmark is as simple as providing path to your YAML file and specifying the class containing the benchmark state.
@JmhBenchmark public class SampleBenchmark { public static class MyState extends CascJmhBenchmarkState { @Nonnull @Override protected String getResourcePath() { return "config.yml"; } @Nonnull @Override protected Class<?> getEnclosingClass() { return SampleBenchmark.class; } } @Benchmark public void benchmark(MyState state) { Jenkins jenkins = state.getJenkins(); // jenkins is configured and ready to be benchmarked. // your benchmark code goes here... } }
More Samples
As a part of this project, a few benchmarks have been created in the Role Strategy Plugin which show configuring the instances for various situations. You can find them here.
Running Benchmarks
Running benchmarks from Maven
To easily run benchmarks from Maven, a Maven profile to run the benchmarks has been created and is available starting Plugin-POM version 3.45. You can then run your benchmarks from the command line using
mvn test -Dbenchmark
.Running benchmarks on ci.jenkins.io
If you have your plugins hosted on ci.jenkins.io, you can easily run benchmarks directly from your Jenkinsfile by using the
runBenchmarks()
method after thebuildPlugin()
step in your which is now available in Jenkins Pipeline library. This function also accepts the path to your generated JMH benchmark reports as an optional parameter and archives the benchmark results. Running benchmarks in pull request builds allows you to constantly monitor the performance implications of a given change. For example, the Jenkinsfile from Role Strategy Plugin:buildPlugin() runBenchmarks('jmh-report.json')
Visualizing benchmark results
Benchmark reports generated (in JSON) can be visualized using the either the JMH Report Plugin or by passing the benchmark reports to the JMH visualizer web service. As an example, here is a visualized report of some benchmarks from the Role Strategy Plugin:
These improvements seen above were obtained through a small pull request to the plugin and shows how even seemingly small changes can bring major performance improvements. Microbenchmarks help to find these hot-spots and estimate the impact of changes.
Some tips and tricks
-
Since
BenchmarkRunner
class name in the example above does not qualify as a test according to Maven surefire plugin’s naming conventions, the benchmarks will not interfere with your JUnit tests. -
Benchmark methods need to be annotated by
@Benchmark
for JMH to detect them. -
Classes containing benchmarks are found automatically by the
BenchmarkFinder
when annotated with@JmhBenchmark
. -
A reference to the Jenkins instance is available through either
JmhBenchmarkState#getJenkins()
or throughJenkins.getInstance()
like you would otherwise do. -
JmhBenchmarkState
providessetup()
andtearDown()
methods which can be overridden to configure the Jenkins instance according to your benchmark’s requirements. -
The benchmark builds on ci.jenkins.io are currently throttled because of the limited availability of
highmem
nodes. -
The benchmark framework was made available in Jenkins Test Harness 2.50, it is recommended to use version 2.51 as it includes some bug fixes.
Links and Feedback
If you have any feedback, comments or questions, please feel free to reach out to me through either the Role Strategy Plugin Gitter chat or through the Jenkins Developer Mailing list.
-
Documentation for the micro-benchmark framework:
http://feedproxy.google.com/~r/ContinuousBlog/~3/19MdMh1JjzQ/
-
© Lightnetics 2024