Automated Performance Tests using JMeter and Maven

Learning how to write and automate performance tests isn't easy. Perhaps that is why I have never actually wrote and automated performance tests; until now. For my latest open source side project, sass4j, I wanted to start the project with some automated performance tests. Since we were already using maven2, I wanted to find a maven2 plugin that allowed me to write complex performance tests and automate them easily in a continuous integration environment such as Hudson. Since I had heard good things about JMeter and there was an existing JMeter plugin, I decided on those technologies. Unfortunately the path this took me down was rather long and annoying, but I eventually figured everything out and that is the reason I would like to document the steps for others to reproduce in less time.

But first, why would anyone want to spend unnecessary time setting this up? The big advantage I think, is having the ability to compare nightly test results with a baseline and compare them with expectations. If my latest changes caused a major decrease in performance when I only added a few lines of code, then perhaps something is wrong. The second advantage is the sooner a performance issue is found the cheaper it costs to fix it. Think about the change set a developer has to look at if nightly performance tests were ran, compared to finding a performance issue 6 months from when the bug was introduced. With the former scenario, I only have to look at what has changed in the last 24 hours.

Now onto how to actually do this. I had two references in getting this done: The Official Apache JMeter Maven Plugin Site and a blog on AMIS by Robbrecht van Amerongen. Both of which are incomplete, but combined provide enough information.

Here is an outline of what you need to do. End to end this should take you about 15 minutes to setup; compared to the hours I spent I think you are getting a deal. Also you need to think about doing this in your artifactory server or company maven repository and not locally (doing it locally only helps you and not your entire company).

1) Download the JMeter maven bundle I created containing all the necessary artifacts
2) Install JMeter Plugin dependencies
3) Install the JMeter Plugin
4) Install JMeter
5) Update your maven project
6) Create jmx files and run mvn verify

The first 3 steps are necessary because there are specific dependencies the JMeter plugin requires that are not available on any public maven repo I can find, nor is the JMeter plugin.

1) Download the JMeter plugin bundle

• Click here to download the JMeter plugin zip file
  
• Unzip it

2) Install JMeter Plugin dependencies

• cd to the extracted folder jmeter
• run the following mvn commands to deploy the jar files locally. Obviously update the location of your local maven2 repository and keep in mind the file paths are specific to linux. Again do this once in your company's maven repository.
  

1. mvn deploy:deploy-file -DgroupId=org.apache.jmeter -DartifactId=jmeter -Dversion=2.2 -Dpackaging=jar -Dfile=jmeter-2.2.jar -DpomFile=jmeter-2.2.pom -Durl=file:///home/jlorenzen/.m2/repository/
2. mvn deploy:deploy-file -DgroupId=jcharts -DartifactId=jcharts -Dversion=0.7.5 -Dpackaging=jar -Dfile=jcharts-0.7.5.jar -Durl=file:///home/jlorenzen/.m2/repository/
3. mvn deploy:deploy-file -DgroupId=org.apache.jorphan -DartifactId=jorphan -Dversion=2.2 -Dpackaging=jar -Dfile=jorphan-2.2.jar -Durl=file:///home/jlorenzen/.m2/repository/
4. mvn deploy:deploy-file -DgroupId=org.mozilla.javascript -DartifactId=javascript -Dversion=1.0 -Dpackaging=jar -Dfile=javascript-1.0.jar -Durl=file:///home/jlorenzen/.m2/repository/
   

3) Install the JMeter Plugin

• Unzip the maven-jmeter-plugin.zip file that was included in the JMeter plugin bundle.
• cd to the maven-jmeter-plugin folder
• run: mvn install
• This will install version 1.0 of the maven-jmeter-plugin. It's important we install a release verses a snapshot because you don't want your project to depend on snapshot plugins because this could have nasty side effects for building and releasing.

4) Install JMeter
Now that you have the plugin installed you can actually start modifying your projects pom to use it. You first need to install JMeter because we will need a jmeter.properties file, some XSL files, and you are going to need it to create the .jmx files.

• Download and install JMeter

5) Update your maven project

• Under your project create the directory: src/test/jmeter and src/test/resources
• Copy the jmeter.properties file from the JMeter bin folder to src/test/jmeter.
• Update the property jmeter.save.saveservice.output_format in the jmeter.properties file from csv to xml.
• Copy the files jmeter-results-detail-report_21.xsl and jmeter-results-report_21.xsl from the JMeter extras folder to src/test/resources
• Add the following to your POMs build/plugins section
  

<build>
   <plugins>
       <plugin>
           <groupId>org.apache.jmeter</groupId>
           <artifactId>maven-jmeter-plugin</artifactId>
           <version>1.0</version>
           <executions>
               <execution>
                   <id>jmeter-tests</id>
                   <phase>verify</phase>
                   <goals>
                       <goal>jmeter</goal>
                   </goals>
                   <configuration>
                       <reportDir>${project.build.directory}/jmeter-reports</reportDir>
                   </configuration>
               </execution>
           </executions>
       </plugin>
       <plugin>
           <groupId>org.codehaus.mojo</groupId>
           <artifactId>xml-maven-plugin</artifactId>
           <version>1.0-beta-2</version>
           <executions>
               <execution>
               <phase>pre-site</phase>
               <goals>
                   <goal>transform</goal>
               </goals>
               </execution>
           </executions>
           <configuration>
               <transformationSets>
                   <transformationSet>
                       <dir>${project.build.directory}/jmeter-reports</dir>
                       <stylesheet>src/test/resources/jmeter-results-detail-report_21.xsl</stylesheet>
                       <outputDir>${project.build.directory}/site/jmeter-results</outputDir>
                       <fileMappers>
                           <fileMapper implementation="org.codehaus.plexus.components.io.filemappers.FileExtensionMapper">
                               <targetExtension>html</targetExtension>
                           </fileMapper>
                       </fileMappers>
                   </transformationSet>
               </transformationSets>
           </configuration>
       </plugin>
   </plugins>
</build>

6) Create jmx files and run mvn verify

• Now use JMeter to create your .jmx files and place them under the src/test/jmeter directory.
• run: mvn verify to execute your performance tests
• run: mvn verify pre-site to execute your performance tests and produce the test results in an HTML report.

If you want to see a real example of this click here. Now the only step I am leaving out is actually automating it which isn't the hard part luckily. All you need to do in Hudson is create a new job and execute the mvn goals mvn verify to get them automated in a CI environment.