Automated Performance Tests using JMeter and Maven

Learning how to write and automate performance tests isn't easy. Perhaps that is why I have never actually wrote and automated performance tests; until now. For my latest open source side project, sass4j, I wanted to start the project with some automated performance tests. Since we were already using maven2, I wanted to find a maven2 plugin that allowed me to write complex performance tests and automate them easily in a continuous integration environment such as Hudson. Since I had heard good things about JMeter and there was an existing JMeter plugin, I decided on those technologies. Unfortunately the path this took me down was rather long and annoying, but I eventually figured everything out and that is the reason I would like to document the steps for others to reproduce in less time.

But first, why would anyone want to spend unnecessary time setting this up? The big advantage I think, is having the ability to compare nightly test results with a baseline and compare them with expectations. If my latest changes caused a major decrease in performance when I only added a few lines of code, then perhaps something is wrong. The second advantage is the sooner a performance issue is found the cheaper it costs to fix it. Think about the change set a developer has to look at if nightly performance tests were ran, compared to finding a performance issue 6 months from when the bug was introduced. With the former scenario, I only have to look at what has changed in the last 24 hours.

Now onto how to actually do this. I had two references in getting this done: The Official Apache JMeter Maven Plugin Site and a blog on AMIS by Robbrecht van Amerongen. Both of which are incomplete, but combined provide enough information.

Here is an outline of what you need to do. End to end this should take you about 15 minutes to setup; compared to the hours I spent I think you are getting a deal. Also you need to think about doing this in your artifactory server or company maven repository and not locally (doing it locally only helps you and not your entire company).

1) Download the JMeter maven bundle I created containing all the necessary artifacts
2) Install JMeter Plugin dependencies
3) Install the JMeter Plugin
4) Install JMeter
5) Update your maven project
6) Create jmx files and run mvn verify

The first 3 steps are necessary because there are specific dependencies the JMeter plugin requires that are not available on any public maven repo I can find, nor is the JMeter plugin.

1) Download the JMeter plugin bundle

• Click here to download the JMeter plugin zip file
  
• Unzip it

2) Install JMeter Plugin dependencies

• cd to the extracted folder jmeter
• run the following mvn commands to deploy the jar files locally. Obviously update the location of your local maven2 repository and keep in mind the file paths are specific to linux. Again do this once in your company's maven repository.
  

1. mvn deploy:deploy-file -DgroupId=org.apache.jmeter -DartifactId=jmeter -Dversion=2.2 -Dpackaging=jar -Dfile=jmeter-2.2.jar -DpomFile=jmeter-2.2.pom -Durl=file:///home/jlorenzen/.m2/repository/
2. mvn deploy:deploy-file -DgroupId=jcharts -DartifactId=jcharts -Dversion=0.7.5 -Dpackaging=jar -Dfile=jcharts-0.7.5.jar -Durl=file:///home/jlorenzen/.m2/repository/
3. mvn deploy:deploy-file -DgroupId=org.apache.jorphan -DartifactId=jorphan -Dversion=2.2 -Dpackaging=jar -Dfile=jorphan-2.2.jar -Durl=file:///home/jlorenzen/.m2/repository/
4. mvn deploy:deploy-file -DgroupId=org.mozilla.javascript -DartifactId=javascript -Dversion=1.0 -Dpackaging=jar -Dfile=javascript-1.0.jar -Durl=file:///home/jlorenzen/.m2/repository/
   

3) Install the JMeter Plugin

• Unzip the maven-jmeter-plugin.zip file that was included in the JMeter plugin bundle.
• cd to the maven-jmeter-plugin folder
• run: mvn install
• This will install version 1.0 of the maven-jmeter-plugin. It's important we install a release verses a snapshot because you don't want your project to depend on snapshot plugins because this could have nasty side effects for building and releasing.

4) Install JMeter
Now that you have the plugin installed you can actually start modifying your projects pom to use it. You first need to install JMeter because we will need a jmeter.properties file, some XSL files, and you are going to need it to create the .jmx files.

• Download and install JMeter

5) Update your maven project

• Under your project create the directory: src/test/jmeter and src/test/resources
• Copy the jmeter.properties file from the JMeter bin folder to src/test/jmeter.
• Update the property jmeter.save.saveservice.output_format in the jmeter.properties file from csv to xml.
• Copy the files jmeter-results-detail-report_21.xsl and jmeter-results-report_21.xsl from the JMeter extras folder to src/test/resources
• Add the following to your POMs build/plugins section
  

<build>
   <plugins>
       <plugin>
           <groupId>org.apache.jmeter</groupId>
           <artifactId>maven-jmeter-plugin</artifactId>
           <version>1.0</version>
           <executions>
               <execution>
                   <id>jmeter-tests</id>
                   <phase>verify</phase>
                   <goals>
                       <goal>jmeter</goal>
                   </goals>
                   <configuration>
                       <reportDir>${project.build.directory}/jmeter-reports</reportDir>
                   </configuration>
               </execution>
           </executions>
       </plugin>
       <plugin>
           <groupId>org.codehaus.mojo</groupId>
           <artifactId>xml-maven-plugin</artifactId>
           <version>1.0-beta-2</version>
           <executions>
               <execution>
               <phase>pre-site</phase>
               <goals>
                   <goal>transform</goal>
               </goals>
               </execution>
           </executions>
           <configuration>
               <transformationSets>
                   <transformationSet>
                       <dir>${project.build.directory}/jmeter-reports</dir>
                       <stylesheet>src/test/resources/jmeter-results-detail-report_21.xsl</stylesheet>
                       <outputDir>${project.build.directory}/site/jmeter-results</outputDir>
                       <fileMappers>
                           <fileMapper implementation="org.codehaus.plexus.components.io.filemappers.FileExtensionMapper">
                               <targetExtension>html</targetExtension>
                           </fileMapper>
                       </fileMappers>
                   </transformationSet>
               </transformationSets>
           </configuration>
       </plugin>
   </plugins>
</build>

6) Create jmx files and run mvn verify

• Now use JMeter to create your .jmx files and place them under the src/test/jmeter directory.
• run: mvn verify to execute your performance tests
• run: mvn verify pre-site to execute your performance tests and produce the test results in an HTML report.

If you want to see a real example of this click here. Now the only step I am leaving out is actually automating it which isn't the hard part luckily. All you need to do in Hudson is create a new job and execute the mvn goals mvn verify to get them automated in a CI environment.

Always specify a version for Maven2 plugins

For the past several months my team has been suffering a lot of downtime because our local builds fail and our Continuous Integration environment fails for what appears to be no reason. What makes these issues difficult to debug is what might fail on your machine, won't fail on mine ("works on my machine"). We eventually found the cause and that was we were not specifying a version for plugins in our maven2 poms. Pretty dumb, eh?

Wrong way

<plugin>
 <groupid>org.apache.maven.plugins</groupid>
 <artifactid>maven-surefire-plugin</artifactid>
</plugin>

Correct way

<plugin>
 <groupid>org.apache.maven.plugins</groupid>
 <artifactid>maven-surefire-plugin</artifactid>
 <version>2.3</version>
</plugin>

This matters because if you leave out the version, maven2 defaults to SNAPSHOT. So what was happening is my project unknowingly became automatic beta testers for third party plugins.

So the moral of the story is always include a version (preferably a stable release version like 2.3 and not 2.3-SNAPSHOT) when defining your plugins. This will go a long ways in making your build more stable. And better yet, define those plugins in your parent pom using the <pluginManagement> section. This keeps your versions in one place so that when you do upgrade intentionally to a newer version you only have to modify it in one place for your entire project. For example:

<pluginmanagement>
  <plugins>
      <plugin>
            <groupid>org.apache.maven.plugins</groupid>
            <artifactid>maven-surefire-plugin</artifactid>
            <version>2.3</version>
        </plugin>
  <plugins>
</plugins>

Specifically the maven-surefire-plugin and maven-war-plugin have cost us a lot already, so recently I updated all our poms to define a version for all the plugins we were using. And one reason I know it would fail for some, but not others, is some prefer to build in offline mode (-o) always which prevented them from seeing the problems.

Avoid installing 2 versions of Netbeans 6

Tonight I installed the latest OpenESB software (Build 20080303) and was rather confused as a lot has changed since I last used it back in October 2007. I found myself being rather unproductive so I decided to revert to an earlier version. So I downloaded Build 20080214 and installed it, but I did not uninstall the previous version. With the older version I repeatedly received exceptions on just about everything I clicked on. Many of my co-workers battle these issues everyday, but now I might have figured out why. For some reason, Netbeans just doesn't like being installed twice on the same system under the same folder (for me was /workspace/java/openesb). Under there I had 2 netbeans and 2 glassfish folders. Once I uninstalled everything and reinstalled the older version (Build 20080214) it worked perfectly.

So when you are receiving all those nasty exceptions every 5 seconds in Netbeans, make sure you only have one installed.

Challenges using Icefaces

I would just like to take a quick moment and inform anyone about Icefaces, and the challenges I have faced using it. Take it from my current experience (version 1.6.1 and 1.6.2) and the words of an Icefaces developer that since Icefaces "is a stateful technology......providing enhanced features for the user....but require server-side resources to do so." That last statement has been our recent struggle.

Unfortunately, on my current project we don't have the luxury of lots of server-side resources. It's disappointing to say we have to share a single server that is already running IIS; then we have to run Jboss and MS SQL on the same machine. Obviously this is not by choice and doesn't look to change in the near future.

It might be Icefaces never intended their framework to run in such a limited environment. But that really doesn't help me now. It would be great if they published information about minimum hardware requirements, especially if they are aware of performance issues. Equally important I am sure we have strayed away from best practices in using Icefaces.

Please don't ask me why we choose to use Icefaces in the first place (I wasn't with the group at the time). But I assume it was a lack of knowledge of how Icefaces works, and we weren't aware of the environment limitations. Either way, we are facing difficult challenges that are very frustrating.

For example, we recently had severe problems with an editable table which was sortable (much like an excel spreadsheet). When users clicked twice on the header to sort, the data appeared to be sent twice and some how our data got corrupted. Consequently we had to disable the ability to sort.

Also, based on my understanding of Icefaces, it appears that all clients maintain a constant connection with the server. I am assuming this is what the Icefaces developer above was referring to when he says Icefaces is a stateful technology. For some reason, I feel this constant connection would cause challenges for 50-100+ concurrent clients with limited server-side resources.

Hope it helps someone.

How to simulate onblur event for a Panel in Extjs

Here is a simple trick in Extjs I borrowed from Ext.layout.BorderLayout that simulates listening for the onblur event of an Ext.Panel. This trick is necessary because unfortunately Ext.Panel does not natively support registering for the onblur event (or at least I don't know how). We typically use this trick when we show a Panel and want to remove it when the user clicks off, or to state it differently, when the Panel looses focus. This is very similar to how menus work.

Step 1: Register for mousedown event

this.panel.show();
Ext.getDoc().on("mousedown", this.handleDocMouseDown, this);

First, we show the panel. Here it is assumed this was created previously. Second, we use Ext.getDoc() to return the current HTML document object, and then we register for the mousedown event. The second parameter, this.handleDocMouseDown, is our function we want called when this event is fired. Notice that we register for the mousedown event as opposed to the click event. This is intentional because listening for the click event will actually call this.handleDocMouseDown before your ready because a click involves both mousedown and mouseup.

Step 2: Handle the event

handleDocMouseDown : function(e) {
if (!e.within(this.panel.getEl())) {
   this.panel.destroy();
   Ext.getDoc().un("mousedown", this.handleDocMouseDown, this);
}
}

In our second step we first check to make sure the user didn't click anywhere in the Panel that was just displayed. If it wasn't then we destroy the panel and unregister the mousedown event.