Friday, March 21, 2008

Automated Performance Tests using JMeter and Maven

Learning how to write and automate performance tests isn't easy. Perhaps that is why I have never actually wrote and automated performance tests; until now. For my latest open source side project, sass4j, I wanted to start the project with some automated performance tests. Since we were already using maven2, I wanted to find a maven2 plugin that allowed me to write complex performance tests and automate them easily in a continuous integration environment such as Hudson. Since I had heard good things about JMeter and there was an existing JMeter plugin, I decided on those technologies. Unfortunately the path this took me down was rather long and annoying, but I eventually figured everything out and that is the reason I would like to document the steps for others to reproduce in less time.

But first, why would anyone want to spend unnecessary time setting this up? The big advantage I think, is having the ability to compare nightly test results with a baseline and compare them with expectations. If my latest changes caused a major decrease in performance when I only added a few lines of code, then perhaps something is wrong. The second advantage is the sooner a performance issue is found the cheaper it costs to fix it. Think about the change set a developer has to look at if nightly performance tests were ran, compared to finding a performance issue 6 months from when the bug was introduced. With the former scenario, I only have to look at what has changed in the last 24 hours.

Now onto how to actually do this. I had two references in getting this done: The Official Apache JMeter Maven Plugin Site and a blog on AMIS by Robbrecht van Amerongen. Both of which are incomplete, but combined provide enough information.

Here is an outline of what you need to do. End to end this should take you about 15 minutes to setup; compared to the hours I spent I think you are getting a deal. Also you need to think about doing this in your artifactory server or company maven repository and not locally (doing it locally only helps you and not your entire company).

1) Download the JMeter maven bundle I created containing all the necessary artifacts
2) Install JMeter Plugin dependencies
3) Install the JMeter Plugin
4) Install JMeter
5) Update your maven project
6) Create jmx files and run mvn verify

The first 3 steps are necessary because there are specific dependencies the JMeter plugin requires that are not available on any public maven repo I can find, nor is the JMeter plugin.

1) Download the JMeter plugin bundle

  • Click here to download the JMeter plugin zip file
  • Unzip it
2) Install JMeter Plugin dependencies
  • cd to the extracted folder jmeter
  • run the following mvn commands to deploy the jar files locally. Obviously update the location of your local maven2 repository and keep in mind the file paths are specific to linux. Again do this once in your company's maven repository.
  1. mvn deploy:deploy-file -DgroupId=org.apache.jmeter -DartifactId=jmeter -Dversion=2.2 -Dpackaging=jar -Dfile=jmeter-2.2.jar -DpomFile=jmeter-2.2.pom -Durl=file:///home/jlorenzen/.m2/repository/
  2. mvn deploy:deploy-file -DgroupId=jcharts -DartifactId=jcharts -Dversion=0.7.5 -Dpackaging=jar -Dfile=jcharts-0.7.5.jar -Durl=file:///home/jlorenzen/.m2/repository/
  3. mvn deploy:deploy-file -DgroupId=org.apache.jorphan -DartifactId=jorphan -Dversion=2.2 -Dpackaging=jar -Dfile=jorphan-2.2.jar -Durl=file:///home/jlorenzen/.m2/repository/
  4. mvn deploy:deploy-file -DgroupId=org.mozilla.javascript -DartifactId=javascript -Dversion=1.0 -Dpackaging=jar -Dfile=javascript-1.0.jar -Durl=file:///home/jlorenzen/.m2/repository/
3) Install the JMeter Plugin
  • Unzip the maven-jmeter-plugin.zip file that was included in the JMeter plugin bundle.
  • cd to the maven-jmeter-plugin folder
  • run: mvn install
  • This will install version 1.0 of the maven-jmeter-plugin. It's important we install a release verses a snapshot because you don't want your project to depend on snapshot plugins because this could have nasty side effects for building and releasing.
4) Install JMeter
Now that you have the plugin installed you can actually start modifying your projects pom to use it. You first need to install JMeter because we will need a jmeter.properties file, some XSL files, and you are going to need it to create the .jmx files.
5) Update your maven project
  • Under your project create the directory: src/test/jmeter and src/test/resources
  • Copy the jmeter.properties file from the JMeter bin folder to src/test/jmeter.
  • Update the property jmeter.save.saveservice.output_format in the jmeter.properties file from csv to xml.
  • Copy the files jmeter-results-detail-report_21.xsl and jmeter-results-report_21.xsl from the JMeter extras folder to src/test/resources
  • Add the following to your POMs build/plugins section
<build>
<plugins>
<plugin>
<groupId>org.apache.jmeter</groupId>
<artifactId>maven-jmeter-plugin</artifactId>
<version>1.0</version>
<executions>
<execution>
<id>jmeter-tests</id>
<phase>verify</phase>
<goals>
<goal>jmeter</goal>
</goals>
<configuration>
<reportDir>${project.build.directory}/jmeter-reports</reportDir>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>xml-maven-plugin</artifactId>
<version>1.0-beta-2</version>
<executions>
<execution>
<phase>pre-site</phase>
<goals>
<goal>transform</goal>
</goals>
</execution>
</executions>
<configuration>
<transformationSets>
<transformationSet>
<dir>${project.build.directory}/jmeter-reports</dir>
<stylesheet>src/test/resources/jmeter-results-detail-report_21.xsl</stylesheet>
<outputDir>${project.build.directory}/site/jmeter-results</outputDir>
<fileMappers>
<fileMapper implementation="org.codehaus.plexus.components.io.filemappers.FileExtensionMapper">
<targetExtension>html</targetExtension>
</fileMapper>
</fileMappers>
</transformationSet>
</transformationSets>
</configuration>
</plugin>
</plugins>
</build>
6) Create jmx files and run mvn verify
  • Now use JMeter to create your .jmx files and place them under the src/test/jmeter directory.
  • run: mvn verify to execute your performance tests
  • run: mvn verify pre-site to execute your performance tests and produce the test results in an HTML report.

If you want to see a real example of this click here. Now the only step I am leaving out is actually automating it which isn't the hard part luckily. All you need to do in Hudson is create a new job and execute the mvn goals mvn verify to get them automated in a CI environment.

37 comments:

Tamoor Abbasi said...

Cool...
I am wondering how would the jmeter test results would be displayed in Hudson... I've seen some cool findbug and cobertura reports in Hudson. Can you please post a snapshot of jmeter test results in hudson.

jlorenzen said...

Sorry, but I haven't done that part of it. It wouldn't be too hard I imagine since other plugins are doing it.

federico wachs said...

hi there, i have followed your tutorial and everything worked out just fine.
the bad thing is that when i run mvn verify or something (doesnt matter what i run), i get this error:
Can't read log file
Embedded error: target/jmeter/jmeter.log (No such file or directory).

and the build fails, i tried to modify the jmeter.properties so it wont use that file but it keeps failing with the same error.

do you know what can i do?

federico wachs said...

well i found out what the problem was.
the jmeter.properties file, comes with the jmeter.log un-commented, all i have to do was comment it and everything worked out just fine.
thanks james for answering my mails

bye bye

Tom said...

I have been working with the JMeter Maven plugin for a recent project and have made some enhancements to it. For the project we wanted to run testNG tests by way of JMeter, as a part of our maven build process. To do that, we enhanced the way the JMeterMojo handles class loading, and wrote a custom JMeter/TestNG integration as well.

I'd like to contribute these changes back to... well someone. The JMeter project will likely accept the TestNG integration, but will not the new Maven mojo class since the original is not part of the JMeter project. In fact I can't find anywhere that is home for the JMeterMojo, outside of the wiki page.

I've been trying to locate the original contributor of the code (I think it's Tim McCune based on the jMeter wiki page), but with no luck so far.

Any ideas?

jlorenzen said...

That is awesome. I love open source.
I think you have a couple of options, if you haven't done so already.

First, try and email any jmeter or maven2 mailing lists. Perhaps someone on those mailing lists will be able to help you out.

Outside of that, I think you have exhausted your options. I think its pretty obvious the jmeter maven plugin isn't your average open source project. Therefore, I think you would be justified in maybe open sourcing your plugin; even though I personally hate this, having two options to decided between and I believe it is frowned upon. Besides that you don't have any other choices.

If I remember correctly, the jmeter maven plugin was like only 1 class.

So here is what I would do if I were you. To ensure people use your plugin verses the original, make sure your maven mojo includes all the functionality of the original (and perhaps more). Once your plugin supports everything the original maven plugin does, plus your TestNG stuff, find a place to put it. I would recommend codehuas.org. There are alot of maven plugins out there and it would be the top one I would recommend. If not that one then googlecode or java.net. What you want are mailing lists, issue tracker, and a SCM, so your project doesn't suffer from the same mistakes of the original plugin.

Once that is all done, create your first 1.0 release and provide a download link with some examples. In the description of your project you can indicate the reasons why you created a second maven jmeter plugin.

Hope it helps and good luck.

developer said...

Hello. Thanks for very useful article. But I have some problems:
Error in NonGUIDriver com.thoughtworks.xstream.converters.ConversionException: null
---- Debugging information ----
cause-exception : java.lang.reflect.UndeclaredThrowableException
cause-message : null
class : org.apache.jmeter.save.ScriptWrapper
required-type : org.apache.jmeter.save.ScriptWrapper
path : /org.apache.jmeter.save.ScriptWrapper/version
line number : 2

Version of my JMeter is 2.3 and jmx files differ from your example. Can you help me, please?

Anonymous said...

Hi

Cool tutorial - it works, but I have one problem the jmeter test do not terminate after "... end of run"

Any one know why?

/Peter

Anonymous said...

Hi

Fix to the hanging problem:

see:

http://wiki.apache.org/jakarta-jmeter/JMeterMavenPlugin

BTW: In my setting the transformation makes an extra entry in the table (equals to the first entry) - adding xalan 2.4.1 to the transformation plugins dependencies fix this problem.

/Peter

Ryan Romanchuk said...

thanks for this post!

Michael said...

Nice tutorial, it helped me a lot. The only thing that isn't quite working are the xsl docs particularly the detailed one. the min and max values come out as NaN. I presume its because of some trnsfo9rmation in the style sheet, but I cannot see it. Does anyone else get this?

Brian Matthews said...

If I hadn't found this post I would probably have given up on using JMeter from Maven.

gvk said...

When i am using maven jmeter plugin for Java Request instead Http request i am getting below error.
Can any one have any idea plz..


Exception creating: jmeter.SampleClient java.lang.ClassCastException: jmeter.SampleClient cannot be cast to org.apache.jmeter.protocol.java.sampler.JavaSamplerClient
at org.apache.jmeter.protocol.java.sampler.JavaSampler.createJavaClient(JavaSampler.java:180)
at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:159)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:247)
at java.lang.Thread.run(Thread.java:619)

Martin Cleaver said...

If you are automating jmeter tests and comparing test runs, are you comparing stddev between runs?

Thanks, Martin.

Omkaram said...

I am getting following exception when I am trying to run Jmeter test from Maven 2.1
I have followed all the steps from http://jlorenzen.blogspot.com/2008_03_01_archive.html
Can you please suggest what could be the problem?

Thanks
Omkar


java.lang.NullPointerException
at org.apache.jmeter.JMeterMojo.createSaveServiceProps(JMeterMojo.java:164)
at org.apache.jmeter.JMeterMojo.initSystemProps(JMeterMojo.java:127)
at org.apache.jmeter.JMeterMojo.execute(JMeterMojo.java:89)
at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:579)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:498)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegmentForProject(DefaultLifecycleExecutor.java:265)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:191)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:149)
at org.apache.maven.DefaultMaven.execute_aroundBody0(DefaultMaven.java:223)
at org.apache.maven.DefaultMaven.execute_aroundBody1$advice(DefaultMaven.java:304)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:1)
at org.apache.maven.embedder.MavenEmbedder.execute_aroundBody2(MavenEmbedder.java:904)
at org.apache.maven.embedder.MavenEmbedder.execute_aroundBody3$advice(MavenEmbedder.java:304)
at org.apache.maven.embedder.MavenEmbedder.execute(MavenEmbedder.java:1)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:176)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:408)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:351)
at org.codehaus.classworlds.Launcher.main(Launcher.java:31)

jlorenzen said...

@Omkaram
I would take a look in JMeterMojo.java around line 164 and find out what is throwing the NPE.

edovale said...

Folks,
I am running into an issue where a parser for html is not being found. Here is the stack trace:


2010/03/22 12:53:05 ERROR - jmeter.threads.JMeterThread: Test failed! org.apache.jmeter.protocol.http.parser.HTMLParseError
at org.apache.jmeter.protocol.http.parser.HTMLParser.getParser(HTMLParser.java:98)
at org.apache.jmeter.protocol.http.parser.HTMLParser.getParser(HTMLParser.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.downloadPageResources(HTTPSamplerBase.java:715)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.resultProcessing(HTTPSamplerBase.java:946)
at org.apache.jmeter.protocol.http.sampler.HTTPSampler.sample(HTTPSampler.java:474)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:658)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:647)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:247)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.ClassNotFoundException: org.apache.jmeter.protocol.http.parser.HtmlParserHTMLParser
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
at org.codehaus.classworlds.RealmClassLoader.loadClassDirect(RealmClassLoader.java:195)
at org.codehaus.classworlds.DefaultClassRealm.loadClass(DefaultClassRealm.java:255)
at org.codehaus.classworlds.DefaultClassRealm.loadClass(DefaultClassRealm.java:274)
at org.codehaus.classworlds.DefaultClassRealm.loadClass(DefaultClassRealm.java:274)
at org.codehaus.classworlds.RealmClassLoader.loadClass(RealmClassLoader.java:214)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.apache.jmeter.protocol.http.parser.HTMLParser.getParser(HTMLParser.java:87)
... 8 more



Have you seen this before?? How did you address it?? I looked at the lib folder for jmeter-2.2 and it has a few jars that called htmlparser*** and my thery is that these are not in maven's classpath.

jlorenzen said...

I'm not exactly sure how to automate it, but ApacheBench or ab would also be another great performance testing tool to use. I am sure others have figured out how to automate it.

hbjastad said...

I have tried to follow every step in detail, but I still get the following error when running the tests:

---- Debugging information ----
message : WebServiceSampler : WebServiceSampler
line number : 85
path : /jmeterTestPlan/hashTree/hashTree/hashTree[4]/WebServiceSampler
cause-message : WebServiceSampler : WebServiceSampler
class : org.apache.jmeter.save.ScriptWrapper
cause-exception : com.thoughtworks.xstream.alias.CannotResolveClassException
required-type : org.apache.jorphan.collections.ListedHashTree

Obviously, the jorphan jar is not included in the classpath. Do you know how it is supposed to get included? Since you deploy it to the repository, I'd expect it to be needed....

Searching on the net only provides a few others with the same problem, but no solution:
http://www.myeclipseide.com/PNphpBB2-viewtopic-t-24417.html
http://old.nabble.com/Re:-JMeter-Ant-Newbie-Question-p23223467.html

jlorenzen said...

@hbjastad
Maybe try adding it as a dependency in the projects POM? Other than that maybe trying running mvn dependency:tree to see what dependencies are being pulled in; I'd expect to see jorphan in the output.

hbjastad said...

Yeah...I wish it was that easy...

I tried adding jorphan to the project's dependencies, and I tried adding it as a dependency to maven-jmeter-plugin. The result is the same.

mvn dependency:tree does NOT include jorphan (unless I put it in as a dependency for the project).

But at least I got one step further: When I use the sass4j jmx file, I don't get this error. It's only when using a WebserviceSampler in the jmx file, that the problem occurs. It doesn't bring me any closer to get my tests working, but it is at least good to know that for HTTPSampler, the whole config setup is working :)

hbjastad said...

OK, just in case someone else runs into the same problem...here is the solution: While not apparent from the error message, it was caused by soap.jar not being part of the classpath. So it was solved by adding the following dependency to maven-jmeter-plugin's POM:


<dependency>
<groupId>soap</groupId>
<artifactId>soap</artifactId>
<version>2.3.1</version>
</dependency>

Tarun K said...

Many many many thanks for ur post, despite being novice to Maven I could easily follow ur post

~ T

Angélica Jimenez said...

Hello,

I’ve done the steps described there, it’s fine. It tries to execute the jmx file, but I receive this error:

Error in NonGUIDriver java.lang.NullPointerException
[ERROR] BUILD FAILURE
And in generated jmeter.log files it says:
2010/01/19 16:42:45 ERROR – jmeter.JMeter: java.lang.NullPointerException
at org.apache.jmeter.gui.tree.JMeterTreeModel.addSubTree(JMeterTreeModel.java:91)
at org.apache.jmeter.JMeter.run(JMeter.java:728)...


Could someone explain me about “Error in NonGUIDriver java.lang.NullPointerException.”

Thanks for the help.

Henri said...

About Error in NonGUIDriver java.lang.NullPointerException
[ERROR] BUILD FAILURE

You probably miss commons-logging in pom.xml.
See here http://blog.hgomez.net/?p=592

Anonymous said...

.....
</transformationSets>
</configuration>
<dependencies>
<dependency>
<groupId>xalan</groupId>
<artifactId>xalan</artifactId>
<version>2.7.1</version>
</dependency>
</dependencies>
</plugin>

Michael said...

I am interested in learning how I can create the jmeter jar for *any* version of jmeter. Does anyone know how the jmeter-2.2.jar file was created? If not, does anyone know who I can contact that created the original?

jlorenzen said...

@Michael
To get any version of the jmeter.jar I think all I did was download jmeter itself and look in its lib directory. It should contain a jmeter.jar. Just rename it to include the version so people don't get confused about which version it is.

Anonymous said...

when the jmeter test are run maven still indicate BUILD SUCCESSFUL although there are some errors.

Is there a parameter to make the test fail on jmeter error

Anonymous said...

hello
I have a problem when I am trying to run my tests with jmeter in maven. I get the following error:
[INFO] BUILD FAILURE
[ERROR] Failed to execute goal org.apache.jmeter:maven-jmeter-plugin:1.0:jmeter
(jmeter-tests) on project testing: There were test errors -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal o
rg.apache.jmeter:maven-jmeter-plugin:1.0:jmeter (jmeter-tests) on project testing: There were test errors
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
.java:213)

I have to say that the tests are executed correctly, the report is created but I still get the same error..

Ardesco said...

The jmeter-maven plugin is in active development again and is now available from the central maven repository so you no longer need to specify a repository in your POM to use it.

Note that the GroupID and ArtifactID have changed:

GroupID - com.lazerycode.jmeter
ArtifactID - jmeter-maven-plugin

Project website is http://jmeter.lazerycode.com

Anonymous said...

Could any one please give me the correct link to download jmeter plugin bundle.zip mentioned at top of this page "1) Download the JMeter plugin bundle"..
As currently this link is broken. If you have in your local machine please feel free to forward it to my email, which is virendra82@hotmail.com

I really need to configured this jmeter - maven as soon as possible...as i have lots of pressure to finish up this project..

jlorenzen said...

@virendra82@hotmail.com
Sorry. I forgot that they moved java.net projects and they must not have migrated that project over as it was dormant.
It shouldn't be hard to find the necessary files though. Most of the dependencies you'll need I think come from the lib directory after you install jmeter. And you can download the jmeter maven plugin here: https://github.com/Ronnie76er/jmeter-maven-plugin

Anonymous said...

Thanks @jlorenzen .....

I am just stuck..and not able to even move a single step...as i have just started doing performance testing using Jmeter... and manager wants me to integrate with maven...i thought your .zip file would be very helpful...as you also mentioned all the steps on it... no other blog got info as yours...

Dzmitry said...

There is one more method of integrating Jmeter into building process.
In this case we do not use local JMeter instance, but upload *.jmx file to jmeter cloud, than add jenkins plugin to our Jenkins server.

Anonymous said...

I am trying to run a .jmx test using this guide but I run into the following problems:

1) I cannot download the .zip file (could be an issue with the company firewall) so I am using the JMeter Maven plugin from as downloaded from here: https://github.com/Ronnie76er/jmeter-maven-plugin

I cannot find a global.properties file, which I imagine is not a big deal

2) I am using JMeter to run a load test against a database. I can run the test both from the JMeter GUI and the command line. When I run it through Maven I get the error that it cannot find the jdbc driver. I read that JMeter is looking by default at the .jar files within /lib and /lib/ext. What am I missing?


Many thanks,
Aris

Innovapath said...

Excellent Post. Also visit http://whiteboxqa.com/#performance.php