A Framework for JMeter using GitHub and Ruby

Version control is not something you often associate with performance testing scripts. The reasons generally vary. A significant one is the lack of any version control support in commercial toolsets like LoadRunner. Another reason is the perceived short-term shelf life of test scripts. You tend to write them once and throw them away right?

If you work with more agile teams/developers you will probably start to require a lot of re-use out of your scripts, especially when the release cycle increases in terms of frequency. It starts to make sense to use version control and some form of scripting framework.

In this post I’m going to use JMeter as the tool of choice here and I’m going to re-introduce concepts which I normally force upon others whilst contracting. That is, use of Application Simulation Models (ASMs) to hep define the load and use of the Application Performance Index (Apdex) to help present results. I’m also going to use GitHub (or git) as my tool for maintaining version control and implementing a wiki as a dashboard replacement for other test management tools.

In short, I’m hosting version controlled scripts on GitHub. You can view my repository here:
http://wiki.github.com/90kts/performance_test

I’m using Google Documents to store my ASMs. You can see an example here.

I’m also using the Wiki functionality of GitHub plus some Ruby scripts as my test runners. In other words, a scripted version of a controller, driven by data from the ASMs.

GitHub can be quite useful. Not only can you version control all your scripts, you can also implement concepts based around a “low-tech testing dashboard” first explored by James Bach. The use of a wiki to support this was recently explored by Matthew at Socialtext. Coupled with that you also get an Issues database which is perfect for tracking those performance defects during test execution. Google Docs is also very useful in that it allows you to centrally store test artifacts like your ASMs which has plenty of neat APIs for programatically accessing content. Start working more with GitHub and less with QualityCentre and watch your circle of geek friends change.

Dashboard
So my dashboard is really just a broad sweep of major system and application components that present some level of risk to application stability and performance. It’s represented in the GitHub wiki as follows:

Main elements are the areas I wish to test, the estimated effort required to script/prepare those areas, an indication of risk to production stability (carried out separately in risk assessments with appropriate technical/business teams), day to day comments on test readiness and a link to the online ASM. I use the keywords “run” or “draft” to distinguish between ASM status’. This helps the test runner determine which tests to execute.

Test Runner
The test runner is a Ruby script which when executed, queries the Dashboard for ASMs to execute (based on the keyword “run”) then fetches the appropriate ASM via Google Documents to parse out relevant volumetrics for the test. Finally, it launches thread groups within JMeter to execute the test according to the ASM. Results are stored in the corresponding logs directory.

Apdex
If you’d like to find out more about the Application Performance Index please see here. I’ve found Apdex to be a very useful format to communicate application performance consistently to non-tech-heads. There’s plenty of room to geek out at low level later, but for a 20,000′ view on things, Apdex is often a good start. I’ve implemented Apdex support for this framework using a Ruby script to parse the results. The Google chart API for Ruby is used to present sparklines. You can do this during or after a test run has complete. It basically picks up results from the logs directory and parses them against response time targets specified in the ASM. The output of the Apdex script is a html page which looks like this:

Application Simulation Models
The ASM is really the brains behind the JMeter test plan. It looks like this:

It has two worksheets. The ASM worksheet is intended for business stakeholders. It’s geared more towards their understanding. I encourage these groups to articulate load in terms of transaction distribution (how many and what types of transactions groups) and user distribution (who are your user groups and what do they consist of). The key metrics I get these groups to enter are highlighted in yellow. Things like different user groups, percentage breakdowns of transactions and transaction/concurrent user quantities. Have a play, it’s pretty easy to follow. Coincidentally in every contract I’ve worked, I make this a mandatory test artifact for both my testers and the teams they’ve engaged with. No ASM = No Performance test…

The Scenario worksheet is really just the reverse of the first. It breaks down all the metrics that the business have provided into a worksheet to help you plug figures into your test scenarios. It helps cut down user-errors when calculating transaction rates etc and makes it much easier to chop and change figures. Want to do a test run with less concurrent users or higher transaction throughput? Just muck around with the percentages in yellow. I’ve used this format for both LoadRunner and JMeter gigs, so the terminology is meant to be familiar to both those types of users.

JMeter Test Plan
Through the beauty of JMeter -J command line parameters I’m able to pass all the relevant metrics from the ASM such as number of threads (vusers), duration, throughput, rampup and action(transaction) distribution percentages. I use this template format quite often and have explained some of these concepts previously.

The test plan gets the required parameters from the command line at run time. You can see the User Defined Variables look something like this:
DURATION = ${__P(duration,60)

Which basically sets the ${DURATION} JMeter variable based on the -Jduration command line variable, or defaults to 60 seconds if it is not passed. Other variables are also defined here and get loaded into the main Thread Group.

I use a Runtime Controller to control how long the test plan executes for. I also keep a separate Transaction Controller called Init to execute login actions etc or things I only want to execute once in the test plan.

Nested inside the Runtime Controller I use Throughput Controllers with percent executions tied to action percentages defined in the ASM. For example, action 1 might be a “view product” sequence of requests which I execute 50% of the time and so forth. Similar to weighted percentages for transaction blocks in LoadRunner. It is best to calculate these percentages using the Per User mode in JMeter for more accurate results.

Nested inside Throughput Controllers will be further Transaction Controllers, with Generate Parent Sample checked to enable similar functionality to transaction timers in LoadRunner around groups of requests.

The whole Thread Group has its throughput limited with a Constant Throughput Timer which basically tries to make all active threads in the thread group stay at a constant rate (defined by samples per minute). This eliminates the need to figure out pacing or sleep times as you might in LoadRunner.

The Summary Report and Gaussian Random Timers are just used for debugging so don’t normally include those in the final script…

Git
Want to get started? Well first you will need a copy of a git client. Have a look at various how-to guides out there to get you started. I’ve found this site pretty useful for understanding and working with git basics. My public clone URL for all the code used in this post is available here:
git://github.com/90kts/performance_test.git

Enjoy!

Social tagging: jmeter > ruby