If you [...]]]> Version control is not something you often associate with performance testing scripts. The reasons generally vary. A significant one is the lack of any version control support in commercial toolsets like LoadRunner. Another reason is the perceived short-term shelf life of test scripts. You tend to write them once and throw them away right?

If you work with more agile teams/developers you will probably start to require a lot of re-use out of your scripts, especially when the release cycle increases in terms of frequency. It starts to make sense to use version control and some form of scripting framework.

In this post I’m going to use JMeter as the tool of choice here and I’m going to re-introduce concepts which I normally force upon others whilst contracting. That is, use of Application Simulation Models (ASMs) to hep define the load and use of the Application Performance Index (Apdex) to help present results. I’m also going to use GitHub (or git) as my tool for maintaining version control and implementing a wiki as a dashboard replacement for other test management tools.

In short, I’m hosting version controlled scripts on GitHub. You can view my repository here:
http://wiki.github.com/90kts/performance_test

I’m using Google Documents to store my ASMs. You can see an example here.

I’m also using the Wiki functionality of GitHub plus some Ruby scripts as my test runners. In other words, a scripted version of a controller, driven by data from the ASMs.

GitHub can be quite useful. Not only can you version control all your scripts, you can also implement concepts based around a “low-tech testing dashboard” first explored by James Bach. The use of a wiki to support this was recently explored by Matthew at Socialtext. Coupled with that you also get an Issues database which is perfect for tracking those performance defects during test execution. Google Docs is also very useful in that it allows you to centrally store test artifacts like your ASMs which has plenty of neat APIs for programatically accessing content. Start working more with GitHub and less with QualityCentre and watch your circle of geek friends change.

Dashboard
So my dashboard is really just a broad sweep of major system and application components that present some level of risk to application stability and performance. It’s represented in the GitHub wiki as follows:

Main elements are the areas I wish to test, the estimated effort required to script/prepare those areas, an indication of risk to production stability (carried out separately in risk assessments with appropriate technical/business teams), day to day comments on test readiness and a link to the online ASM. I use the keywords “run” or “draft” to distinguish between ASM status’. This helps the test runner determine which tests to execute.

Test Runner
The test runner is a Ruby script which when executed, queries the Dashboard for ASMs to execute (based on the keyword “run”) then fetches the appropriate ASM via Google Documents to parse out relevant volumetrics for the test. Finally, it launches thread groups within JMeter to execute the test according to the ASM. Results are stored in the corresponding logs directory.

Apdex
If you’d like to find out more about the Application Performance Index please see here. I’ve found Apdex to be a very useful format to communicate application performance consistently to non-tech-heads. There’s plenty of room to geek out at low level later, but for a 20,000′ view on things, Apdex is often a good start. I’ve implemented Apdex support for this framework using a Ruby script to parse the results. The Google chart API for Ruby is used to present sparklines. You can do this during or after a test run has complete. It basically picks up results from the logs directory and parses them against response time targets specified in the ASM. The output of the Apdex script is a html page which looks like this:

Application Simulation Models
The ASM is really the brains behind the JMeter test plan. It looks like this:

It has two worksheets. The ASM worksheet is intended for business stakeholders. It’s geared more towards their understanding. I encourage these groups to articulate load in terms of transaction distribution (how many and what types of transactions groups) and user distribution (who are your user groups and what do they consist of). The key metrics I get these groups to enter are highlighted in yellow. Things like different user groups, percentage breakdowns of transactions and transaction/concurrent user quantities. Have a play, it’s pretty easy to follow. Coincidentally in every contract I’ve worked, I make this a mandatory test artifact for both my testers and the teams they’ve engaged with. No ASM = No Performance test…

The Scenario worksheet is really just the reverse of the first. It breaks down all the metrics that the business have provided into a worksheet to help you plug figures into your test scenarios. It helps cut down user-errors when calculating transaction rates etc and makes it much easier to chop and change figures. Want to do a test run with less concurrent users or higher transaction throughput? Just muck around with the percentages in yellow. I’ve used this format for both LoadRunner and JMeter gigs, so the terminology is meant to be familiar to both those types of users.

JMeter Test Plan
Through the beauty of JMeter -J command line parameters I’m able to pass all the relevant metrics from the ASM such as number of threads (vusers), duration, throughput, rampup and action(transaction) distribution percentages. I use this template format quite often and have explained some of these concepts previously.

The test plan gets the required parameters from the command line at run time. You can see the User Defined Variables look something like this:
DURATION = ${__P(duration,60)

Which basically sets the ${DURATION} JMeter variable based on the -Jduration command line variable, or defaults to 60 seconds if it is not passed. Other variables are also defined here and get loaded into the main Thread Group.

I use a Runtime Controller to control how long the test plan executes for. I also keep a separate Transaction Controller called Init to execute login actions etc or things I only want to execute once in the test plan.

Nested inside the Runtime Controller I use Throughput Controllers with percent executions tied to action percentages defined in the ASM. For example, action 1 might be a “view product” sequence of requests which I execute 50% of the time and so forth. Similar to weighted percentages for transaction blocks in LoadRunner. It is best to calculate these percentages using the Per User mode in JMeter for more accurate results.

Nested inside Throughput Controllers will be further Transaction Controllers, with Generate Parent Sample checked to enable similar functionality to transaction timers in LoadRunner around groups of requests.

The whole Thread Group has its throughput limited with a Constant Throughput Timer which basically tries to make all active threads in the thread group stay at a constant rate (defined by samples per minute). This eliminates the need to figure out pacing or sleep times as you might in LoadRunner.

The Summary Report and Gaussian Random Timers are just used for debugging so don’t normally include those in the final script…

Git
Want to get started? Well first you will need a copy of a git client. Have a look at various how-to guides out there to get you started. I’ve found this site pretty useful for understanding and working with git basics. My public clone URL for all the code used in this post is available here:
git://github.com/90kts/performance_test.git

Enjoy!

I’m currently using a wamp box to serve up content for performance metrics. What I needed was a Ruby script [...]]]> If you are a rails fan you will probably recommend using active record instead, but if you just want to hack away at your mysql db from within your Ruby code, you may find this handy…

I’m currently using a wamp box to serve up content for performance metrics. What I needed was a Ruby script which could interrogate data on this installation.

You can download the binary required for this from here. After you install the gem (I do this off-line as my command prompt doesn’t allow me internet access here at work) you may find that you get an error similar to the following:
"NameError: uninitialized constant Mysql"

See what’s happening?

To find out what’s happening, you can jump into an irb terminal and type the following:
irb(main):002:0> require 'mysql'

When you do you will probably get a popup dialog like this:

Basically if you read the README that comes with the gem you will notice:

Dependencies …
For this to work, you must have libmysql.dll in your PATH environmental variable. If you have MySQL installed locally, just make sure that \bin is in your path. If you don’t have MySQL installed locally, you can install one or more of the MySQL tools, find the libmysql.dll included in their bin directory, and copy it to the %SystemRoot%\System32 folder.

So all you need to do is find out where your libmysql.dll lives:
D:\wamp>dir /b /s LIBMYSQL.dll
D:\wamp\bin\apache\apache2.2.8\bin\libmysql.dll
D:\wamp\bin\mysql\mysql5.0.51a\bin\libmySQL.dll
D:\wamp\bin\php\php5.2.5\libmysql.dll

And then add that bin directory to your environment path:
D:\wamp\bin\mysql\mysql5.0.51a\bin

Hope that saves some of you some time in getting connectivity on your windows boxes!
Attached below is some sample code to test your config.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

require 'rubygems'
require 'mysql'
 
def with_db
    dbh = Mysql.real_connect('localhost', 'myuser', 'mypass', 'mydb', 3306)
    begin
        yield dbh
    ensure
        dbh.close
    end
end
 
with_db do |db|
    res = db.query('select * from tablename')
    res.each do |row|
        puts row[1]
    end
end

Despite having some initial issues with the code, the author of the jmx4r module turned around [...]]]> After getting nowhere with lack luster HP support, I turned to the power of the Open Source community and got a very simple script up and running to remotely monitor Weblogic JVM Performance and JMS queues using JMX and JRuby.

Despite having some initial issues with the code, the author of the jmx4r module turned around a fix inside 24 hours so that I could connect to a Weblogic 9.2 server and look at its domainruntime MBeans. SiteScope can’t handle domainruntimes of these size (short of being told to ‘add more memory’ to a 3GB+ machine!). So exit stage left Mercury, enter stage right JRuby…

This script will enumerate JVM performance and also JMS queue depths in around 65 lines of code and spits those results into a MySQL database using the mysqlimport CLI.
I’ve formatted the output so I can still input the data into LoadRunner Analysis.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

require 'rubygems'
require 'jmx4r'
groupid = "mycluster"
JMX::MBean.establish_connection :url => "service:jmx:rmi:///jndi/iiop://hostname:7001/weblogic.management.mbeanservers.domainruntime", :username => "username", :password => "password"
 
open( 'D:\perf_jvm.csv', 'a') do |file|
   runtime_mbeans = JMX::MBean.find_all_by_name "com.bea:Type=JRockitRuntime,*"
   runtime_mbeans.each do | mbean |
		now  = Time.now
		date = "#{now.year}-#{now.month}-#{now.day}"
		time = "#{now.hour}:#{now.min}:#{now.sec}"
 
		data  = []
		if(mbean.object_name.get_key_property("Name") =~ /server/) then
			data << ""
			data << date
			data << time
			data << groupid
			data << mbean.object_name.get_key_property("Name")
			data << mbean.used_physical_memory
			data << mbean.heap_size_max
			data << mbean.free_physical_memory
			data << mbean.total_garbage_collection_time
			data << mbean.total_garbage_collection_count
			data << mbean.total_nursery_size
			data << mbean.heap_free_percent
			file.puts data.compact.join(", ")
		end
   end
end
 
cmd = 'D:\\wamp\\bin\\mysql\\mysql5.0.51a\\bin\\mysqlimport -h localhost -P 3390 -s -u root --fields-optionally-enclosed-by=""" --fields-terminated-by=, --lines-terminated-by="\n" --local sensis D:\perf_jvm.csv'
puts %x{#{cmd}}
cmd = 'cat D:\perf_jvm.csv >> D:\perf_jvm_archive.csv'
puts %x{#{cmd}}
File.delete 'D:\perf_jvm.csv'
 
 
open( 'D:\perf_queues.csv', 'a') do |file|
   runtime_mbeans = JMX::MBean.find_all_by_name "com.bea:Type=JMSDestinationRuntime,*"
   runtime_mbeans.each do | mbean |
		now  = Time.now
		date = "#{now.year}-#{now.month}-#{now.day}"
		time = "#{now.hour}:#{now.min}:#{now.sec}"
 
		data  = []
		if(mbean.object_name.get_key_property("Name") =~ /Module/) then
			data << ""
			data << date
			data << time
			data << groupid
			data << mbean.object_name.get_key_property("Name")
			data << mbean.messages_received_count
			data << mbean.messages_high_count
			file.puts data.compact.join(", ")
		end
   end
end
 
cmd = 'D:\\wamp\\bin\\mysql\\mysql5.0.51a\\bin\\mysqlimport -h localhost -P 3390 -s -u root --fields-optionally-enclosed-by=""" --fields-terminated-by=, --lines-terminated-by="\n" --local sensis D:\perf_queues.csv'
puts %x{#{cmd}}
cmd = 'cat D:\perf_queues.csv >> D:\perf_queues_archive.csv'
puts %x{#{cmd}}
sleep 10
File.delete 'D:\perf_queues.csv'