We were lucky to be selected by the development team at Cordelta to help them automate and performance test a high profile public website called MyHospitals. We were able to test millions of hits per hour from domestic and international locations in a wide variety of load scenarios. The success of this approach was underpinned by the following:

1. An open minded project / development team not coupled to a ‘must-have-most-expensive-toolset-to-do-job’ mentality
2. An open minded performance test analyst (me!) who believes Excel really is the grandpa of charting, R is the grandma of stats, Sparklines are the only way to present time series data to management, ANY tool can simulate load via HTTP/S and that there is no real distinction between good software testers or developers (only hard work separates the best from the worst).
3. That 2010 come-no-doubt 2011 buzz word… Cloud

As I sit here pluggin’ our alternative approach at Altentee, I can see the rise of other more successful punters taking on the big kids. I sit and [continue to] chuckle at the reaction to LR pricing and LR zealots who will fall on their swords over LR itself. I have come to realise one thing though, it is not about the tool, or even the alternate tool like perhaps I first thought. It is more about the freedom of choice.

To tackle MyHospitals I was free to choose and implement the following tools:

1. WatirGrid to orchestrate a small flotilla of IE and FireFox based browsers based on Watir
2. JMeter to add more at the protocol level of performance testing
3. httperf to do some basic benchmarking, similar to my front page
4. numbrcrunchr to pull together system metrics and make for easier analysis
5. A variety of Australian cloud providers and of course Amazon EC2 to host the test environment

It has been a great year. I’m not entirely free of the commercial chains just yet and am still needing the LR type work to prop up this approach, but I hope 2011 brings about some fresh thinking in performance and test automation with hopefully me somewhere amidst that space.

The solution as it turns out is quite simple. Google Analytics is well documented here in terms of what query parameters constitute the GIF request that registers traffic.

The key parameters which need to be catered for when spoofing traffic are:
utmn => Unique ID generated for each GIF request to prevent caching of the GIF image.
utmhid => Another unique ID not documented but I think is related to AdSense
plus all the GA cookies including utma, utmb, utmc and utmz

The best way to parametize this is to observe real traffic with a tool like FireBug in the Net(work) panel. That way you can make a copy of a real GIF request. Essentially you are going to fake the call to __utm.gif. An example request looks like this:

http://www.google-analytics.com/__utm.gif?utmwv=4.7.2&utmn=123456789&utmhn=www.altentee.com&utmcs=UTF-8&utmsr=1920×1200&utmsc=24-bit&utmul=en-us&utmje=1&utmfl=10.0%20r45&utmdt=AlteneeBlog&utmhid=1697634062&utmr=-&utmp=%2F&utmac=UA-12345678-3&utmcc=__utma%3D199946558.28502083.1280297456.1280637882.1281056979.6%3B%2B__utmz%3D199946558.1280297456.1.1.utmcsr%3D(direct)%7Cutmccn%3D(direct)%7Cutmcmd%3D(none)%3B&gaq=1

All you need to do to spoof traffic or generate page views to your GA account is make repeated calls to this URL after parametizing key fields previously mentioned. An example request in JMeter with corresponding parametized functions in bold (random and time) looks like this:

/__utm.gif?utmwv=4.7.2&utmn=${__Random(10000000,60000000,utmn)}&utmhn=www.altentee.com&utmcs=UTF-8&utmsr=1366×768&utmsc=24-bit&utmul=en-us&utmje=1&utmfl=10.1%20r53&utmdt=PerformanceTest&utmhid=${__Random(100000000,900000000,utmhid)}&utmr=-&utmp=%2F&utmac=UA-12345678-3&utmcc=__utma%3D199946558.1448353202.${__time()}.${__time()}.${__time()}.4%3B%2B__utmz%3D199946558.${__time()}.1.1.utmcsr%3D(direct)%7Cutmccn%3D(direct)%7Cutmcmd%3D(none)%3B&gaq=1

You could also build this URL from scratch although it’s much easier to copy and modify the target URL for the site you’re actually testing. Please don’t use the URLs above as I’ve faked other things like the Google Analytics account code (utmac). It’s also worth changing the page title (utmdt) so you can separate real traffic from the fake traffic generated during performance testing.

Build a simple test plan in JMeter with a HTTP sampler generating requests to the aforementioned URL, put in a constant timer (or similar) so you don’t spam GA’s servers and away you go. You can then measure response time to this service for as many threads as you like. I generated 10K page views in 5 minutes which was more than sufficient to satisfy the client’s requirements.

In this example we’ll need to add a JDBC Connection Configuration to our test plan. This will define and control the way in which JMeter will access your database.

Notice I’ve left the defaults for connection pool properties, although you may need to tweak this depending on how you’ll access the data. I’ve used a JDBC connection URL that accesses a local mysql server listening on port 3306. The database in use is called watirgrid

I’ve also defined the JDBC Driver class and specified the username / password to connect with.

Now we can make a JDBC request using the JDBC Request sampler. It will look something like this:

The Variable Name matches the JDBC Connection Config Variable Name i.e. mysql

I’m using a select statement that will print out results in a JSON-like format. That query is as follows:

SELECT
CONCAT("[",
GROUP_CONCAT(
CONCAT("{email:'",email,"'"),
CONCAT(",login_count:'",login_count),"'}")
,"]")
AS json FROM users

In doing this, it will return a result set that looks like this:

+-------------------------------------------------+
| json                                            |
+-------------------------------------------------+
| [{email:'tim.koops@gmail.com',login_count:'1'}] |
+-------------------------------------------------+
1 row IN SET (0.00 sec)

Now we need to add a Post Processor Regex Extractor to the JDBC request as follows:

The reason we do this is so we can extract the results from the JSON string returned by the database query. For example, to extract and populate a variable called ${email} our PPRE would look like this:

Now we’re cooking! Add a Debug Sampler to confirm everything works as you’d expect. You should see results like this in the response data

email=tim.koops@gmail.com
email_g=1
email_g0=email:'tim.koops@gmail.com'
email_g1=tim.koops@gmail.com
mysql=org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl@10d69502

Attached is a demo test plan: jdbc_example.jmx

If you work with more agile teams/developers you will probably start to require a lot of re-use out of your scripts, especially when the release cycle increases in terms of frequency. It starts to make sense to use version control and some form of scripting framework.

In this post I’m going to use JMeter as the tool of choice here and I’m going to re-introduce concepts which I normally force upon others whilst contracting. That is, use of Application Simulation Models (ASMs) to hep define the load and use of the Application Performance Index (Apdex) to help present results. I’m also going to use GitHub (or git) as my tool for maintaining version control and implementing a wiki as a dashboard replacement for other test management tools.

In short, I’m hosting version controlled scripts on GitHub. You can view my repository here:
http://wiki.github.com/90kts/performance_test

I’m using Google Documents to store my ASMs. You can see an example here.

I’m also using the Wiki functionality of GitHub plus some Ruby scripts as my test runners. In other words, a scripted version of a controller, driven by data from the ASMs.

GitHub can be quite useful. Not only can you version control all your scripts, you can also implement concepts based around a “low-tech testing dashboard” first explored by James Bach. The use of a wiki to support this was recently explored by Matthew at Socialtext. Coupled with that you also get an Issues database which is perfect for tracking those performance defects during test execution. Google Docs is also very useful in that it allows you to centrally store test artifacts like your ASMs which has plenty of neat APIs for programatically accessing content. Start working more with GitHub and less with QualityCentre and watch your circle of geek friends change.

Dashboard
So my dashboard is really just a broad sweep of major system and application components that present some level of risk to application stability and performance. It’s represented in the GitHub wiki as follows:

Main elements are the areas I wish to test, the estimated effort required to script/prepare those areas, an indication of risk to production stability (carried out separately in risk assessments with appropriate technical/business teams), day to day comments on test readiness and a link to the online ASM. I use the keywords “run” or “draft” to distinguish between ASM status’. This helps the test runner determine which tests to execute.

Test Runner
The test runner is a Ruby script which when executed, queries the Dashboard for ASMs to execute (based on the keyword “run”) then fetches the appropriate ASM via Google Documents to parse out relevant volumetrics for the test. Finally, it launches thread groups within JMeter to execute the test according to the ASM. Results are stored in the corresponding logs directory.

Apdex
If you’d like to find out more about the Application Performance Index please see here. I’ve found Apdex to be a very useful format to communicate application performance consistently to non-tech-heads. There’s plenty of room to geek out at low level later, but for a 20,000′ view on things, Apdex is often a good start. I’ve implemented Apdex support for this framework using a Ruby script to parse the results. The Google chart API for Ruby is used to present sparklines. You can do this during or after a test run has complete. It basically picks up results from the logs directory and parses them against response time targets specified in the ASM. The output of the Apdex script is a html page which looks like this:

Application Simulation Models
The ASM is really the brains behind the JMeter test plan. It looks like this:

It has two worksheets. The ASM worksheet is intended for business stakeholders. It’s geared more towards their understanding. I encourage these groups to articulate load in terms of transaction distribution (how many and what types of transactions groups) and user distribution (who are your user groups and what do they consist of). The key metrics I get these groups to enter are highlighted in yellow. Things like different user groups, percentage breakdowns of transactions and transaction/concurrent user quantities. Have a play, it’s pretty easy to follow. Coincidentally in every contract I’ve worked, I make this a mandatory test artifact for both my testers and the teams they’ve engaged with. No ASM = No Performance test…

The Scenario worksheet is really just the reverse of the first. It breaks down all the metrics that the business have provided into a worksheet to help you plug figures into your test scenarios. It helps cut down user-errors when calculating transaction rates etc and makes it much easier to chop and change figures. Want to do a test run with less concurrent users or higher transaction throughput? Just muck around with the percentages in yellow. I’ve used this format for both LoadRunner and JMeter gigs, so the terminology is meant to be familiar to both those types of users.

JMeter Test Plan
Through the beauty of JMeter -J command line parameters I’m able to pass all the relevant metrics from the ASM such as number of threads (vusers), duration, throughput, rampup and action(transaction) distribution percentages. I use this template format quite often and have explained some of these concepts previously.

The test plan gets the required parameters from the command line at run time. You can see the User Defined Variables look something like this:
DURATION = ${__P(duration,60)

Which basically sets the ${DURATION} JMeter variable based on the -Jduration command line variable, or defaults to 60 seconds if it is not passed. Other variables are also defined here and get loaded into the main Thread Group.

I use a Runtime Controller to control how long the test plan executes for. I also keep a separate Transaction Controller called Init to execute login actions etc or things I only want to execute once in the test plan.

Nested inside the Runtime Controller I use Throughput Controllers with percent executions tied to action percentages defined in the ASM. For example, action 1 might be a “view product” sequence of requests which I execute 50% of the time and so forth. Similar to weighted percentages for transaction blocks in LoadRunner. It is best to calculate these percentages using the Per User mode in JMeter for more accurate results.

Nested inside Throughput Controllers will be further Transaction Controllers, with Generate Parent Sample checked to enable similar functionality to transaction timers in LoadRunner around groups of requests.

The whole Thread Group has its throughput limited with a Constant Throughput Timer which basically tries to make all active threads in the thread group stay at a constant rate (defined by samples per minute). This eliminates the need to figure out pacing or sleep times as you might in LoadRunner.

The Summary Report and Gaussian Random Timers are just used for debugging so don’t normally include those in the final script…

Git
Want to get started? Well first you will need a copy of a git client. Have a look at various how-to guides out there to get you started. I’ve found this site pretty useful for understanding and working with git basics. My public clone URL for all the code used in this post is available here:
git://github.com/90kts/performance_test.git

Enjoy!

Some people get confused with the additional protocols that LoadRunner sports for SAP, particularly the SAP GUI. But if your SAP portal uses plain old HTTP (or HTTPS) then JMeter can do the job.

There are a few gotchas which I will continue to update on this post, namely around correlation. In JMeter, correlation is particularly *easier* because you don’t have to worry about where to position a correlation rule. The equivalent function to use is a Post Processor -> Regular Expression Extractor.

I used the following rules to help correlate the SAP portal:

Response Field to check: Body
Reference Name: sap-ext-sid
Regular Expression: sap-ext-sid=”(.+?)”
Template: $1$
Match No.: 1

Response Field to check: Body
Reference Name: sap-wd-tstamp
Regular Expression: name=”sap-wd-tstamp” value=”(.+?)”
Template: $1$
Match No.: 1

Response Field to check: Body
Reference Name: sap-ep-iviewhandle
Regular Expression: name=”sap-ep-iviewhandle” value=”(.+?)”
Template: $1$
Match No.: 1

Just scope those globally in your thread group and they will find and populate the necessary variables with ${variable_name}
All you need to do to your final script is find and replace all instances of those variables with the JMeter variable in your favourite text editor. On windows I find SciTE is free (normally installed with Ruby) and powerful. e.g.

Replace:
7QQ8jk_TEknhboPm9fmgqw–Mce5By08HC1RDy9m4LImcA–
With:
${sap-ext-sid}

In our particular portal a cookie value is used in the context of a query parameter (or form value). So to read a cookie value from the HTTP Cookie Manager element at run time (make sure you include it in your Thread Group) you just use another regular expression extractor:

Response Field to check: Header
Reference Name: MYSAPS02
Regular Expression: MYSAPSSO2=(.+?);
Template: $1$
Match No.: 1

Note in this case I am checking the header rather than the body for this value. Then use the variable as before.

Another tip when recording SAP portals, you might want to use the BadBoy recorder which can export to JMeter test plans. I had more success using this over the traditional recording proxy in JMeter.

java -version
java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)

Then you are likely to encounter this error when using the standard HTTP Request sampler:

Response code: Non HTTP response code: java.net.SocketException
Response message: Non HTTP response message: Unconnected sockets not implemented

If you have the opportunity, use the HTTP Request HTTPClient sampler instead and this problem should be resolved. Otherwise use Java 1.5 instead. If you’ve already recorded your test plan in JMeter or other tools (like BadBoy) then some simple regular expressions on the jmx file will fix the problem manually.

Windows:

ruby -pi.bak -e "gsub(/

ruby -pi.bak -e "gsub(/HTTPSampler>/, 'HTTPSampler2>')" *.jmx


Mac:

ruby -pi.bak -e "gsub(/

ruby -pi.bak -e "gsub(/HTTPSampler>/, 'HTTPSampler2>')" *.jmx

I had not heard of this approach before and am sure it will be useful in future, provided you have sufficient access to create the bindings in the first place.

However, if it is vital to your script (for example, a login page) then you can use an alternative recording tool called Badboy Software. I frequently use this to record HTTPS then export the test plan to JMeter. Alternatively you can manually create the HTTPS request sample yourself.

JMeter can run HTTPS request samplers, it just can’t record them using the native proxy recorder.

In practice, correlation requirements relate to things like session ids, viewstate, cookies, date time stamps and other transactional information. So more often than not you will see this data changing following a post back or response from the server. With the introduction of AJAX calls, correlation requirements probably increase (per page) as the number of transactional requests probably increases as well.

In LoadRunner, you typically correlate data manually using the web_reg_save_param function. This in effect is a prepared statement, placed before the request that will return the data in the response you are looking for. For example, your function will specify a left and right boundary of the data that it is looking for before you make the request. Then the request is made and if a match is found, then the data is saved in a parameter which you can later call as a variable in subsequent requests. LoadRunner also has a feature called Correlation Studio which takes affect during recording of your script, using a similar approach (left and right boundaries) and then dynamically swapping corresponding matches with the variabilzed data. All this is quite painful though, and whenever I’m specifying primitive left and right boundaries or using the flaky Correlation Studio (I can explain if you like), I’m left wishing I could just use regular expressions …

The trick to correlating data easily with JMeter is to first know what data it is that needs correlating! (duh…) Experience will make this easier as a lot of common web apps have common dynamic data requirements as already mentioned. Another trick is to always record the script twice (and diff to see differences between unique user sessions), then more often than not record 2+ iterations of an action within a single script (and diff to see differences between user transactions/actions). If you do this before anything else, and diff the results, you will save bucketloads of heartache in getting your script harnesses running, no matter which tool you choose.

But back to JMeter, once you do know what data you need to correlate, simply create a Post Processor -> Regular Expression Extractor to ‘variablize’ that data. Huh? Read on for a more detailed explanation of this.

In my following example, I have 4 dummy pages that return unique jsessionid’s for each page as follows:

I’ve done this to simulate the changing session information typically found in any jsp based web app. To look for a jsessionid I simply add a new Post Processor called the Regular Expression Extractor. The following screenshot shows an example:

In the example provided, the reference name is what I can later call upon as a variable within the script, for example, ${jsessionid_trans}. The regular expression is the pattern I am looking for. Note in using a parenthesis I’m specifying a subexpression which I can get to with the match result number of n(1). If you’re new to regex, my example is equivalent to saying look for a left boundary of jsessionid= then match any character until you hit an ampersand. Even if you know nothing about regex, this will probably get you out of hot water 90% of the time. The real power in regex though is the ability to manipulate and narrow in on search results, or search for varying patterns (like telephone numbers, alphanumeric sequences and so on). I still find the Perl regular expression tutorials the best:
http://perldoc.perl.org/perlretut.html

So that’s it, you’ve found the jsessionid and to use it later on in your script, you simply call on it as a variable i.e. ~koops/test_jmeter/page2.html?value=${jsessionid}
I often find it is easier to open up your test plan (.jmx) in your favourite text editor and then use the same regular expression to search and replace all instances of that pattern with your newly set variable.

However there is a twist to all this, which I refer to as scope. You really need to understand the scope of your regular expression extractors and how they are applied. By scope I’m loosely referring to variable scope in a normal programming language (which JMeter is not). I think the best way to understand the scope of your extractors, is to identify which ‘container’ they are placed in. For a JMeter test plan, there are multiple containers you might place these extractors in:

If you’re familiar with programming languages and scoping, then you’ll probably identify the pecking order of the scope I’ve just mentioned, from global right down to transaction level (I’ve made up a few names there). Global variables will override local variables and so on down the chain. So you need to be careful where you place your regular expression extractors and what reference (variable) names you use, as these can interfere with each other. Sometimes it is useful to limit your extractor in scope to a particular transaction, thread group or test plan.

In my example listed above, I’ve written a test plan to explore the different scopes identified. Using the debug sampler, you can see that the trans scope is limited to single request sampler, because it is added as a child element. This means the variable jsessionid_trans gets set only once. The local scope is limited to a transaction controller. I placed it before the second request sampler but not as a child to the first, to highlight how it sets the variable jsessionid_local twice (once for each request contained in that transaction controller). The thread scope is limited to the entire thread group, so you will see the jsessionid_thread variable set three times as each page is requested. The same applies to the global scope in which case the jsessionid_global variable is set on each occurrence of the pattern. I’ve also added a second thread group with its own debug sampler to highlight how only global variables are accessible by this thread group. You can run this test plan (regex_test.jmx) yourself and check out the results as shown below:

Finally, here are some example regular expressions which have worked well for me for different types of web apps:
jsessionid=(.+?)&
jsessionid=(.+?)'
id="__VIEWSTATE" value="(.+?)"
jsd=(\d+)
strEntry=(.+?)'
CdzSession' value='(.+?)'
WebiSession' value='(.+?)'
DOCTOKEN VALUE='(.+?)'
etc ...

If you roll-up your transactions into separate transaction controllers as I demonstrated in a previous post, you will get useful summary views for all your transactions:

Alternatively, drop a stat agg visualizer listener under a single controller to get a graph for a single transaction, useful if clients are only interested in certain transactions.