Load testing does not work for web as it is today primarily because… 1. Load testing tool [...]]]> I was going to write why LoadRunner does not work for the web as it is today and into the foreseeable future, but decided against that because I don’t think the following points are necessarily limited to a specific toolset.

Load testing does not work for web as it is today primarily because…

1. Load testing tool languages are not flexible enough.
Load testing tools are in need of a major overhaul. Consider LoadRunner, the majority of your scripts will be specified in C or proprietary API calls such as lr_ statements. String manipulation for example, whilst possible is extremely clunky and requires oodles of C code. Lack of proper regex or xpath support limits script flexibility in terms of parsing the necessary contents of web page and ajax calls. Don’t even get me started on ajax! Similar complaints for JMeter. Although better support for regex you can really only edit test plans via the GUI. You’ll go cross-eyed looking at test plans in xml. I think the Grinder is the closest product I’ve seen that supports flexibility as you can write tests in Jython which provides more flexibility in terms of scripting.

2. We’re still stuck in record and playback mode.
We really haven’t advanced like our automation brethren. Part of the big sell by commercial load testing companies is “Look it’s easy, just hit the red button and play it back when you’re finished!”. Bollocks to that, we know that never works. A dream load testing tool would support the ability to hand-code your statements and be editable in a text editor! In a similar fashion to watir.

The dream tool would look like:

1
2
3
4

vusers.start(100)
  browser.goto('url')
  browser.button(:text, /Continue/).click
end

Instead load testing scripts in LoadRunner look something like this:

1
2
3
4
5
6
7
8
9
10

web_submit_data("transaction",
"Action=https://someurl/I/have/to/keep/correlating/damn/it",
"Method=POST"
"RecContentType="text/html",
"Mode=HTML",
"ITEMDATA,
"Name=L_3_0_0_1_0_1", "Value={another_value_I_have_to_correlate}",
"Name=L_3_0_0_2_0_1", "Value={maybe_paramatize_this_one}",
ENDITEM,
LAST);

We are of course talking about very different technologies. Watir automation scripts driving real browsers with win32ole, whereas most load testing tools are injecting traffic at the protocol layer. If a typical end to end scenario has 30 steps, this can amount to some 2000+ lines of code thanks to the extra verbosity of having to specify all the form values and so forth. This also means you have oodles of code to correlate and parametize. It’s not something you want to hand code, the quickest path is to record (unfortunately). BrowserMob is the closest online provider I’ve seen that overcomes this, supporting load tests written as selenium scripts. Problem is, we need a solution for all those non-public facing companies, the bulk of where I do my work at!

3. No real code versioning.
Changes to the UI will break your script! One can implement their own code versioning as I have recently begun to do with git but there really should be built in support for this. That aside, you revision your code but effectively blast it away every time you hit the red record button. Reason? Each recording can look vastly different to the last. It only takes one form value L_3_0_0_1_0_1 to change to L_3_0_0_1_0_2 or one block of code executed by an ajax call to shift to cause mass confusion. Versioning your code *may* pick this change up, but only if you got the power of the diff! Diffing just got a little harder. Shoot back 10 years ago and diffing for correlation would have been a cinch. Wind forward to today with hundreds of ajax or json calls and correlation can be a nightmare. Windiff struggles. Daisydiff does a much better job highlighting differences by line and by column. This approach kind of works in a waterfall test environment. Throw in a twice weekly code drop in your performance test environment and watch your scripts break. Agile anyone?

So if I was an uber-elite programmer what would I do to move loadtesting into the current decade?
1. Develop a load-testing tool that is based on a flexible scripting language such as Ruby.
2. Get rid of the record button and provide free text editors to the world!
3. Provide a framework that can be versioned and easily detect changes in the system under test.

In short this is a great (free) opportunity to attend a conference using an open session format. By that I mean the conference is run/organized like a wiki, where [...]]]> I recently attended the 2008 conference for Continuous Integration in Melbourne, and mixed with like minded professionals involved with all aspects of CI and testing in general.

In short this is a great (free) opportunity to attend a conference using an open session format. By that I mean the conference is run/organized like a wiki, where the attendees nominate topics they’d like to discuss or facilitate and then a user vote system organizes and makes it happen.

I avoided the performance testing oriented sessions and went for sessions that spoke about exploratory testing, defect management, benefits of test driven development, skills required for test automation and the last one titled ‘do we still need testers?’ …

They all generated a healthy amount of discussion and it was great to hear other people’s experienced opinions on the subject matter. It was also a great opportunity to network, find out who’s working on similar problems in the local industry and from afar. There was a healthy interest in using JMeter as a performance test tool to support CI, so I’ll be posting some more targeted blogs about this in the near future, as well as some consolidated info on performance metrics collection (without SiteScope) which a few people were also interested in.

Anyway, if you haven’t heard about the conference, keep an eye out for next years version. It’s well worth the zero dollar commitment, free beer and t-shirt. All you have to do is ‘use your feet’.
=)

RailsCamp is a get together of mostly Rails developers, but also a fairly eclectic mix of Ruby hackers and geeks in general. 3 [...]]]> Went to the third RailsCamp held in a scout hall near Gosford NSW last weekend; my brain has just about got back to normal after 3 solid days of drinking and coding =)

RailsCamp is a get together of mostly Rails developers, but also a fairly eclectic mix of Ruby hackers and geeks in general. 3 nights of drinking, camping and coding with NO internet. Possibly the hardest habit to kick is to stop checking for emails every 5 minutes.

For those with harder to kick habits such as Twittering, there was a ‘Twetter’ clone amongst other cool apps such as the ‘Duke’. The Duke was a user vote playlist driver for winamp (I think) and I got to observe just how close the geek community is when it comes to music tastes. Daft Punk was on fairly high rotation. Of mention was the first time we got Rick Rolled, which then led to a flurry of hack attempts at the guy hosting it on his Mac.

Guitar Hero was in force, as was Urban Terror and a touch of BZFlag. Nice.

Accommodation, setup and food was awesome, the organisers really did do a good job in getting this together. Big thanks to those guys! Perhaps the best part was the ability to network with like-minded people and discuss all things related to Ruby or Rails in a collaborative manner. I managed to get a little work done relating to a test framework for use with FireWatir which I’ll post later.

IF you didn’t or couldn’t make it, keep an eye out for the next one to occur. Definitely worth the $100 to nerd on for a full weekend …

eval{ &yourSubRoutine($parameters); }; $@ ? $yourLogObject->yourErrorMethod("I failed with reason: $@") ): $yourLogObject->yourTraceMethod("Everything was good: $?") );