I was going to write why LoadRunner does not work for the web as it is today and into the foreseeable future, but decided against that because I don’t think the following points are necessarily limited to a specific toolset.
Load testing does not work for web as it is today primarily because…
1. Load testing tool languages are not flexible enough.
Load testing tools are in need of a major overhaul. Consider LoadRunner, the majority of your scripts will be specified in C or proprietary API calls such as lr_ statements. String manipulation for example, whilst possible is extremely clunky and requires oodles of C code. Lack of proper regex or xpath support limits script flexibility in terms of parsing the necessary contents of web page and ajax calls. Don’t even get me started on ajax! Similar complaints for JMeter. Although better support for regex you can really only edit test plans via the GUI. You’ll go cross-eyed looking at test plans in xml. I think the Grinder is the closest product I’ve seen that supports flexibility as you can write tests in Jython which provides more flexibility in terms of scripting.
2. We’re still stuck in record and playback mode.
We really haven’t advanced like our automation brethren. Part of the big sell by commercial load testing companies is “Look it’s easy, just hit the red button and play it back when you’re finished!”. Bollocks to that, we know that never works. A dream load testing tool would support the ability to hand-code your statements and be editable in a text editor! In a similar fashion to watir.
The dream tool would look like:
1 2 3 4 | vusers.start(100) browser.goto('url') browser.button(:text, /Continue/).click end |
Instead load testing scripts in LoadRunner look something like this:
1 2 3 4 5 6 7 8 9 10 | web_submit_data("transaction", "Action=https://someurl/I/have/to/keep/correlating/damn/it", "Method=POST" "RecContentType="text/html", "Mode=HTML", "ITEMDATA, "Name=L_3_0_0_1_0_1", "Value={another_value_I_have_to_correlate}", "Name=L_3_0_0_2_0_1", "Value={maybe_paramatize_this_one}", ENDITEM, LAST); |
We are of course talking about very different technologies. Watir automation scripts driving real browsers with win32ole, whereas most load testing tools are injecting traffic at the protocol layer. If a typical end to end scenario has 30 steps, this can amount to some 2000+ lines of code thanks to the extra verbosity of having to specify all the form values and so forth. This also means you have oodles of code to correlate and parametize. It’s not something you want to hand code, the quickest path is to record (unfortunately). BrowserMob is the closest online provider I’ve seen that overcomes this, supporting load tests written as selenium scripts. Problem is, we need a solution for all those non-public facing companies, the bulk of where I do my work at!
3. No real code versioning.
Changes to the UI will break your script! One can implement their own code versioning as I have recently begun to do with git but there really should be built in support for this. That aside, you revision your code but effectively blast it away every time you hit the red record button. Reason? Each recording can look vastly different to the last. It only takes one form value L_3_0_0_1_0_1 to change to L_3_0_0_1_0_2 or one block of code executed by an ajax call to shift to cause mass confusion. Versioning your code *may* pick this change up, but only if you got the power of the diff! Diffing just got a little harder. Shoot back 10 years ago and diffing for correlation would have been a cinch. Wind forward to today with hundreds of ajax or json calls and correlation can be a nightmare. Windiff struggles. Daisydiff does a much better job highlighting differences by line and by column. This approach kind of works in a waterfall test environment. Throw in a twice weekly code drop in your performance test environment and watch your scripts break. Agile anyone?
So if I was an uber-elite programmer what would I do to move loadtesting into the current decade?
1. Develop a load-testing tool that is based on a flexible scripting language such as Ruby.
2. Get rid of the record button and provide free text editors to the world!
3. Provide a framework that can be versioned and easily detect changes in the system under test.
The Microsoft load testing tools included with VS2008 Team Suite Testing Edition are decent as the virtual users can be coded in C# which is much more flexible than the Load Runner scripting but you run into the same problem as with any other protocol only testing, lack of AJAX integration.
Also, with the Microsoft tool you are limited to HTTP/S only, unlike LoadRunner and JMeter which have much broader support for other goodies like FTP, SMTP, etc.
Well, Mercury was attempting to make this easier when they were bought by HP.
If you use HTML mode, LoadRunner should capture most form submits as WEB_SUBMIT_FORM with much easier data values that don’t have to be correlated as closely, if at all. It also records clicks as web_click(“ButtonNamedFoozle”); rather than URLs you have to correlate.
I honestly don’t see how it could get any better or easier without losing the flexibility.
In my opinion, the fact that I can do anything I want in ANSI C is a FEATURE, not a bug.
To each their own!
I also completely disagree with the title of this post, but I’m assuming you’re overstating the case to make it sound interesting.
G
Some very good points Tim. Sounds like an opportunity for a new breed of tool!
I dont know how we will ever get away from correlation but I guess having a more flexible scripting language can only assist.
You’re not wrong Koops. I find much larger problems, not only in the tools we use, but the general knowledge that people have in performance testing. Test managers, in general, who don’t know ‘much’ about our task, think its easy to just ‘record a new script’. The sales men who sell these products simplify the process too much.
I also find, that due to the sheer cost of applications (like LoadRunner/QC/any mercury/HP app) has made them no longer listen to the users of the app, for future changes. Ajax is not that ‘new’ of a technology, and LR struggles so hard to support it. And don’t get me started on ‘click and script’ – Recording made even easier – sold as “no need to correlate” – No people, it does not work!
Watir, is defiantly the way to go, although I don’t have much experience in the open source tools you mention, its clear to see that open source is the peoples software, we make the changes and enhancements that WE want. Instead of waiting for the big wigs to help us out.
Hopefully this will flush out the fakers in the industry…but selling performance testing as an easy job makes it harder for us….I heard someone once say “when will LR release an auto analyse tool” – I decided to walk away.
Good article Koop.
LOL, yes the title was deliberately tongue-in-cheek. You’re just the recipient of my weekend’s frustration spent going cross-eyed looking at LR scripts…
Also interesting about HTML-Mode. We are using that here but web_submit_forms are not being recorded, only web_submit_data which I’m not sure why. (We use HTML-based scripts with HTML advanced as ‘a script describing user actions (e.g. web_ink, web_submit_form). I use web_link whenever I can because that is definitely easier. Shame you can’t pass in regex to the string match parameters.
Also web_click, I think you’re talking about the Click and Script protocol which costs more money, and I haven’t heard enough good (anecdotal) advice to switch to that.
I’ll leave ANSI C for you to enjoy
Great post! I agree. Some parts of this type of work do seem stuck in the past. The first vendor to come out with a ground-breaking improvement in #s 1 & 2 above will make a killing, imho.
Tim,
“You’ll go cross-eyed looking at test plans in xml”
LOL
I’ve worked with these tools and I agree with you Tim, there are a lot of features to add to the performance tools. I think the next step is to combine the automation tools like Watir or Selenium with performance tools to make it easier the task to create performance scenarios and we “humans with brains and experience” can make a deeper analyze of the results.
Best Regards.
José
Agreed José,
It would be nice to spend less time scripting and more time analysing performance results… Stay tuned for more open source alternatives in the near future!
Dear Tim,
Thank you for your articles on performance testing, I have found them very enlightening but the last one is strangely daunting.
Let me explain, you see I have recently applied for a position where I belive you have worked in the recent past. I have heard of your expertise from many people yet you seem very frustrated.
I would have thought that a national institution hosting one of the largest performance test teams in Australia would be a great place for a budding newbie performance tester to really see how the big end of town goes about its business.
I am thinking of moving from Perth and will take only a small paycut from my permanent position. Can you please confirm that it will be a wonderfully rewarding experience for me??
Yours
Budd
Budd, it will definitely be a rewarding experience for you no doubt.
Despite this blog’s grievances towards commercial load testing tools, I acknowledge that they’ve kept me employed for the last 5 years consulting as a performance tester. Please don’t be put off by them.
I don’t think the frustrations are reflective of the environments I’ve worked in. The people I’ve worked with have been great and I’ve loved the challenge each different workplace offers. The frustrations lies more with an ageing toolset which hasn’t really kept pace with the maturing software testing field in general.
Also, bigger is not necessarily better! Some of the most enjoyable work I’ve done has been with very small teams. Be sure to consider Conway’s law! [1]. However if you’re looking for experience and variety in terms of protocols tested with, larger team sizes etc then I strongly encourage you to consider the role on the other side of Australia =)
Regards,
Tim
[1] http://en.wikipedia.org/wiki/Conway‘s_Law
Yes, the tools fall short. What are you doing about it?
I’m going to get a fuller response to this on my blog. Let me know if you want to organise a hack day to start work on a simple tool.
well, i’ve started this side-project => http://github.com/90kts/watirgrid
Let me know about your hack day, I’d be interested…
Cheers,
Tim
Hi,
First of all Thanks very much for your useful post.
I would like to introduce another good blog which is having free software testing ebooks and technical content, Have a look.
Hey! Just wanted to tell you that you own a super little website and I would like you understand how sweet it looks. Will come back on a regular basis.