Wow, wasn’t that long ago when I was promising things that were harder, better, faster and stronger. I think I got part of the way there. The core reason I set up Altentee was to provide reliable (and potentially cheaper) alternatives to traditional licensed performance testing tool sets. There’s no doubt Altentee can test at the limits of your typical web app using tools that cost zilch. The bigger challenge I’ve found this year is convincing potential clients that they really don’t need to spend that much. I’ve even offered a free load test on our homepage to help illustrate this point.
We were lucky to be selected by the development team at Cordelta to help them automate and performance test a high profile public website called MyHospitals. We were able to test millions of hits per hour from domestic and international locations in a wide variety of load scenarios. The success of this approach was underpinned by the following:
- An open minded project / development team not coupled to a ‘must-have-most-expensive-toolset-to-do-job’ mentality
- An open minded performance test analyst (me!) who believes Excel really is the grandpa of charting, R is the grandma of stats, Sparklines are the only way to present time series data to management, ANY tool can simulate load via HTTP/S and that there is no real distinction between good software testers or developers (only hard work separates the best from the worst).
- That 2010 come-no-doubt 2011 buzz word… Cloud
As I sit here pluggin’ our alternative approach at Altentee, I can see the rise of other more successful punters taking on the big kids. I sit and [continue to] chuckle at the reaction to LR pricing and LR zealots who will fall on their swords over LR itself. I have come to realise one thing though, it is not about the tool, or even the alternate tool like perhaps I first thought. It is more about the freedom of choice.
To tackle MyHospitals I was free to choose and implement the following tools:
- WatirGrid to orchestrate a small flotilla of IE and FireFox based browsers based on Watir
- JMeter to add more at the protocol level of performance testing
- httperf to do some basic benchmarking, similar to my front page
- numbrcrunchr to pull together system metrics and make for easier analysis
- A variety of Australian cloud providers and of course Amazon EC2 to host the test environment
It has been a great year. I’m not entirely free of the commercial chains just yet and am still needing the LR type work to prop up this approach, but I hope 2011 brings about some fresh thinking in performance and test automation with hopefully me somewhere amidst that space.
Read More
In working with SiteScope of late, I’ve found that it doesn’t always collect performance metrics the way I want to. More importantly, it can often turn a simple monitoring activity into a complex disaster. Take monitoring via JMX for example. In SiteScope, it has a rather complicated (and sometimes broken) interface when trying to communicate with a busy MBean Server. One can quite easily roll your own JMX monitor using open source tools in about 65 lines of code as I demonstrated here.
But we still all use tools like LoadRunner in these commercial 9-5 contracts right? Wouldn’t it be nice, if you could roll your own custom monitors in Ruby, Perl or whatever language you like, store that data in a simple repository, let’s say a MySQL database, and still be able to hook into those metrics from a LoadRunner Controller during test execution!?
It is possible, with one PHP file and a simple WAMP (or LAMP) installation all wrapped up in a SiteScope-like alternative.
Read More
Google have got some charting code worth looking at, which basically allows you to create charts by modifying parameters in the query string.
Something as simple as this:
http://chart.apis.google.com/chart?chtt=Tim's%20Beer%20Intake&cht=p3&chd=s:9h&chs=250x100&chco=00ff00&chl=Unemployed|Employed produces this:
Being on holidays at the moment, I’m lacking the um, ‘project’ on which to apply this technology, but am curious to see how functional it is. They give some examples on how to serialize datasets, and I think up to 4095 data points can be used. Not bad for those dashboard type applications …
Read More
I’ve explored in previous posts the use of tools such as onboard Analytics (LoadRunner), off-the-shelf tools (Excel) and custom web based implementations (JGraph, ChartDirector) used to analyze the nitty gritty of performance metrics.
All of these tool’s use are governed by some common factors being:
the Expediency factor – the timeliness of data being analyzed as measured between capturing and analysis of raw data.
the Pimp factor – the ‘prettiness’ in the presentation of data. Particularly important for benchmarking or external (public) facing documents. Never underestimate the importance in presentation of your results.
the Share-ability factor – how portable the analysis needs to be. Particularly important when working with different technology groups such as middleware, storage or network subject matter experts.
the Proprietary factor – sometimes you just can’t escape this. Your heart may lie with support open source, but often your pay check dictates that you must use proprietary formats, templates and the like as already setup by the client. Particularly pertinent with the use of tools like Load Runner, QALoad and the like.
Read More
I’ve been looking for some easy ways to enumerate all of the available performance counters on a Win32 platform programatically.
Rather than trawl through the PerfMon counter GUI, or regress back into the millions of raw counters available via WMI, I was looking for a solution somewhere in between.
The ActiveState version of Perl has a Win32::PerfMon module which lets you enumerate cooked performance counter metrics on a Win32 platform. So I started to write a simple script that would list all Objects->Instances->Counters recursively. Sounds simple right?
Unfortunately I found that the module appears faulty when trying to enum Objects that have multiple instances, such as PhysicalDisk and LogicalDisk. Naturally, these are the counters I’m most interested in! After googling to the end of the known universe, I couldn’t find a solution to this problem.
Windows comes with a command line utility called Typeperf which is great for what I ultimately need to script (an unattended batch job that collects performance metrics).
But in order to develop a list of ALL available counters I wrote a mashup in Perl using the Win32::Perfmon module and calls to Typeperf to discover performance counters related to any particular object. That way, I can top and tail the output to my chosen performance counters and not have to worry about the exact typing of these finicky little buggers.
Read More
