Wow, wasn’t that long ago when I was promising things that were harder, better, faster and stronger. I think I got part of the way there. The core reason I set up Altentee was to provide reliable (and potentially cheaper) alternatives to traditional licensed performance testing tool sets. There’s no doubt Altentee can test at the limits of your typical web app using tools that cost zilch. The bigger challenge I’ve found this year is convincing potential clients that they really don’t need to spend that much. I’ve even offered a free load test on our homepage to help illustrate this point.
We were lucky to be selected by the development team at Cordelta to help them automate and performance test a high profile public website called MyHospitals. We were able to test millions of hits per hour from domestic and international locations in a wide variety of load scenarios. The success of this approach was underpinned by the following:
- An open minded project / development team not coupled to a ‘must-have-most-expensive-toolset-to-do-job’ mentality
- An open minded performance test analyst (me!) who believes Excel really is the grandpa of charting, R is the grandma of stats, Sparklines are the only way to present time series data to management, ANY tool can simulate load via HTTP/S and that there is no real distinction between good software testers or developers (only hard work separates the best from the worst).
- That 2010 come-no-doubt 2011 buzz word… Cloud
As I sit here pluggin’ our alternative approach at Altentee, I can see the rise of other more successful punters taking on the big kids. I sit and [continue to] chuckle at the reaction to LR pricing and LR zealots who will fall on their swords over LR itself. I have come to realise one thing though, it is not about the tool, or even the alternate tool like perhaps I first thought. It is more about the freedom of choice.
To tackle MyHospitals I was free to choose and implement the following tools:
- WatirGrid to orchestrate a small flotilla of IE and FireFox based browsers based on Watir
- JMeter to add more at the protocol level of performance testing
- httperf to do some basic benchmarking, similar to my front page
- numbrcrunchr to pull together system metrics and make for easier analysis
- A variety of Australian cloud providers and of course Amazon EC2 to host the test environment
It has been a great year. I’m not entirely free of the commercial chains just yet and am still needing the LR type work to prop up this approach, but I hope 2011 brings about some fresh thinking in performance and test automation with hopefully me somewhere amidst that space.
Read More
I was going to write why LoadRunner does not work for the web as it is today and into the foreseeable future, but decided against that because I don’t think the following points are necessarily limited to a specific toolset.
Load testing does not work for web as it is today primarily because…
Read More
Confused about the subtle differences between a stress test, performance test or a load test? Don’t get lost in the technobabble! Altentee uses the following terminology to help describe the different types of tests we conduct:
- Shakeout – single script, single user, low volume scenarios whose main purpose is to confirm a correctly configured test harness (script, parameters, data, correlation, iterations) and environment.
- Baseline – arbitrary user/volumes (typically low with multiple iterations) used to establish a baseline or reference point for further testing. Can also be used to explore/set SLAs if they have not been defined by the business using minimum sample sizes appropriate to establish a baseline.
- Load / Volume – often expressed in terms of intended production load at 100% or variations of depending on the desired test outcome. Can also incorporate growth scenarios e.g. 200%. Can also be structured to incrementally increase and sustain load up to a target so that utilization metrics can be collected for capacity modelling at each increment (e.g. ramp up from 100 to 200 users at increments of 20 additional users, with sustained execution of 20 minutes at each increment).
- Stress / Stress to Break – loads greater than intended production loads or with the specific purpose to identify component failure or ‘break’ points. Other variants include Surge / Spike testing simulating peak demands over shorter timeframes.
- Soak / Endurance – long running tests to establish performance over longer timeframes and any anomalies over time.
- Failover / Availability – targeted tests under load in failover conditions. Can also be used to test disaster recovery or availability scenarios.
- Component Based – Targeted tests designed to isolate and examine components (functional and/or specific technologies/platforms) under load.
- Penetration / Security – with the increasing inclusion of security related non-functional requirements, targeted load tests which focus on application level and/or hardware level security NFRs.
- Tuning – cyclic / iterative performance testing using any number of variables in a process of test, tune, retest and compare (for improvement).
Still confused? Contact Altentee and we’ll help explain our approach.
Read More
I recently attended the 2008 conference for Continuous Integration in Melbourne, and mixed with like minded professionals involved with all aspects of CI and testing in general.
In short this is a great (free) opportunity to attend a conference using an open session format. By that I mean the conference is run/organized like a wiki, where the attendees nominate topics they’d like to discuss or facilitate and then a user vote system organizes and makes it happen.
I avoided the performance testing oriented sessions and went for sessions that spoke about exploratory testing, defect management, benefits of test driven development, skills required for test automation and the last one titled ‘do we still need testers?’ …
They all generated a healthy amount of discussion and it was great to hear other people’s experienced opinions on the subject matter. It was also a great opportunity to network, find out who’s working on similar problems in the local industry and from afar. There was a healthy interest in using JMeter as a performance test tool to support CI, so I’ll be posting some more targeted blogs about this in the near future, as well as some consolidated info on performance metrics collection (without SiteScope) which a few people were also interested in.
Anyway, if you haven’t heard about the conference, keep an eye out for next years version. It’s well worth the zero dollar commitment, free beer and t-shirt. All you have to do is ‘use your feet’.
=)
Read More
Went to the third RailsCamp held in a scout hall near Gosford NSW last weekend; my brain has just about got back to normal after 3 solid days of drinking and coding =)
RailsCamp is a get together of mostly Rails developers, but also a fairly eclectic mix of Ruby hackers and geeks in general. 3 nights of drinking, camping and coding with NO internet. Possibly the hardest habit to kick is to stop checking for emails every 5 minutes.
For those with harder to kick habits such as Twittering, there was a ‘Twetter’ clone amongst other cool apps such as the ‘Duke’. The Duke was a user vote playlist driver for winamp (I think) and I got to observe just how close the geek community is when it comes to music tastes. Daft Punk was on fairly high rotation. Of mention was the first time we got Rick Rolled, which then led to a flurry of hack attempts at the guy hosting it on his Mac.
Guitar Hero was in force, as was Urban Terror and a touch of BZFlag. Nice.
Accommodation, setup and food was awesome, the organisers really did do a good job in getting this together. Big thanks to those guys! Perhaps the best part was the ability to network with like-minded people and discuss all things related to Ruby or Rails in a collaborative manner. I managed to get a little work done relating to a test framework for use with FireWatir which I’ll post later.
IF you didn’t or couldn’t make it, keep an eye out for the next one to occur. Definitely worth the $100 to nerd on for a full weekend …
Read More
