The last few days have been busy for me as I had the privilege to present WatirGrid at the Watirday conference in San Francisco. You can catch the slide deck here in case you couldn’t make it. WatirGrid lets you test Watir in parallel, why not give it a shot yourself?
<sudo> gem install watirgrid
I’m also attending the Selenium Conference which is giving me lots of great ideas and inspiration in the performance & automation space. It’s great to see a greater shift to open source testing tools and some may argue that QTP and the like perhaps could die a graceful death. It’s also been fun meeting the god-father of Selenium and confirming that his choice of product-name really was intended as an antidote to Mercury!
Read More
A quick poll on wtr-development indicated a number of different applications to achieve such a task, each having their own pros and cons. It really depends on what features appeal to you. My personal experience has led me to try a lot of different editors and development environments in the past including:
Editors
SciTE
NotePad++
TextMate
VIM
Emacs (actually I haven’t used this one)
IDEs
RubyMine
NetBeans
Pros and Cons for each (subject to personal opinion) are as follows.
Read More
Here’s a little dashboard (private for now, email me for a peek at the code) I wrote for my team that we use during our performance tests. There’s so many out there I just can’t help myself from writing my own. The banking application under test naturally had a lot of valuable data that could be extracted from the application database in the form of response times and volumetrics for different payment types. Where we didn’t have enough data, we wrote our own transactions logs within a separate MySQL database.
I used Sinatra on top of JRuby (for stability), jQuery Sparklines to present response time data (green is the non functional requirement, blue are the actuals) and Application Simulation Model data driven from a YAML file, making it easy for the test analysts to maintain.
e.g. ASM YAML
# this is just an example of how to specify
# parent, child, grandchild transaction relationships
:Custom: # parent
:Logon: # child
:Password: # grandchild
:nfr: 5 # response time non functional requirement
:regex: "PW_Login" # regex match from transaction log
:av2011: 1,339 # volumetric target average 2011
:av2012: 1,913 # volumetric target average 2012
:Smartcard:
:nfr: 5
:regex: "SM_Login"
:av2011: 446
:av2012: 638All up we have a simple app that gives us a consistent view of application performance as measured by things like response time and volumetrics against targets (non functional requirements) and actuals. We use commercial tools to drive the load, however through use of simple tools like Ruby you can quickly put a UI onto any form of application performance data.
Read More
For some time now I’ve been working in the background on watirgrid which allows for distributed testing across a grid network using Watir. The engine (gem) itself is open source and you can get it as follows:
One of the questions I often get asked (via email) are what ports / network communications are required by Watirgrid, especially in the setup and establishment of a grid. Watirgrid uses both UDP and TCP to establish comms between the providers and a controller as such. Consider the following bounce diagram.
- First we set up a controller listening on a public WAN IP of 150.133.123.51. We tell the controller that the ring server should listen on this IP with the -h flag. We also tell the controller to start its DRb server on the same IP with the -H flag. If we don’t specify a port for the ring server or the DRb server (using -r and -d flags) then the controller will use the defaults; 12358 and 11235 respectively.
- Next we start up a provider and point it to the Ring server IP of 150.133.123.51 with the -h flag. We also tell the provider to start its DRb server on its own public IP with the -H flag, which in this case is 150.133.123.8
- When the provider starts in the previous step, it will broadcast using UDP to all hosts on the controller’s subnet, on port 12358 asking if there are any Ring servers available.
- The controller will subsequently respond, letting the provider know which port its own DRb server can be be contacted on using TCP. In this case it will send a TCP packet to the provider’s port 11236 after which the provider will respond back via TCP to the controller’s port of 11235.
Essentially the ports you need open on the controller, are UDP (default 12358) which is the Ring server and TCP (default 11235) which is the controller’s DRb server. The ports you need open on the provider are TCP (default 11236) which is the provider’s DRb server.
All communications between controller and provider are first marshalled via the Ring server (using UDP) and then directed between controller and provider(s) via the DRb servers (using TCP). If you have established a grid out in the wild, or on your own intranet, you should be using the external WAN or LAN IP of your host machines (not local / loopback addresses).
I’ll be posting more information on using watirgrid, including how to execute tests via different testing frameworks such as cucumber in the near future. I’ll also make an effort to monitor watir-general for any questions related to watirgrid.
Happy bot’netting!
Read More
Wow, wasn’t that long ago when I was promising things that were harder, better, faster and stronger. I think I got part of the way there. The core reason I set up Altentee was to provide reliable (and potentially cheaper) alternatives to traditional licensed performance testing tool sets. There’s no doubt Altentee can test at the limits of your typical web app using tools that cost zilch. The bigger challenge I’ve found this year is convincing potential clients that they really don’t need to spend that much. I’ve even offered a free load test on our homepage to help illustrate this point.
We were lucky to be selected by the development team at Cordelta to help them automate and performance test a high profile public website called MyHospitals. We were able to test millions of hits per hour from domestic and international locations in a wide variety of load scenarios. The success of this approach was underpinned by the following:
- An open minded project / development team not coupled to a ‘must-have-most-expensive-toolset-to-do-job’ mentality
- An open minded performance test analyst (me!) who believes Excel really is the grandpa of charting, R is the grandma of stats, Sparklines are the only way to present time series data to management, ANY tool can simulate load via HTTP/S and that there is no real distinction between good software testers or developers (only hard work separates the best from the worst).
- That 2010 come-no-doubt 2011 buzz word… Cloud
As I sit here pluggin’ our alternative approach at Altentee, I can see the rise of other more successful punters taking on the big kids. I sit and [continue to] chuckle at the reaction to LR pricing and LR zealots who will fall on their swords over LR itself. I have come to realise one thing though, it is not about the tool, or even the alternate tool like perhaps I first thought. It is more about the freedom of choice.
To tackle MyHospitals I was free to choose and implement the following tools:
- WatirGrid to orchestrate a small flotilla of IE and FireFox based browsers based on Watir
- JMeter to add more at the protocol level of performance testing
- httperf to do some basic benchmarking, similar to my front page
- numbrcrunchr to pull together system metrics and make for easier analysis
- A variety of Australian cloud providers and of course Amazon EC2 to host the test environment
It has been a great year. I’m not entirely free of the commercial chains just yet and am still needing the LR type work to prop up this approach, but I hope 2011 brings about some fresh thinking in performance and test automation with hopefully me somewhere amidst that space.
Read More
