Gridin’it again

Mar 26 12 under Altentee, Gridinit | 2 Comments

It’s been a while and I’ve been reasonably busy starting a new family and getting on with Altentee client work. I’m still galavanting about in the commercial performance testing space working on all sorts of target systems with a mixed bag of commercial and open source tools.

In the meantime, and the purpose of this post, I’d like to re-introduce you to Gridinit.com. In its third iteration (or third re-invention) it serves as a collection of performance and test automation tools, bound together in the cloud and reporting as a single entity. The tools I’m using are:

JMeter for high volume protocol based load testing. I favour this tool because it allows for more complex test design (emulate user cache, session cookies, header manipulation, conditional logic) whilst still being able to achieve reasonably high concurrency and throughput from a single machine.
Siege which is basically a high volume url basher. Sometimes it’s easier just to whack a front page and associated resources to get a very basic indication of performance. Tools such as Siege are good for this high volume type load. Soon to be added will be Httperf, which is of a similar vein, but also obeys things like TCP keepalives so you can also test high concurrency as well as volume targets.
Watir WebDriver for browser based testing. I always keep a small pool of real browsers running under load, as you’d be surprised what types of issues this control sample can weasel out. This is where the first iteration of Gridinit was born, until I accepted that browser based load testing is not economically viable when executed without some form of protocol based load to support it. I’m also looking at some more efficient full browser (but headless) implementations, particularly around phantomjs.

No surprises there, three stable workhorses that get that job done.

But testing site after site and using early versions of the grid, I realised something was missing. It took a step back into the commercial world of tools to realise what that was. Canned reporting.

I was tired of re-inventing the collection and reporting mechanism for each test effort. Whether that was man-draulic-ally sucking in humongous CSV files, writing exorbitant parsers, or flipping through different charting packages, trust me, I tried them all.

I’m a big fan of templating the test effort, so you can get on with the job of analysing results, rather than worrying about the formatting of them, so I settled on the following tools for the onerous task of data collection and reporting:

Logstash to munge and grok pretty much any log format and ship them off to grid central via Zero MQ. This tool is awesome. Available as a jar file, I use its agents to tail all the logs of the different performance and test automation tools, grok their results into a useable format and store them in elasticsearch.
Elasticsearch just makes sense really. I have all this disparite data, seemingly impossible to index yet elasticsearch lets me index using JSON over HTTP, scale out to the near horizon and search in real time. You just have to use it to see what I mean.

The results look something like this …

Hopefully clean, simple, powerful and ready to paste into any report.

So now I have the means to consistently generate the load, the mechanism to index and search the data generated, and glue, being a simple rails application, to tie it all together. It’s hosted on the cloud, runs on EC2 or any Debian based image making it pretty damn portable and cheap. And best of all, it’s all open source. Yep, no monthly fees, no pay per VU, just try out the demo on Gridinit.com and if you like, fork it, or bootstrap your own image. If there’s enough interest I’ll make a snapshot of the AMI publicly available for EC2 users.

I’ll be tracking issues on github and making regular updates to the site as I use it for more clients. How to use it will be blogged about here. Let me know how you go.

Visualizing Web Performance

Nov 4 11 under Altentee | Comments Off

This really is a cool info graphic from strangeloopnetworks.com …

Split panes and ‘remembered’ sessions in iTerm2

Sep 29 11 under Altentee | Comments Off

If you’re a mac user and frequently use the terminal, then you should probably have a look at iTerm2.

“Its focus is on performance, internationalization, and supporting innovative features that make your life better.”

Amongst its many features, I love the ability to slit a pane horizontally or vertically. (Command-Shift-D or d) It’s great being able to be logged in to multiple backend servers at once.

Another cool feature is the ability to send commands to all panes in a tab (Alt-Command-I), useful when editing multiple files on multiple servers or starting and stopping services etc.

But the coolest hack I reckon is using applescript to start up iTerm windows and sessions within them. I use something like the following to launch my workspace.

launch "iTerm"
 
tell application "iTerm"
 activate
 
 -- my project workspace
 set myterm to (make new terminal)
 tell myterm
  launch session "varnish"
  set number of columns to 100
  set number of rows to 30
  tell the last session to write text "cd Code/myproject &amp;&amp; git pull origin master"
 end tell
 
 -- ssh in split panes to my varnish stack
 set myterm to (make new terminal)
 tell myterm
  launch session "varnish"
  set number of columns to 100
  set number of rows to 60
  tell the last session to write text "ssh vara.cbr"
  tell the last session to write text "varnishlog"
  tell i term application "System Events" to keystroke "D" using command down
  tell the last session to write text "ssh varb.cbr"
  tell i term application "System Events" to keystroke "D" using command down
  tell the last session to write text "ssh varc.cbr"
 end tell
 
 -- ssh in split panes to my apache stack
 set myterm to (make new terminal)
 tell myterm
  launch session "appserver"
  set number of columns to 100
  set number of rows to 60
  tell the last session to write text "ssh appa.cbr"
  tell i term application "System Events" to keystroke "D" using command down
  tell the last session to write text "ssh appb.cbr"
  tell i term application "System Events" to keystroke "D" using command down
  tell the last session to write text "ssh appc.cbr"
 end tell
 
end tell

I did try other session management tools such as tmux, but whilst on OSX, am finding iTerm2 + Applescript perfect for my needs. YMMV.

WatirGrid 1.1.3 Released

May 26 11 under Altentee, Gridinit | Comments Off

WatirGrid just passed 3,400 downloads. Help it get to 4,000 downloads in June 2011 and Altentee will donate a flock of 10 sheep to a respectable cause.

1.1.3 has been released which includes the following changes:

Improved command line –help and more simple instructions in the Readme. There’s also a mention at watir.com/frameworks
Deprecated support for -b flag on provider where you used to specify a browser.
Now just specify -d DRIVER flag to specify which driver you want to provide (watir, firewatir, safariwatir, webdriver or selenium)
Updated basic cucumber examples
.control and .iterate methods to yield browser objects so you don’t need to mess around with threads. Your test cases can start looking more like normal Watir code rather than a jungle of threads and remote DRb objects.
Put the code base on a diet with 400+ lines deleted, and cut out a lot of extraneous ‘features’

On that last point, I’ve decided to keep WatirGrid as slim as possible. This means you won’t see development of extraneous features like grid management tasks such as starting/stopping VMs, killing browser processes and the like. For full fledged ‘grid management’ you should of course check out Gridinit or Selenium Grid.

Gridinit is also getting some complimentary changes. Things to look out for in June include:

BYOG – bring your own grid, will let you use your own EC2 account, radically reducing costs and making Gridinit one of the most cost effective solutions out there.
Zombies – currently working on a remote WebDriver server implementation using Node.js and Zombie.js for insanely fast headless browsers. This means you’ll get much more bang for your buck from a single core, and overcome many of the JavaScript woes that affect other headless solutions like HtmlUnit. Zombies will also be available for WatirGrid.

Gridinit Beta Released

May 14 11 under Altentee, Gridinit | Comments Off

The Gridinit beta has been officially released this weekend. ”Gridinit makes it incredibly simple to control hundreds of browsers from a single test case, with easy configuration and control of your own cloud based grid network.”

You can see a quick overview and video here.

Gridinit supports execution of Watir or Selenium test cases using Ruby in parallel on real browsers across a grid network, allowing you to easily distribute your tests.

Spend Less Time Scripting

Watir and Selenium scripts let you interact with the browser, not at the protocol level. You can use the same scripts from acceptance testing right through to performance testing.

Simply Powerful

Gridinit gives you options to control the grid as you prefer:

A command line interface available as a gem
A RESTful web service
A simple web based interace

Our powerful yet simple API lets you incorporate the grid into your existing test frameworks including RSpec, Capybara, Cucumber and others. Have a look at our examples or write your own!

Up Next

Gridinit is the new kid on the block for cloud based distributed testing, with an eye for the Ruby community. We’ll be adding plenty more examples and integrating with more Ruby test frameworks as we listen to and explore user feedback through the beta. We’re also looking at BYO EC2 accounts, packaged installations for your own private networks, support for htmlunit, webrat and much more …

Learn more or sign up for an account now.

Email us at support@gridinit.com or follow us on twitter @gridinit