In the meantime, and the purpose of this post, I’d like to re-introduce you to Gridinit.com. In its third iteration (or third re-invention) it serves as a collection of performance and test automation tools, bound together in the cloud and reporting as a single entity. The tools I’m using are:

No surprises there, three stable workhorses that get that job done.

But testing site after site and using early versions of the grid, I realised something was missing. It took a step back into the commercial world of tools to realise what that was. Canned reporting.

I was tired of re-inventing the collection and reporting mechanism for each test effort. Whether that was man-draulic-ally sucking in humongous CSV files, writing exorbitant parsers, or flipping through different charting packages, trust me, I tried them all.

I’m a big fan of templating the test effort, so you can get on with the job of analysing results, rather than worrying about the formatting of them, so I settled on the following tools for the onerous task of data collection and reporting:

Hopefully clean, simple, powerful and ready to paste into any report.

So now I have the means to consistently generate the load, the mechanism to index and search the data generated, and glue, being a simple rails application, to tie it all together. It’s hosted on the cloud, runs on EC2 or any Debian based image making it pretty damn portable and cheap. And best of all, it’s all open source. Yep, no monthly fees, no pay per VU, just try out the demo on Gridinit.com and if you like, fork it, or bootstrap your own image. If there’s enough interest I’ll make a snapshot of the AMI publicly available for EC2 users.

I’ll be tracking issues on github and making regular updates to the site as I use it for more clients. How to use it will be blogged about here. Let me know how you go.

“Its focus is on performance, internationalization, and supporting innovative features that make your life better.”

Amongst its many features, I love the ability to slit a pane horizontally or vertically. (Command-Shift-D or d) It’s great being able to be logged in to multiple backend servers at once.

Another cool feature is the ability to send commands to all panes in a tab (Alt-Command-I), useful when editing multiple files on multiple servers or starting and stopping services etc.

But the coolest hack I reckon is using applescript to start up iTerm windows and sessions within them. I use something like the following to launch my workspace.

launch "iTerm"
 
tell application "iTerm"
 activate
 
 -- my project workspace
 set myterm to (make new terminal)
 tell myterm
  launch session "varnish"
  set number of columns to 100
  set number of rows to 30
  tell the last session to write text "cd Code/myproject && git pull origin master"
 end tell
 
 -- ssh in split panes to my varnish stack
 set myterm to (make new terminal)
 tell myterm
  launch session "varnish"
  set number of columns to 100
  set number of rows to 60
  tell the last session to write text "ssh vara.cbr"
  tell the last session to write text "varnishlog"
  tell i term application "System Events" to keystroke "D" using command down
  tell the last session to write text "ssh varb.cbr"
  tell i term application "System Events" to keystroke "D" using command down
  tell the last session to write text "ssh varc.cbr"
 end tell
 
 -- ssh in split panes to my apache stack
 set myterm to (make new terminal)
 tell myterm
  launch session "appserver"
  set number of columns to 100
  set number of rows to 60
  tell the last session to write text "ssh appa.cbr"
  tell i term application "System Events" to keystroke "D" using command down
  tell the last session to write text "ssh appb.cbr"
  tell i term application "System Events" to keystroke "D" using command down
  tell the last session to write text "ssh appc.cbr"
 end tell
 
end tell

I did try other session management tools such as tmux, but whilst on OSX, am finding iTerm2 + Applescript perfect for my needs. YMMV.

1.1.3 has been released which includes the following changes:

• Improved command line –help and more simple instructions in the Readme. There’s also a mention at watir.com/frameworks
• Deprecated support for -b flag on provider where you used to specify a browser.
  Now just specify -d DRIVER flag to specify which driver you want to provide (watir, firewatir, safariwatir, webdriver or selenium)
• Updated basic cucumber examples
• .control and .iterate methods to yield browser objects so you don’t need to mess around with threads. Your test cases can start looking more like normal Watir code rather than a jungle of threads and remote DRb objects.
• Put the code base on a diet with 400+ lines deleted, and cut out a lot of extraneous ‘features’

On that last point, I’ve decided to keep WatirGrid as slim as possible. This means you won’t see development of extraneous features like grid management tasks such as starting/stopping VMs, killing browser processes and the like. For full fledged ‘grid management’ you should of course check out Gridinit or Selenium Grid.

Gridinit is also getting some complimentary changes. Things to look out for in June include:

• BYOG – bring your own grid, will let you use your own EC2 account, radically reducing costs and making Gridinit one of the most cost effective solutions out there.
• Zombies – currently working on a remote WebDriver server implementation using Node.js and Zombie.js for insanely fast headless browsers. This means you’ll get much more bang for your buck from a single core, and overcome many of the JavaScript woes that affect other headless solutions like HtmlUnit. Zombies will also be available for WatirGrid.

The Gridinit beta has been officially released this weekend. ”Gridinit makes it incredibly simple to control hundreds of browsers from a single test case, with easy configuration and control of your own cloud based grid network.”

You can see a quick overview and video here.

Gridinit supports execution of Watir or Selenium test cases using Ruby in parallel on real browsers across a grid network, allowing you to easily distribute your tests.

Spend Less Time Scripting

Watir and Selenium scripts let you interact with the browser, not at the protocol level. You can use the same scripts from acceptance testing right through to performance testing.

Simply Powerful

Gridinit gives you options to control the grid as you prefer:

• A command line interface available as a gem
• A RESTful web service
• A simple web based interace

Our powerful yet simple API lets you incorporate the grid into your existing test frameworks including RSpec, Capybara, Cucumber and others. Have a look at our examples or write your own!

Up Next

Gridinit is the new kid on the block for cloud based distributed testing, with an eye for the Ruby community. We’ll be adding plenty more examples and integrating with more Ruby test frameworks as we listen to and explore user feedback through the beta. We’re also looking at BYO EC2 accounts, packaged installations for your own private networks, support for htmlunit, webrat and much more …

Learn more or sign up for an account now.

Email us at support@gridinit.com or follow us on twitter @gridinit

This family of tools lets you easily distribute watir test cases onto a grid network, the latter being a commercial implementation on EC2 of the former. I’ll be posting more about Gridinit soon as we enter into our beta!

http://watirpodcast.com/44-tim-koopmans-on-watirgrid-and-gridinit/

1. Document templates: Everything from a test strategy through to detailed plans, application simulation models, status updates or summary reports are all based on a template for me now. A template view of things is also a good way of culling back all the unnecessary guff that you often find in these documents. Unless you’re legally obliged to, cut back on the guff and tell them what you’re planning to test, how you’re going to measure it, and what you’re going to do if things fall outside your hypotheses…

2. Look beyond LoadRunner: I’ve gained much self satisfaction and saved employers large sums along the way by looking at tools other than LoadRunner. You may feel like you’re shafting yourself initially as you fumble about in strange GUIs or delve into the command line /code but in reality, you’re making yourself a better tester by understanding what it is you’re trying to test/simulate, and the multitude of ways you can accomplish it. Time to wean yourself off that expensive teat and start looking at the alternatives.

3. Business intelligence: You need to be able to understand machine events and transform them back into business speak. Most managers don’t give a hoot about number of IO’s per second or the vagaries of GC algorithms. You need to transform this data into a language that you can communicate up the chain. Avoid the FUD (Fear, Uncertainty & Doubt) and instead speak in terms of risk (probability, frequency, impact).

4. Monitor and measure: If you can’t monitor then you probably can’t measure. I’ve heard so many times that “you need this one tool because … ” it pulls everything together in one interface … Really, you need to know what you can measure and monitor, what tools support capture (most platforms have native commands to do so) and how you’re going to pull it all together. Timestamps are your friend (for correlating events) and so too are databases for storage. More often than not I’ve employed some form of RRD or RDMS such as MySql and more recently have been looking at the virtues of NoSQL with things like CouchDB or MongoDB.

5. The web server bone’s connected to the, database bone: You need to understand the stack you’re playing in. Most prod environments don’t give you the luxury of prodding around, but VMs are your friend and you’ll probably find most platforms you’re working on have free or developer versions of their products to play with. Chances are if you’ve never installed and configured it, you’re probably not going to have the fuzziest on where to start debugging or tuning it!

6. Adopt a scripting language: You’ll no doubt need to munge data, fake stubs, mock services, crawl for information and the like. I’ve seen some hilarious examples of this using heavyweight tools such as QTP/LoadRunner and some downright painful (although closer to zen) alternatives in shell script. But honestly, adopt a scripting language like Ruby, Python or even Perl, buy the corresponding cookbook, and you’ll have an answer to just about every housekeeping task you might find yourself faced with.

7. Learn from others: Go on, be humble, get yourself an RSS reader and learn from others. My personal favourites to name a few are:

Scott Barber http://www.perftestplus.com/scott_blog.php
Neil Gunther http://perfdynamics.blogspot.com/
Stuart Moncrieff http://www.myloadtest.com/
Corey Coldberg http://www.goldb.org/
Jared Quinert http://www.software-testing.com.au/blog/
Sam Abdelhamid http://mishmashmoo.com/blog/

8. Write your own blog: Got something to share? Want to improve your writing skills? Go on, write your own blog. I often find that in showing someone else either in person or the written form, I get to understand the subject matter better. So what are you waiting for?

Thanks for listening =)

User latency is an important quality benchmark for Web Applications. While JavaScript-based mechanisms can provide comprehensive instrumentation for user latency measurements within an application, in many cases, they are unable to provide a complete end-to-end latency picture.

To address the need for complete information on user experience, this document introduces the PerformanceTiming interfaces. This interface allows JavaScript mechanisms to provide complete client-side latency measurements within applications. With the proposed interface, the previous example can be modified to measure a user’s perceived page load time.

You can read more about the PerformanceTiming specification here. Essentially this gem collects all available metrics from the browser and summarises them in a format similar to the following diagram:

To use this gem simply install:

 gem install watir-webdriver-performance

And use the browser.performance helper method to access the metrics.

>> require 'watir-webdriver'
>> require 'watir-webdriver-performance'
>> b = Watir::Browser.new :chrome
>> b.goto "watirgrid.com"
=> "https://github.com/90kts/watirgrid"
>> # End user response time was
?> b.performance.summary[:response_time]/1000
=> 8
>> #seconds ...
?> # Time to first byte i.e. "server time" was
?> b.performance.summary[:time_to_first_byte]
=> 3341
>> #milliseconds ...
?> # Summary timings available
{
    :summary => {
                   :redirect=>0,
                  :app_cache=>0,
                        :dns=>0,
            :tcp_connection=>982,
     :tcp_connection_secure=>721,
                  :request=>1222,
                    :response=>4,
           :dom_processing=>4293,
            :response_time=>7298,
       :time_to_first_byte=>2205,
        :time_to_last_byte=>2209
    },
    :navigation => {
                     :type => 0,
        :type_back_forward => 2,
           :redirect_count => 0,
            :type_reserved => 255,
            :type_navigate => 0,
              :type_reload => 1
    },
        :memory => {
        :total_js_heap_size => 0,
        :js_heap_size_limit => 0,
         :used_js_heap_size => 0
    },
        :timing => {
                   :domain_lookup_start => 1303180421599,
                        :load_event_end => 0,
                           :connect_end => 1303180421642,
                          :response_end => 1303180421853,
                           :dom_loading => 1303180421840,
                      :navigation_start => 0,
                          :redirect_end => 0,
                    :unload_event_start => 0,
               :secure_connection_start => 0,
                         :connect_start => 1303180421600,
          :dom_content_loaded_event_end => 1303180421934,
                     :domain_lookup_end => 1303180421600,
                       :dom_interactive => 1303180421934,
                      :load_event_start => 0,
                         :request_start => 1303180421642,
                        :response_start => 1303180421838,
                          :dom_complete => 0,
                           :fetch_start => 1303180421598,
        :dom_content_loaded_event_start => 1303180421934,
                        :redirect_start => 0,
                      :unload_event_end => 0
    }
  }

To use Selenium via WatirGrid …

Get it

<sudo> gem install watirgrid

Start a controller …

controller
I, INFO -- : Controller started on : druby://10.0.1.3:11235

Start a provider with -b selenium …

provider -b selenium
I, INFO -- : Provider started on   : druby://10.0.1.3:11236
I, INFO -- : Controller found on   : druby://10.0.1.3:12358
I, INFO -- : Provider registered   : druby://10.0.1.3:12358

Execute your Selenium code in Ruby!

>> require 'watirgrid'
=> true
>> Watir::Grid.control(
>>   :controller_uri=>'druby://10.0.1.3:11235') do |driver, id|
>>   driver.navigate.to "http://google.com"
>>   element = driver.find_element(:name, 'q')
>>   element.send_keys "Hello WebDriver!"
>>   element.submit
>>   driver.quit
>> end

So what’s the difference between WatirGrid and SeleniumGrid?

They have radically different architectures in terms of establishing a Grid network. WatirGrid is built on distributed programming paradigms such as Linda and is a pure Ruby implementation. SeleniumGrid is perhaps a more heavy weight Java implementation (you can still write your tests in most languages) and packs a lot more features for grid control into its design. Both have their own merits, you be the judge!

Next steps for WatirGrid is to look at implementing some type of Capybara adapter, since this tool (and WebRat) appear to dominate the Ruby test automation space. I’m also working on Gridinit, a commercial implementation on EC2 which is available for beta testing. Sign up and give it a try!

What an awesome experience. Despite spending most of it jet-lagged, travelling to San Francisco was such a great experience for me, and helped reaffirm just how strong the Watir and Selenium community is. Without dropping names, some of the experiences I got to enjoy were:

1. Sharing late night beers and early morning burgers with the leader of Watir development
2. Pair coding on watir-webdriver performance extensions with the smartest kid in Norway
3. Finally meeting perhaps the leader of Australian based Watir testing
4. Eating chinese with the god-father of Watir and hearing some great anecdotes on the origins of Selenium
5. Shown the light on page objects, opera and superbad with a Polish born American
6. Hearing about the inner workings of ‘that other browser’ from a genius at Opera
7. Learning more about the Watir community and getting a thirst for Croatian micro breweries
8. Meeting my role model in person and perhaps feeling a bit shied by his awesomeness!
9. Getting some great advice and tips for my own startup
10. Being awed and perhaps jealous (in a slightly convict vs. colonial way) of the greatest mind behind webdriver

I met many other people whilst there, did tons of late night coding and got to experience public speaking for the first time. A big thank you to all the organisers of #watirday and #seconf. Looking forward to the next!