This family of tools lets you easily distribute watir test cases onto a grid network, the latter being a commercial implementation on EC2 of the former. I’ll be posting more about Gridinit soon as we enter into our beta!

http://watirpodcast.com/44-tim-koopmans-on-watirgrid-and-gridinit/

User latency is an important quality benchmark for Web Applications. While JavaScript-based mechanisms can provide comprehensive instrumentation for user latency measurements within an application, in many cases, they are unable to provide a complete end-to-end latency picture.

To address the need for complete information on user experience, this document introduces the PerformanceTiming interfaces. This interface allows JavaScript mechanisms to provide complete client-side latency measurements within applications. With the proposed interface, the previous example can be modified to measure a user’s perceived page load time.

You can read more about the PerformanceTiming specification here. Essentially this gem collects all available metrics from the browser and summarises them in a format similar to the following diagram:

To use this gem simply install:

 gem install watir-webdriver-performance

And use the browser.performance helper method to access the metrics.

>> require 'watir-webdriver'
>> require 'watir-webdriver-performance'
>> b = Watir::Browser.new :chrome
>> b.goto "watirgrid.com"
=> "https://github.com/90kts/watirgrid"
>> # End user response time was
?> b.performance.summary[:response_time]/1000
=> 8
>> #seconds ...
?> # Time to first byte i.e. "server time" was
?> b.performance.summary[:time_to_first_byte]
=> 3341
>> #milliseconds ...
?> # Summary timings available
{
    :summary => {
                   :redirect=>0,
                  :app_cache=>0,
                        :dns=>0,
            :tcp_connection=>982,
     :tcp_connection_secure=>721,
                  :request=>1222,
                    :response=>4,
           :dom_processing=>4293,
            :response_time=>7298,
       :time_to_first_byte=>2205,
        :time_to_last_byte=>2209
    },
    :navigation => {
                     :type => 0,
        :type_back_forward => 2,
           :redirect_count => 0,
            :type_reserved => 255,
            :type_navigate => 0,
              :type_reload => 1
    },
        :memory => {
        :total_js_heap_size => 0,
        :js_heap_size_limit => 0,
         :used_js_heap_size => 0
    },
        :timing => {
                   :domain_lookup_start => 1303180421599,
                        :load_event_end => 0,
                           :connect_end => 1303180421642,
                          :response_end => 1303180421853,
                           :dom_loading => 1303180421840,
                      :navigation_start => 0,
                          :redirect_end => 0,
                    :unload_event_start => 0,
               :secure_connection_start => 0,
                         :connect_start => 1303180421600,
          :dom_content_loaded_event_end => 1303180421934,
                     :domain_lookup_end => 1303180421600,
                       :dom_interactive => 1303180421934,
                      :load_event_start => 0,
                         :request_start => 1303180421642,
                        :response_start => 1303180421838,
                          :dom_complete => 0,
                           :fetch_start => 1303180421598,
        :dom_content_loaded_event_start => 1303180421934,
                        :redirect_start => 0,
                      :unload_event_end => 0
    }
  }

To use Selenium via WatirGrid …

Get it

<sudo> gem install watirgrid

Start a controller …

controller
I, INFO -- : Controller started on : druby://10.0.1.3:11235

Start a provider with -b selenium …

provider -b selenium
I, INFO -- : Provider started on   : druby://10.0.1.3:11236
I, INFO -- : Controller found on   : druby://10.0.1.3:12358
I, INFO -- : Provider registered   : druby://10.0.1.3:12358

Execute your Selenium code in Ruby!

>> require 'watirgrid'
=> true
>> Watir::Grid.control(
>>   :controller_uri=>'druby://10.0.1.3:11235') do |driver, id|
>>   driver.navigate.to "http://google.com"
>>   element = driver.find_element(:name, 'q')
>>   element.send_keys "Hello WebDriver!"
>>   element.submit
>>   driver.quit
>> end

So what’s the difference between WatirGrid and SeleniumGrid?

They have radically different architectures in terms of establishing a Grid network. WatirGrid is built on distributed programming paradigms such as Linda and is a pure Ruby implementation. SeleniumGrid is perhaps a more heavy weight Java implementation (you can still write your tests in most languages) and packs a lot more features for grid control into its design. Both have their own merits, you be the judge!

Next steps for WatirGrid is to look at implementing some type of Capybara adapter, since this tool (and WebRat) appear to dominate the Ruby test automation space. I’m also working on Gridinit, a commercial implementation on EC2 which is available for beta testing. Sign up and give it a try!

What an awesome experience. Despite spending most of it jet-lagged, travelling to San Francisco was such a great experience for me, and helped reaffirm just how strong the Watir and Selenium community is. Without dropping names, some of the experiences I got to enjoy were:

1. Sharing late night beers and early morning burgers with the leader of Watir development
2. Pair coding on watir-webdriver performance extensions with the smartest kid in Norway
3. Finally meeting perhaps the leader of Australian based Watir testing
4. Eating chinese with the god-father of Watir and hearing some great anecdotes on the origins of Selenium
5. Shown the light on page objects, opera and superbad with a Polish born American
6. Hearing about the inner workings of ‘that other browser’ from a genius at Opera
7. Learning more about the Watir community and getting a thirst for Croatian micro breweries
8. Meeting my role model in person and perhaps feeling a bit shied by his awesomeness!
9. Getting some great advice and tips for my own startup
10. Being awed and perhaps jealous (in a slightly convict vs. colonial way) of the greatest mind behind webdriver

I met many other people whilst there, did tons of late night coding and got to experience public speaking for the first time. A big thank you to all the organisers of #watirday and #seconf. Looking forward to the next!

<sudo> gem install watirgrid

I’m also attending the Selenium Conference which is giving me lots of great ideas and inspiration in the performance & automation space. It’s great to see a greater shift to open source testing tools and some may argue that QTP and the like perhaps could die a graceful death. It’s also been fun meeting the god-father of Selenium and confirming that his choice of product-name really was intended as an antidote to Mercury!

Recently I decided to lighten the load with my commercial contract work and contribute / commit more to the open source space. Part of that effort involves further strengthening and stabilising the WatirGrid project, which I’ll definitely keep open source. I’d also like to get more into watir-webdriver as I see this as the future for the Watir project.

The other part (half?) of my time involves getting my own startup GRIDinit.com up and running. The GRID is a commercial implementation of WatirGrid, which simplifies management of a GRID network across different cloud providers. Basically it’s a RESTful API for WatirGrid. I’m presently running a private alpha on EC2 with 50 instances but I hope to open the doors for a public beta with both International (US, EU, Tokyo and Singapore) and Domestic (Australia) cloud providers in May/June! If you’d like to participate send an email to support@gridinit.com

Having a commercial implementation will help pay the bills and let me channel necessary improvements back into WatirGrid itself. If you’re keen you can keep track of the pre-release gems via:

<sudo> gem install watirgrid --pre

A recent improvement is the addition of a GRID control method, to take the somewhat confusing (for first timers) Thread calls out of the picture. For example:

1. Start a controller (from a command line)

controller

2. Start your providers (from a command line) on remote machines (typically same LAN as the controller). It supports all major browsers/platforms including webdriver. I hope to add Opera soon as well.

provider -b webdriver|safari|ie|firefox

3. Control your GRID from your code (the pre-release let’s us contact our controller direct via the controller_uri, no need to broadcast via UDP anymore)

Watir::Grid.control({
  :controller_uri => 'druby://127.0.0.1:12357'}) do |browser, id|
  3.times do |iteration| # because we love to iterate right!?
    browser.goto "http://your.awesome.test.environment"
    sleep 2
  end
  browser.close
end
=>
DRb server started on : druby://127.0.0.1:61325
Controller found on   : druby://127.0.0.1:12357
Found 2 tuples.
Grid size                         : 2
Grid rampup                       : 0.5 secs
Browser 1#2171119800 start         : Tue Apr 05 01:39:03 -0700 2011
Browser 1#2171119800 architecture  : universal-darwin10.0
Browser 1#2171119800 type          : webdriver
Browser 1#2171119800 hostname      : air.local
Browser 2#2172047580 start         : Tue Apr 05 01:39:04 -0700 2011
Browser 2#2172047580 architecture  : universal-darwin10.0
Browser 2#2172047580 type          : webdriver
Browser 2#2172047580 hostname      : air.local
Browser 1#2171119800 stop          : Tue Apr 05 01:39:18 -0700 2011
Browser 1#2171119800 elapsed       : 15 secs
Browser 2#2172047580 stop          : Tue Apr 05 01:39:20 -0700 2011
Browser 2#2172047580 elapsed       : 16 secs

In other words, I’ll be abstracting a lot of the WatirGrid’isms and making it easier for you to wrap in your own test frameworks be they rspec, cukes, plain old ruby or whatever. This is my philosophy when designing GRIDinit. I won’t be dictating how you should run your tests or what you should monitor, I’ll leave that to your control with as few lines as possible required from GRIDinit. Good times ahead!

Still a couple more days left at Selenium Conference, so if you’d like to catch up send us a hoy on Twitter @90kts.

Cheers,
Tim

gem install watirgrid

One of the questions I often get asked (via email) are what ports / network communications are required by Watirgrid, especially in the setup and establishment of a grid. Watirgrid uses both UDP and TCP to establish comms between the providers and a controller as such. Consider the following bounce diagram.

1. First we set up a controller listening on a public WAN IP of 150.133.123.51. We tell the controller that the ring server should listen on this IP with the -h flag. We also tell the controller to start its DRb server on the same IP with the -H flag. If we don’t specify a port for the ring server or the DRb server (using -r and -d flags) then the controller will use the defaults; 12358 and 11235 respectively.
2. Next we start up a provider and point it to the Ring server IP of 150.133.123.51 with the -h flag. We also tell the provider to start its DRb server on its own public IP with the -H flag, which in this case is 150.133.123.8
3. When the provider starts in the previous step, it will broadcast using UDP to all hosts on the controller’s subnet, on port 12358 asking if there are any Ring servers available.
4. The controller will subsequently respond, letting the provider know which port its own DRb server can be be contacted on using TCP. In this case it will send a TCP packet to the provider’s port 11236 after which the provider will respond back via TCP to the controller’s port of 11235.

Essentially the ports you need open on the controller, are UDP (default 12358) which is the Ring server and TCP (default 11235) which is the controller’s DRb server. The ports you need open on the provider are TCP (default 11236) which is the provider’s DRb server.

All communications between controller and provider are first marshalled via the Ring server (using UDP) and then directed between controller and provider(s) via the DRb servers (using TCP). If you have established a grid out in the wild, or on your own intranet, you should be using the external WAN or LAN IP of your host machines (not local / loopback addresses).

I’ll be posting more information on using watirgrid, including how to execute tests via different testing frameworks such as cucumber in the near future. I’ll also make an effort to monitor watir-general for any questions related to watirgrid.

Happy bot’netting!

We were lucky to be selected by the development team at Cordelta to help them automate and performance test a high profile public website called MyHospitals. We were able to test millions of hits per hour from domestic and international locations in a wide variety of load scenarios. The success of this approach was underpinned by the following:

1. An open minded project / development team not coupled to a ‘must-have-most-expensive-toolset-to-do-job’ mentality
2. An open minded performance test analyst (me!) who believes Excel really is the grandpa of charting, R is the grandma of stats, Sparklines are the only way to present time series data to management, ANY tool can simulate load via HTTP/S and that there is no real distinction between good software testers or developers (only hard work separates the best from the worst).
3. That 2010 come-no-doubt 2011 buzz word… Cloud

As I sit here pluggin’ our alternative approach at Altentee, I can see the rise of other more successful punters taking on the big kids. I sit and [continue to] chuckle at the reaction to LR pricing and LR zealots who will fall on their swords over LR itself. I have come to realise one thing though, it is not about the tool, or even the alternate tool like perhaps I first thought. It is more about the freedom of choice.

To tackle MyHospitals I was free to choose and implement the following tools:

1. WatirGrid to orchestrate a small flotilla of IE and FireFox based browsers based on Watir
2. JMeter to add more at the protocol level of performance testing
3. httperf to do some basic benchmarking, similar to my front page
4. numbrcrunchr to pull together system metrics and make for easier analysis
5. A variety of Australian cloud providers and of course Amazon EC2 to host the test environment

It has been a great year. I’m not entirely free of the commercial chains just yet and am still needing the LR type work to prop up this approach, but I hope 2011 brings about some fresh thinking in performance and test automation with hopefully me somewhere amidst that space.

If you want to get on with the business of parallel testing across a grid network with Watir or Watir-WebDriver then have a look at the following code:

Firstly, update your gem installation if it’s been a while

gem update --system

Now install the prerelease version of Watirgrid

gem install watirgrid --prerelease

At this stage I haven’t updated my gemspec with dependencies, so you’ll also need uuid

gem install uuid

To test a simple 1 controller / 1 provider scenario using IE

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

require 'rubygems'
require 'watirgrid'
 
# setup a controller on port 12351 for your new grid
controller = Controller.new(
        :ring_server_port => 12351,
  :loglevel => Logger::ERROR)
controller.start
 
# add a provider to your grid
# :browser_type => 'webdriver' if using webdriver or
# :browser_type => 'ie' if using watir...
provider = Provider.new(
        :ring_server_port => 12351,
  :loglevel => Logger::ERROR, :browser_type => 'webdriver')
provider.start
 
# connect to the grid and take all providers from it (this time only one)
grid = Watir::Grid.new(:ring_server_port => 12351,
    :ring_server_host => '127.0.0.1')
 
grid.start(:take_all => true)
 
# for each provider on the grid, launch a new thread to start multiple browsers
threads = []
grid.browsers.each do |browser|
        threads << Thread.new do
                p browser[:hostname]
                p browser[:architecture]
                p browser[:browser_type]
                # in this case we are starting a new IE browser
                b = browser[:object].new_browser(:ie)
                b.goto("http://www.google.com")
                b.text_field(:name, 'q').set("watirgrid")
                b.button(:name, "btnI").click
        end
end
threads.each {|thread| thread.join}

There’s some more examples of using watirgrid with watir-webdriver here.

The problem here is that you cannot auto authenticate to a proxy server using Chrome –proxy-server=my.proxy.com:3128 or using Firefox.

Chrome will always prompt you for a user name and password on an authenticating proxy when it starts up (via Webdriver or manually). Firefox will not prompt (when launched via WebDriver) and will silently fail.

I *almost* got this to work by modifying headers on the fly with a Firefox add on (Modify Headers). Basic authentication is just base-64 encoded so you can force each request to authorize itself by modifying the Proxy-Authorization request header e.g.:
Proxy-Authorization Basic asHJksaKHs87akhkjah7

However whilst this works for manually launched instances of Firefox, Webdriver launched instances would complain that the add on is not compatible with FF3.6.3 and so once again would fail. The other problem with this approach is that I couldn’t find a similar plugin for Chrome.

Eventually I gave in and went with a slightly more complicated workaround. That is, chaining an authenticated proxy (with squid) running on localhost with a cache_peer relationship to the authenticated proxy running out there on the ‘tubes’. I had thought about using Webrick as a proxy (as per previous posts) but lack of SSL support is a gotcha for me. Squid seemed most appropriate in this case:

To install squid from source:
cd /opt/src
sudo curl -O http://www.squid-cache.org/Versions/v2/2.7/squid-2.7.STABLE9.tar.gz
sudo tar xjf squid-2.7.STABLE9.tar.gz
cd squid-2.7.STABLE9
# I wanted SSL support for this
./configure --enable-ssl
make
sudo make install

Edit the squid.conf (location is normally /etc/squid/squid.conf on linux):
sudo vim /usr/local/squid/etc/squid.conf

# Don't want to run squid as root
# TAG: cache_effective_user
cache_effective_user nobody
# TAG: cache_effective_group
cache_effective_group wheel
# Point this proxy to a peer cache, and provide the logon credentials
# TAG: cache_peer
cache_peer proxy.altentee.com parent 3128 0 default no-query login=username:mypassword
never_direct allow all
# Allow access from my local private network
# TAG: cache_peer
acl localnet src 10.0.0.0/0

Save changes and change ownership as appropriate:
sudo chown -R nobody:wheel /usr/local/squid

Initialize the cache and startup squid:
sudo /usr/local/squid/sbin/squid -z
sudo /usr/local/squid/sbin/squid -D

Now you can configure firefox (using -ProfileManager) or chrome (using –proxy-server=127.0.0.1) to point to your local proxy server. This proxy will then authenticate on your behalf and peer all requests via the original parent.

A little long winded but achieves the outcome. Added side benefit is you’ve now got a local proxy (with access logs) that you can start doing interesting things with in your automation / performance tests like modifying headers and so forth.

Hope this helps!

Part of my grievance for that was based on frustration in dealing with different load testing platforms. Not frustration born out of incompetence, more the frustration born out writing tests for increasingly complex browsers with an antiquated toolset.

That’s not to say that protocol based application load testing (or the languages they sit on) aren’t powerful in their own right. LoadRunner, JMeter et al. has their place in the load testing world. But I would like to relegate them to the background as useful, sometimes expensive tools to spin up the wheels so to speak. Nothing like a bit of JMeter infused fun to get the wheels smoking on your system under test. But for the real part, measuring user experience on the web platform?

If you can’t wait to get started then I’d like to introduce BrowserMob.com. Been around since mid 2008 and offers real browser testing using Selenium scripts distributed across a grid of EC2 computers. Reasonably priced, well documented and supported by the legendary Patrick Lightbody, founder of things like OpenQA and Selenium Remote Control.

But hang on a second, you don’t like Selenium right? Have zero budget? Love Watir? Well I’m working on a similar construct aptly called WatirGrid which also allows for distributed testing across a grid network using Watir. The engine (gem) itself is open source and you can get it as follows:

gem install watirgrid

If you’d like to contribute please follow the WatirGrid project on github.

The Basics
WatirGrid is built on Rinda which implements the Linda distributed computing paradigm in Ruby. According to Wikipedia: “Linda is a model of coordination and communication among several parallel processes operating upon objects stored in and retrieved from shared, virtual, associative memory.”

In other words, WatirGrid allows multiple parallel processes to provide remote Watir objects in the form of tuple spaces across a grid network. This grid network is controlled by a ring server and looks a bit like this:

Key Terminology
The controller implements a repository of tuples (tuple space) that can be accessed concurrently. The controller hosts the ring server which advertises these tuples across a grid network. Typically you will host one controller on a central machine.

The providers make remote Watir objects available to the tuple space hosted by the ring server. Typically you will host one or many providers on your grid network, for example, each PC may become a single provider of a Watir tuple in the form of an Internet Explorer, Firefox or Safari browser object.

Simple Example
Pick a server to host the controller for the ring server and execute the following:

  $ controller

This should find your external facing IP and start the corresponding DRb and Ring Servers:

  DRb server started on : druby://143.238.105.61:11235
  Ring server started on: druby://143.238.105.61:12358

On each client PC, host the provider for the distributed (DRb) Watir objects

  $  provider

This should find the recently started Ring Server and register a tuple for the Watir object:

  DRb server started on : druby://143.238.105.61:11236
  Ring server found on  : druby://143.238.105.61:12358
  New tuple registered  : druby://143.238.105.61:12358

You will now be able to execute commands across remote browsers on your grid network.
e.g.

  grid = Watir::Grid.new(:ring_server_port => 12358)
  grid.start(:quantity => 1, :read_all => true)
  threads = []
  grid.browsers.each do |browser|
    threads << Thread.new do
      b = browser[:object].new_browser
      b.goto("http://www.google.com")
      b.text_field(:name, 'q').set("watirgrid")
      b.button(:name, "btnI").click
    end
  end
  threads.each {|thread| thread.join}

You may wish to host the controller and providers on different machines in a real scenario. You can specify things like server hostnames, ports and access control lists for each of the different types of servers. For more help see:

  $  controller --help
  Usage: controller [options]
  Specific options:
      -H, --drb-server-host HOST       Specify DRb Server interface to host on
      -d, --drb-server-port PORT       Specify DRb Server port to listen on
      -h, --ring-server-host HOST      Specify Ring Server interface to host on
      -r, --ring-server-port PORT      Specify Ring Server port to listen on
      -a, --access-control-list ACLS   Specify a comma separated Access Control List
      -l, --log-level LEVEL            Specify log level {DEBUG|INFO|ERROR}
          --help                       Show this message
 
  $  provider --help
  Usage: provider [options]
  Specific options:
      -H, --drb-server-host HOST       Specify DRb Server interface to host on
      -d, --drb-server-port PORT       Specify DRb Server port to listen on
      -h, --ring-server-host HOST      Specify Ring Server host to connect to
      -r, --ring-server-port PORT      Specify Ring Server port to broadcast on
      -b, --browser-type TYPE          Specify browser type to register {ie|firefox|safari}
      -a, --access-control-list ACLS   Specify a comma separated Access Control List
      -l, --log-level LEVEL            Specify log level {DEBUG|INFO|ERROR}
          --help                       Show this message

Need more help?

Keep an eye on the wiki and the rspec test cases!