Altentee » Performance & Test Automation Experts » tools

Instant Website Tests with BrowserMob

Tim Koopmans — Wed, 06 Jan 2010 00:40:28 +0000

Can you afford a slow landing page?

Google AdWords page quality criteria are many, a prominent check being your landing page load time. A slow load time can lead to a lower quality score.

Make sure you’re not wasting effort on advertising campaigns. Altentee uses BrowserMob to help check website performance for clients from multiple locations. You can run a free check using the form below.

If you need help optimizing your landing page, or help with performance testing in general feel free to contact Altentee today.

My Favourite SCiTE settings …

Tim Koopmans — Tue, 22 Dec 2009 20:05:16 +0000

Well the subject goes without saying. I’ve been working on WinXP again lately and when I’m not around my favourite text editor on the mac (TextMate) I opt for SCiTE on windows. The following settings I find useful for this simple but powerful text editor. YMMV.

# auto complete word autocompleteword.automatic=1

# advanced find and replace find.replace.advanced=1

# force monospaced fonts font.base=$(font.monospace) font.small=$(font.monospace) font.comment=$(font.monospace) font.text=$(font.monospace) font.text.comment=$(font.monospace) font.embedded.base=$(font.monospace) font.embedded.comment=$(font.monospace) font.vbs=$(font.monospace) font.monospace=font:DejaVu Sans Mono,size:8


# split horizontal instead of vertical

split.vertical=0
# Wrapping of long lines

wrap=1
# one instance of SciTE only

check.if.already.open=1
# save on deactivate

save.on.deactivate=1
# full path in title

title.full.path=1
# indentation

tabsize=2

indent.size=2

use.tabs=1
# max number of tabs

buffers=20
# reload file and prompt on activation

load.on.activate=1
# prompt me for any suspicous action

are.you.sure.on.reload=1

# show the status bar statusbar.visible=1

Why Load Testing Does Not Work for the Web `Date::today >> 36`

Tim Koopmans — Sun, 18 Oct 2009 21:20:50 +0000

I was going to write why LoadRunner does not work for the web as it is today and into the foreseeable future, but decided against that because I don’t think the following points are necessarily limited to a specific toolset.

Load testing does not work for web as it is today primarily because…

1. Load testing tool languages are not flexible enough.
Load testing tools are in need of a major overhaul. Consider LoadRunner, the majority of your scripts will be specified in C or proprietary API calls such as lr_ statements. String manipulation for example, whilst possible is extremely clunky and requires oodles of C code. Lack of proper regex or xpath support limits script flexibility in terms of parsing the necessary contents of web page and ajax calls. Don’t even get me started on ajax! Similar complaints for JMeter. Although better support for regex you can really only edit test plans via the GUI. You’ll go cross-eyed looking at test plans in xml. I think the Grinder is the closest product I’ve seen that supports flexibility as you can write tests in Jython which provides more flexibility in terms of scripting.

2. We’re still stuck in record and playback mode.
We really haven’t advanced like our automation brethren. Part of the big sell by commercial load testing companies is “Look it’s easy, just hit the red button and play it back when you’re finished!”. Bollocks to that, we know that never works. A dream load testing tool would support the ability to hand-code your statements and be editable in a text editor! In a similar fashion to watir.

The dream tool would look like:

vusers.start(100)
  browser.goto('url')
  browser.button(:text, /Continue/).click
end

Instead load testing scripts in LoadRunner look something like this:

web_submit_data("transaction",
"Action=https://someurl/I/have/to/keep/correlating/damn/it",
"Method=POST"
"RecContentType="text/html",
"Mode=HTML",
"ITEMDATA,
"Name=L_3_0_0_1_0_1", "Value={another_value_I_have_to_correlate}",
"Name=L_3_0_0_2_0_1", "Value={maybe_paramatize_this_one}",
ENDITEM,
LAST);

We are of course talking about very different technologies. Watir automation scripts driving real browsers with win32ole, whereas most load testing tools are injecting traffic at the protocol layer. If a typical end to end scenario has 30 steps, this can amount to some 2000+ lines of code thanks to the extra verbosity of having to specify all the form values and so forth. This also means you have oodles of code to correlate and parametize. It’s not something you want to hand code, the quickest path is to record (unfortunately). BrowserMob is the closest online provider I’ve seen that overcomes this, supporting load tests written as selenium scripts. Problem is, we need a solution for all those non-public facing companies, the bulk of where I do my work at!

3. No real code versioning.
Changes to the UI will break your script! One can implement their own code versioning as I have recently begun to do with git but there really should be built in support for this. That aside, you revision your code but effectively blast it away every time you hit the red record button. Reason? Each recording can look vastly different to the last. It only takes one form value L_3_0_0_1_0_1 to change to L_3_0_0_1_0_2 or one block of code executed by an ajax call to shift to cause mass confusion. Versioning your code *may* pick this change up, but only if you got the power of the diff! Diffing just got a little harder. Shoot back 10 years ago and diffing for correlation would have been a cinch. Wind forward to today with hundreds of ajax or json calls and correlation can be a nightmare. Windiff struggles. Daisydiff does a much better job highlighting differences by line and by column. This approach kind of works in a waterfall test environment. Throw in a twice weekly code drop in your performance test environment and watch your scripts break. Agile anyone?

So if I was an uber-elite programmer what would I do to move loadtesting into the current decade?
1. Develop a load-testing tool that is based on a flexible scripting language such as Ruby.
2. Get rid of the record button and provide free text editors to the world!
3. Provide a framework that can be versioned and easily detect changes in the system under test.

Vim Cheat Sheets on a Mug

Tim Koopmans — Fri, 05 Jun 2009 07:39:27 +0000

I’ve been thinking about some promotional material for the company I work for. What inspired me was seeing a vi coffee mug… Unfortunately only available in the US. I’ve been endeavouring to wean myself off textmate and the like so thought this was a brilliant idea. But with limited space, what essential commands should go on a vim cheat sheet? What other cheat sheets would you like to see in this format?

:e filename Open a new file. :w filename Save changes to a file. :q Quit Vim. :q! Exit Vim without saving changes. :wq Write the file and exit. Exits without writing the file. j Move the cursor up one line. k Down one line. l Right one character. h Left one character. e To the end of a word. E To the end of a whitespace-delimited word. b To the beginning of a word. B To the beginning of a whitespace-delimited word. 0 To the beginning of a line. ^ To the first non-whitespace character of a line. $ To the end of a line. H To the first line of the screen. M To the middle line of the screen. L To the the last line of the screen. :n Jump to line number n. i Insert before cursor. I Insert to the start of the current line. a Append after cursor. A Append to the end of the current line. o Open a new line below and insert. O Open a new line above and insert. C Change the rest of the current line. r Overwrite one character. R Enter insert mode but replace characters. x Delete characters under the cursor. X Delete characters before the cursor. dd Delete the current line. v Start highlighting characters. V Start highlighting lines. ~ Change the case of characters. > (V) Shift right. < (V) Shift left. c (V) Change the highlighted text. y (V) Yank the highlighted text. d (V) Delete the highlighted text. yy Yank the current line. p Put characters after the cursor. P Put characters before the cursor. u Undo the last action. U Undo all the latest changes. Ctrl + r Redo. /pattern Search the file for pattern. n Scan for next search match N Scan for next search match opposite direction. :%s/foo/bar/a Substitute foo with bar.

Sparklines

Tim Koopmans — Fri, 05 Jun 2009 07:17:45 +0000

“Sparklines: Intense, Simple, Word-Sized Graphics”

If you’ve been reading this blog for sometime, you will know I’ve got a certain penchant for sparklines, a term coined by Edward Tufte and referred to in his book titled “Beautiful Evidence”

True sparklines are meant to be word-like graphics, such that they fit inline with the body of the text, however I’ve had difficulty implementing this with tools such as Excel or custom chart libraries like JFreeChart. The Google Chart API makes this relatively simple.

To that end, my choice of implementation is to present inline with a paragraph. Say for example I wanted to display the amount of jobs advertised over 4 months, then my version of a sparkline would look like this:

Sure, it’s pixelated, the colours are a little hard to differentiate, and the vertical axis is lacking a scale or unit of measurement. However this simple column chart does convey quite succinctly, when jobs are advertised, whether it is the start, middle of end of a month, and how many jobs are available per month.

So what makes a perfect sparkline? To be honest, I don’t ever think sparklines should be used as a supplement for detail. I believe sparklines are best used in situations where you want to convey a lot of data in a small space. They really should be thought of as triggers, not from which you conduct a detailed analysis, but something which triggers the thought to pursue more information in the first place.

Here’s another example recently used to show size of allocation failures on an IBM JVM heap.

Cool? Go get em: http://en.wikipedia.org/wiki/Sparkline

Gantt Charts in Open Office

Tim Koopmans — Tue, 24 Mar 2009 09:51:41 +0000

This might help you out when analyzing time series data.

In this particular situation I’m looking at a bunch of SOA web services that get called synchronously in the backend infrastructure. The logs have entries with a unix timestamp, service description and duration. What I want to do is view the data in a Gantt Chart format. This link from Microsoft put me on the right track.

Here’s how to do it in Open Office.

The original data highlighted in green has a unix timestamp, service name and duration in milliseconds.

I’ve inserted some additional columns as follows:
elapsed_secs = timestamp – minimum of timestamp column
duration_secs = duration_msecs / 1000

Then I’ve created a chart using columns B, C & D as the data source i.e. B1:E10
Create a stacked horizontal bar chart, remove the line colour and fill for elapsed seconds data series and there you have it. A nice little gantt chart demonstrating when services were called and a visual indication of their duration.

I know it’s really simple, but often the simple things demonstrate a point better than bucket loads of data. Here’s an ODS and XLS template.

Microsoft Log Parser to the Rescue

Tim Koopmans — Wed, 04 Mar 2009 19:15:09 +0000

I hate to say it… but Microsoft really have produced an excellent tool for sysadmins and performance testers alike. That tool is called LogParser. I’ve mentioned it before but that was more IIS specific. LogParser can be pointed at just about any type of log file. So with a bit of Perl (or equivalent) to cleanup, and LogParser I was able to analyze Gigabytes of IBM WebSEAL access logs on a production system.

Go to my fileshare at http://90kts.com/uploads and download/install the following files, or google for the latest binaries, using the defaults.
LogParser.msi gzip-1.3.12-1-setup.exe

Create a working directory e.g. C:\working
Copy the parser.pl to your working directory

parser.pl code looks like this:

open(IN, "<$ARGV[0]");
open(OUT, ">processed.csv");
 
print OUT "host, user, datetime, method, file, http, code, bytes";
while()
{
  my($line) = $_;
  chomp($line);
  $line =~ s/\[|\]|\+1100|"//g;
  $line =~ s/\s+/,/g;
  $line =~ s/2009:/2009 /g;
  $line =~ s/,\-,/,/g;
  $line =~ s/,$//g;
 
  print OUT "$line\n";
}
close(IN);
close(OUT);

Copy a gz’d log fie to the working directory and open a cmd shell there
WebSEAL logs look a little like this:
10.1.2.3 - TKOOPS [05/Feb/2009:06:54:11 +1100] "GET HTTP://productionsite.com/wps/CacheProxyServlet/colorPalette/default/browserVendor/Microsoft/browserName/Internet+Explorer/browserVersion/6.0/locale/en/forwardurl/wps/themes/html/images/portlet-table-heading-bg-right.gif HTTP/1.1" 304 0

Run the following commands:
gunzip request.log.2009-02-06-06-59-36_DIR.gz parser.pl request.log.2009-02-06-06-59-36_DIR

To view the number of requests grouped by 5 minute intervals and HTTP result code run this from the cmd shell:
"C:\Program Files\Log Parser 2.2\LogParser.exe" "SELECT QUANTIZE(datetime,300) AS TimeRange, code AS HttpCode, Count(*) AS Count FROM C:\working\processed.csv GROUP BY TimeRange, code" -i:CSV -iTsFormat:"dd/MMM/yyyy hh:mm:ss" -o:DATAGRID

To view the number of requests grouped by 5 minute intervals and User ID run this from the cmd shell:
"C:\Program Files\Log Parser 2.2\LogParser.exe" "SELECT QUANTIZE(datetime,300) AS TimeRange, user AS User, Count(*) AS Count FROM C:\working\processed.csv GROUP BY TimeRange, user" -i:CSV -iTsFormat:"dd/MMM/yyyy hh:mm:ss" -o:DATAGRID

Note these two commands will send results to a GUI datagrid, from which you can copy results into excel and chart if you wish. If you don’t want the datagrid and just want stdout then omit the -o:DATAGRID tag from the command. Example output is shown as follows:

LogParser also has its own charting equivalent, which I haven’t explored. My parser is the bit that cleans up the file and adds the headers etc. Hopefully you can get enough from these instructions to tackle it on your own. For all you perl haters out there, I already know the answer to this question. Is perl dead yet? Feel free to post your python, ruby, shell equivalents.

PS. we ended up modifying the parser.pl used above to iterate through a directory full of gz files as such:

$dirname = $ARGV[0] || 'C:\working';
 
opendir(DIR, $dirname) or die "can't open directory $dirname: $!";
while (defined($file = readdir(DIR))) {
    if($file =~ /\.gz/) {
        $unzip = `c:\\PROGRA~1\\GnuWin32\\bin\\gzip -d $dirname\\$file`;
        ($filename) = ($file =~ /(.+?)\.gz/);
        open(IN, "<$dirname\\$filename");
        open(OUT, ">$filename.log");
 
        print OUT "host, user, datetime, method, file, http, code, bytes";
        while()
        {
          my($line) = $_;
          chomp($line);
          $line =~ s/\[|\]|\+1100|"//g;
          $line =~ s/\s+/,/g;
          $line =~ s/2009:/2009 /g;
          $line =~ s/,\-,/,/g;
          $line =~ s/,$//g;
 
          print OUT "$line\n";
        }
        close(IN);
        close(OUT);
 
        unlink("$filename");
    }
 
}
closedir(DIR);

Data Driven AutoIt Scripts

Tim Koopmans — Mon, 12 Jan 2009 05:16:52 +0000

I’ve recently been playing around with AutoIt in an effort to steer clear of QTP and the like. AutoIt is a simple but effective way in which to automate pretty much any Windows GUI application. To that end I needed a simple data driven framework to include in the script. AutoIt comes with some handy Excel user defined functions which allow you to do the same.

For example, the following spreadsheet:

Can be read into an array and looped through as follows:

#include 
#include 
 
$file = @ScriptDir & "\excel.xls"
$oExcel = _ExcelBookOpen($file, 0) ; open excel in the background
 
$aArray = _ExcelReadSheetToArray($oExcel, 2) ; ignoring header row
_ArrayDisplay($aArray, "Array using Default Parameters")
 
For $i = 1 to UBound($aArray)-1
	$name 	 = $aArray[$i][1]
	$surname = $aArray[$i][2]
 
	; do application stuff
	MsgBox(0,'debug', $name &' '& $surname )

	; write result to the book
	_ExcelWriteCell($oExcel, "PASS", $i+1, 5)
 
Next
 
_ExcelBookClose($oExcel) ; close excel

Pretty simple stuff!

Establishing Concurrency from your Web Logs

Tim Koopmans — Fri, 09 Jan 2009 03:25:23 +0000

I recently had a conversation with a colleague about how one might determine the concurrency for a given page or transaction at any point in time. The system under test was a typical web server and we had access to the web logs for analysis.

That is, we wanted to know how many virtual users were hitting a web page at the same time without using a sledgehammer to open the walnut so to speak… To that end we fell back to simple unix commands like awk, sort and uniq.

The typical web log entry we were looking for looked like this:
10.1.20.10 - Unauth [16/Dec/2008:11:59:16 +1100] "GET HTTP://system.under.test/secure.html HTTP/1.1" 200 692

On windows we would use a command similar to this to generate our report:
gawk "{print $4,$7}" web01-request.log | grep "\/secure" | uniq -c | sort -T C:\temp | gawk "{print $1}" | uniq -c

The first awk prints the date time column and url column which we then grep for any entries containing our target word ’secure’ (we could have done this in the awk statement too).

Then we use uniq -c to generate a unique count of rows that have the same date time, sort that (using a temp file), awk again on the first column and uniq -c again. The output looks like this:
2218 1 1483 2 263 3 258 4 38 5 36 6 2 7 5 8 1 12
So we can then determine for example that there were 1,483 occurrences where 2 virtual users hit the target at the same time. There were 258 occurrences where 4 virtual users hit the target at the same time and so on.

You can then graph this using the google chart API. No excel required =)

Performance Testing with Apdex

Tim Koopmans — Thu, 28 Aug 2008 04:03:41 +0000

Often when performance testing in a black box environment, you are left with the onerous responsibility to report against response time performance.

A typical approach by performance testers is to rely on 95th percentiles, which is effectively a Service Level Agreement (SLA) saying that 95 percent of all my samples have a response time below 5 seconds. This is often specified as a Non Functional Requirement (NFR).

But so what? What does this really tell us? More importantly, if the SLA fails, you’re probably left hanging in the wind trying to explain how “bad” it failed to a (now interested) Project Manager. On the flip side, if everything is green, how close were you to failing? Are you an n’th degree away from failing? What about comparing two different application’s response time performance when they have different NFRs to begin with? Getting confused?

Enter the Apdex performance index. Apdex is a numerical measure of user satisfaction that can be built from metrics expressed via more traditional SLAs and/or NFRs. The fundamental objective of Apdex is to simplify the reporting of application response time measurements by making it possible to represent any such measurement using a common metric. This can be reported on by extracting data from Load Runner / JMeter and analysing within Excel as demonstrated below.

Read on to find out more about these scores and how they are calculated.

The apdex score is a unified measurement between 1.00 – 0.00, or in plain English, Excellent, Good, Fair, Poor and Unacceptable. This colour chart represents those values:

You calculate the apdex index by first counting the total number of samples, the number of samples that have ’satisified’ a criteria (e.g. your SLA of 5 seconds), the number of samples that are ‘tolerating’ (e.g. greater than 5 seconds but less than 10 seconds) and the number of samples that have failed (e.g. greater than 10 seconds). You then whack them into a formula as such:

And out pops an index between 0 and 1. Compare it to your colour chart and whammo! You now know how good or bad your application performance (in terms of response time) really is…

Apdex really is just a granular traffic light, but it also provides a single scale which traffic lights or SLA red/green status do not. So for example, one app has an NFR of 30 secs and another has one at 5 secs, apdex provides the same unit of measurement for both. So you could possibly trend for improvement/degradation of that index over time if you sampled frequently enough i.e.
app 1 was 0.85, 0.84, 0.83, 0.82 and app 2 was 0.85, 0.80, 0.79, 0.87, is a better unit of comparison than NFR/SLA secs if app1 was 30, 31, 32, 34, and app 2 was 15, 13, 10, 13. Hard to compare the latter as it is a different scale.

You can also subjectively differentiate with T, so if you see APDEX[subscript T] score of 0.85[30] you know that there is possibly a huge buffer factored into the tolerating score, perhaps reducing the effectiveness of the index. I prefer to set satisfying = NFR, then tolerating to 1.5 x NFR.

Another technique I like to apply is to use the apdex index score in a risk management matrix, so I can use that as a weighting factor i.e traditional risk score / adpex index = revised risk score.
So low apdex scores are duly considered in a so-what decision about risk.

Plus in generating an apdex you get information on the count that was satisfied, tolerating, or failing, which audiences such as Project Managers can better understand. People really want to know how badly they failed a NFR/SLA and apdex is user friendly in this regard.

So how do you use Apdex with LoadRunner (or JMeter, or Grinder etc)?
Pretty simple actually. Export the response times from your measured transactions into a spreadsheet, then with a little array formula magic, spit out an automated report for inclusion in your test summaries.

I am attaching an example spreadsheet in Excel 2002 format, as this is probably the most common denominator in your work environment. For the ultra snazzy client, I prefer 2007 with its new ability to do multiple COUNTIFS (multiple criteria) and snappy looking traffic lights. You be the judge.

Download: apdex excel spreadsheet template
If you’re feeling generous at the end of all this thinking “thank God I don’t have to go through the pain of figuring this out in Excel” then feel free to: