Something as simple as this: http://chart.apis.google.com/chart?chtt=Tim's%20Beer%20Intake&cht=p3&chd=s:9h&chs=250x100&chco=00ff00&chl=Unemployed|Employed produces this:

Being on holidays at the moment, I’m lacking the um, ‘project’ on which to apply this technology, but am curious to see how functional it is. They [...]]]> Google have got some charting code worth looking at, which basically allows you to create charts by modifying parameters in the query string.

Something as simple as this:
http://chart.apis.google.com/chart?chtt=Tim's%20Beer%20Intake&cht=p3&chd=s:9h&chs=250x100&chco=00ff00&chl=Unemployed|Employed produces this:

Being on holidays at the moment, I’m lacking the um, ‘project’ on which to apply this technology, but am curious to see how functional it is. They give some examples on how to serialize datasets, and I think up to 4095 data points can be used. Not bad for those dashboard type applications …

All of these tool’s use are governed by some common factors being:

Rather than trawl through the PerfMon counter GUI, or regress back into the millions of raw counters available via WMI, I was looking for a solution somewhere in between.

The ActiveState version of Perl has a Win32::PerfMon [...]]]> I’ve been looking for some easy ways to enumerate all of the available performance counters on a Win32 platform programatically.

Rather than trawl through the PerfMon counter GUI, or regress back into the millions of raw counters available via WMI, I was looking for a solution somewhere in between.

The ActiveState version of Perl has a Win32::PerfMon module which lets you enumerate cooked performance counter metrics on a Win32 platform. So I started to write a simple script that would list all Objects->Instances->Counters recursively. Sounds simple right?

Unfortunately I found that the module appears faulty when trying to enum Objects that have multiple instances, such as PhysicalDisk and LogicalDisk. Naturally, these are the counters I’m most interested in! After googling to the end of the known universe, I couldn’t find a solution to this problem.

Windows comes with a command line utility called Typeperf which is great for what I ultimately need to script (an unattended batch job that collects performance metrics).

But in order to develop a list of ALL available counters I wrote a mashup in Perl using the Win32::Perfmon module and calls to Typeperf to discover performance counters related to any particular object. That way, I can top and tail the output to my chosen performance counters and not have to worry about the exact typing of these finicky little buggers.

To use:
discoverCounters.pl

Output:
.NET CLR Data
.NET CLR Networking
.NET Data Provider for Oracle
.NET Data Provider for SqlServer
.NET CLR Memory
.NET CLR Memory(_Global_)\# Gen 0 Collections
.NET CLR Memory(_Global_)\# Gen 1 Collections
.NET CLR Memory(_Global_)\# Gen 2 Collections
...

Code:
use Win32::PerfMon;
my $server = $ARGV[0] || "VIRTUAL";
my $PerfObj = Win32::PerfMon->new("\\\\$server");
my $Objects = $PerfObj->ListObjects();

# Clean up and print available counters to display
if($val=~/No valid counters/) {
print "$Object\n";
} elsif ($val=~/$Object/) {
print "$Object\n";
@values = split(',', $val);
foreach $value(@values) {
if($value=~/$Object/) {
$value=~s/"\\\\$server\\//;
$value=~s/"\n.*//;
$value=~s/"$//;
print "\t$value\n";
}
}
}
}
}

As a performance tester, I find that I often need to provide snapshot summaries of systems performance ala dashboards. The key requirement imposed is normally timeliness and relevance of data being presented.

So sticking with Jason’s requirements, how do [...]]]> Jason Gorman describes in his article 6 requirements for the practical use of dashboards in performance related testing.

As a performance tester, I find that I often need to provide snapshot summaries of systems performance ala dashboards. The key requirement imposed is normally timeliness and relevance of data being presented.

So sticking with Jason’s requirements, how do you implement good dashboard design?

1. Cognitive Simplicity – you should be able to digest the meaning of it in a matter of seconds …
I think the key here is to understand your audience. Generally the higher up the management food chain you go, the more simple they like the presentation. For a tech audience, I like to present in something they might relate to, such as transaction distributions represented by a miniature line chart and vertical box plot. Take for example the following graphical representation:

I find that to be a popular format. The line chart gives you an instant snapshot of data over time (in this case asvc_t response time on the active queue) and you can easily see where response times spiked in the sample. More useful IMHO though is the use of box plots. The vertical box plot to the right indicates that the top 25% of transactions measured ranged from 0 – 60 msecs indicating a problem with 25% of my results. Couple with the use of visual indicators such as colour, where red represents an exceeded threshold, I find these snapshots very useful and popular with other tech peeps.

A bit higher up the food chain, I find people tend to get suckered by corny dashboard widgets as in the following:

2. Timeliness – the information should, as much as possible, be a reflection of the reality now …
This is actually a difficult requirement to achieve, and normally the limiting factor for me is the size of the data that you need to analyze in order to provide concise, meaningful snapshots of information. Particularly when that data exists on multiple providers/sources, that are separate to your analysis solution. In any case, the best way to achieve timeliness is to automate the processing as much as possible. To that end, I normally collect data via scheduled cron jobs, and facilitate some sort of batch processing into my dashboard solution. Currently I generate data into csv files, then have a web based front end where I can upload that data for further processing. Sometimes I am dealing with 300MB+ size files of raw data, so you may wish to introduce some form of data aggregation or thinning in your solution. An example of the web front end I use is available at numbrcrunchr.com but this is really not suitable for Internet use. You need high speed bandwidth to upload (and process) such huge files.

3. Actionability – (yes, that’s a made-up word) – you should be able to take some kind of action to correct any problems revealed by the dashboard …
I’m afraid this is something that my own implementation lacks and I fear it could be quite difficult to provide based on the permutations and combinations of advice you may need to provide for a given scenario. However, more often than not, the use of thresholds in your dashboards can often provide the necessary triggers to instigate corrective action. Whether that be as simple as sending an email or sms, restarting a service or flagging the problem in some sort of log is the start of your action plan. Actually telling people how to resolve it, now that’s a challenge!

4. Constructiveness – it should point you towards practical improvements, rather than just damning current performance …
I think as a performance tester you always need to be cognizant of this, as being overly negative both in your analysis and presentation of information can relegate you to the too hard basket. I try to remove any personal comments in my analysis as in:
“the sub system performance of the SAN was very poor and the application could not provide a consistent response time” – this only isolates yourself from the production support and application development teams
Try and concentrate on the aspects which you may be able to tune or improve such as:
“disk io wait was observed to be high, it is recommended that cache hit ratios and network latencies are checked by the ABC support team for the period 03.00-04.00 am”

5. Approachability – it should not be threatening or put people off seeking feedback …
Once again I think this relates to your intended audience, and being aware of what they expect. One performance measurement I particularly hate is RAG (red amber green traffic lights) but I find it commonplace in most places I work. Over simplification of dashboards can be just as bad as over the top detailed widgets! Imagine people’s reaction to your assessment of their overall application performance as ‘red’. I think you need to be granular enough in your approach, and descriptive enough that it encourages feedback or more questions. “Do you know why response times peaked at 3.00am? Why yes, we run an Oracle backup at that time etc”

6. Practicality – a dashboard needs to not be a burden on the people who manage and use it …
I find web based tools are the best. At the end of the day it depends on what the company already has in place, try not to re-invent the wheel as there are a lot of great applications out there (SiteScope, OpenView, SAS etc). But if you get the urge, try some of the multitudes of charting APIs available. I personally prefer ChartDirector but there are tons of options on offer!

The log parser comes for free with the IIS resource pack but you can also [...]]]> IIS logs if setup properly can be an extremely useful source of information during load testing and for other activities such as capacity planning. Another load tester sings its praise here and provides good justification of why you should use it.

The log parser comes for free with the IIS resource pack but you can also get it individually here.

There are some gotchas that you need to be aware, mostly to do with the amount of extended info you are logging. If you hope to get into the nitty gritty and use extended logging info you must make sure these options are selected in the advanced properties for your log files.

You probably also want to consolidate your log files into a dedicated partition or volume, for easier administration further down the track. If you’re operating in a web farm type environment, you may also want to consolidate log files onto a single volume. You can specify these type of locations using UNC paths / shares etc. There’s a very good blog post that explains how to control the seemingly random instance naming convention that Microsoft assigns to each additional web server in your farm.

And finally, this tool runs great via cygwin/ssh so you can remotely log in and run queries as required. Here’s a query to get you started, there are many more out there on the web. I use this to show me how many total Bytes are being transmitted for any given day grouped by file extension, and also the average time taken in milliseconds to serve up each of the file extensions.


./LogParser.exe "SELECT EXTRACT_TOKEN (cs-uri-stem, -1, '.') AS Extension, AVG(time-taken) AS AvgTimeMs, SUM(sc-bytes) AS SumBytesTx, SUM(cs-bytes) AS SumBytesRx FROM D:\LOGS\HTTP\w3svc1\ex070719*.log WHERE Extension IN ('php';'html';'swf';'js'; 'jpg';'flv';'xml') GROUP BY Extension"


So what are my favourite descriptive statistics I like to [...]]]> The screenshot you see is an example of the graphs you can output in my descriptive statistics Excel template without using any macros. I often find moving averages and boxplots / histograms to be extremely useful in analyzing raw data for stress and volume testing …

So what are my favourite descriptive statistics I like to analyse data with?

1. Box plots and histograms are great for displaying distribution and give you a good insight to characteristics of the data you are analyzing. A standard box-whisker chart plots up to 5 data sets using box-whisker symbols. Variants are box plots without the whiskers. The 5 data sets are sometimes called the maximum, 3rd quartile, median, 1st quartile and minimum, although they can represent any kind of quantities.
2. Trend line charts. A trend line is a straight line that fits a number of data points computed using linear regression (the least square method). This is excellent for predictive analysis of data, especially if you get roped into capacity planning activities. You do however need a signficant empirical baseline from which to draw your results, which usually means you have to execute many test runs in order to establish the data.
3. Good old line charts. A line chart with thinned results (for easier presentation) can give immediate insights to where your transaction response times went wrong. This is good for coarse analysis of rows of data. Some variations of the line chart is the moving average as provided by excel. I’m not sure on the accuracy of moving averages though. Can be useful when trying to smooth the curve.
4. In a spline charts, the data points are connected together using cardinal spline curves (as opposed to straight lines). The “tension” of the curve can normally be configured.