I’m using MySQL’s AES which is better than older standards. Basically 128 bit key encryption …

To begin, let’s create a [...]]]> There’s been a bit of buzz lately where MySQL databases have been compromised and user passwords being stored in plain text. Bad fu. It’s pretty simple to encrypt passwords in MySQL, here’s how you do it.

I’m using MySQL’s AES which is better than older standards. Basically 128 bit key encryption …

To begin, let’s create a demonstration table. e.g.
CREATE TABLE users ( user_name VARCHAR(20) NOT NULL, user_password BLOB );

So now we have a simple table with a user_name field and user_password field. e.g.

mysql> DESCRIBE users;
+---------------+-------------+------+-----+---------+-------+
| FIELD         | Type        | NULL | KEY | DEFAULT | Extra |
+---------------+-------------+------+-----+---------+-------+
| user_name     | varchar(20) | NO   |     | NULL    |       |
| user_password | blob        | YES  |     | NULL    |       |
+---------------+-------------+------+-----+---------+-------+

If we were creating a new user we would insert using the AES_ENCRYPT function. e.g.

INSERT INTO users VALUES ('tim.koopmans', AES_ENCRYPT('mypassword', 'Q4QjOyKANXj3qO'));

In this case I’ve used AES_ENCRYPT which encrypts a string and returns a binary string. The AES (Advanced Encryption Standard) algorithm is encoded with a 128-bit key. The second parameter is the key string, important to keep that secure!

On its own, this offers a good level of security. If the database was compromised the hacker will see information like this. e.g.

mysql> SELECT * FROM users;
+--------------+------------------+
| user_name    | user_password    |
+--------------+------------------+
| tim.koopmans | ??iu?hQ:???Ti?} |
+--------------+------------------+

The problem with this, is that the user_password field can still be ‘guessed’ via brute force methods. Every password that is the same i.e. ‘mypassword’ will also have the same binary string.

You can add a salt to this by prefixing random digits to the plain text password before you encrypt. For example, we can use the first 12 digits of a UUID. e.g.

INSERT INTO users VALUES ('joe.blow', AES_ENCRYPT(CONCAT(LEFT(UUID(),12), 'mypassword'), 'Q4QjOyKANXj3qO'));

The result set now looks like this if compromised.

mysql> SELECT * FROM users;
+--------------+----------------------------------+
| user_name    | user_password                    |
+--------------+----------------------------------+
| tim.koopmans | ??iu?hQ:???Ti?}                  |
| joe.blow     | ?M}69?H?0?H}?/{.?we|?A?P7ZD?R?   |
+--------------+----------------------------------+

Every password that is encrypted should look different which will further complicate dictionary attacks.

To determine a password from a stolen hash, an attacker cannot simply try common passwords (such as English language words or names). Rather, they must calculate the hashes of random characters (at least for the portion of the input they know is the salt), which is much slower.

For best results, the value/length of your salt should be kept secret (and separate from your database). You should also keep the key secret of course!

Now if we ever need to extract the password we use the AES_DECRYPT function. e.g.

mysql> SELECT SUBSTR(AES_DECRYPT(user_password, 'Q4QjOyKANXj3qO'),13) AS password FROM users WHERE user_name = 'joe.blow';
+------------+
| password   |
+------------+
| mypassword |
+------------+

In this case we know the first 12 digits were the salt so I used SUBSTR after decrypting the hash to get the original password.

A couple of caveats, encrypting passwords in this fashion is reversible i.e. If your key gets compromised, then the hacker has access to all your passwords. Another option is to reverse the plain text and key parameters of AES_ENCRYPT, where the user’s password now becomes the key, and the key becomes the plain text. Even if the database and key is compromised, it is hard to crack user passwords. You would not be able to recover lost passwords of course … Security through obscurity!

In this [...]]]> A common requirement when load testing is to populate subsequent requests with dynamic data from previous requests. If you’re thinking about using a database to store static test data, such as usernames, paswords, credit card numbers etc then read on to find out how to extract this data at runtime and populate JMeter variables.

In this example we’ll need to add a JDBC Connection Configuration to our test plan. This will define and control the way in which JMeter will access your database.

Notice I’ve left the defaults for connection pool properties, although you may need to tweak this depending on how you’ll access the data. I’ve used a JDBC connection URL that accesses a local mysql server listening on port 3306. The database in use is called watirgrid

I’ve also defined the JDBC Driver class and specified the username / password to connect with.

Now we can make a JDBC request using the JDBC Request sampler. It will look something like this:

The Variable Name matches the JDBC Connection Config Variable Name i.e. mysql

I’m using a select statement that will print out results in a JSON-like format. That query is as follows:

SELECT
CONCAT("[",
GROUP_CONCAT(
CONCAT("{email:'",email,"'"),
CONCAT(",login_count:'",login_count),"'}")
,"]")
AS json FROM users

In doing this, it will return a result set that looks like this:

+-------------------------------------------------+
| json                                            |
+-------------------------------------------------+
| [{email:'tim.koops@gmail.com',login_count:'1'}] |
+-------------------------------------------------+
1 row IN SET (0.00 sec)

Now we need to add a Post Processor Regex Extractor to the JDBC request as follows:

The reason we do this is so we can extract the results from the JSON string returned by the database query. For example, to extract and populate a variable called ${email} our PPRE would look like this:

Now we’re cooking! Add a Debug Sampler to confirm everything works as you’d expect. You should see results like this in the response data

email=tim.koops@gmail.com
email_g=1
email_g0=email:'tim.koops@gmail.com'
email_g1=tim.koops@gmail.com
mysql=org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl@10d69502

Attached is a demo test plan: jdbc_example.jmx

Essentially the EC2 snapshot is triggered like this:

This method is based on advice from here

Notice I [...]]]> So I’ve been working on a system where the MySQL instance on EC2 sporadically locks up (mysqld is zombied) during a snapshot process =)

Essentially the EC2 snapshot is triggered like this:

system("xfs_freeze -f /vol")                        and die;
system("ec2-create-snapshot $vol -K $key -C $crt ") and die;
system("xfs_freeze -u /vol")                        and die;

This method is based on advice from here

Notice I am doing an xfs_freeze of the entire volume on which the mysql data sits. It is intended to be used with volume managers and hardware RAID devices that support the creation of snapshots such as EBS.

The -f flag requests the specified XFS filesystem to be frozen from new modifications. When this is selected, all ongoing transactions in the filesystem are allowed to complete, new write system calls are halted, other calls which modify the filesystem are halted, and all dirty data, metadata, and log information are written to disk. Any process attempting to write to the frozen filesystem will block waiting for the filesystem to be unfrozen.

Note that even after freezing, the on-disk filesystem can contain information on files that are still in the process of unlinking. These files will not be unlinked until the filesystem is unfrozen or a clean mount of the snapshot is complete.

The -u flag is used to un-freeze the filesystem and allow operations to continue. Any filesystem modifications that were blocked by the freeze are unblocked and allowed to complete.

EC2 recommends this for ec2-create-snapshot:

When taking a snapshot of a file system, we recommend unmounting it first. This ensures the file system metadata is in a consistent state, that the ‘mounted indicator’ is cleared, and that all applications using that file system are stopped and in a consistent state. Some file systems, such as xfs, can freeze and unfreeze activity so a snapshot can be made without unmounting.

Clearly we don’t want to unmount the whole drive, so freezing the XFS volume is our best bet.

On a couple of occassions, we’ve seen the whole mysql server crash when a backup was taking place. The first symptoms displayed in logs is a long semaphore wait. This message is repeated for different threads:

mysqld[26852]: InnoDB: Warning: a long semaphore wait:

mysqld[26852]: --Thread 46912644213072 has waited at ./../include/trx0sys.ic line 101 for 241.00 seconds the semaphore:

mysqld[26852]: X-lock on RW-latch at 0x2aaaaf01cc08 created in file buf0buf.c line 497

mysqld[26852]: a writer (thread id 46912644213072) has reserved it in mode  wait exclusive

After about 10 minutes of this behaviour the server crashes. This is because 101 threads have made 101 connections where the max_connections=100. This only occurs whenever we are doing a snapshot in the background which makes me thing its a deadlock condition on the underlying filesystem.

mysqld[26852]: InnoDB: We intentionally crash the server, because it appears to be hung.

mysqld[26852]: 091207 20:16:41InnoDB: Assertion failure in thread 46912625965392 in file srv0srv.c line 2097

mysqld[26852]: InnoDB: We intentionally generate a memory trap.

mysqld[26852]: InnoDB: Submit a detailed bug report to http://bugs.mysql.com.

mysqld[26852]: InnoDB: If you get repeated assertion failures or crashes, even

mysqld[26852]: InnoDB: immediately after the mysqld startup, there may be

mysqld[26852]: InnoDB: corruption in the InnoDB tablespace. Please refer to

mysqld[26852]: InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html

mysqld[26852]: InnoDB: about forcing recovery.

mysqld[26852]: 091207 20:16:41 - mysqld got signal 11 ;

mysqld[26852]: This could be because you hit a bug. It is also possible that this binary

mysqld[26852]: or one of the libraries it was linked against is corrupt, improperly built,

mysqld[26852]: or misconfigured. This error can also be caused by malfunctioning hardware.

mysqld[26852]: We will try our best to scrape up some info that will hopefully help diagnose

mysqld[26852]: the problem, but since we have already crashed, something is definitely wrong

mysqld[26852]: and this may fail.

In panic mode, we restarted the whole server which was able to then recover from this error. But is there a more graceful recovery open to us?

Alestic have released a revised version of the ec2-create-snapshot called ec2-consistent-snapshot

Here are some of the ways in which the ec2-consistent-snapshot program has improved over the original:

Command line options for passing in AWS keys, MySQL access information, and more.
Can be run with or without a MySQL database on the file system. This lets you use the command to initiate snapshots for any EBS volume.
Can be used with or without XFS file systems, though if you don’t use XFS, you run the risk of not having a consistent file system on EBS volume restore.
Instead of using the painfully slow ec2-create-snapshot command written in Java, this Perl program accesses the EC2 API directly with orders of magnitude speed improvement.
A preliminary FLUSH is performed on the MySQL database before the FLUSH WITH READ LOCK. This preparation reduces the total time the tables are locked.
A preliminary sync is performed on the XFS file system before the xfs_freeze. This preparation reduces the total time the file system is locked.
The MySQL LOCK now has timeouts and retries around it. This prevents horrible blocking interactions between the database lock, long running queries, and normal transactions. The length of the timeout and the number of retries are configurable with command line options.
The MySQL FLUSH is done in such a way that the statement does not propagate through to slave databases, negatively impacting their performance and/or causing negative blocking interactions with long running queries.
Cleans up MySQL and XFS locks if it is interrupted, if a timeout happens, or if other errors occur. This prevents a number of serious locking issues when things go wrong with the environment or EC2 API.
Can snapshot EBS volumes in a region other than the default (e.g., eu-west-1).
Can initiate snapshots of multiple EBS volumes at the same time while everything is consistently locked. This has been used to create consistent snapshots of RAIDed EBS volumes.

Here’s hoping that this release fixes the problems with locking of the filesystem! Notice we don’t attempt to flush and lock since we are using the InnoDB engine.

Revised snapshot code looks like this:

# create snapshot
system("xfs_freeze -f /vol")                        and die;
system("ec2-consistent-snapshot --aws-access-key-id $key --aws-secret-access-key $secret --xfs-filesystem /vol $vol") and die;
system("xfs_freeze -u /vol")                        and die;

# auto complete word autocompleteword.automatic=1

# advanced find and replace find.replace.advanced=1

# force [...]]]> Well the subject goes without saying. I’ve been working on WinXP again lately and when I’m not around my favourite text editor on the mac (TextMate) I opt for SCiTE on windows. The following settings I find useful for this simple but powerful text editor. YMMV.


# auto complete word
autocompleteword.automatic=1

# advanced find and replace
find.replace.advanced=1

# force monospaced fonts
font.base=$(font.monospace)
font.small=$(font.monospace)
font.comment=$(font.monospace)
font.text=$(font.monospace)
font.text.comment=$(font.monospace)
font.embedded.base=$(font.monospace)
font.embedded.comment=$(font.monospace)
font.vbs=$(font.monospace)
font.monospace=font:DejaVu Sans Mono,size:8

# show the status bar
statusbar.visible=1


Part of my grievance for that was based on frustration in dealing with different load testing platforms. Not frustration born out of incompetence, more the frustration born out writing tests for increasingly [...]]]> In my last post I made a tongue-in-cheek observation that load testing really doesn’t work by today’s perceived standards for web automation and testing.

Part of my grievance for that was based on frustration in dealing with different load testing platforms. Not frustration born out of incompetence, more the frustration born out writing tests for increasingly complex browsers with an antiquated toolset.

That’s not to say that protocol based application load testing (or the languages they sit on) aren’t powerful in their own right. LoadRunner, JMeter et al. has their place in the load testing world. But I would like to relegate them to the background as useful, sometimes expensive tools to spin up the wheels so to speak. Nothing like a bit of JMeter infused fun to get the wheels smoking on your system under test. But for the real part, measuring user experience on the web platform?

If you can’t wait to get started then I’d like to introduce BrowserMob.com. Been around since mid 2008 and offers real browser testing using Selenium scripts distributed across a grid of EC2 computers. Reasonably priced, well documented and supported by the legendary Patrick Lightbody, founder of things like OpenQA and Selenium Remote Control.

But hang on a second, you don’t like Selenium right? Have zero budget? Love Watir? Well I’m working on a similar construct aptly called WatirGrid which also allows for distributed testing across a grid network using Watir. The engine (gem) itself is open source and you can get it as follows:

gem install watirgrid

If you’d like to contribute please follow the WatirGrid project on github.

The Basics
WatirGrid is built on Rinda which implements the Linda distributed computing paradigm in Ruby. According to Wikipedia: “Linda is a model of coordination and communication among several parallel processes operating upon objects stored in and retrieved from shared, virtual, associative memory.”

In other words, WatirGrid allows multiple parallel processes to provide remote Watir objects in the form of tuple spaces across a grid network. This grid network is controlled by a ring server and looks a bit like this:

Key Terminology
The controller implements a repository of tuples (tuple space) that can be accessed concurrently. The controller hosts the ring server which advertises these tuples across a grid network. Typically you will host one controller on a central machine.

The providers make remote Watir objects available to the tuple space hosted by the ring server. Typically you will host one or many providers on your grid network, for example, each PC may become a single provider of a Watir tuple in the form of an Internet Explorer, Firefox or Safari browser object.

Simple Example
Pick a server to host the controller for the ring server and execute the following:

  $ controller

This should find your external facing IP and start the corresponding DRb and Ring Servers:

  DRb server started on : druby://143.238.105.61:11235
  Ring server started on: druby://143.238.105.61:12358

On each client PC, host the provider for the distributed (DRb) Watir objects

  $  provider

This should find the recently started Ring Server and register a tuple for the Watir object:

  DRb server started on : druby://143.238.105.61:11236
  Ring server found on  : druby://143.238.105.61:12358
  New tuple registered  : druby://143.238.105.61:12358

You will now be able to execute commands across remote browsers on your grid network.
e.g.

  grid = Watir::Grid.new(:ring_server_port => 12358)
  grid.start(:quantity => 1, :read_all => true)
  threads = []
  grid.browsers.each do |browser|
    threads << Thread.new do
      b = browser[:object].new_browser
      b.goto("http://www.google.com")
      b.text_field(:name, 'q').set("watirgrid")
      b.button(:name, "btnI").click
    end
  end
  threads.each {|thread| thread.join}

You may wish to host the controller and providers on different machines in a real scenario. You can specify things like server hostnames, ports and access control lists for each of the different types of servers. For more help see:

  $  controller --help
  Usage: controller [options]
  Specific options:
      -H, --drb-server-host HOST       Specify DRb Server interface to host on
      -d, --drb-server-port PORT       Specify DRb Server port to listen on
      -h, --ring-server-host HOST      Specify Ring Server interface to host on
      -r, --ring-server-port PORT      Specify Ring Server port to listen on
      -a, --access-control-list ACLS   Specify a comma separated Access Control List
      -l, --log-level LEVEL            Specify log level {DEBUG|INFO|ERROR}
          --help                       Show this message
 
  $  provider --help
  Usage: provider [options]
  Specific options:
      -H, --drb-server-host HOST       Specify DRb Server interface to host on
      -d, --drb-server-port PORT       Specify DRb Server port to listen on
      -h, --ring-server-host HOST      Specify Ring Server host to connect to
      -r, --ring-server-port PORT      Specify Ring Server port to broadcast on
      -b, --browser-type TYPE          Specify browser type to register {ie|firefox|safari}
      -a, --access-control-list ACLS   Specify a comma separated Access Control List
      -l, --log-level LEVEL            Specify log level {DEBUG|INFO|ERROR}
          --help                       Show this message

Need more help?

Keep an eye on the wiki and the rspec test cases!

Load testing does not work for web as it is today primarily because… 1. Load testing tool [...]]]> I was going to write why LoadRunner does not work for the web as it is today and into the foreseeable future, but decided against that because I don’t think the following points are necessarily limited to a specific toolset.

Load testing does not work for web as it is today primarily because…

1. Load testing tool languages are not flexible enough.
Load testing tools are in need of a major overhaul. Consider LoadRunner, the majority of your scripts will be specified in C or proprietary API calls such as lr_ statements. String manipulation for example, whilst possible is extremely clunky and requires oodles of C code. Lack of proper regex or xpath support limits script flexibility in terms of parsing the necessary contents of web page and ajax calls. Don’t even get me started on ajax! Similar complaints for JMeter. Although better support for regex you can really only edit test plans via the GUI. You’ll go cross-eyed looking at test plans in xml. I think the Grinder is the closest product I’ve seen that supports flexibility as you can write tests in Jython which provides more flexibility in terms of scripting.

2. We’re still stuck in record and playback mode.
We really haven’t advanced like our automation brethren. Part of the big sell by commercial load testing companies is “Look it’s easy, just hit the red button and play it back when you’re finished!”. Bollocks to that, we know that never works. A dream load testing tool would support the ability to hand-code your statements and be editable in a text editor! In a similar fashion to watir.

The dream tool would look like:

1
2
3
4

vusers.start(100)
  browser.goto('url')
  browser.button(:text, /Continue/).click
end

Instead load testing scripts in LoadRunner look something like this:

1
2
3
4
5
6
7
8
9
10

web_submit_data("transaction",
"Action=https://someurl/I/have/to/keep/correlating/damn/it",
"Method=POST"
"RecContentType="text/html",
"Mode=HTML",
"ITEMDATA,
"Name=L_3_0_0_1_0_1", "Value={another_value_I_have_to_correlate}",
"Name=L_3_0_0_2_0_1", "Value={maybe_paramatize_this_one}",
ENDITEM,
LAST);

We are of course talking about very different technologies. Watir automation scripts driving real browsers with win32ole, whereas most load testing tools are injecting traffic at the protocol layer. If a typical end to end scenario has 30 steps, this can amount to some 2000+ lines of code thanks to the extra verbosity of having to specify all the form values and so forth. This also means you have oodles of code to correlate and parametize. It’s not something you want to hand code, the quickest path is to record (unfortunately). BrowserMob is the closest online provider I’ve seen that overcomes this, supporting load tests written as selenium scripts. Problem is, we need a solution for all those non-public facing companies, the bulk of where I do my work at!

3. No real code versioning.
Changes to the UI will break your script! One can implement their own code versioning as I have recently begun to do with git but there really should be built in support for this. That aside, you revision your code but effectively blast it away every time you hit the red record button. Reason? Each recording can look vastly different to the last. It only takes one form value L_3_0_0_1_0_1 to change to L_3_0_0_1_0_2 or one block of code executed by an ajax call to shift to cause mass confusion. Versioning your code *may* pick this change up, but only if you got the power of the diff! Diffing just got a little harder. Shoot back 10 years ago and diffing for correlation would have been a cinch. Wind forward to today with hundreds of ajax or json calls and correlation can be a nightmare. Windiff struggles. Daisydiff does a much better job highlighting differences by line and by column. This approach kind of works in a waterfall test environment. Throw in a twice weekly code drop in your performance test environment and watch your scripts break. Agile anyone?

So if I was an uber-elite programmer what would I do to move loadtesting into the current decade?
1. Develop a load-testing tool that is based on a flexible scripting language such as Ruby.
2. Get rid of the record button and provide free text editors to the world!
3. Provide a framework that can be versioned and easily detect changes in the system under test.

Now I can tag emails in my inbox when I view them with the keyboard shortcut l->a for action, w for [...]]]> I really like the concept of Getting Things Done. I’m also a fan of Inbox Zero. Well, at least try to be. Here’s how I implement GTD using my gmail account…

First I create four labels:
label:g-action
label:g-waiting-on
label:g-some-day
label:g-finished

Now I can tag emails in my inbox when I view them with the keyboard shortcut l->a for action, w for waiting-on and so forth.

I make a habit of reading, deleting or tagging each email that comes through my inbox. Hence I get closer to Inbox Zero.

I also use the Quick Links feature available through GMail Labs.

I then create links to the 4 labels by searching on the syntax described previously.

There are of course plenty of apps and additional plugins for specific browsers that achieve the same thing. I just like this approach because it is generally OS/browser independent which means I can take it with me from workplace to workplace (provided they let me access my gmail account of course!)

Even worse, have you had to write those scripts against a development environment with an unstable code base and shocking performance? Sound familiar? Ever felt the frustration of re-recording [...]]]> Here’s an idea… Ever been in a LoadRunner contract where you’ve got a ton of scripts to write for business transactions that have many many design steps?

Even worse, have you had to write those scripts against a development environment with an unstable code base and shocking performance? Sound familiar? Ever felt the frustration of re-recording manual test cases back into LoadRunner in said environments. Why wouldn’t you just automate the script recording process?

Well if you’re in a LoadRunner shop there’s probably a QTP license floating around. However there’s a great alternative with Ruby, specifically Web Application Testing In Ruby.

The gist is that you setup your manual test case using a watir script, then re-record your LoadRunner scripts using those watir scripts as the driver. The benefits are many:

1. Reduce user input errors when recording the LoadRunner scripts.
2. Have a consistent way of re-recording LoadRunner scripts based on the same input delivered by the watir script.
3. Diffing for correlation becomes easier because of consistency between multiple recordings of the same script.
4. Watir scripts are easier to maintain than LoadRunner scripts.
5. You can incorporate other Ruby gems into your watir scripts to assist with repetitive tasks during recording. For example, I copy transaction names to the clipboard between each step so I can easily paste them into transaction start timers.
6. Watir scripts can double as a smoke test which you can run in parallel with load tests that use only virtual users.
7. Watir scripts (and sampling response time from a real browser) will incorporate other factors such as client render time. Something which vusers really can’t do.

Downside is that you need to do a little more preparation than you might have to if you’re just following a manual test case. Once you have a framework for creating your watir script, this becomes easier. I figure it takes me about 1 hour to script up an end to end transaction that might have around 30 sub transactions (or design steps).

So what do you need to get started?

Firstly to install watir, follow these instructions here.

Come up with a framework for driving your manual test cases. There’s plenty of examples out there on how to make bullet proof frameworks driven from a variety of sources. Alister shares some good tips here.

Here’s a simple framework to get you started.

I have a helpers class that fires up the browser (attaches to the window that LoadRunner spawns) and some common methods for starting/stopping transaction timers and writing out information to a logfile. It looks like this.
helpers.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83

require 'rubygems'
require 'watir'
require 'logger'
require 'win32/clipboard'
require 'dl'
require 'win32/sound'
include Win32
 
logfile = 'C:\temp\output.log'
@log = Logger.new(logfile, 'daily')
@log.info("\nNew test started at " + Time.now.to_s)
@log.datetime_format = "%H:%M:%S"
 
# if I want each step to start on the minute
# useful when tracing back through profile logs
# on the target server
@staggered = false
 
# pause after each step to prompt you to
# stop a transaction time or start a new one
@pause = false
 
# setup a clipboard object to copy transaction names to
@clipboard ||= Win32::Clipboard
 
# setup my browser object and attach to a running browser
# spawned by the loadrunner recording engine
@b ||=Watir::IE.attach(:title, /FOOBAR/)
 
# alternatively start a new browser
#~ @b ||=Watir::Browser.new
@b.visible = true
@b.speed = :fast
@b.bring_to_front
 
# display a simple message box with a dodgy sound
# to grab your attention
def message_box(txt, title=APP_TITLE, buttons=0)
  5.upto(10) { |i| Sound.beep(i*1000,30) }
  user32 = DL.dlopen('user32')
  msgbox = user32['MessageBoxA', 'ILSSI']
  r, rs = msgbox.call(0, txt, title, buttons)
  return r
end
 
# a start timer method
def timer_start
 
  # figure out how long to sleep if staggered start is true
  now = DateTime.now
  diff = 60 - now.sec
  sleep diff if @staggered
 
  # empty clipboard and copy current transaction name to it
  @clipboard.empty()
  @clipboard.set_data(@t)
 
  # prompt you to start a new transaction if pause is true
  message_box(@t, "Start Transaction", 0) if @pause
 
  @start_time = Time.now
end
 
# end timer method
def timer_end
  @end_time = Time.now
 
  # prompt you to stpo end a transaction if pause is true
  message_box(@t, "End Transaction", 0) if @pause
 
  # figure out elapsed time
  @elapsed_time = (@end_time - @start_time) * 1000
  tab = "\t"
 
  # write it all to you logger
  @log.info(@start_time.to_f.to_s + tab + @end_time.to_f.to_s  + tab \
  + tab + @start_time.to_s + tab + sprintf('%-60s', @t) + tab + sprintf('%10s', @elapsed_time.to_s) )
end
 
# a simple sync method to wait for text on a scren
def sync(text)
  Watir::Waiter.wait_until{ @b.text.include?(text) }
end

Now your manual test case in ruby should look pretty simple. I typically rely on things like the :name, :text or :id attributes of html elements depending on how the application was written by the developers. The manual test case will look something like this.
001_test_case.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

require 'helpers.rb'
 
begin
 
  @t = "INIT"
  timer_start
  @b.goto('https://foobar/wps/myportal')
  timer_end
 
  @t = "LOGIN"
  timer_start
  @b.text_field(:name, "username").set("yoohoo")
  @b.text_field(:name, "password").set("foobar")
  @b.button(:index, 1).click
  timer_end
 
  @t = "FOOBAR_03a_03_Tab"
  timer_start
  @b.link(:text, /FOOBAR/).click
  timer_end
 
  @t = "FOOBAR_03a_04_FOOBAR_Tab"
  timer_start
  @b.link(:text, /FOOBAR/).click
  sync("New FOOBAR")
  timer_end
 
  @t = "FOOBAR_03a_27_Record_Notes"
  @b.text_field(:id, "PC_7_C5L40JR2001F002J5MT98S28H6__L_8_0_1_0_14_1_1").set("Justification")
  timer_start
  @b.button(:text, /Save/).click
  timer_end
 
  @t = "FOOBAR_03a_28_Save_And_Foobaz"
  timer_start
  @b.link(:text, /Save & Foobaz/).click
  timer_end
 
  # etc ...
 
rescue Exception => e
  @log.error(@start_time.to_i.to_s + "\t" + sprintf('%-60s', @trans) + "\tError\n" + e)
end

Running this test case will output a log file to C:\temp\output.log that looks like this.

New test started at Thu Oct 08 20:37:48 +1100 2009
1254994670.936  1254994671.576          Thu Oct 08 20:37:50 +1100 2009  INIT                                                                 640.0
1254994671.576  1254994678.233          Thu Oct 08 20:37:51 +1100 2009  LOGIN                                                               6657.0
1254994678.233  1254994694.123          Thu Oct 08 20:37:58 +1100 2009  FOOBAR_03a_03_Tab                                          15890.0
1254994694.123  1254994696.014          Thu Oct 08 20:38:14 +1100 2009  FOOBAR_03a_04_FOOBAR_Tab                                      1891.0
1254994717.171  1254994727.608          Thu Oct 08 20:38:37 +1100 2009  FOOBAR_03a_05_Enter_FOOBAR_Type                              10437.0
1254994729.28   1254994743.499          Thu Oct 08 20:38:49 +1100 2009  FOOBAR_03a_06_Search_for_Primary_Person                      14219.0
1254994748.093  1254994767.265          Thu Oct 08 20:39:08 +1100 2009  FOOBAR_03a_07_Associate_Primary_Person                       19172.0
1254994767.265  1254994780.796          Thu Oct 08 20:39:27 +1100 2009  FOOBAR_03a_08_Enter_Primary_Person_Address_Details           13531.0
1254994780.796  1254994818.765          Thu Oct 08 20:39:40 +1100 2009  FOOBAR_03a_09_Enter_Primary_Person_Doc_Details        37969.0
1254994818.765  1254994835.906          Thu Oct 08 20:40:18 +1100 2009  FOOBAR_03a_16_Add_PoC _Link                                     17141.0
1254994838.828  1254994843.515          Thu Oct 08 20:40:38 +1100 2009  FOOBAR_03a_17_Add_PoC                                            4687.0
1254994852.078  1254994855.469          Thu Oct 08 20:40:52 +1100 2009  FOOBAR_03a_18_Search_Agent                                       3391.0
1254994855.469  1254994859.203          Thu Oct 08 20:40:55 +1100 2009  FOOBAR_03a_19_View_Agent                                         3734.0
1254994863.953  1254994876.203          Thu Oct 08 20:41:03 +1100 2009  FOOBAR_03a_20_Associate_Agent                                   12250.0
1254994876.203  1254994879.484          Thu Oct 08 20:41:16 +1100 2009  FOOBAR_03a_21_Add_Payment_Link                                   3281.0
1254994894.094  1254994894.703          Thu Oct 08 20:41:34 +1100 2009  FOOBAR_03a_22_Record_Payment_Details                              609.0
1254995196.611  1254995196.611          Thu Oct 08 20:46:36 +1100 2009  SAP                                                                    0.0
1254995202.486  1254995258.517          Thu Oct 08 20:46:42 +1100 2009  FOOBAR_03a_25_ Run Validity Assessment                          56031.0
1254995267.736  1254995280.017          Thu Oct 08 20:47:47 +1100 2009  FOOBAR_03a_27_Record_Notes                                      12281.0
1254995280.017  1254995346.579          Thu Oct 08 20:48:00 +1100 2009  FOOBAR_03a_28_Save_And_Foobaz                                    66562.0

Which gives you things like start and stop times, elapsed times and action names to help you debug things further down the track.

All up I think it’s a pretty neat solution for re-re-re-recording LoadRunner scripts (or your tool of choice) you inevitably face in fast changing environments. Whilst writing this post I had a script ticking away in the background recording some 3000+ LoadRunner events. YMMV.

By definition, the first two Fibonacci numbers are 0 and 1, and each remaining number is the sum of the previous two. This naturally occurring sequence is useful for determining the amount of time to sleep whenever you [...]]]>

In mathematics, Fibonacci numbers are the following sequence of numbers:
0, 1, 1, 2, 3, 5, 8, 13, …

By definition, the first two Fibonacci numbers are 0 and 1, and each remaining number is the sum of the previous two. This naturally occurring sequence is useful for determining the amount of time to sleep whenever you are polling a resource.

I’m often required to poll a database resource in some sort of daemon. For example, check a table for some asynchronous tasks to complete. If you have a lot of threads polling this resource at quick intervals, the impact on the resource can be excessive/high. Rather than poll at a set interval, I use a Fibonacci sequence to determine what the next sleep interval will be. Following are code examples of this polling technique hosted on GitHub in my most commonly used programming languages…

Ruby
Perl
PHP

Essentially what each script does is the following …

Initialise a thread with a sleep interval of 5 seconds (or F5 in reference to the Fibonacci sequence number)

The thread will then:
1. Check if there is work to do by polling the resource
2. Complete any work and indicate if work has been done
3. If work has been done then next sleep interval will be 3 seconds (or F5-1)
4. If work has not been done then next sleep interval will be 8 seconds (or F5+1)
5. Sleep for the desired interval and loop

Enjoy!

If you [...]]]> Version control is not something you often associate with performance testing scripts. The reasons generally vary. A significant one is the lack of any version control support in commercial toolsets like LoadRunner. Another reason is the perceived short-term shelf life of test scripts. You tend to write them once and throw them away right?

If you work with more agile teams/developers you will probably start to require a lot of re-use out of your scripts, especially when the release cycle increases in terms of frequency. It starts to make sense to use version control and some form of scripting framework.

In this post I’m going to use JMeter as the tool of choice here and I’m going to re-introduce concepts which I normally force upon others whilst contracting. That is, use of Application Simulation Models (ASMs) to hep define the load and use of the Application Performance Index (Apdex) to help present results. I’m also going to use GitHub (or git) as my tool for maintaining version control and implementing a wiki as a dashboard replacement for other test management tools.

In short, I’m hosting version controlled scripts on GitHub. You can view my repository here:
http://wiki.github.com/90kts/performance_test

I’m using Google Documents to store my ASMs. You can see an example here.

I’m also using the Wiki functionality of GitHub plus some Ruby scripts as my test runners. In other words, a scripted version of a controller, driven by data from the ASMs.

GitHub can be quite useful. Not only can you version control all your scripts, you can also implement concepts based around a “low-tech testing dashboard” first explored by James Bach. The use of a wiki to support this was recently explored by Matthew at Socialtext. Coupled with that you also get an Issues database which is perfect for tracking those performance defects during test execution. Google Docs is also very useful in that it allows you to centrally store test artifacts like your ASMs which has plenty of neat APIs for programatically accessing content. Start working more with GitHub and less with QualityCentre and watch your circle of geek friends change.

Dashboard
So my dashboard is really just a broad sweep of major system and application components that present some level of risk to application stability and performance. It’s represented in the GitHub wiki as follows:

Main elements are the areas I wish to test, the estimated effort required to script/prepare those areas, an indication of risk to production stability (carried out separately in risk assessments with appropriate technical/business teams), day to day comments on test readiness and a link to the online ASM. I use the keywords “run” or “draft” to distinguish between ASM status’. This helps the test runner determine which tests to execute.

Test Runner
The test runner is a Ruby script which when executed, queries the Dashboard for ASMs to execute (based on the keyword “run”) then fetches the appropriate ASM via Google Documents to parse out relevant volumetrics for the test. Finally, it launches thread groups within JMeter to execute the test according to the ASM. Results are stored in the corresponding logs directory.

Apdex
If you’d like to find out more about the Application Performance Index please see here. I’ve found Apdex to be a very useful format to communicate application performance consistently to non-tech-heads. There’s plenty of room to geek out at low level later, but for a 20,000′ view on things, Apdex is often a good start. I’ve implemented Apdex support for this framework using a Ruby script to parse the results. The Google chart API for Ruby is used to present sparklines. You can do this during or after a test run has complete. It basically picks up results from the logs directory and parses them against response time targets specified in the ASM. The output of the Apdex script is a html page which looks like this:

Application Simulation Models
The ASM is really the brains behind the JMeter test plan. It looks like this:

It has two worksheets. The ASM worksheet is intended for business stakeholders. It’s geared more towards their understanding. I encourage these groups to articulate load in terms of transaction distribution (how many and what types of transactions groups) and user distribution (who are your user groups and what do they consist of). The key metrics I get these groups to enter are highlighted in yellow. Things like different user groups, percentage breakdowns of transactions and transaction/concurrent user quantities. Have a play, it’s pretty easy to follow. Coincidentally in every contract I’ve worked, I make this a mandatory test artifact for both my testers and the teams they’ve engaged with. No ASM = No Performance test…

The Scenario worksheet is really just the reverse of the first. It breaks down all the metrics that the business have provided into a worksheet to help you plug figures into your test scenarios. It helps cut down user-errors when calculating transaction rates etc and makes it much easier to chop and change figures. Want to do a test run with less concurrent users or higher transaction throughput? Just muck around with the percentages in yellow. I’ve used this format for both LoadRunner and JMeter gigs, so the terminology is meant to be familiar to both those types of users.

JMeter Test Plan
Through the beauty of JMeter -J command line parameters I’m able to pass all the relevant metrics from the ASM such as number of threads (vusers), duration, throughput, rampup and action(transaction) distribution percentages. I use this template format quite often and have explained some of these concepts previously.

The test plan gets the required parameters from the command line at run time. You can see the User Defined Variables look something like this:
DURATION = ${__P(duration,60)

Which basically sets the ${DURATION} JMeter variable based on the -Jduration command line variable, or defaults to 60 seconds if it is not passed. Other variables are also defined here and get loaded into the main Thread Group.

I use a Runtime Controller to control how long the test plan executes for. I also keep a separate Transaction Controller called Init to execute login actions etc or things I only want to execute once in the test plan.

Nested inside the Runtime Controller I use Throughput Controllers with percent executions tied to action percentages defined in the ASM. For example, action 1 might be a “view product” sequence of requests which I execute 50% of the time and so forth. Similar to weighted percentages for transaction blocks in LoadRunner. It is best to calculate these percentages using the Per User mode in JMeter for more accurate results.

Nested inside Throughput Controllers will be further Transaction Controllers, with Generate Parent Sample checked to enable similar functionality to transaction timers in LoadRunner around groups of requests.

The whole Thread Group has its throughput limited with a Constant Throughput Timer which basically tries to make all active threads in the thread group stay at a constant rate (defined by samples per minute). This eliminates the need to figure out pacing or sleep times as you might in LoadRunner.

The Summary Report and Gaussian Random Timers are just used for debugging so don’t normally include those in the final script…

Git
Want to get started? Well first you will need a copy of a git client. Have a look at various how-to guides out there to get you started. I’ve found this site pretty useful for understanding and working with git basics. My public clone URL for all the code used in this post is available here:
git://github.com/90kts/performance_test.git

Enjoy!