In this [...]]]> A common requirement when load testing is to populate subsequent requests with dynamic data from previous requests. If you’re thinking about using a database to store static test data, such as usernames, paswords, credit card numbers etc then read on to find out how to extract this data at runtime and populate JMeter variables.

In this example we’ll need to add a JDBC Connection Configuration to our test plan. This will define and control the way in which JMeter will access your database.

Notice I’ve left the defaults for connection pool properties, although you may need to tweak this depending on how you’ll access the data. I’ve used a JDBC connection URL that accesses a local mysql server listening on port 3306. The database in use is called watirgrid

I’ve also defined the JDBC Driver class and specified the username / password to connect with.

Now we can make a JDBC request using the JDBC Request sampler. It will look something like this:

The Variable Name matches the JDBC Connection Config Variable Name i.e. mysql

I’m using a select statement that will print out results in a JSON-like format. That query is as follows:

SELECT
CONCAT("[",
GROUP_CONCAT(
CONCAT("{email:'",email,"'"),
CONCAT(",login_count:'",login_count),"'}")
,"]")
AS json FROM users

In doing this, it will return a result set that looks like this:

+-------------------------------------------------+
| json                                            |
+-------------------------------------------------+
| [{email:'tim.koops@gmail.com',login_count:'1'}] |
+-------------------------------------------------+
1 row IN SET (0.00 sec)

Now we need to add a Post Processor Regex Extractor to the JDBC request as follows:

The reason we do this is so we can extract the results from the JSON string returned by the database query. For example, to extract and populate a variable called ${email} our PPRE would look like this:

Now we’re cooking! Add a Debug Sampler to confirm everything works as you’d expect. You should see results like this in the response data

email=tim.koops@gmail.com
email_g=1
email_g0=email:'tim.koops@gmail.com'
email_g1=tim.koops@gmail.com
mysql=org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl@10d69502

Attached is a demo test plan: jdbc_example.jmx

If you [...]]]> Version control is not something you often associate with performance testing scripts. The reasons generally vary. A significant one is the lack of any version control support in commercial toolsets like LoadRunner. Another reason is the perceived short-term shelf life of test scripts. You tend to write them once and throw them away right?

If you work with more agile teams/developers you will probably start to require a lot of re-use out of your scripts, especially when the release cycle increases in terms of frequency. It starts to make sense to use version control and some form of scripting framework.

In this post I’m going to use JMeter as the tool of choice here and I’m going to re-introduce concepts which I normally force upon others whilst contracting. That is, use of Application Simulation Models (ASMs) to hep define the load and use of the Application Performance Index (Apdex) to help present results. I’m also going to use GitHub (or git) as my tool for maintaining version control and implementing a wiki as a dashboard replacement for other test management tools.

In short, I’m hosting version controlled scripts on GitHub. You can view my repository here:
http://wiki.github.com/90kts/performance_test

I’m using Google Documents to store my ASMs. You can see an example here.

I’m also using the Wiki functionality of GitHub plus some Ruby scripts as my test runners. In other words, a scripted version of a controller, driven by data from the ASMs.

GitHub can be quite useful. Not only can you version control all your scripts, you can also implement concepts based around a “low-tech testing dashboard” first explored by James Bach. The use of a wiki to support this was recently explored by Matthew at Socialtext. Coupled with that you also get an Issues database which is perfect for tracking those performance defects during test execution. Google Docs is also very useful in that it allows you to centrally store test artifacts like your ASMs which has plenty of neat APIs for programatically accessing content. Start working more with GitHub and less with QualityCentre and watch your circle of geek friends change.

Dashboard
So my dashboard is really just a broad sweep of major system and application components that present some level of risk to application stability and performance. It’s represented in the GitHub wiki as follows:

Main elements are the areas I wish to test, the estimated effort required to script/prepare those areas, an indication of risk to production stability (carried out separately in risk assessments with appropriate technical/business teams), day to day comments on test readiness and a link to the online ASM. I use the keywords “run” or “draft” to distinguish between ASM status’. This helps the test runner determine which tests to execute.

Test Runner
The test runner is a Ruby script which when executed, queries the Dashboard for ASMs to execute (based on the keyword “run”) then fetches the appropriate ASM via Google Documents to parse out relevant volumetrics for the test. Finally, it launches thread groups within JMeter to execute the test according to the ASM. Results are stored in the corresponding logs directory.

Apdex
If you’d like to find out more about the Application Performance Index please see here. I’ve found Apdex to be a very useful format to communicate application performance consistently to non-tech-heads. There’s plenty of room to geek out at low level later, but for a 20,000′ view on things, Apdex is often a good start. I’ve implemented Apdex support for this framework using a Ruby script to parse the results. The Google chart API for Ruby is used to present sparklines. You can do this during or after a test run has complete. It basically picks up results from the logs directory and parses them against response time targets specified in the ASM. The output of the Apdex script is a html page which looks like this:

Application Simulation Models
The ASM is really the brains behind the JMeter test plan. It looks like this:

It has two worksheets. The ASM worksheet is intended for business stakeholders. It’s geared more towards their understanding. I encourage these groups to articulate load in terms of transaction distribution (how many and what types of transactions groups) and user distribution (who are your user groups and what do they consist of). The key metrics I get these groups to enter are highlighted in yellow. Things like different user groups, percentage breakdowns of transactions and transaction/concurrent user quantities. Have a play, it’s pretty easy to follow. Coincidentally in every contract I’ve worked, I make this a mandatory test artifact for both my testers and the teams they’ve engaged with. No ASM = No Performance test…

The Scenario worksheet is really just the reverse of the first. It breaks down all the metrics that the business have provided into a worksheet to help you plug figures into your test scenarios. It helps cut down user-errors when calculating transaction rates etc and makes it much easier to chop and change figures. Want to do a test run with less concurrent users or higher transaction throughput? Just muck around with the percentages in yellow. I’ve used this format for both LoadRunner and JMeter gigs, so the terminology is meant to be familiar to both those types of users.

JMeter Test Plan
Through the beauty of JMeter -J command line parameters I’m able to pass all the relevant metrics from the ASM such as number of threads (vusers), duration, throughput, rampup and action(transaction) distribution percentages. I use this template format quite often and have explained some of these concepts previously.

The test plan gets the required parameters from the command line at run time. You can see the User Defined Variables look something like this:
DURATION = ${__P(duration,60)

Which basically sets the ${DURATION} JMeter variable based on the -Jduration command line variable, or defaults to 60 seconds if it is not passed. Other variables are also defined here and get loaded into the main Thread Group.

I use a Runtime Controller to control how long the test plan executes for. I also keep a separate Transaction Controller called Init to execute login actions etc or things I only want to execute once in the test plan.

Nested inside the Runtime Controller I use Throughput Controllers with percent executions tied to action percentages defined in the ASM. For example, action 1 might be a “view product” sequence of requests which I execute 50% of the time and so forth. Similar to weighted percentages for transaction blocks in LoadRunner. It is best to calculate these percentages using the Per User mode in JMeter for more accurate results.

Nested inside Throughput Controllers will be further Transaction Controllers, with Generate Parent Sample checked to enable similar functionality to transaction timers in LoadRunner around groups of requests.

The whole Thread Group has its throughput limited with a Constant Throughput Timer which basically tries to make all active threads in the thread group stay at a constant rate (defined by samples per minute). This eliminates the need to figure out pacing or sleep times as you might in LoadRunner.

The Summary Report and Gaussian Random Timers are just used for debugging so don’t normally include those in the final script…

Git
Want to get started? Well first you will need a copy of a git client. Have a look at various how-to guides out there to get you started. I’ve found this site pretty useful for understanding and working with git basics. My public clone URL for all the code used in this post is available here:
git://github.com/90kts/performance_test.git

Enjoy!

Some people get confused with the additional protocols that LoadRunner sports for SAP, particularly the SAP GUI. But if your SAP portal uses plain old HTTP (or HTTPS) then JMeter can do the job.

There are a [...]]]> A colleague recently asked me if it was possible to test SAP portals using JMeter. The short answer is yes.

Some people get confused with the additional protocols that LoadRunner sports for SAP, particularly the SAP GUI. But if your SAP portal uses plain old HTTP (or HTTPS) then JMeter can do the job.

There are a few gotchas which I will continue to update on this post, namely around correlation. In JMeter, correlation is particularly *easier* because you don’t have to worry about where to position a correlation rule. The equivalent function to use is a Post Processor -> Regular Expression Extractor.

I used the following rules to help correlate the SAP portal:

Response Field to check: Body
Reference Name: sap-ext-sid
Regular Expression: sap-ext-sid=”(.+?)”
Template: $1$
Match No.: 1

Response Field to check: Body
Reference Name: sap-wd-tstamp
Regular Expression: name=”sap-wd-tstamp” value=”(.+?)”
Template: $1$
Match No.: 1

Response Field to check: Body
Reference Name: sap-ep-iviewhandle
Regular Expression: name=”sap-ep-iviewhandle” value=”(.+?)”
Template: $1$
Match No.: 1

Just scope those globally in your thread group and they will find and populate the necessary variables with ${variable_name}
All you need to do to your final script is find and replace all instances of those variables with the JMeter variable in your favourite text editor. On windows I find SciTE is free (normally installed with Ruby) and powerful. e.g.

Replace:
7QQ8jk_TEknhboPm9fmgqw–Mce5By08HC1RDy9m4LImcA–
With:
${sap-ext-sid}

In our particular portal a cookie value is used in the context of a query parameter (or form value). So to read a cookie value from the HTTP Cookie Manager element at run time (make sure you include it in your Thread Group) you just use another regular expression extractor:

Response Field to check: Header
Reference Name: MYSAPS02
Regular Expression: MYSAPSSO2=(.+?);
Template: $1$
Match No.: 1

Note in this case I am checking the header rather than the body for this value. Then use the variable as before.

Another tip when recording SAP portals, you might want to use the BadBoy recorder which can export to JMeter test plans. I had more success using this over the traditional recording proxy in JMeter.

java -version java version "1.6.0_11" Java(TM) SE Runtime Environment (build 1.6.0_11-b03) Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)

Then you are likely to encounter this error when using the standard HTTP Request sampler:

Response code: Non HTTP response code: java.net.SocketException Response message: Non [...]]]> If you’re testing HTTPS with JMeter 2.3.2 and a current version of Java greater than 1.5 e.g.

java -version
java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)

Then you are likely to encounter this error when using the standard HTTP Request sampler:

Response code: Non HTTP response code: java.net.SocketException
Response message: Non HTTP response message: Unconnected sockets not implemented

If you have the opportunity, use the HTTP Request HTTPClient sampler instead and this problem should be resolved. Otherwise use Java 1.5 instead. If you’ve already recorded your test plan in JMeter or other tools (like BadBoy) then some simple regular expressions on the jmx file will fix the problem manually.

Windows:

ruby -pi.bak -e "gsub(/

ruby -pi.bak -e "gsub(/HTTPSampler>/, 'HTTPSampler2>')" *.jmx


Mac:

ruby -pi.bak -e "gsub(/

ruby -pi.bak -e "gsub(/HTTPSampler>/, 'HTTPSampler2>')" *.jmx

I had not heard of this approach before and am sure it will be useful in future, provided [...]]]> This is a relatively new example of how to use JMeter with WebSphere MQ. This approach is a clever way of inserting messages into MQ using a JMS sampler (point to point) with a JNDI binding within MQ.

I had not heard of this approach before and am sure it will be useful in future, provided you have sufficient access to create the bindings in the first place.

However, if it is vital to your script (for [...]]]> This is most likely because the JMeter recording proxy does not support the https protocol. If it’s a screen you can live without, you can just stop/start the proxy recorder around that page (I use the Firefox plugin called FoxyProxy to quickly switch between different proxy configurations).

However, if it is vital to your script (for example, a login page) then you can use an alternative recording tool called Badboy Software. I frequently use this to record HTTPS then export the test plan to JMeter. Alternatively you can manually create the HTTPS request sample yourself.

JMeter can run HTTPS request samplers, it just can’t record them using the native proxy recorder.

If you roll-up your transactions into separate transaction controllers as I demonstrated in a previous post, you will get useful [...]]]> This plugin has been around for a while and it’s definitely worth including in your JMeter library if you want to go for a glossy finish! The StatAggVisualizer will transform your graphs
from this to this

If you roll-up your transactions into separate transaction controllers as I demonstrated in a previous post, you will get useful summary views for all your transactions:

Alternatively, drop a stat agg visualizer listener under a single controller to get a graph for a single transaction, useful if clients are only interested in certain transactions.

One of the first things I like to do when performance testing is establish in reasonable [...]]]> I’ve been working with JMeter of late and after speaking with some people at conferences and the like, I thought it would be good to share what I’ve learned along the way in building, structuring and executing JMeter test plans.

One of the first things I like to do when performance testing is establish in reasonable detail what the Application Simulation Model (ASM) will entail. You can create an ASM visually using something like UCML. This is a good starting point for performance testers looking to structure their test planning efforts.

UCMLâ„¢ is a set of symbols that can be used to create visual system usage models and depict associated parameters. When applied to Performance Testing these symbols can serve to represent the workload distributions, operational profiles, pivot tables, matrixes, and Markov chains that performance testers often employ to determine what activities are to be included in a test and with what frequency they’ll occur.

More often than not though, I create an ASM using a spreadsheet format as I find this easiest to communicate with people from the business teams. If people are interested I will share some of the spreadsheet templates. The following screenshot gives you an idea of what I’m talking about:

The pertinent points for an ASM, is that it should:

e.g. “we have 3 user groups with a total of 150 users, 80% of them are helpdesk support, 10% are team leads and 10% are management”

e.g. “we’re aiming for around 240 trans/hour, of which 50% are searches, 25% are updates and 25% are reports”

Read on to discover how to turn ASMs into working JMeter test plans.

One of things I’m aiming for is to use relative random distributions of transactions within a loop controller which simulates the transaction distributions in real life. The following screenshot shows a Loop Controller, with three children Throughput Controllers (renamed to 20%, 50% and 30%). Each Throughput Controller then has a Transaction Controller, which is the parent container for all subsequent requests that make up that transaction (there are typically tens or hundreds of requests here depending on the complexity of the web application). Note I have also renamed the Transaction Controllers to something more representative of the transactions being simulated.

The Loop Controller will effectively control the number of iterations and I often set this to run forever when threads are shut down manually during test runs. I also use a Constant Throughput Timer, which alleviates the need to set iteration counts and pacing between iterations which LoadRunner users would be familiar with. When using a Constant Throughput Timer, you can simply set the targeted transaction rate in terms of samples/minute (in this case it’s measured on collective Transaction Controller’s throughput). JMeter will then self-throttle the load to try and match that transaction rate. If you want more granular control, you can move CTT to be child elements of selected Transaction Controllers, thus applying different transaction rates for different transactions. The benefit in using Throughput Controllers with Percent Execution settings specified, is that transactions will be randomly distributed according to those percentages down to each subsequent Transaction Controller. The following screenshot demonstrates those settings used for the 20% Throughput Controller:

This means that if I specified 100 iterations in my Loop Controller, then 20 transactions will be distributed to the 20% Throughput Controller. This is done in a relatively random fashion, it does not process transactions in sequence which is good for random simulations.

Once you’ve established your transaction groups with percentage distribution, you can then create your user groups using the Thread Group element. In my example I’m using two user groups (Group A and B), each with differing user quantities. The purpose of these is just to pigeon hole my user groups and quantities as detailed in the corresponding ASM. Each user group is run in parallel, and is subject to the startup (ramp-up) and loop count settings for the threads (users) created. The following screenshot shows some typical settings:

A more complete test plan looks like the following screenshot:

You can see two user groups in there, along with Loop Controllers and Throughput Controllers. You can also see all the sample requests under each of the Transaction Controllers. In my example I’m using HTTP request samplers, but this could just as easily be a web service, ftp or some other form of JMeter sampler. I often start with this template in mind, then record and edit transactions in the Workbench before importing back into my Transaction Controllers.

Basically JMeter will run through the test plan from the top to the bottom. For this reason I often keep my configuration elements such as HTTP cookie managers, user parameters and reporting listeners towards the top of the test plan so they get initialized. Then, as it hits the Thread Groups it will launch those according to the settings specified. You can also see in the screenshot some results from the View Results Tree listener for Thread Group A. Essentially this demonstrates the randomized distribution of Transaction Controllers according to their parent Throughput Controllers.

In coming posts I’ll walk through some other features of JMeter that help you control thread execution and delve deeper into the reporting functionality that comes with JMeter. You can download a copy of the jmeter test plan referred to in this post here.