We were lucky to be selected by the development team at Cordelta to help them automate and performance test a high profile public website called MyHospitals. We were able to test millions of hits per hour from domestic and international locations in a wide variety of load scenarios. The success of this approach was underpinned by the following:

1. An open minded project / development team not coupled to a ‘must-have-most-expensive-toolset-to-do-job’ mentality
2. An open minded performance test analyst (me!) who believes Excel really is the grandpa of charting, R is the grandma of stats, Sparklines are the only way to present time series data to management, ANY tool can simulate load via HTTP/S and that there is no real distinction between good software testers or developers (only hard work separates the best from the worst).
3. That 2010 come-no-doubt 2011 buzz word… Cloud

As I sit here pluggin’ our alternative approach at Altentee, I can see the rise of other more successful punters taking on the big kids. I sit and [continue to] chuckle at the reaction to LR pricing and LR zealots who will fall on their swords over LR itself. I have come to realise one thing though, it is not about the tool, or even the alternate tool like perhaps I first thought. It is more about the freedom of choice.

To tackle MyHospitals I was free to choose and implement the following tools:

1. WatirGrid to orchestrate a small flotilla of IE and FireFox based browsers based on Watir
2. JMeter to add more at the protocol level of performance testing
3. httperf to do some basic benchmarking, similar to my front page
4. numbrcrunchr to pull together system metrics and make for easier analysis
5. A variety of Australian cloud providers and of course Amazon EC2 to host the test environment

It has been a great year. I’m not entirely free of the commercial chains just yet and am still needing the LR type work to prop up this approach, but I hope 2011 brings about some fresh thinking in performance and test automation with hopefully me somewhere amidst that space.

To cut through the chaff, these are some of the questions I like to ask potential candidates. What’s your favourite questions? How might you answer these?

I also like to follow up these questions with a half hour practical scripting in VuGen (or your tool of choice). You can learn alot about someone by the way they write their scripts.

The point? Determine if the person has recently used the tool (you’d be surprised) and suss out general knowledge of performance testing…

1. What are essential components of a performance test plan? Give an
example of a non functional requirement.
Test objectives, scope, scripts, scenarios, management, schedule, data,
environment, access, defect management, traceability requirements, roles &
responsibilities.

An example NFR might be response time and/or volumetric requirements. e.g.
this application/transaction is targeting a 5 second response time, with 20
concurrent users and peak 20 trans/min.

2. Describe steps you’d take to setup a script / load harness and confirm
correct operation / configuration.
Document manual test case, dry run, record 1st time, record 2nd time to
help diff dynamic data, parametize & correlate, replay. Validate using
server logs, database or other areas which demonstrate correct application
functionality.

3. What OS level resource metrics would you use to detect bottlenecks?
CPU, memory, disk, network

4. Which OS commands would you typically use to collect those metrics
(Solaris, AIX etc)?
sar, prstat, vmstat, iostat, netstat etc

5. What are 2 advantages of using Performance Center?
Provide a central repository of scripts and their results
Simplify licensing and distribution of software

6. Describe differences between correlation and parameterization in a LR
script.

Correlation
To simplify or optimize your code
For example, if you perform a series of dependent queries one after
another, your code may become very long. To reduce the size of the code,
you can nest the queries, but then you lose precision and the code becomes
complex and difficult to understand. Correlating the statements enables you
to link queries without nesting.

Parameterization
When you record a business process, VuGen generates a script that contains
the actual values used during recording. Suppose you want to perform the
script’s actions (query, submit, and so forth) using different values from
those recorded. To do this, you replace the recorded values with
parameters. This is known as parameterizing the script.

7. What LR function would you use to save a parameter in a web-based
script?
web_reg_save_param()

8. Name 3 types of performance test types or methodologies you might employ
in a typical test effort?
Shakeout, Baseline, Load, Volume, Stress, Soak, Tuning

9. Which blogs/user groups/forums would you use to help solve VUGen
scripting difficulties?
Google
HP support site
SQA forums
Yahoo or Google user groups
90kts

10. Broadly describe typical run-time settings you’d employ in your VUGen
script.
Run logic set to number of iterations
Pacing to control start of iterations
Log settings appropriate for test type
Think time as recorded or random percentage
Browser emulation with/without cache

11. What other types of resource/application level metrics might you
monitor? Which tools would you require?
Web Servers – availability, bytes tx/rx, workers, requests, memory, cpu
Application Servers – JDBC pools, thread pools, servlet sessions, EJBs,
JVM (memory util, garbage collection)
Database Servers – server, tablespace and database metrics, locks, buffer
ppols, reads, writes, slow queries
Messaging Servers – queue stats, message flows, node statistics
Network Devices – packets/bytes rx/tx, errors, discards

12. How does SiteScope facilitate agentless monitoring of unix hosts?
Uses ssh sessions to log in and retrieve OS and app level metrics from
command line.

13. Define the differences between response time, throughput and latency.
Response time is normally end to end time taken for a round trip response. Can include many sub-components of the system under test.Normally expressed in milliseconds or seconds.

Latency is a period of delay. Normally expressed in milliseconds or
seconds.

Before we proceed to discuss Performance Management in the world of Web 2.0 solutions, let us briefly examine where the core performance management disciplines evolved from. By doing so we can see that the fundamentals of Performance Management have not changed but in fact are required more than ever.

I do not want to highlight my age, neither promote myself as a mainframe bigot. But the mainframe did establish many mature software engineering disciplines, such as release, change, configuration, environment, problem and performance management. All of these disciplines fed into mature processes such as the Unified Process, which are still used today for the development of new software platforms.

When talking specifically about performance management, in the days when mainframes were the sole core delivery source of customer data, there was a strong performance management discipline. This discipline provided assurance and risk mitigation to the owners of these systems that the customers would not receive any negative experiences when interacting with these systems such as bad response times or system outages. Using the IBM MVS mainframe as an example, there were such tools as OMEGAMON for monitoring queues, cpu, disk and memory, RMFMON and SMF for performance sampling response times, SAS for statistical analysis and TPNS for load simulation. Proactive processes were in place to ensure that the IT departments were aware of any potential performance impacts prior to the customer finding them. Large financial institutions were the major supporters of performance management in this area as they understood system outages could lead to large monetary losses. Today, mainframes continue to be supported by a strict performance management regime.

As the move was made into developing client/server solutions, and progressed to web based solutions, these disciplines, specifically Performance Management, seemed to become temporarily forgotten or maybe just received less priority. This was short lived after a large number of massive failures and subsequent monetary losses were encountered as a result of web site failures caused by halts and excessive response times. Soon the once forgotten discipline of performance testing was once again included as a core phase of the software development life cycle. Once again test managers, project managers and even senior executive management members were talking about performance testing activities, results, success or failure of KPIs or Service Level Agreements (SLAs) prior to deployment. However, a lot of companies, primarily due to costs involved in performance testing activities, still omitted these activities and suffered the consequences. Although, even those companies who performance tested their applications prior to deployment suffered later as they neglected to encompass the whole performance management package, that used to protect the mainframe from system failures. For performance testing, although a major contributor to the Performance Management solution, is only part of an overall holistic performance management solution.
As there is more hype and talk about Web 2.0 solutions, there is even more necessity for IT departments to deliver a complete Performance Management package. The end user experience is now more complicated to gauge using the old Load Testing 1.0 philosophy of neglecting those components past an external firewall, even the client performance itself had been previously ignored. We will explain in the next section why performance risk areas in Web 2.0 solutions are more prevalent .

This whitepaper will explain to the reader what is required to deliver a complete holistic approach when mitigating performance risks. It will explain what should be incorporated in a Performance Management package and also highlight why companies should ensure they encompass the complete set of Performance Management disciplines, rather than a small subset of them when preparing to embark on any new software development project.

Web 2.0 Performance Risk Areas

Before we get into the detail of what performance management is lets just stop here for a minute and discuss at a high level some performance risk areas within Web 2.0 solutions.

Please refer to the many whitepapers on the internet that will provide a more in depth explanation of what is meant by Web 2.0 solutions. For our purpose here, we will provide a brief description of what is meant by it, highlighting why this has more performance impacts if not properly managed.

Client Rendering

Web 2.0 solutions now deliver more functionally rich capabilities to the web browser. Technologies such as Flex from Macromedia, as an example, take the processing responsibilities away from the server and place them on the client.

Ajax controls and more client powered java scripts, empower the client browser, but now place more importance on the performance measurement of what we call client rendering. Client rendering is the process that the web browser takes to present the complete refreshed response to the end user.

Client rendering could just be the populating of an Ajax drop down list control, or it could be the time take to completely draw that all important Sales graph. Performance testing for Web 1.0 solutions has normally focussed on server performance, although this is still important we need to ensure that we now measure response times on the client as well. A complete end to end user response needs to be measured when the applications servers are not under load, and then when they are under load, to determine whether or not the the response time is acceptable.

Web Services

More Web 2.0 solutions are utilising the flexibility of using external web services as defined by the World Wide Web Consortium (W3C). There are many obvious benefits in utilising external web services, specifically in saving development time and costs. However, your application now has a dependency on performance of an external site. How do you ensure that this performance is adequate?

Geographical Locations

As we know, the benefit of the internet is that users of web site applications can be geographically dispersed. However the downside of geographically dispersed users is determining how to measure the response times of a new web site under test, when so many variables are involved.

With the development of Web 2.0 solutions, the client becomes more functionally rich, and more resource intensive. The complete end to end user experience for every user, no matter their geographical location, now has to be managed by a performance management solution.

RSS Feeds

There has been a rapid increase in the amount of web sites using RSS (Rich Site Summary) feeds. Basically the RSS feed provides a format to deliver regularly changing content to your website such as news, weather and stock prices as an example.

Again, we now have external dependencies on performance. Also the feeds into the web site will have an unknown performance impact to the other processes running on the clients computer, which will impact the users experience with the web site.
These are some areas with Web 2.0 solutions that magnify the necessity to performance manage your web site. Together with the existing issues surrounding web 1.0 solutions, there is now more than ever a need to encompass the performance management regime.

What do we mean by Performance Management?

Now we have highlighted some of the issues associated with the deployment of new web site applications specifically Web 2.0 solutions, lets now elaborate on performance management.

When it comes to discussing performance most IT practitioners are now aware of what we refer to as performance testing, although a lot still do not understand how this is done in a properly focussed approach, but that is another white paper in itself. However, as we mentioned previously, performance testing is one aspect of performance management.

A complete performance management solution, will performance test in a controlled approach, in an appropriate environment, before live deployment. It will then continue to performance regression test subsequent releases, to ensure no performance degradation has occurred between releases. A capacity planning discipline should be adopted for the live site, where statistics and models are reviewed on a regular basis. The user end experience should be proactively monitored using an appropriate dashboard. Lastly all server metrics should be monitored and fed into a viewable dashboard realtime.

That in a nut shell is what we mean when we talk about Performance Management. Now lets discuss some of these aspects in further detail.

Performance Testing

This deserves a white paper of its own. So without spending too much time on it here, check our website www.altentee.com for more on the different types of performance test types.

The most important ingredients into successfully performance testing is focussing on the high volume and business critical functionality of the web site under test. With the emphasis on end user response times, it is now necessary to not only performance test the appropriate servers, but to include the client rendering response time measurement in overall end to end response time measurement.

By utilising some open source alternatives to reduce costs, such as JMeter or OpenSTA, web traffic can be successfully emulated in varying loads on the web, file, application and database servers, by mimicking the browser incoming traffic. Whilst the servers are being loaded in this manner, geographically dispersed real browsers can emulate real user response times from varying locations. This can be achieved via cloud computing and such tools as BrowserMob, which altentee has an alliance with.
For the test environment and load generators, the Amazon Elastic Cloud Computing (EC2) facilities can be leveraged to again massively reduce the associated costs.

Throughout the various performance tests, server metrics and user end response times should be collected and examined. Refer to the monitoring section discussed further on.

To ensure no performance degradation is encountered between code releases or any other changes, rapid performance tests should be conducted prior to any changes being deployed to production. Technical impact assessments can be conducted to determine what specific performance tests should be executed.

Sometimes it is forgotten that only one bad line of code, or one incorrect value of a configuration file can cause serious application performance problems, and can be a pest to find in production.

Capacity Planning

A proper capacity planning discipline should be adopted for any new website. This can be as simple as a projection model which can determine when links need upgrading, or new hard disk or memory will be required.

A projection model can be prepared based on ballpark projection figures, either sourced from historical data or from business projections. By regularly reviewing real statistics from the usage of the site by examining web logs and other sources, the model can be updated to ensure more realistic projections. Figures from the server monitoring solution, discussed later can feed into the model as well.

Capacity Planning is a logical exercise that when done correctly can save your web site a lot of downtime.

Proactive Performance Monitoring

The performance monitoring solution is the glue that holds the performance management solution together. It is vital that you proactively monitor the performance of both your production and test environments.

By proactively monitoring performance we mean consistently collecting server metrics and user end response times from all your environments, and reviewing these on a regular basis. Better still, setting up various thresholds and automatically receiving alerts once these thresholds are breached. This approach is in contrast to the passive monitoring approach, where performance metrics are only examined once a problem has been reported by an end user.

The key to performance monitoring is not only monitoring all the vital server metrics, but also monitoring the critical business processes from a users perspective. These can be done by automating these processes and running them on a regular basis.

Conclusion

By adopting the three key performance management areas of performance testing, capacity planning and proactive performance monitoring, you will be well on the way to guarding your website from costly outages. You can also safely support and encourage any business campaign or promotion that leads to increased traffic to your web site.

Nick Singh is an Australian based Performance Testing practitioner. He is a co-founder of Altentee, a company that prides itself on focused and cost effective performance management solutions. Nick has 28 years experience in IT of which 23 years have been dedicated to his passion for performance testing.

Altentee is a company based in Melbourne, Australia that specialises in performance management solutions. It prides itself on expertise in performance management and has expert knowledge in all toolsets both commercial and open source. This allows the company not only to provide quality performance management solutions, but cost effective solutions as well.
For further information on how Altentee can help your company, please contact Altentee on
Phone 03 9005 NTEE
Or +61 3 9005 6833

Suite 71, Building 2, 574 Plummer St
Port Melbourne VIC 3207
Australia

Here is a great analogy of throughput and latency. I don’t pretend to understand queuing theory (just yet), but am working on it

Given a list of n integers and another integer called m, determine (true / false) if there exist 2 numbers in that list which sum up to m.
Example: 2,6,4,9,1,12,7 and m=14 -> 2 and 12 sum up to 14, so the answer is true.
Provide the best algorithm in both manners: performance and memory to solve this puzzle. Don’t forget to mention the complexity of your solution!

Read on for my thoughts in solving this challenge.

Looking at the problem, you obviously start with an array of numbers called n. If the sum of two numbers from the array is m, then you know that the array *may* contain two elements which are equal to m, such as n[a] + n[b] = m.

However you don’t know which two elements of the array might achieve this, nor do you know how many pairs in the array would satisfy the same conditions.

So my first thoughts were to iterate through each element of the array, then for each element, subtract that value from m, and iterate through remaining elements in the array looking for that value.

def test_nosort(m,n)
  found=0
  n.each do |a|
    b = m - a
    n.each do |x|
      found += 1 if (x.to_i == b)
    end
  end
   puts found if @debug
end

The problem with this approach, is that for a starting array size of n, you have to check all possible combinations or n*n-1. Say for an array of size 100 elements, you have to check each value in the array 100-1 times (excluding current value), which means 100*99 = 9,900 possible combinations!

In the example provided above, I used the #each iterator for the array believing this to be the fastest operation when iterating through an array in Ruby.

This approach is suitable for small sized arrays (< 100 elements), but quickly becomes unusable when scaled up. In hindsight, this approach seems very similar to a bubble sort algorithm, where the list to be sorted has two items compared at a time and swapping them if they’re in the wrong order; iterating as many times until no more swaps are needed. It has a worst-case complexity described as n^2 where n is the number of items being sorted.

In the next approach, I decided to give regex a try ( I love regex ), where in this case I still use an #each iterator to loop through the array, but use a regex grep comparison against the original array. The following code demonstrates:

def test_regex(m,n)
  found=0
  n.each do |a|
    b = m - a
    found += n.grep(b).length
  end
  puts found if @debug
end

This approach performed better than the first approach. I also experimented with some other array methods such as #select, #detect and #partition all of which gave varying but slow results.

After a much needed coffee and re-think, I decided to build a hash, where each key was based on the original values in my array, with the key-value equaling a count of all occurrences for that key… Then I could just use the array #each iterator and jump straight to the number of values that match that key. The following code demonstrates:

def test_hash(m,n)
  found=0
  hash = {}
  n.each do |a|
    hash.has_key? a ? hash[a] = hash[a].to_i + 1 : hash[a] = 1
  end
  n.each do |a|
    b = m - a
    found += hash[b] if hash.has_key? b
  end
  puts found if @debug
end

This approach was a marked success over previous approaches and it scales well for large arrays of numbers. You can see the differences in time (using the benchmark gem) for each approach when using an array size of 100 elements as follows:

      user     system      total        real
test_nosort results :
  0.530000   0.000000   0.530000 (  0.534364)
test_sort results   :
  0.530000   0.000000   0.530000 (  0.534992)
test_regex results  :
  0.360000   0.000000   0.360000 (  0.357073)
test_detect results :
  6.390000   0.040000   6.430000 (  6.480277)
test_select results :
  0.400000   0.000000   0.400000 (  0.403032)
test_partn results  :
  0.600000   0.010000   0.610000 (  0.623047)
test_hash results   :
  0.020000   0.000000   0.020000 (  0.021390)
test_binary results :
  0.270000   0.000000   0.270000 (  0.273587)

Even with array sizes of 10,000+ elements the benchmark was in the order of seconds.

However with an even larger array size of say 100,000+ elements, the memory usage of this hash search would be significant. I also tried a binary search but found this had worse performance than a hash. From reading wikipedia though it would appear that a binary tree search might be the best bet in terms of memory usage for large arrays.

The Answer?

For now I am satisfied that a hash key search is the least complex way to deal with this particular challenge, with the only caveat being a ceiling limit on the amount of elements that are in your initial array to begin with e.g. 1 – 10,000 elements. For larger arrays, binary tree algorithms might offer better memory performance but would be more complex to implement in a scripting language like Ruby.

The code I used to test this is attached below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136

require 'benchmark'
 
# m = 14
m = rand(100)
# n = [2,6,4,9,1,12,7]
n = Array.new(100) { |e| e = rand(100) }
@debug = false
 
def test_nosort(m,n)
  found=0
  n.each do |a|
    b = m - a
    n.each do |x|
      found += 1 if (x.to_i == b)
    end
  end
   puts found if @debug
end
 
def test_sort(m,n)
  found=0
  n.sort!
  n.each do |a|
     b = m - a
    n.reverse_each do |x|
      found += 1 if (x.to_i == b)
    end
  end
  puts found if @debug
end
 
def test_regex(m,n)
  found=0
  n.each do |a|
    b = m - a
    found += n.grep(b).length
  end
  puts found if @debug
end
 
def test_detect(m,n)
  found=0
  n.each do |a|
    b = m - a
    found += 1 if n.detect { |x| /\b#{b}\b/ =~ x.to_s }
  end
  puts found if @debug
end
 
def test_select(m,n)
  found=0
  n.each do |a|
    b = m - a
    found_array = n.select { |x| x == b }
    found += found_array.length
  end
   puts found if @debug
end
 
def test_partn(m,n)
  found=0
  n.each do |a|
    b = m - a
    found_array = n.partition { |x| x == b }
    found += found_array.length
  end
   puts found if @debug
end
 
def test_hash(m,n)
  found=0
  hash = {}
  n.each do |a|
    hash.has_key? a ? hash[a] = hash[a].to_i + 1 : hash[a] = 1
  end
  n.each do |a|
    b = m - a
    found += hash[b] if hash.has_key? b
  end
  puts found if @debug
end
 
def test_binary(m,n)
  found=0
  n.each do |a|
    b = m - a
    found += 1 if binary_search(n, b)
  end
  puts found if @debug
end
 
def binary_search(array, val, left = 0, right = nil)
  array.sort!
  right = array.size unless right;
  mid = (left + right) / 2;
 
  return nil if left > right
 
  if val == array[mid]
    return mid
  elsif val > array[mid]
    binary_search(array, val, mid + 1, right)
  else
    binary_search(array, val, left, mid - 1)
  end
end
 
Benchmark.bm() do |timer|
  @debug = false
  timer.report("test_nosort results :\n") { 100.times { test_nosort(m,n) }  }
  timer.report("test_sort results   :\n") { 100.times { test_sort(m,n)   }  }
  timer.report("test_regex results  :\n") { 100.times { test_regex(m,n)   }  }
  timer.report("test_detect results :\n") { 100.times { test_detect(m,n)   }  }
  timer.report("test_select results :\n") { 100.times { test_select(m,n)   }  }
  timer.report("test_partn results  :\n") { 100.times { test_partn(m,n)   }  }
  timer.report("test_hash results   :\n") { 100.times { test_hash(m,n)   }  }
  timer.report("test_binary results :\n") { 100.times { test_binary(m,n)   }  }
end
 
# >>       user     system      total        real
# >> test_nosort results :
# >>   0.530000   0.000000   0.530000 (  0.534364)
# >> test_sort results   :
# >>   0.530000   0.000000   0.530000 (  0.534992)
# >> test_regex results  :
# >>   0.360000   0.000000   0.360000 (  0.357073)
# >> test_detect results :
# >>   6.390000   0.040000   6.430000 (  6.480277)
# >> test_select results :
# >>   0.400000   0.000000   0.400000 (  0.403032)
# >> test_partn results  :
# >>   0.600000   0.010000   0.610000 (  0.623047)
# >> test_hash results   :
# >>   0.020000   0.000000   0.020000 (  0.021390)
# >> test_binary results :
# >>   0.270000   0.000000   0.270000 (  0.273587)