1. Shakeout – single script, single user, low volume scenarios whose main purpose is to confirm a correctly configured test harness (script, parameters, data, correlation, iterations) and environment.
2. Baseline – arbitrary user/volumes (typically low with multiple iterations) used to establish a baseline or reference point for further testing. Can also be used to explore/set SLAs if they have not been defined by the business using minimum sample sizes appropriate to establish a baseline.
3. Load / Volume – often expressed in terms of intended production load at 100% or variations of depending on the desired test outcome. Can also incorporate growth scenarios e.g. 200%. Can also be structured to incrementally increase and sustain load up to a target so that utilization metrics can be collected for capacity modelling at each increment (e.g. ramp up from 100 to 200 users at increments of 20 additional users, with sustained execution of 20 minutes at each increment).
4. Stress / Stress to Break – loads greater than intended production loads or with the specific purpose to identify component failure or ‘break’ points. Other variants include Surge / Spike testing simulating peak demands over shorter timeframes.
5. Soak / Endurance – long running tests to establish performance over longer timeframes and any anomalies over time.
6. Failover / Availability – targeted tests under load in failover conditions. Can also be used to test disaster recovery or availability scenarios.
7. Component Based – Targeted tests designed to isolate and examine components (functional and/or specific technologies/platforms) under load.
8. Penetration / Security – with the increasing inclusion of security related non-functional requirements, targeted load tests which focus on application level and/or hardware level security NFRs.
9. Tuning – cyclic / iterative performance testing using any number of variables in a process of test, tune, retest and compare (for improvement).

Still confused? Contact Altentee and we’ll help explain our approach.

On a basic level, the honeybee’s dilemma is a tale of two flower patches. If one patch is yielding better nectar than the other, how can the hive use its workforce most efficiently to retrieve the best supply at the moment? The solution, which earned Austrian zoologist Karl von Frisch a Nobel Prize, is a communication system called the waggle dance.

Saw this on Slashdot … The waggle dance

The first behaviour I wanted to describe, was whether the use of MQ persistence creates random or sequential disk activity, on the assumption that random disk access to block locations on the disk would incur a greater time penalty than sequential access.

In order to setup a test environment, I installed MQ version 6 on winXP platform using the default settings. This test was setup using a custom java load harness to put/get messages from a default queue which had persistence enabled with DEFPSIST(YES). File and disk statistics were collected on a Win32 platform using filemon and diskmon respectively.

The following graphs depict the block size written over time (x axis) relative to the sector position (y axis) of the disk drive being monitored. I was trying to achieve a similar effect to the Solaris internals TAZ disk trace utility, albeit in a Windows environment. File stats such as number and size of write operations were collected using filemon. Transaction throughput was measured by dividing the total amount of transactions (1,000) by the total processing time. A number of different configurations were tested, but I chose to present a common setup which uses circular logging with single write, and a file page size of 16,384 x 4KB = 64MB per primary log file. The two variations I have presented are writes (MQPUT) only as well as writes (MQPUT) and reads (MQGET) concurrently. The latter being a more familiar situation for me in the workplace.

It can be determined from the previous test results that MQ accesses the disk in a sequential behaviour. On windows, depending on the concurrent load, it will tend to allocate 16 – 32 x 512B blocks ranging from 8 – 16KB in size, although smaller less frequent writes will favour smaller 8 x 512B blocks 4KB in size. As a result, a high number of disk write IOs are observed.

The environment used to test was virtual (parallels), so limited disk cacacity was available, however the change in amplitude for trans/sec is considered relative. As a result, over an 80% reduction in throughput was observed when reading (MQGET) and writing (MQPUT) to the same queue. This will have significant impact on disk subsystem performance in a production environment.

I was unable to perform this test on a Solaris environment due to lack of privileged access, so would be keen to see what the results are there. I believe Solaris uses the same 512B block size, so am expecting similar results. Although analysing an MQ server’s impact on disk performance using sar, produces the following information:
sar -d 1 10 | egrep 'md72'
md72 60 0.7 1867 16749 0.0 0.4
md72 58 0.6 1955 17272 0.0 0.3
md72 52 0.6 1768 15619 0.0 0.3
md72 52 0.6 1764 15577 0.0 0.3
md72 54 0.6 1817 16045 0.0 0.3
md72 56 0.6 1861 16447 0.0 0.3
From that information I can then infer that the total reads + writes/sec (r+w/s) divided by the number of 512B blocks per second (blk/s) would give me an average size of around 8-9KB/sec being written to the filesystem, which correlates my assumptions on block/file sizes. Using sar and other tools like iostat though is thought to be fraught with danger, especially when monitoring a rather complex SAN subsystem; but without access to any other tools, I’m stuck with the basics. To date I’ve been relying on perceived throughput (Mr+Mw/s) reported by iostat -xM

Worth noting, is that when MQ performs an MQGET on a persistent queue, it also causes a write IO to update the active log file, so the number of total write IOs effectively doubles.

Out of all this I’m working on the following assumptions:
1. MQ uses sequential disk access for persistence logging.
2. The OS is typically allocating between 16 – 32 x 512B blocks ranging between 8 – 16KB in size.
3. MQGETs are a contributor to overall writes (I’m assuming o remove the transactions from the log files)
4. Transaction throughput is significantly reduced when an application is reading (MQGET) and writing (MQPUT) to the same queue.

Attached below is a copy of the qm.ini I am working with, which is based on recommended performance tuning considerations from IBM
#*******************************************************************#
#* Module Name: qm.ini *#
#*******************************************************************#
#ExitPath:
ExitsDefaultPath=/var/mqm/exits/
ExitsDefaultPath64=/var/mqm/exits64/
#Log:
LogPrimaryFiles=15
LogSecondaryFiles=15
LogFilePages=16384
LogType=CIRCULAR
LogBufferPages=512
LogPath=/var/mqm/log/QUEUEMANGERNAME/
LogWriteIntegrity=SingleWrite
#Service:
Name=AuthorizationService
EntryPoints=13
#ServiceComponent:
Service=AuthorizationService
Name=MQSeries.UNIX.auth.service
Module=/opt/mqm/lib64/amqzfu
ComponentDataSize=0
#Channels:
MaxChannels=400
MaxActiveChannels=400
#TCP:
KeepAlive=Yes
#TuningParameters:
DefaultQBufferSize=1048576
DefaultPQBufferSize=1048576

LogFilePages is set to the maximum 16384 x 4KB making each S0000[n].log file about 64MB in size. You must set this parameter as part of your Queue Manager creation i.e. crtmqm -lf 16384 Hopefully that decreases the frequency MQ needs to loop around the ring of log files. LogPrimaryFiles and LogSecondary files are set to 15, and I haven’t run out of space in my circular log files yet. It is worth reading up on the MQ system administration manual for this, in terms of planning how much space you need. As a planning figure I add the size of MQPUT messages with DEFPSIST + 750B overhead as IBM state, then multiply that amount by my expected throughput. Checkpoint behaviour as part of circular logging activity should hopefully release space as it checkpoints (once every 10,000 transactions I believe), so if you plan properly, you shouldn’t hit the tail end of your ring too soon

The DefaultQBufferSize is 64KB which I increased to 1MB (1048576). Apparently this increases the size of the buffer before writing to the file system, but I haven’t seen the evidence of that with the tools I’m using to monitor disk writes. This area is a little cloudy for me.

I have also modified the LogBufferPages to 512 x 4KB giving me a 2MB buffer for similar reasons.

And finally, I’ve been using a combination of either MQ, a custom perl script, or EMC iorate to benchmark message persistence on different mounts/filesystems etc which has been useful. Especially when you don’t have one or the other available, and you need to determine if the space you’ve been allocated is up to the task of supporting MQ persistence.

In order to truss the process on a Solaris 9 environment, we first had to establish which pid was writing to the log mount. Our sysadmin was able to do this for me with the following:

# fuser -cu /var/mqm/log
/var/mqm/log: 27306c(root) 27197c(root) 19735o(sr123456)

In this case the sr user was the account under which mqm was running, so we were then able to determine the corresponding process name:

# top -s sr123456
2756 sr123456 11 59 0 0K 0K sleep 313:05 12.08% amqzlaa0_nd
20162 sr123456 10 59 0 0K 0K cpu45 406:47 9.49% amqrmppa
19735 sr123456 6 59 0 0K 0K sleep 718:45 3.47% amqzmuc0
19742 sr123456 14 59 0 0K 0K sleep 22:22 0.00% amqzlaa0_nd
19744 sr123456 1 59 0 0K 0K sleep 1:36 0.00% amqpcsea
19752 sr123456 3 59 0 0K 0K sleep 0:16 0.00% runmqlsr_nd
19734 sr123456 2 59 0 0K 0K sleep 0:10 0.00% amqzfuma
19733 sr123456 14 59 0 0K 0K sleep 0:06 0.00% amqzxma0_nd
19751 sr123456 1 59 0 0K 0K sleep 0:00 0.00% runmqtrm
19749 sr123456 1 59 0 0K 0K sleep 0:00 0.00% runmqtrm
19739 sr123456 4 59 0 0K 0K sleep 0:00 0.00% amqzmgr0
19746 sr123456 1 59 0 0K 0K sleep 0:00 0.00% runmqtrm
19747 sr123456 1 59 0 0K 0K sleep 0:00 0.00% runmqtrm
19737 sr123456 2 59 0 0K 0K sleep 0:00 0.00% amqrrmfa
19738 sr123456 2 59 0 0K 0K sleep 0:00 0.00% amqzdmaa

So in this case, the process named amqzmuc0 was the process performing the logging.

For reference, here is a list of processes and what they typically control for MQ:

1. RUNMQLSR - MQ TCP listener (multi-threaded)
2. AMQCLMAA - MQ TCP listener (single-threaded)
3. AMQRMPPA - Channel process pooling job
4. RUNMQCHI - MQ channel initiator
5. AMQCRSTA - MQ receiving MCA jobs
6. RUNMQCHL - MQ sending MCA jobs
7. AMQCRS6B - LU62 receiver channel
8. AMQPCSEA - MQ command server
9. RUNMQTRM - Application trigger monitor
10. RUNMQDLQ - Dead letter queue handler
11. AMQFCXBA - MQ Broker Worker Job
12. RUNMQBRK - MQ Broker Control Job
13. AMQZMUC0 - MQ Utility Manager
14. AMQZMUR0 - MQ Utility Manager
15. AMQZMGR0 - MQ Process Controller
16. AMQRRMFA - MQ cluster repository manager
17. AMQZDMAA - MQ deferred message manager
18. AMQALMPX - MQ Log Manager
19. AMQZFUMA - MQ Object Authority Manager
20. AMQZLAS0 - MQ Local Queue Manager agents
21. AMQZLAA0 - MQ Local Queue Manager agents
22. AMQZXMA0 - MQ Execution Controller


With this info at hand we could then truss the process as per the following:

# truss -D -p 19735


Which produces the following output:

12677624.0012 lseek(18, 2543616, SEEK_SET) = 2543616
12677624.0017 write(18, "0FEE0315\001\0\0\0D6A4 f".., 4096) = 4096
12677624.0018 lwp_cond_broadcast(0xFFFFFFFF7B46CAC0) = 0
12677624.0018 lwp_mutex_wakeup(0xFFFFFFFF7B46CAA8) = 0
12677624.0019 lseek(18, 2547712, SEEK_SET) = 2547712
12677624.0023 write(18, "0FEE02FD\001\0\0\0D6A4 f".., 4096) = 4096
12677624.0024 lwp_cond_broadcast(0xFFFFFFFF7B46C3D8) = 0
12677624.0024 lwp_mutex_wakeup(0xFFFFFFFF7B46C3C0) = 0
12677624.0024 lwp_cond_broadcast(0xFFFFFFFF7B46CCE0) = 0
12677624.0025 lwp_mutex_wakeup(0xFFFFFFFF7B46CCC8) = 0
12677624.0025 lwp_cond_broadcast(0xFFFFFFFF7B46BE88) = 0
12677624.0026 lwp_mutex_wakeup(0xFFFFFFFF7B46BE70) = 0
12677624.0026 lseek(18, 2551808, SEEK_SET) = 2551808
12677624.0030 write(18, "\tD10206\001\0\0\0D6A4 f".., 4096) = 4096


What we can see from this test case is that Solaris is writing 4KB sized blocks of data to the file system in support of MQ persistent logging. I would prefer a higher size (say 8 – 16KB) as the SAN under test would be more efficient and capable of higher throughput, but I guess that is a limitation of the size of messages I am writing (185B per message) and the way Solaris breaks up the IO.

There is a comment from an IBM performance tuning guide for message persistence that states:It is unlikely that poor persistent message throughput will be attributed to the 2MB limit of the queue manager log. It is possible to fill and empty the log buffer several times each second and reach a CPU limit writing data into the log buffer, before a log disk bandwidth limit is reached.

In this, they are referring to the LogBufferPages parameter which I have increased to its maximum configurable size of 512 x 4K pages = 2MB. At this point in time I am still working with the sysadmin in an effort to prove this statement provided by IBM. For the time being we are investigating the seemingly high number of syscalls and context switching which is evident from vmstat:

# vmstat 1
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m1 m3 m4 in sy cs us sy id
0 1 0 27901544 13821744 0 6 8 0 0 0 0 0 0 0 0 11369 173717 83335 16 24 60
0 1 0 27901544 13821688 0 0 8 0 0 0 0 0 0 0 0 11201 175563 84711 14 20 66
0 1 0 27901536 13821648 635 4573 0 0 0 0 0 0 0 0 0 11841 179265 84000 14 26 59
0 0 0 27900496 13821472 847 6045 8 8 8 0 0 0 0 0 0 13003 178012 82634 17 25 58
2 0 0 27901552 13821528 0 0 8 0 0 0 0 0 0 0 0 10994 174223 83744 15 20 64
0 1 0 27901544 13821480 0 0 0 0 0 0 0 0 0 0 0 11210 175509 83986 15 18 66
0 1 0 27901544 13821440 0 0 8 0 0 0 0 0 0 0 0 11178 174778 83952 17 19 63


Another aspect I am investigating is the use of threads my the amqzmuc0 process. It would seem that for a given load amqzmuc0 is spawns around XX threads for that process. Looking at the prstat for that process confirms the following:

# prstat -mL -p 29314
PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
29314 sr53186 0.4 0.7 - - - - 3.2 - 564 157 1K 0 amqzmuc0/3
29314 sr53186 0.0 0.0 - - - - 0.4 - 1 0 1 0 amqzmuc0/4
29314 sr53186 0.0 0.0 - - - - 0.0 - 0 0 0 0 amqzmuc0/6
29314 sr53186 0.0 0.0 - - - - 0.0 - 0 0 0 0 amqzmuc0/5
29314 sr53186 0.0 0.0 - - - - 100 - 0 0 0 0 amqzmuc0/2
29314 sr53186 0.0 0.0 - - - - 0.0 - 0 0 0 0 amqzmuc0/1


Because the architecture had recently changed to a cluster, and my load harness was no longer getting the desired response times, we all went down the track of investigating issues with clustering as this was the new kid on the block. Whilst the developer and sys admin revealed many other interesting tid bits of information regarding correct configuration and the like, we still could not collectively resolve the problem: the COBOL code was throwing a 2195 reason code on multiple MQCONN calls to the queue manager.

Some key lessons I learned here though, was to always simplify the problem by going back to a known configuration. We ended up doing that at the end of the day (using a standard queue manager configuration) and found that the 2195 errors persisted. So this was a good indicator that we had some sort of environment/configuration error.

The second key lesson was to always trust the API documentation. In this case a 2195 error has the following info:

2195 MQRC_UNEXPECTED_ERROR
The call was rejected because an unexpected error occurred.
Corrective action: Check the application’s parameter list to ensure, for example, that the correct number of parameters was passed, and that data pointers and storage keys are valid.

If the problem cannot be resolved, contact your system programmer.
v On OS/390, check whether any information has been displayed on the console. If this error occurs on an MQCONN or MQCONNX call, check that the subsystem named is an active MQSeries subsystem. In particular, check that it is not a DB2 subsystem. If the problem cannot be resolved, rerun the application with a CSQSNAP DD card (if you have not already got a dump) and send the resulting dump to IBM.
v On OS/2 and AS/400, consult the FFST record to obtain more detail about the problem.
v On Compaq (DIGITAL) OpenVMS, Tandem NonStop Kernel, and UNIX systems, consult the FDC file to obtain more detail about the problem.

To be honest, I got put off by the “unexpected error” bit, and had the same reaction I do when I see server 500 errors for http. It’s hard not to put your head in the sand in this situation. But on closer inspection, and now with the benefit of hindsight, the corrective action does mention how to resolve this, as well as a key tip to look at the FDC file to obtain more detail about the problem. Never having seen a FDC file before I simply ignored it, but the collective nouse of the sys admin and developer later figured it all out, with the recommended solutions posted here and here. Well done Ted

I have already posted a few times about presenting stats using common tools like Microsoft Excel. I am still learning the stats package ‘r’ as I see that to be more powerful and extensible. In the interim I use a combination of Excel, MySQL and PHP to prepare and analyse stats. As this contract winds down and I prepare for the next, I’ve come across a useful application of stats to keep me entertained.

A friend of mine recently stumbled across a well known roulette strategy called the martingale strategy.

Originally, martingale referred to a class of betting strategies popular in 18th century France. The simplest of these strategies was designed for a game in which the gambler wins his stake if a coin comes up heads and loses it if the coin comes up tails. The strategy had the gambler double his bet after every loss , so that the first win would recover all previous losses plus win a profit equal to the original stake. Since a gambler with infinite wealth will with probability of 1 eventually flip heads, the martingale betting strategy was seen as a sure thing by those who practiced it…

So how can you lose? My friend and I put this to test after a few bourbons and a couple of beers. In the two times we’ve met since, we have both walked away winners, smug in our victory and laughing all the way back to the ATM… Obviously, casinos would have a counter to this though?

Stacked Odds
The first point that will work against us in the long run is basic probability. On an American roulette table (which predominates here in Australia) there are 18 red, 18 black and 2 green positions. A total of 38 possible outcomes. The addition of the 2 green positions are the first part of the casino’s counter attack. It tips the probability in their favour, since if you land on a green and you are betting on red or black (which is the strategy), you will lose regardless.

So if we were betting on either red or black, there is a probability of 18/38 that we will win, and 37/38 that we will lose:
18/38 = 47.36% chance to win
20/38 = 52.64% chance to lose

So from the outset, probability wise, we are more likely to lose than win, simply by the casino adding two green numbers (0 and 00). However, it is still likely to be the most fairest bet in the casino if you are hankering to gamble.

Coincidentally, if you add the winning numbers 1 to 36, you come up with the number 666. Hmmm.

Expected Value
This is also known as the House’s Edge, but in probability theory, the expected value of a random variable is the sum of the probability of each possible outcome multiplied by the outcome value (or in this case your winnings). So for the American roulette table with probability of losing being 20/38 and probability of winning being 18/38, the expected outcome value for a $1 bet is:
(-$1 x 20/38) + ($1 x 18/38) = -$0.0526316

Therefore on average for every dollar bet the expected value of one dollar is:
$1 - $0.0526316 = $0.95

So your net gain is less than what you started with. Statistically, this is not a fair bet.

Gambler’s Fallacy
Perhaps the next biggest problem with this strategy, is the simple human tendency to try and recognize patterns. You will hear my friend say that it is extremely unlikely for the play to fall red, black, red, black, red, black and so on. If this happens, and you follow the strategy you will lose quickly, because you will be forced to double each losing bet ($5, $10, $20, $40, $80, $160, $320) …

Despite human intuition telling us that the next play can’t possibly be another black, or it can’t possibly be another red, or it can’t possibly alternate between red and black I mean, come on, how likely is that? The reality is, each roll is statistically independent. That is, between two spins of the wheel, the occurence of one black or one red is neither more nor less proabable than the other occurring. The human trait of identifying lucky streaks, or patterns in play is precisely what casinos prey upon. In fact, casinos no doubt go through painstaking measures to ensure that the roulette tables are balanced, because if not, then they risk people using more detailed statistical analysis to recognise patterns for a particular table. This is probably why roulette calculators are banned. In any case, don’t be fooled by the hype. Or at least that’s what I try and convince myself.

The Maximum Bet
It is possible to lose more individual games than win but still win money overall. Assuming you make bets that are a small percentage of your bankroll (<1%) so for a $500 bankroll you should stick to bets of $5. The bigger your bankroll the better. This is what makes the martingale strategy seemingly unbeatable as my friend and I chuckled to ourselves with suspicious glances from the croupier ... However, perhaps the most effective counter attack at the casino's disposal is the application of a house rule that specifies what the maximum bet for a table can be. For instance, on a $5 minimum bet table, you will often see $500 as the maximum bet. Why? because this limits the amount of money that a player can progressively double after losses. For example, if you start with $5 and lose, then following the strategy until you win again you will place $10, $20, $40, $80, $160 and $320 before you hit the maximum bet limit. That means you only have to lose around 6 times in a row before your strategy falls apart. Coincidentally, the odds of losing 6 straight games is quite high where 56.2% ^ 6 is approx 1/47. In the short term this doesn't seem too bad, but over a long term, coupled with the other points highlighted in this post I can guarantee one thing.

Statistically you will lose.

However, in the interests of having a great night out, and not throwing your money away on the pokies, or pointlessly trying to count cards on the black jack table, the martingale strategy keeps the night entertaining. Now try to explain all of this next time you’re drunk.

If you’re still not convinced, I wrote a perl script to help simulate the martingale strategy. In the following program, the defaults can be controlled via command line arguments, or you can follow my strategy which I plan to unleash on Queenstown casinos during my upcoming holidays to New Zealand. I’m not too sure which has the greater probability, me winning, or decent snow falls in July…


#!/usr/bin/perl -w
# --------------------------------------------------------------------------
# Martingale Roulette Betting System Simulator
#
# Usage : Roulette.pl [options]
# Options :
# -bet minimum bet amount
# -atm amount of money to withdraw from ATM each night
# -maxBets maximum number of bets to place each night
# -maxBet maximum bet amount permitted by house rules
# -nightsOut number of nights to visit the casino
# -verbose optional [true] flag for verbose results
# -quit quite when you're [n] times ahead
# -fair optional [false] flag to remove casino edge
#
# $Revision: 1.3 $
# $Date: 2007/06/07 08:43:11 $
#
# Author: tim.koopmans@90kts.com [http://90kts.com]
# --------------------------------------------------------------------------
use Getopt::Long;

# Summary statistics
print "\n------------------------------------------------------\n";
print "$bet \tMinimum bet\n";
print "$maxBet \tMaximum bet permitted by house rules\n";
print "$maxBets \tMaximum number of bets placed in one night\n";
print "".($n-1)." \tTotal nights out at casino\n";
print "\$$totalWinnings.00 \tTotal winnings less ATM debt\n";
print "------------------------------------------------------\n";


What to do?

In order to find this error I generated a snapshot on error in the run time settings for LoadRunner, then browsed the html source for the vuser typically available at C:\Documents and Settings\UserName\Local Settings\Temp\res

There were a number of incorrect conclusions that I jumped to here:

1. That google would save the day and spoon feed me the answer! The only information I could find was here but alas they did not have the answer. Hence my decision to post here, as no doubt some other performance tester out there is going to hit this problem. I would post on the BOB forum, but hey, you need a login to comment. Meh.

2. My kludge’s approach would be good enough. I thought if I want 50 concurrent users, all I need to do is make sure that I have at least 100 simultaneous connections available. Right?

Wrong …

As it turns out, a user opens more than one session on the web intelligence server. It seems they need one to login, one as they start to navigate around the folder structure and another to open reports. I’m not too sure if this behaviour is caused by the browser (IE or Mozilla) but it seems to be consistent when using a browser. You can see this information by monitoring a user session in the central management console.

So that seems to fix the RWI 00239 errors. Now that I’m past that, on to investigating why the web intelligence servers fails under concurrent user load. RWI 00236 is my starting point, let’s revisit google to see what others have found …

The Web Intelligence server could not be reached. Contact your Business Objects administrator. (Error: RWI 00236)(Error: INF)

So let’s start with a diagram of what I’m talking about. This is what I am trying to achieve …

There’s many reference’s to this but I couldn’t for the life of me figure out how to
a. Demonstrate this in something like excel, and
b. Create the same harmonic damped effect in a simple java load harness.

Well I couldn’t at first, but here is something that kind of works.

In the following graph, the green line is my SLA of 2,000 milliseconds. The dark blue line is response time that can vary according to load. The light blue line is the varying pacing (or throughput) that is required in order to achieve the SLA. In an excel spreadsheet, the meaning (and calculation) of these values went something like this:

So putting it all together in excel works ok. You can download the curve dampening spreadsheet here.

// LOAD BALANCER CALCULATIONS FOR DISCOVER MODE
// Pacing mods based on response time
double diff = (double)sla - (double)responseTime;
double fract = diff/1000;
double inv = 1.00 - fract;
double fractSq = fract * fract;
double factor = (sla/100) * (sla/100);

if(diff<0){
adjustedPacing = (int)( (fractSq * inv * 100) + factor);
} else {
adjustedPacing = (int)( (1-(fractSq * 100))+ factor);
}