Despite having some initial issues with the code, the author of the jmx4r module turned around [...]]]> After getting nowhere with lack luster HP support, I turned to the power of the Open Source community and got a very simple script up and running to remotely monitor Weblogic JVM Performance and JMS queues using JMX and JRuby.

Despite having some initial issues with the code, the author of the jmx4r module turned around a fix inside 24 hours so that I could connect to a Weblogic 9.2 server and look at its domainruntime MBeans. SiteScope can’t handle domainruntimes of these size (short of being told to ‘add more memory’ to a 3GB+ machine!). So exit stage left Mercury, enter stage right JRuby…

This script will enumerate JVM performance and also JMS queue depths in around 65 lines of code and spits those results into a MySQL database using the mysqlimport CLI.
I’ve formatted the output so I can still input the data into LoadRunner Analysis.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

require 'rubygems'
require 'jmx4r'
groupid = "mycluster"
JMX::MBean.establish_connection :url => "service:jmx:rmi:///jndi/iiop://hostname:7001/weblogic.management.mbeanservers.domainruntime", :username => "username", :password => "password"
 
open( 'D:\perf_jvm.csv', 'a') do |file|
   runtime_mbeans = JMX::MBean.find_all_by_name "com.bea:Type=JRockitRuntime,*"
   runtime_mbeans.each do | mbean |
		now  = Time.now
		date = "#{now.year}-#{now.month}-#{now.day}"
		time = "#{now.hour}:#{now.min}:#{now.sec}"
 
		data  = []
		if(mbean.object_name.get_key_property("Name") =~ /server/) then
			data << ""
			data << date
			data << time
			data << groupid
			data << mbean.object_name.get_key_property("Name")
			data << mbean.used_physical_memory
			data << mbean.heap_size_max
			data << mbean.free_physical_memory
			data << mbean.total_garbage_collection_time
			data << mbean.total_garbage_collection_count
			data << mbean.total_nursery_size
			data << mbean.heap_free_percent
			file.puts data.compact.join(", ")
		end
   end
end
 
cmd = 'D:\\wamp\\bin\\mysql\\mysql5.0.51a\\bin\\mysqlimport -h localhost -P 3390 -s -u root --fields-optionally-enclosed-by=""" --fields-terminated-by=, --lines-terminated-by="\n" --local sensis D:\perf_jvm.csv'
puts %x{#{cmd}}
cmd = 'cat D:\perf_jvm.csv >> D:\perf_jvm_archive.csv'
puts %x{#{cmd}}
File.delete 'D:\perf_jvm.csv'
 
 
open( 'D:\perf_queues.csv', 'a') do |file|
   runtime_mbeans = JMX::MBean.find_all_by_name "com.bea:Type=JMSDestinationRuntime,*"
   runtime_mbeans.each do | mbean |
		now  = Time.now
		date = "#{now.year}-#{now.month}-#{now.day}"
		time = "#{now.hour}:#{now.min}:#{now.sec}"
 
		data  = []
		if(mbean.object_name.get_key_property("Name") =~ /Module/) then
			data << ""
			data << date
			data << time
			data << groupid
			data << mbean.object_name.get_key_property("Name")
			data << mbean.messages_received_count
			data << mbean.messages_high_count
			file.puts data.compact.join(", ")
		end
   end
end
 
cmd = 'D:\\wamp\\bin\\mysql\\mysql5.0.51a\\bin\\mysqlimport -h localhost -P 3390 -s -u root --fields-optionally-enclosed-by=""" --fields-terminated-by=, --lines-terminated-by="\n" --local sensis D:\perf_queues.csv'
puts %x{#{cmd}}
cmd = 'cat D:\perf_queues.csv >> D:\perf_queues_archive.csv'
puts %x{#{cmd}}
sleep 10
File.delete 'D:\perf_queues.csv'

You can find out more info about WebLogic MBeans and accessing them with JMX here …

Following are some augmented instructions for setting up JMX monitors within SiteScope. I used this [...]]]> Using JMX (Java Management Extensions) it is possible to monitor Managed Beans published by MBean servers. Weblogic (WL) provides MBeans for both admin and managed servers.

You can find out more info about WebLogic MBeans and accessing them with JMX here …

Following are some augmented instructions for setting up JMX monitors within SiteScope. I used this excellent article as a starting point.

Types
There are two types of MBeans which we’re interested in namely:

1. Configuration MBeans, which expose attributes and operations
for configuring WebLogic Server resources.
2. Runtime MBeans, which provide information about the runtime state of its resources.

In terms of performance monitoring we’re more interested in the latter, but from an overall system level monitoring point of view, sysadmins may also be interested in static config MBeans.

In reality there are 1,000’s of runtime MBeans available for WebLogic and navigating/finding the correct bean inside SiteScope can be difficult due to it’s clunky web interface …

As a starting point refer to the MBean reference previously mentioned to get an overview of runtime beans available. In this case I’m interested in the JMSRuntime bean …

Using the WL Scripting Tool
If you are working on the WL server you can use the WL Scripting Tool to browse the available Runtime beans:
D:\Data\bea\wlserver_10.0\server\bin>setWLSEnv.cmd
D:\Data\bea\wlserver_10.0\server\bin>java weblogic.WLST
connect('username','password','t3://host:8003')
serverRuntime()
ls()
cd("JMSRuntime")
ls()
cd("server01_01.jms")
ls()

-r-\- ConnectionsCurrentCount 14
-r-\- ConnectionsHighCount 14
-r-\- ConnectionsTotalCount 18
-r-\- HealthState State:HEALTH_OK,ReasonCode:\[\]
-r-\- JMSServersCurrentCount 2
-r-\- JMSServersHighCount 2
-r-\- JMSServersTotalCount 2
-r-\- Name server01_01.jms
-r-\- Type JMSRuntime

This will help identify the name of the MBean (in this case server01_01.jms) and the indicative path
to that MBean when using JConsole or SiteScope.

Setting up IIOP access on monitored machines
SiteScope relies on the IIOP protocol to talk to the WL server. In order to get remote access with JMX via IIOP you need to provide a default username and password for the IIOP user.

This can be done via the WL admin console as in:
domServiceBus->Environment->Servers->server01_nn->Protocols [tab]\->IIOP->advanced

Then provide a default username and password for the IIOP user.

Note: IIOP is already enabled by default, but a username and pwd is not. This change requires a server restart for WL. You must make sure that the username and password is >= 8 digits, otherwise the CORBA connection will fail when using SiteScope.

Testing connectivity from JConsole
Before setting up a SiteScope monitor it is worth using JConsole to familiarize yourself with the published MBeans.
Assuming you have the Java SDK installed, launch JConsole as per the following:
C:\Program Files\Java\jdk1.5.0_14\bin\jconsole.exe \-J-Dcom.sun.CORBA.transport.ORBTCPReadTimeouts=10:60000:500:10

Note: you must provide the additional read timeout arguments otherwise you will get a connection error like this:
Unable to retrieve counters: Error receiving browse data: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696 Maybe; nested exception is: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe

To connect to the target server, click on New Connection->advanced and provide the following details:
JMX URL: service:jmx:rmi:///jndi/iiop://hostname:8003/weblogic.management.mbeanservers.runtime
Username:
Password:

You will then get a tree-like broswer in which you can look at all available MBeans.

Setting up a SiteScope JMX monitor
Before setting up the monitor, make sure you change the following registry key for SiteScope to avoid the aforementioned timeout errors:
@="-server -Xmx256m -showversion -Dcom.sun.CORBA.transport.ORBTCPReadTimeouts=10:30000:500:10
-Dcatalina.home=\"D:\\SiteScope\\Tomcat\" -Dflipper.param.file=..\\conf\\flipperSiteScope.properties
-Dtopaz.home=.. -cp \"D:\\SiteScope\\Tomcat\\bin\\bootstrap.jar\" org.apache.catalina.startup.Bootstrap start"

Note the added parameter argument:
-Dcom.sun.CORBA.transport.ORBTCPReadTimeouts=10:30000:500:10

You will need to stop/start the SiteScope windows service for this change to take affect.

Now create a new JMX monitor:
Name: 8003 - Managed server
Frequency: 2 minutes
JMX URL: service:jmx:rmi:///jndi/iiop://host:8003/weblogic.management.mbeanservers.runtime
Domain filter:
Username: weblogic
Password: password

Then click Get Counters …

If you used the WLST you’d now know what path the MBean exists on. In this case:
com.bea/server01_01/server01_01.jms/JMSRuntime

And suggested counters to collect might be:
ConnectionsCurrentCount
ConnectionsHighCount
ConnectionsTotalCount
JMSServersCurrentCount
JMSServersHighCount
JMSServersTotalCount

Now you will be able to use SiteScope as the central repository for collecting metrics (long term monitoring). When running a load test scenario, you will be able to poll these same metrics using the SiteScope monitor available to the Controller.

Java Virtual Machine Version mismatch: Required version: 1.3

To overcome this try the following steps …

Rename the jmxri.jar file to jmxri.bak in the \LoadRunner\classes directory

Copy the weblogic.jar from your application server directory typically from \bea\wlserver_\server\lib to the \LoadRunner\classes [...]]]> When configuring the WebLogic (JMX) monitor (Add Measurements via the Run tab in Controller) you may be presented with the following error:

Java Virtual Machine Version mismatch: Required version: 1.3

To overcome this try the following steps …

Rename the jmxri.jar file to jmxri.bak in the \LoadRunner\classes directory

Copy the weblogic.jar from your application server directory typically from
\bea\wlserver_\server\lib
to the
\LoadRunner\classes directory

Edit the \LoadRunner\dat\monitors\WebLogicMon.ini
as per the following:


[WebLogicMon]
JVM=javaw.exe
JavaVersion=1.5
;JavaVendor=Sun Microsystems Inc.

ActivateCounters=1
QueryLoginInfo=1
RefreshMetricList=1
DefaultPort=1112
SourceHost=localhost
PingHostURL=/?type=0&id=201


Making sure that the JavaVersion and Port match your client/server settings!

Restart the Controller and you should now be able to add the appropriate counters using the WebLogic (JMX) monitor.