WWW::Mechanize, or Mech for short, helps you automate interaction with a website. It supports performing a sequence of page fetches including following links and submitting forms. Each fetched page is parsed and its links and forms are extracted. A link or a form can be selected, form fields can be filled and the next page can be fetched. Mech also stores a history of the URLs you’ve visited, which can be queried and revisited.

This post details how to install the module as well as a quick example of automation that logs in to a secure website (MyThree.com.au) and extracts information (Bill Info & Data Usage) without ever having to leave your console …

Installation
I like to use the ActiveState package of Perl, which comes pre-installed with LWP (if on Windows). If however you are installing on a non-windows platform, you may need to update your Perl installation with the LWP module. I think the easiest way to do this is to download the module direct from CPAN here. If you don’t do this, then your cpan console will most likely return the following type of error whenever you try to install a new module:
cpan> Could not fetch authors/ .gz
LWP not available

Once downloaded you can install libwww-perl using the normal Perl module distribution drill:
sudo perl Makefile.PL
sudo make
sudo make test
sudo make install

Now you can use the CPAN console to update modules in your perl installation easily.
If you are using a win32 platform, I wouldn’t bother with the cpan console, I would just use the Perl Package Manager and install the win32 bundles direct from the following repository:
ppm install http://theoryx5.uwinnipeg.ca/ppms/WWW-Mechanize.ppd
ppm install http://theoryx5.uwinnipeg.ca/ppms/Crypt-SSLeay.ppd

However if you’re on a *nix environment, then the cpan console is perhaps your best bet:
>sudo cpan
install Bundle::CPAN #this isn't mandatory, but it's good to update in general
install IO::Socket::SSL
install HTML::TokeParser
install WWW::Mechanize
install Crypt::SSLeay #if you're going to automate https transactions


Example Code
Now that you’ve installed the module, you can write some code that will automate actions performed on a typical web site that uses https communications. In the following example, I automate a user logging in to a secure website (www.my.three.com.au) who then obtains summary billing and usage information.

Make sure you include the use statements required to add this module’s functionality to your code:
use WWW::Mechanize;
use Crypt::SSLeay;

Create a new mech object:
$mech = WWW::Mechanize->new();


Open a URL and check the http return code:
$url = 'https://www.my.three.com.au/My3/jfn';
$result = $mech->get( $url );
die "GET failed\n" unless $result->is_success;

Submit a form on the web page after providing some html field details and check the http return code:
$result = $mech->submit_form(
form_name => 'login',
fields => {
login => $mobile,
password => $password
}
);
die "SUBMIT failed\n" unless $result->is_success;

Parse the string that is returned from a successful login for relevant information:
$content = $result->as_string();
if($content =~ /\\$ ([0-9\.]+)/)
{
$value = $1;
print "Total Balance Due \t\t\$$value \n";
}

Simulate clicking on an embedded link within a frame. To figure out the URL query parameters, and sub frame page information I like to use Firefox’s page info feature to determine the correct parameters and sequence. Another handy plugin is LiveHTTP headers, which lets you scrutinize transactions, capture them, replay them and so on.

$params="selectedLineNumber=$mobile&&oid=L7%3A3714843&ctx=L&mfunc=1055&jfnRC=7";
$url="https://www.my.three.com.au/My3/jfn?$params";
$result = $mech->get( $url );
die "GET failed\n" unless $result->is_success;

Parse for further content such as your data usage summary:
$content = $result->as_string();
if($content =~ /Mobile Broadband \(2048.*\n.*\n.*\n\s+(\d+\.\d+)/)
{
$value = $1;
print "Mobile Broadband Remaining \t$value GB \n";
}

Execution
Test and execute your code. The example code below prints billing and usage operation from the console. You can then automate this functionality into other aspects of your operating system (on Mac OSX I like to make widgets that achieve similar outcomes):
/Users/Koops/Documents/working/90KTS>./MyThree.pl
Opened login page ...
Passed 1st login ...
Passed 2nd login ...
Total Balance Due $49.00
Mobile Broadband Remaining 1108.083 MB

Complete Code Example
#!/usr/bin/perl
# $Id: MyThree.pl,v 1.0 2007/09/24 03:29:19 Koops Exp $
use WWW::Mechanize;
use Crypt::SSLeay;

$content = $result->as_string();
if($content =~ /Mobile Broadband \(2048.*\n.*\n.*\n\s+(\d+\.\d+)/)
{
$value = $1;
print "Mobile Broadband Remaining \t$value MB \n";
}

To do that through your ssh session, you can use the native Win32::EventLog module that ships with the ActiveState version of Perl. I can’t remember where I got the majority of this code from, but I have since modified it to work within a Win32 environment along with some simple command line arguments. The version I copied was more in line with *ix systems I believe.

If I can remember where I got the code from I’ll acknowledge it, but feel free to use my modified version in the interim

use Win32::EventLog;

print "$mday/",$mon+1,"/",$year+1900,"\t$hour:$min\t".$event->{Strings}."\n";
}


To rehash what many others have already stated, here are my own favourites:

Print HEX value of a string good for encoding those MQMD headers. If you want to reverse use pack instead of unpack… You can also use B* for binary.
perl -e 'print unpack "H*", "AMQNUS725PQ"'


Replace 8 byte value which is offset at position 128 with the ‘NEWVALUE’:
perl -pi -e 'substr($_, 128, 8)="NEWVALUE"' /root/path/to/your/files/*


To copy posn 55 len 8 to posn 128 len 8:
perl -pi -e 'substr($_, 128, 8)=substr($_, 55, 8)' /root/path/to/your/files/*


To replace posn 128 len 8 in each file with first 8 bytes of the filename:
perl -pi -e 'substr($_,128,8)=substr($ARGV,0,8)' /root/path/to/your/files/*


To access your shell environment variables:
ACCOUNTF10=OUTSIDE
perl -e 'print "I AM INSIDE MY EXPRESSION AND NOW I AM '$ACCOUNTF10' MUHAHAHAHA"'


To allow for variable interpolation, use double quoted strings which substitutes the value of a variable into a string:
perl -pi -e 'substr($_,6162,10)="'$PRODUCTEND'"' $OUTPUTFILEFULL


Global search and replace using regex for search string (could also use fixed values instead of regex):
search="2007[0-9]{4}"
replace="20070321"
perl -pi -e 's/'$search'/'$replace'/g' /root/path/to/your/files/*


Find alternate search values and replace them with a single value:
search="KENANR|SIEBEL"
replace="SIEBEL"
perl -pi -e 's/'$search'/'$replace'/g' /root/path/to/your/files/*


Find anything that starts with a search value and replace them with a single value (Notice I pad strings out as the application in use depends on fixed format):
search="^Christina"
replace="Timothy "
perl -pi -e 's/'$search'/'$replace'/g' /root/path/to/your/files/*


Find anything that ends with a search value and replace them with a single value:
search="Bank Deposit$"
replace="Bank Credit "
perl -pi -e 's/'$search'/'$replace'/g' /root/path/to/your/files/*


Transliteration (swaps characters around) of specific characters:
search="cd"
replace="dc"
perl -pi -e 'tr/'$search'/'$replace'/' /root/path/to/your/files/*


And here are some metacharaters/character classes you can use in your search strings if you don’t want to use literals:
[a-z] any charater between a to z
[A-Z] any charater between A to Z
[0-9] any number between 0 to 9
\d same as [0-9]
\D inverse of [0-9] ie. any non digit
\w same as [a-zA-Z0-9_]
\W inverse of \w
\s same as [ \r\t\n\f] ie. any white space
\S inverse of \s


A good way to achieve this is my using the PCF support package provided by IBM.

The WebSphere MQ Programmable Command Formats (PCF) provide the capability to perform administration tasks on a queue manager by sending and receiving WebSphere MQ messages of a special format. PCF request messages are sent to the queue manager’s command queue, where they are processed by the command server and replies returned to the designated reply-to queue.

This SupportPac contains a set of Java classes representing PCF header structures as well as an agent that can be used to simplify the task of communicating with a target queue manager and thus enable the use of WebSphere MQ Programmable Command Formats for queue manager administration.

Using a simple java object as documented in sample code, you can query queue stats of your choice. My implementation simply prints out the queue you are connected to, along with enqueue, dequeue and total queue size stats over an arbitrary time frame (eg. last 2 seconds). This can easily be called from a simple shell script.

ResetQStats.java

/**
* Reset MQ Queue Statistics (ResetQStats)
*
* Compile : javac -classpath com.ibm.mq.jar:connector.jar:jta.jar:com.ibm.mq.pcf-6.0.jar ResetQStats.java

catch (IOException ioe)
{
System.err.println (ioe);
}
}
}


Simple Shell Script

#~/bin/sh
javac -classpath $HOME/harness/CDE/com.ibm.mq.jar:$HOME/harness/CDE/connector.jar:$HOME/harness/CDE/jta.jar:/home/strata/d281447/harness/CDE/com.ibm.mq.pcf-6.0.jar $HOME/harness/CDE/ResetQStats.java
cd $HOME/harness/CDE
jar cfm ResetQStats.jar ResetQStats.MF *.class
rm $HOME/harness/CDE/*.class

done
printf "\n"



eval{ &yourSubRoutine($parameters); };
$@ ? $yourLogObject->yourErrorMethod("I failed with reason: $@") ):
$yourLogObject->yourTraceMethod("Everything was good: $?") );