Using SOAP::Lite With Perl

I’ve been trying to access a SOAP service using Perl for a project, so I took a quick look at CPAN to see what was available. The answer seemed to be to use SOAP::Lite.

This module turned out to be a real devil to use, and my problems with it were probably compounded by my lack of SOAP experience.

SOAP (incase you didn’t know) is a method of accessing remote services using XML. In my case, I was trying to access content supplied by a third party ringtone provider using SOAP over HTTP.

Using SOAP::Lite you have to specify a uri and a proxy. What stumped me for a while is that the proxy is the URI of service you wish to access, and uri is the namespace of the service.

So if the provider I was using had their SOAP service available at ws.robstones-services.co.uk/external.asmx, I would use this as the proxy, and in this case I would use ws.robstones-services.co.uk/External as the urinamespace.

Great, I knew the service I needed was called getCallList, and required a Username and Password to be passed to it. In return it would give me a list of valid content types. The SOAPAction header to be added to the call to proxy, letting the remote SOAP service I wanted to use the service http://ws.robstones-services.co.uk/External/getCallList.

My first attempt was the following code…

#!/usr/bin/perl -w
use strict;
use SOAP::Lite 'trace', 'debug';
my $server = SOAP::Lite
->uri('http://ws.robstones-services.co.uk/External')
->proxy('http://ws.robstones-services.co.uk/external.asmx');
my $returned = $server
->getCallList({
'Username' => 'RobsUser',
'Password' => 'RobsPassword'
});
foreach my $type ($returned->valueof('//getCallListResult/string')) {
next unless ($type); ## ignore any undefs
print "$typen";
}

You’ll notice I’m using trace and debug to see the SOAP messages being sent and received. The dialogue from this script was…

> perl -w test.pl
SOAP::Transport::HTTP::Client::send_receive: POST http://ws.robstones-services.co.uk/external.asmx HTTP/1.1
Accept: text/xml
Accept: multipart/*
Content-Length: 615
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://ws.robstones-services.co.uk/External#getCallList"
<?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Envelope xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/1999/XMLSchema" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Body><namesp1:getCallList xmlns:namesp1="http://ws.robstones-services.co.uk/External"><c-gensym3><Username xsi:type="xsd:string">RobsUser</Username><Password xsi:type="xsd:string">RobsPassword</Password></c-gensym3></namesp1:getCallList></SOAP-ENV:Body></SOAP-ENV:Envelope>
SOAP::Transport::HTTP::Client::send_receive: HTTP/1.1 500 (Internal Server Error) Internal Server Error.
Cache-Control: private
Connection: close
Date: Tue, 29 Nov 2005 15:23:38 GMT
Server: Microsoft-IIS/6.0
Content-Length: 508
Content-Type: text/xml; charset=utf-8
Client-Date: Tue, 29 Nov 2005 15:23:09 GMT
Client-Response-Num: 1
X-AspNet-Version: 1.1.4322
X-Powered-By: ASP.NET
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<soap:Fault>
<faultcode>soap:Client</faultcode>
<faultstring>Server did not recognize the value of HTTP Header SOAPAction: http://ws.robstones-services.co.uk/External#getCallList.</faultstring>
<detail />
</soap:Fault>
</soap:Body>
</soap:Envelope>

The call didn’t work, you’ll see the SOAPAction variable is wrong, it’s http://ws.robstones-services.co.uk/External#getCallList instead of http://ws.robstones-services.co.uk/External/getCallList.

I needed to get SOAP::Lite to use the correct URL. The secret turned out to be to change the use SOAP::Lite line to the following.

use SOAP::Lite on_action => sub {sprintf '%s/%s', @_},
'trace', 'debug';

on_action is a parameter SOAP::Lite uses to separate the URI from the action, here we’re telling it to use /.

This gave me the following SOAP dialogue…

> perl -w test.pl
SOAP::Transport::HTTP::Client::send_receive: POST http://ws.robstones-services.co.uk/external.asmx HTTP/1.1
Accept: text/xml
Accept: multipart/*
Content-Length: 615
Content-Type: text/xml; charset=utf-8
SOAPAction: http://ws.robstones-services.co.uk/External/getCallList
<?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Envelope xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/1999/XMLSchema" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Body><namesp1:getCallList xmlns:namesp1="http://ws.robstones-services.co.uk/External"><c-gensym3><Username xsi:type="xsd:string">RobsUser</Username><Password xsi:type="xsd:string">RobsPassword</Password></c-gensym3></namesp1:getCallList></SOAP-ENV:Body></SOAP-ENV:Envelope>
SOAP::Transport::HTTP::Client::send_receive: HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Connection: close
Date: Tue, 29 Nov 2005 15:22:53 GMT
Server: Microsoft-IIS/6.0
Content-Length: 405
Content-Type: text/xml; charset=utf-8
Client-Date: Tue, 29 Nov 2005 15:22:25 GMT
Client-Response-Num: 1
X-AspNet-Version: 1.1.4322
X-Powered-By: ASP.NET
<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><getCallListResponse xmlns="http://ws.robstones-services.co.uk/External"><getCallListResult><string xsi:nil="true" /></getCallListResult></getCallListResponse></soap:Body></soap:Envelope>

Bugger, it still didn’t work. I looked again at the SOAP message being sent. It was sending a lot of namespace information and a wrapper around the Username and Password. The wrapper was definitely wrong, and the remote .NET service didn’t seem to like the namespaces.

I needed to sort that out, so I had to tell SOAP::Lite to not to include namespaces or wrap up the Username and Password. The way to do this is to use SOAP::Data to hardcode the data being sent.

This meant changing the call to getCallList to the following…

my $returned = $server
->getCallList(
SOAP::Data->name('Username')->value('RobsUser')->type(''),
SOAP::Data->name('Password')->value('RobsPassword')->type('')
);

The SOAP dialogue after these changes looked like this..

SOAP::Transport::HTTP::Client::send_receive: POST http://ws.robstones-services.co.uk/external.asmx HTTP/1.1
Accept: text/xml
Accept: multipart/*
Content-Length: 548
Content-Type: text/xml; charset=utf-8
SOAPAction: http://ws.robstones-services.co.uk/External/getCallList
<?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Envelope xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/1999/XMLSchema" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Body><namesp1:getCallList xmlns:namesp1="http://ws.robstones-services.co.uk/External"><Username>RobsUser</Username><Password>RobsPassword</Password></namesp1:getCallList></SOAP-ENV:Body></SOAP-ENV:Envelope>
SOAP::Transport::HTTP::Client::send_receive: HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Connection: close
Date: Tue, 29 Nov 2005 15:19:49 GMT
Server: Microsoft-IIS/6.0
Content-Length: 405
Content-Type: text/xml; charset=utf-8
Client-Date: Tue, 29 Nov 2005 15:19:21 GMT
Client-Response-Num: 1
X-AspNet-Version: 1.1.4322
X-Powered-By: ASP.NET
<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><getCallListResponse xmlns="http://ws.robstones-services.co.uk/External"><getCallListResult><string xsi:nil="true" /></getCallListResult></getCallListResponse></soap:Body></soap:Envelope>

Close, but still not right. The problem was with the getCallList itself, it was still trying to use a namespace.

The final solution was to this problem was to not use getCallList, but instead to use SOAP::Lite’s call method to implicitly set how the function was being called.

The code after making this change was the following…

my $returned = $server
->call(SOAP::Data->name('getCallList')->attr({xmlns => 'http://ws.robstones-services.co.uk/External'}) =>
SOAP::Data->name('Username')->value('RobsUser')->type(''),
SOAP::Data->name('Password')->value('RobsPassword')->type('')
);

This gave me the SOAP dialogue…

> perl -w test_soap.pl
SOAP::Transport::HTTP::Client::send_receive: POST http://ws.robstones-services.co.uk/external.asmx HTTP/1.1
Accept: text/xml
Accept: multipart/*
Content-Length: 524
Content-Type: text/xml; charset=utf-8
SOAPAction: http://ws.robstones-services.co.uk/External/getCallList
<?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Envelope xmlns:xsi="http://www.w3.org/1999/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/1999/XMLSchema" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Body><getCallList xmlns="http://ws.robstones-services.co.uk/External"><Username>RobsUser</Username><Password>RobsPassword</Password></getCallList></SOAP-ENV:Body></SOAP-ENV:Envelope>
SOAP::Transport::HTTP::Client::send_receive: HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Connection: close
Date: Tue, 29 Nov 2005 15:16:07 GMT
Server: Microsoft-IIS/6.0
Content-Length: 540
Content-Type: text/xml; charset=utf-8
Client-Date: Tue, 29 Nov 2005 15:15:39 GMT
Client-Response-Num: 1
X-AspNet-Version: 1.1.4322
X-Powered-By: ASP.NET
<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><getCallListResponse xmlns="http://ws.robstones-services.co.uk/External"><getCallListResult><string>ROBS_JAVA</string><string>ROBS_WALL</string><string>ROBS_POLY</string><string>ROBS_MONO</string><string>ROBS_REAL</string><string xsi:nil="true" /></getCallListResult></getCallListResponse></soap:Body></soap:Envelope>
ROBS_JAVA
ROBS_WALL
ROBS_POLY
ROBS_MONO
ROBS_REAL

Hurrah, after all that effort, I have a list of 5 types of content offered by my third party supplier. I can now go ahead and build my service.

SOAP::Lite is a powerful module, but the lack of simple, easy to follow documentation and examples holds it back. I hope this small article helps out other programmers just starting on the SOAP path.

Using del.icio.us Bookmarks With Perl

Earlier this week I put live some Perl code that took bookmarks I was posting to del.icio.us and added them automatically to my blog.

I’ve had a few people ask how I was able to do this, and it’s no big secret.

del.icio.us expose an API that anyone can use to interact with their service. I’m using this with some Perl glue to aggregate the previous days posts and put them up on my own site.

The function I’m using is GET, which can take a date as an optional parameter. I use this parameter to get all postings from yesterday. So if the date today is 25th November 2005, to get yesterdays posts from del.icio.us I call the URL http://del.icio.us/api/posts/get?dt=2005-11-24.

All API calls to del.icio.us use HTTP-Auth, these are your site login and password.

To access this data using perl, we can use the LWP modules from CPAN.

my $ua = LWP::UserAgent->new;
$ua->agent('robslinkbot/0.1 (+http://www.robertprice.co.uk)');
my $get_url = 'http://del.icio.us/api/posts/get?dt=' . $yesterday;
my $req = HTTP::Request->new(GET => $get_url);
$req->authorization_basic(USERNAME, PASSWORD);
my $res = $ua->request($req);
if ($res->is_success) {
my $xml = $res->content;
} else {
warn "unable to get content from del.icio.usn";
}

In this case USERNAME and PASSWORD are both two constant values with my username and password defined in. You’ll need to put your own details in here.

You’ll also notice that I’m setting the agent to be robslinkbot/0.1 (+http://www.robertprice.co.uk). del.icio.us specicially request that a user-agent is set as they tend to ban generic user-agents from time to time. If I didn’t set this, it would be set to something like lwp-perl.

So now we have this code, and if it’s worked correctly, I should have my posts from the previous day in the variable $xml, and if it’s not worked, I should have seen a warning informing me that the script was unable to get content from del.icio.us.

We can parse the XML provided very easily using one of Perl’s many XML modules. In this case, I’m going to use XML::XPath.

As we’ll have multiple bookmarks (hopefully), we’ll create a list of hashrefs. The hashrefs will contain the information relating to the post.

Ok, so firstly we create a variable called @posts to store the hashrefs in.

my @posts;

Now we can create our XML::XPath object, and get it to parse the xml we’ve already downloaded from del.icio.us and have in the $xml variable.

my $xp = XML::XPath->new(xml => $xml);

We need to see if the XPath /posts/post exists, if it does it means we have posts to parse.

if ($xp->exists('/posts/post')) {

Now we have to find al the posts and iterate over them

foreach my $posts ($xp->find('//post')->get_nodelist) {

Now all we do is to extract the information we need from each post, store it in our hashref and finally store it in the @posts list.

my $post_hashref;
## we use .= to stringify the find value, must be a better way to do that.
$post_hashref->{'href'} .= $posts->find('@href');
$post_hashref->{'description'} .= $posts->find('@description');
$post_hashref->{'time'} .= $posts->find('@time');
$post_hashref->{'hash'} .= $posts->find('@hash');
$post_hashref->{'extended'} .= $posts->find('@extended');
my @tags = split(/ /, $posts->find('@tag'));
$post_hashref->{'tags'} = @tags;
push @posts, $post_hashref;

You may have noticed that we have split the tags and stored them as a list. I just find this easier to work with.

And that is about it. You should have list you can iterate over, or pass to something like Template Toolkit for displaying. This is a process I use on robertprice.co.uk.

Using RDF::Redland On Debian

This had me fooled for a few hours, so I thought i’d just record it here incase anyone else is having the same problem.

When installing the RDF::Redland Perl modules to use Dave Beckett’s Redland on Debian with a custom compiled version of Perl, the libraries on CPAN will not work.

Instead, the version of Redland on the current Debian (11th September 2005) is version 1.0.0.2, so you need to download the redland-bindings for 1.0.0.2 on librdf.org, and compile them yourself by running perl Makefile.PL and make, make test and finally make install.

You should now be able to use the RDF::Redland modules successfully.

In London, But OK

It looks like there have been more attacks in London on a bus, and at 3 tube stations.

Although I’m working close to Warren Street, I’m fine.

We’ve been told to stay in the office and not leave.

London 7th July 2005

It’s eerie walking through London trying to get home today.

As Liverpool Street is obviously shut following the terrorist bombings, i decided to try going to Eastbourne from Victoria instead.

Buckingham Palace is partially sealed off, and armed soldiers in combat gear are patrolling instead of the usual dress sentries standing outside.

As buses are suspended, the bus station at Victoria was deserted. It was also strange seeing Oxford Street without a single bus running down it.

Trains are a nightmare. I’m on a special Brighton service as very few trains are running. There are no tubes or buses running so it’s very busy.

Strangely, despite all the trains being messed up, the public information board cheerfully informs us there are no reported problems today, and tries to sell us balloon rides, gay pride and Party in the Park tickets.

Thu 07/07/2005 15:31 - Buckingham Palace
Thu 07/07/2005 15:43 - Victoria Bus Station
Thu 07/07/2005 16:01 - no delays?

I’m In London, But OK

Just so you all know, I’m in central London, but safe.

Looks like I’ll be stuck here as the entire transport system has shut down here after todays terrorist attacks.

It’s a good job I didn’t come via Liverpool Street today!

Whatson.com – The Best In UK Commercial Radio

Whatson.com - Kerrang

I’m really pleased as I made the new Whatson.com site live today. It’s the result of several months hard work by a small team, with me as the lead developer.

Currently it’s the home for 8 radio players at present, with more to follow soon!

Kerrang Radio, Kerrang 105.2, Kiss 100 and Magic 105.4 contain a mix of live radio streams, the best of the weeks previous shows and some other special content.

The BBC has been doing something similar for a while with the BBC Radio Player, and ourselves with the Kiss player, but this a first for UK commercial radio.

All the players cross link to each other and we try to offer highlights from other stations to give visitors a real choice of listening.

Anyway, there’s more to come, so keep tuned in.

Whatson.com - Kiss

Whatson.com - Heat

Whatson.com - Smash Hits

Using Symbian Perl

I’ve been playing with Jarkko Hietaniemi’s port of Perl for Series 60 Symbian devices on my Nokia 7610.

There is a lot of work to be done on it still, but it works, and I can run simple Perl scripts.

Perl running on a Nokia 7610 phone

The screenshot above show’s me running a simple script to list the contents of the Lifeblog directory on my series 7610 phone. Here’s the code that was run on the phone to generate the listing.


my $directory = "C:\system\Data\lifeblog\";
print "$directoryn";
opendir(DIR,$directory);
foreach my $file (readdir(DIR)) {
print "$filen";
}
closedir(DIR);
sleep(5);

As you can see it’s simple stuff. I added the sleep(5) at the end just to allow me to take a screenshot as when the script has been run it prompts you to press any key to exit. In this case, the screenshot key, caused an exit. 🙂

It may not look much, but it’s bloody exciting stuff for me. The possibility of running decent Perl applications on my phone has me salivating in anticipation of the next release. At present the version out there (0.1.0) doesn’t offer any interface into the native Symbian libraries from Perl, but I’m sure that will be changing soon.

It’s easy to write simple scripts, as all I had to do for this one was to just drop it into the Perl directory on my memory card (via Nokai PC Suite) and use the PerlApp application supplied in the Symbian distribution.

The Python guys are luckier to have a fuller featured release of their favourite scripting language, but watch out, the Perl Mongers are coming!