Category: Dev

Dynamically Setting Domain Names For Webtrends

We use Webtrends at work to track usage of our websites.

We use the on demand version which relies on embedding a JavaScript tag onto the page. Recently, we’ve moved to using 1st party cookies (cookies set by the site) instead of 3rd party cookies (cookies set externally by webtrends). These are set in the JavaScript tag and are hard coded to the site, a real pain if you happen to move domain names as you need to code them up specifically each time. We got caught out recently by using a tag with the wrong domain name set in it, meaning we were unable to track the page impressions. We needed a solution…

As the tag is being set in JavaScript, we can use JavaScript to set the domain dynamically. Not all our sites are served dynamically so using JavaScript should mean it would work for anyone using this tag.

Here’s the offending bit of code…

// Code section for Set the First-Party Cookie domain
var gFpcDom=".kerrang.com";

This works for anything served from a Kerrang.com domain. However, what if we want to use a different domain for some reason?

We’ll JavaScript has a property in the document object called domain, we can use this at runtime to find out the full domain name the page was served from.

However, we don’t need the full domain name, just parent domain. For example, www.kerrang.com should become kerrang.com.

We can fix this by using several approaches, but the easiest is the split, splice, join method.

Firstly we split the domain into it’s components by splitting on the periods.

document.domain.split('.');

This gives us the array (“www”,”kerrang”,”com”).

Secondly we splice this to remove the first element.

document.domain.split('.').splice(1.3);

This is saying to take 3 elements after the 1st element in the array. This should cover us if we want to use something like kerrang.co.uk. So after running this we need to should have an array looking like this, (“kerrang”,”com”).

Finally we join this back together again, making sure a period is between each joined element.

document.domain.split('.').splice(1,3).join('.');

Running this should give us kerrang.com, which is acceptable to use in the Webtrends tag.

So our final code looks like this…

var gFpcDom = document.domain.split('.').splice(1,3).join('.');

Movable Type Comment Problems

Earlier on the guys at Kerrang posted a rather hot topic on the lovely Mr Manson and his thoughts on My Chemical Romance.

This caused a fever of activity on the website, and unfortunately caused it to corrupt with the following messages appearing in the Movable Type activity logs.

Comment save failed with Insertion test failed on SQL error Duplicate entry ‘6692’ for key 1

Now it took a while digging in the code to work out what was wrong as I’ve not seen this before.

Eventually logging into the MySQL database, showed up the problem.

mysql> check table mt_comment;
+———————+——-+———-+———————————————————–+
| Table | Op | Msg_type | Msg_text |
+———————+——-+———-+———————————————————–+
| kerrang2.mt_comment | check | warning | 12 clients are using or haven’t closed the table properly |
| kerrang2.mt_comment | check | warning | Size of datafile is: 2999316 Should be: 2998080 |
| kerrang2.mt_comment | check | error | Found 6561 keys of 6559 |
| kerrang2.mt_comment | check | error | Corrupt |
+———————+——-+———-+———————————————————–+

Ahha! The comments table is screwed. The solution is to run the following…

mysql> repair table mt_comment;
+———————+——–+———-+——————————————+
| Table | Op | Msg_type | Msg_text |
+———————+——–+———-+——————————————+
| kerrang2.mt_comment | repair | warning | Number of rows changed from 6559 to 6561 |
| kerrang2.mt_comment | repair | status | OK |
+———————+——–+———-+——————————————+

Comments on Movable Type now work fine.

Using Perl To Send Heartbeat To A Python Server

I’ve been brushing up on my networking and Python skills lately.

One example in the Python Cookbook is recipe 13.11 Detecting Inactive Computers. Here there are two scripts, one that sends a heartbeat, and another that listens for heartbeats. The idea is a simple UDP packet containing the short string “PyHB” is sent every 5 seconds or so by a client. A server then listens for these datagrams, stores the addresses of the clients which it receives messages from, and if it fails to receive a message in a certain time period it flags this up on the console.

I thought I’d just check how easy it would be to create a Perl client to talk to the Python backend. As it’s all through sockets it should be fairly easy.

To start with we define a few constants that cover the address of the server we want to connect to, the port to send the datagram to, how often we want to send it and if we want debugging information displayed or not.

use constant SERVER_IP => '127.0.0.1';
use constant SERVER_PORT => 43278;
use constant BEAT_PERIOD => 5;
use constant DEBUG => 1;

Now we need to create the socket, in this case we’ll create a new IO::Socket object.

my $hbsocket = IO::Socket::INET->new(Proto => 'udp',
PeerPort => SERVER_PORT,
PeerAddr => SERVER_IP)
or die "Creating socket: $!n";

Next we need to send our message, the string “PyHB” over the socket.

$hbsocket->send('PyHB') or die "send: $!";

Finally we need to wrap this up in a loop and sleep BEAT_PERIOD seconds before repeating our message.

Here’s the final script.

#!/usr/bin/perl -w
use strict;
use IO::Socket;
use constant SERVER_IP => '127.0.0.1';
use constant SERVER_PORT => 43278;
use constant BEAT_PERIOD => 5;
use constant DEBUG => 1;
print "Sending heatbeat to IP " . SERVER_IP ." , port " . SERVER_PORT . "n";
print "press Ctrl-C to stopn";
while (1) {
my $hbsocket = IO::Socket::INET->new(Proto => 'udp',
PeerPort => SERVER_PORT,
PeerAddr => SERVER_IP)
or die "Creating socket: $!n";
$hbsocket->send('PyHB') or die "send: $!";
print "Time: " . localtime(time) . "n" if (DEBUG);
sleep(BEAT_PERIOD);
}

Just set the value of DEBUG to 0 if you don’t want to see the time each time we send a message to the server.

To test this you’ll have to be running the server from recipe 13.11 in the Python Cookbook. You can download a zip file of the examples in the Python Cookbook from the O’Reilly website to test this.

Atom WSSE Authentication Using C# .NET

I received an email asking for a hand with a C# .NET implementation of the WSSE atom authentication for Nokia’s Lifeblog.

I took a look at this as I’m brushing up on my C# at the moment for work, and I relish a challenge.

As there are two different methods of authentication depending on the version of Lifeblog being used, the code has to handle this.

The first method is to append the nonce, timestamp and password together, create a SHA1 hash with this and Base64 encode the result. The second way is almost the same, but the nonce has to be Base64 decoded first before being appended with the timestamp and password.

See my previous article on WSSE Authentication For Atom Using Perl for more details on how this works.

The first method can be calculate a digest value using code like this.

string mystring = nonce + created + password;
SHA1Managed SHhash = new SHA1Managed();
string mydigest = Convert.ToBase64String(SHhash.ComputeHash(System.Text.Encoding.ASCII.GetBytes(mystring)));

My first attempt at the second method looked very similar.

string mystring = (System.Text.Encoding.ASCII.GetString(Convert.FromBase64String(nonce))) + created + password;
SHA1Managed SHhash = new SHA1Managed();
string mydigest = Convert.ToBase64String(SHhash.ComputeHash(System.Text.Encoding.ASCII.GetBytes(mystring)));

Looks OK on the surface, but it fails to validate when posting from my Nokia N93 phone.

What’s gone wrong?

Well looking in the debugger, the process of converting the nonce back from the array of bytes Convert.FromBase64String() produces and appending them to form mystring corrupts the data compared with the Perl version in my previous article.

We know the approach is basically right, so we need to cut out this corrupting step.

The best way to do this is to keep the data in byte arrays as we need the data in this format anyway to computer the SHA1.


System.Text.Encoding enc = System.Text.Encoding.UTF8;
byte[] noncebytes = enc.GetBytes(nonce);
byte[] passwordbytes = enc.GetBytes(password);
byte[] createdbytes = enc.GetBytes(created);
byte[] code = new byte[nonce.Length + password.Length + created.Length];
Array.Copy(nonce, code, nonce.Length);
Array.Copy(created, 0, code, nonce.Length, created.Length);
Array.Copy(password, 0, code, nonce.Length + created.Length, password.Length);
System.Security.Cryptography.SHA1Managed SHhash = new System.Security.Cryptography.SHA1Managed();
string digest = Convert.ToBase64String(SHhash.ComputeHash(code));

I’m sure there is a nicer way to achieve the appending of the 3 arrays together in C#, but this works for now.

We can tie this up to form a nice static class to handle our WSSE authentication.

/// <summary>
/// Computes WSSE codes from input for validation.
/// </summary>
public static class WSSE {
/// <summary>
/// Calculate the digest.
/// </summary>
/// <param name="password">A string with your unencrypted password.</param>
/// <param name="nonce">The nonce the digest is to be created with as a string.</param>
/// <param name="created">The timestamp as a string.</param>
/// <returns>A string containing the digest.</returns>
public static string GetDigest(string password, string nonce, string created)
{
return GetDigest(password, nonce, created, System.Text.Encoding.ASCII);
}
/// <summary>
/// Calculate the digest.
/// </summary>
/// <param name="password">A string with your unencrypted password.</param>
/// <param name="nonce">The nonce the digest is to be created with as a string.</param>
/// <param name="created">The timestamp as a string.</param>
/// <param name="enc">The System.Text.Encoding to use.</param>
/// <returns>A string containing the digest.</returns>
public static string GetDigest(string password, string nonce, string created, System.Text.Encoding enc)
{
byte[] noncebytes = enc.GetBytes(nonce);
byte[] passwordbytes = enc.GetBytes(password);
byte[] createdbytes = enc.GetBytes(created);
return generatecode(noncebytes, passwordbytes, createdbytes);
}
/// <summary>
/// Calculate the alternative digest.
/// </summary>
/// <param name="password">A string with your unencrypted password.</param>
/// <param name="nonce">The nonce the digest is to be created with as a string.</param>
/// <param name="created">The timestamp as a string.</param>
/// <returns>A string containing the digest.</returns>
public static string GetDigestAlt(string password, string nonce, string created)
{
return GetDigestAlt(password, nonce, created, System.Text.Encoding.ASCII);
}
/// <summary>
/// Calculate the alternative digest.
/// </summary>
/// <param name="password">A string with your unencrypted password.</param>
/// <param name="nonce">The nonce the digest is to be created with as a string.</param>
/// <param name="created">The timestamp as a string.</param>
/// <param name="enc">The System.Text.Encoding to use.</param>
/// <returns>A string containing the digest.</returns>
public static string GetDigestAlt(string password, string nonce, string created, System.Text.Encoding enc)
{
byte[] noncebytes = Convert.FromBase64String(nonce);
byte[] passwordbytes = enc.GetBytes(password);
byte[] createdbytes = enc.GetBytes(created);
return generatecode(noncebytes, passwordbytes, createdbytes);
}
private static string generatecode(byte[] nonce, byte[] password, byte[] created)
{
byte[] code = new byte[nonce.Length + password.Length + created.Length];
Array.Copy(nonce, code, nonce.Length);
Array.Copy(created, 0, code, nonce.Length, created.Length);
Array.Copy(password, 0, code, nonce.Length + created.Length, password.Length);
System.Security.Cryptography.SHA1Managed SHhash = new System.Security.Cryptography.SHA1Managed();
return Convert.ToBase64String(SHhash.ComputeHash(code));
}
/// <summary>
/// Validates password, nonce and created create the same digest code as digest.
/// </summary>
/// <param name="password">A string with your unencrypted password.</param>
/// <param name="nonce">The nonce the digest is to be created with as a string.</param>
/// <param name="created">The timestamp as a string.</param>
/// <param name="digest">The digest to validate the password, nonce and created strings against as a string.</param>
/// <returns>true or false depending on wether the digest validates</returns>
public static bool IsValid(string password, string nonce, string created, string digest)
{
return digest == GetDigest(password, nonce, created) || digest == GetDigestAlt(password, nonce, created);
}
/// <summary>
/// Validates password, nonce and created create the same digest code as digest.
/// </summary>
/// <param name="password">A string with your unencrypted password.</param>
/// <param name="nonce">The nonce the digest is to be created with as a string.</param>
/// <param name="created">The timestamp as a string.</param>
/// <param name="digest">The digest to validate the password, nonce and created strings against as a string.</param>
/// <param name="enc">A System.Text.Encoding encoding.</param>
/// <returns>true or false depending on wether the digest validates.</returns>
public static bool IsValid(string password, string nonce, string created, string digest, System.Text.Encoding enc)
{
return digest == GetDigest(password, nonce, created, enc) || digest == GetDigestAlt(password, nonce, created, enc);
}
}

We can test the code using the following code snippet.

string digest = "nvvHvNuLb+7wGFrop+cC2tjgQqs=";
string nonce = "bgZ4BHcjmWcg7gVhCxyQOg==";
string timestamp = "2006-04-25T19:40:45Z";
string password = "a";
if (WSSE.IsValid(password, nonce, timestamp, digest, System.Text.Encoding.UTF8))
{
Console.WriteLine("valid");
}
else
{
Console.WriteLine("invalid");
}

This gives us the result “valid” as we’d hope.

To save you time, feel free to download the C# code
WSSE.cs.

Python, XML and iTunes

As I’ve taken the week off work, I thought as well as spending time with my family, I’d brush up my Python skills as they’ve been a bit neglected of late.

I’ve never tried XML parsing with Python so thought I’d cover that. Apple’s iTunes has the ability to export information about your music in XML and I’d been meaning to take a look at that for a while. Why not combine the two, so here’s my take on parsing iTunes export information with Python.

I thought i’d work on a small subset of my library, the ones I’ve actually paid to download from iTunes compared to the ones converted from CD.

Rob's purchased iTunes tracks

The exported XML data is a bit peculiar. I would have assumed it to be values enclosed by sensible tag names e.g <artist>Human League</artist>. However, it’s actually a bunch of neighbouring tags and values like this <key>Artist</key><string>Sheb Wooley</string>

Here’s a snippet from the actual data export I ran…

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Major Version</key><integer>1</integer>
<key>Minor Version</key><integer>1</integer>
<key>Application Version</key><string>6.0.5</string>
<key>Features</key><integer>1</integer>
<key>Music Folder</key><string>file://localhost/D:/Documents%20and%20Settings/Windows%20User/My%20Documents/My%20Music/iTunes/iTunes%20Music/</string>
<key>Library Persistent ID</key><string>C5DD29C89369B278</string>
<key>Tracks</key>
<dict>
<key>312</key>
<dict>
<key>Track ID</key><integer>312</integer>
<key>Name</key><string>The Purple People Eater</string>
<key>Artist</key><string>Sheb Wooley</string>
<key>Album</key><string>20th Century Rocks: 50's Rock 'n Roll - At the Hop</string>
<key>Genre</key><string>Pop</string>
<key>Kind</key><string>Protected AAC audio file</string>
<key>Size</key><integer>2260837</integer>
<key>Total Time</key><integer>135533</integer>
<key>Disc Number</key><integer>1</integer>
<key>Disc Count</key><integer>1</integer>
<key>Track Number</key><integer>5</integer>
<key>Year</key><integer>2001</integer>
<key>Date Modified</key><date>2006-09-28T09:54:23Z</date>
<key>Date Added</key><date>2006-09-28T09:54:10Z</date>
<key>Bit Rate</key><integer>128</integer>
<key>Sample Rate</key><integer>44100</integer>
<key>Play Count</key><integer>11</integer>
<key>Play Date</key><integer>-1042489964</integer>
<key>Play Date UTC</key><date>2007-01-24T08:55:32Z</date>
<key>Normalization</key><integer>7764</integer>
<key>Compilation</key><true/>
<key>Artwork Count</key><integer>1</integer>
<key>Persistent ID</key><string>302B45E87F01479F</string>
<key>Track Type</key><string>File</string>
<key>Protected</key><true/>
<key>Location</key><string>file://localhost/D:/Documents%20and%20Settings/Windows%20User/My%20Documents/My%20Music/iTunes/iTunes%20Music/Compilations/20th%20Century%20Rocks_%2050's%20Rock%20'n%20Roll%20-/05%20The%20Purple%20People%20Eater.m4p</string>
<key>File Folder Count</key><integer>4</integer>
<key>Library Folder Count</key><integer>1</integer>
</dict>
<key>313</key>
<dict>
<key>Track ID</key><integer>313</integer>
<key>Name</key><string>Daisy Daisy</string>
<key>Artist</key><string>Johnny O'Tolle &#38; His Naughty Band</string>
<key>Album</key><string>Gay 90's</string>
<key>Genre</key><string>Vocal</string>
<key>Kind</key><string>Protected AAC audio file</string>
<key>Size</key><integer>2346412</integer>
<key>Total Time</key><integer>125084</integer>
<key>Disc Number</key><integer>1</integer>
<key>Disc Count</key><integer>1</integer>
<key>Track Number</key><integer>2</integer>
<key>Track Count</key><integer>10</integer>
<key>Year</key><integer>2006</integer>
<key>Date Modified</key><date>2006-09-28T09:59:52Z</date>
<key>Date Added</key><date>2006-09-28T09:59:38Z</date>
<key>Bit Rate</key><integer>128</integer>
<key>Sample Rate</key><integer>44100</integer>
<key>Play Count</key><integer>6</integer>
<key>Play Date</key><integer>-1038647848</integer>
<key>Play Date UTC</key><date>2007-03-09T20:10:48Z</date>
<key>Artwork Count</key><integer>1</integer>
<key>Persistent ID</key><string>302B45E87F01490F</string>
<key>Track Type</key><string>File</string>
<key>Protected</key><true/>
<key>Location</key><string>file://localhost/D:/Documents%20and%20Settings/Windows%20User/My%20Documents/My%20Music/iTunes/iTunes%20Music/Johnny%20O'Tolle%20&#38;%20His%20Naughty%20Band/Gay%2090's/02%20Daisy%20Daisy.m4p</string>
<key>File Folder Count</key><integer>4</integer>
<key>Library Folder Count</key><integer>1</integer>
</dict>

This makes parsing the data a bit trickier than I had hoped for. I was hoping to use a nice simple XPath expression, but data like this looks like it’s more a job for a SAX based approach.

I took a look in O’Reilly’s excellent Programming Python, and found a nice SAX parser example to modify.

As it’s just a quick test, I’m making a few assumptions on the XML data that a production system would have to handle. In this case, I’m assume a tag order of Track ID, Name and Artist. Using this order, each time we see one of those tags come past, we can make up a Track object and store the relevant data. In this case, when we see Track ID we need a new Track object to store the data in. When we see Name, we store the track name in the object and when we see Artist we save the artist, push the Track object to our list of Tracks and clear the current Track object.

That’s a bit long winded, so here’s the code.

import xml.sax.handler
class ITunesHandler(xml.sax.handler.ContentHandler):
def __init__(self):
self.parsing_tag = False
self.tag = ''
self.value = ''
self.tracks = []
self.track = None
def startElement(self, name, attributes):
if name == 'key':
self.parsing_tag = True
def characters(self, data):
if self.parsing_tag:
self.tag = data
self.value = ''
else:
# could be multiple lines, so append data.
self.value = self.value + data
def endElement(self,name):
if name == 'key':
self.parsing_tag = False
else:
if self.tag == 'Track ID':
# start of a new track, so a new object
# is needed.
self.track = Track()
elif self.tag == 'Name' and self.track:
self.track.track = self.value
elif self.tag == 'Artist' and self.track:
self.track.artist = self.value
# assume this is all the data we need
# so append the track object to our list
# and reset our track object to None.
self.tracks.append(self.track)
self.track = None
class Track:
def __init__(self):
self.track = ''
self.artist = ''
def __str__(self):
return "Track = %snArtist = %s" % (self.track,self.artist)

In the real world, the Track class would offer a lot more functionality, in this case, it’s just for holding data and providing a pretty printer.

Now we need to parse the XML and display the results, here’s the code…

parser = xml.sax.make_parser()
handler = ITunesHandler()
parser.setContentHandler(handler)
parser.parse('D:\Documents and Settings\Windows User\Desktop\Purchased.xml')
for track in handler.tracks:
print track

Let’s run that code and see what we get…

Track = The Purple People Eater
Artist = Sheb Wooley
Track = Daisy Daisy
Artist = Johnny O'Tolle & His Naughty Band
Track = Don't Dilly Dally
Artist = Kidzone
Track = Jump In My Car
Artist = David Hasselhoff
Track = Puff, the Magic Dragon
Artist = Peter, Paul And Mary
Track = You Give Love a Bad Name
Artist = Bon Jovi
Track = Heart of Glass
Artist = Blondie
Track = Grace Kelly
Artist = Mika
Track = Standing In the Way of Control
Artist = Gossip
Track = Physical
Artist = Olivia Newton-John
Track = Don't You Want Me
Artist = The Human League
Track = Have a Drink On Me
Artist = Lonnie Donegan
Track = My Old Man's a Dustman
Artist = Lonnie Donegan

That’s great! OK, I’m not going to win any awards for my taste in music, but at least I can now think about building music services that use this data.

Using Twitter From Perl

The world and his dog is currently looking at Twitter and eyeing up the possibilities it offers.

I thought i’d jump on the bandwagon, and have a look at the Twitter API.

I wanted to post to a timeline, so the solution is to use one of the update methods. I chose to the XML one, though there is a JSON one also available.

To post to the timeline, Twitter expects an HTTP POST request with a status parameter containing the message you want to post. It associates this to your account by using HTTP’s basic authorization functionality.

It’s simple to throw together a spot of Perl code to post messages to Twitter knowing this. Have a look at this example…


my $message = "A test post from Perl";
my $req = HTTP::Request->new(POST =>'http://twitter.com/statuses/update.xml');
$req->content_type('application/x-www-form-urlencoded');
$req->content('status=' . $message);
$req->authorization_basic($username, $password);
my $resp = $ua->request($req);
my $content = $resp->content;
print $content;

You need to set $username and $password to your username and password, and $message to whatever message you want to appear on your timeline (in this case, “A test post from Perl”).

Using A Bluetooth GPS From Python

Following on from my earlier posting on programming bluetooth from Python, I tried talking to a bluetooth GPS unit from Python.

I have a cheap MSI Starfinder SF100 bluetooth GPS that transmits NMEA format positioning information over a bluetooth socket. So, the first thing we have to do is to is to open a bluetooth socket.

import bluetooth
# bluetooth address of the GPS device.
addr = "00:08:1B:C2:AA:6D"
# port to use.
port = 1
# create a socket and connect to it.
socket = bluetooth.BluetoothSocket(bluetooth.RFCOMM)
socket.connect((addr, port))

Fingers crossed we should now be connected to the GPS unit. In the real world we’d also be checking for exceptions to make sure we really have been able to connect, but this is just a simplified example.

Now we need to receive data from the GPS.

data = ""
while True:
data = socket.recv(1024)

Here we’re initialising a holding variable called data where the received data from the bluetooth socket goes, then we create an infinate loop and continually receive data from the bluetooth socket we creataed earlier.

We should now have some data coming in, so to test it we can just print it out.

print data

Here’s what we get back. 

$GPRMC,225406.537,A,5046.3972,N,00017.
3365,E,0.00,,300107,,,A*7E
$GPGGA,225407.537,5046.3972,N,00017.3365,E,1,05,1.7,55.5,
M,,,,0000*33
$GPGSA,A,3,27,10,29,28,08,,,,,,,,3.9,1.7,3.5*35
$GPRMC,225407.537,A,5046.3972,N,00017.3365,E,0.00,,300107,,,A*7F
$GPGGA,225408.537,5046.3972,N,00017.3365,E,1,05,1.7,55.5,M,,,,0000*3C
$GPGSA,A,3,27,1
0,2
9,28,08,,,,,,,,3.9,1.7,3.5*35
$GPGSV,2,1,08,10,70,218,31,08,64,067
,35,29,49,287,30,27,37,058,40*70
$GPGSV,2,2,08,26,35,283,,28,34,133,45,24,27,254,22,21,12,324,*72
$GPRMC,225408.537,A,5046.3972,N,00017.3365,E,0.00,,300107,,,A*70
$GPGGA,225409.537,5046.3972,N,00017.3365,E,1,06,1.4,55.5,M,,,
,0000*3D

Oh dear, although we are receiving data over the bluetooth socket, it’s in random packet sizes. The NMEA data we want is sent in lines terminated with a carriage return and a line feed. So how can we get the data in this format?

Well we need to use Python’s string manipulation methods. We can use the splitlines method to split the data into a list of lines. As we can’t be sure we have a complete line at the end of the list we need to check there is a carriage return and linefeed there. By default splitlines helpfully removes these, so we need to tell it to keep them in place, after all we can use the strip method later to remove them manually. So all we need to do now is to check the last line in the list has the carriage return and linefeed, if it doesn’t we need to make sure the next data received from the bluetooth socket is appended to the end before we repeat the process again. Imagine we create a holding variable called olddata at the same time as data and this is to hold a copy of the last line. Here’s the code…

# make sure we actually have some data.
if len(data) > 0:
# append the old data to the front of data.
data = olddata + data
# split the data into a list of lines, but make
# sure we preserve the end of line information.
lines = data.splitlines(1)
# iterate over each line
for line in lines:
# if the line has a carriage return and a
# linefeed, we know we have a complete line so
# we can remove those characters and print it.
if line.find("rn") != -1 :
line = line.strip()
print line
# empty the olddata variable now we have
# used the data.
olddata = ""
# else we need to keep the line to add to data
else :
olddata = line

We now have some useful data coming in… 

$GPRMC,225406.537,A,5046.3972,N,00017.3365,E,0.00,,300107,,,A*7E
$GPGGA,225407.537,5046.3972,N,00017.3365,E,1,05,1.7,55.5,M,,,,0000*33
$GPGSA,A,3,27,10,29,28,08,,,,,,,,3.9,1.7,3.5*35
$GPRMC,225407.537,A,5046.3972,N,00017.3365,E,0.00,,300107,,,A*7F
$GPGGA,225408.537,5046.3972,N,00017.3365,E,1,05,1.7,55.5,M,,,,0000*3C
$GPGSA,A,3,27,10,29,28,08,,,,,,,,3.9,1.7,3.5*35
$GPGSV,2,1,08,10,70,218,31,08,64,067,35,29,49,287,30,27,37,058,40*70
$GPGSV,2,2,08,26,35,283,,28,34,133,45,24,27,254,22,21,12,324,*72
$GPRMC,225408.537,A,5046.3972,N,00017.3365,E,0.00,,300107,,,A*70
$GPGGA,225409.537,5046.3972,N,00017.3365,E,1,06,1.4,55.5,M,,,,0000*3D

As you can see the NMEA strings are simply comma seperated blocks of data.

The most useful string is the one with the actual position in. This string is the one starting with $GPRMC. If we look out for this line and split it’s data, we can get our latitude and longitude and use it as we please.

gpsstring = line.split(',')
if gpsstring[0] == '$GPRMC' :
print "Lat: " + gpsstring[3] + gpsstring[4]
print "Long: " + gpsstring[5] + gpsstring[5]

This gives us… 

Lat:  5046.3972N
Long: 000017.3365

Which translates to 50 deg 46.3972′ N and 0 deg 17.3365 E.

There you have it, it’s time to start geocoding you data.

Programming Bluetooth Using Python

I discovered the pybluez project that brings bluetooth connectivity to Python today. It’s been around for a while and is compatible with both Windows (running XP) and Linux (running the bluez stack).

I installed it using the ready made windows installer and it ran first time. It’s very simple and easy to use.

I tweaked the example inquiry.py file by Albert Huang and set it up to show all the services on nearby devices.

Here’s the code…

import bluetooth
print "looking for nearby devices..."
nearby_devices = bluetooth.discover_devices(lookup_names = True, flush_cache = True, duration = 20)
print "found %d devices" % len(nearby_devices)
for addr, name in nearby_devices:
print " %s - %s" % (addr, name)
for services in bluetooth.find_service(address = addr):
print " Name: %s" % (services["name"])
print " Description: %s" % (services["description"])
print " Protocol: %s" % (services["protocol"])
print " Provider: %s" % (services["provider"])
print " Port: %s" % (services["port"])
print " Service id: %s" % (services["service-id"])
print ""
print ""

Here’s a quick run through of the code.

Firstly we need to import the bluetooth class to give us access to all that lovely bluetooth functionality.

We now look for nearby devices using bluetooth.discover_devices. I’m adding a few parameters as well. lookup_names is set to True so we can get the devices names instead of the just their addresses, and flush_cache is also set to True to make sure we always look for fresh information. Finally we set duration to 20, meaning we look for devices for up to 20 seconds. This is a bit excessive, but useful when testing.

We should have a list of devices now, assuming some were found, so we now loop over them.

We print out the name and address then look for services offered by the device by calling the bluetooth.find_service method and passing in the device’s address.

This returns a list of dictionaries describing the services available. We iterate over this and print out details.

That’s the code, now here’s a few results. I ran this on the train home and discovered 7 active bluetooth devices offereing a variety of services. I won’t bore you with them all, so here’s the entry for my Nokia N93. 

Perl - 00:16:BC:30:D8:76
Name:        AVRCP Target
Description: Audio Video Remote Control
Protocol:    L2CAP
Provider:    Symbian Software Ltd.
Port:        23
Service id:  None
Name:        Hands-Free Audio Gateway
Description:
Protocol:    RFCOMM
Provider:    None
Port:        28
Service id:  None
Name:        Headset Audio Gateway
Description:
Protocol:    RFCOMM
Provider:    None
Port:        29
Service id:  None
Name:        SyncMLClient
Description:
Protocol:    RFCOMM
Provider:    None
Port:        10
Service id:  None
Name:        OBEX File Transfer
Description:
Protocol:    RFCOMM
Provider:    None
Port:        11
Service id:  None
Name:        Nokia OBEX PC Suite Services
Description:
Protocol:    RFCOMM
Provider:    None
Port:        12
Service id:  None
Name:        SyncML DM Client
Description:
Protocol:    RFCOMM
Provider:    None
Port:        13
Service id:  None
Name:        Nokia SyncML Server
Description:
Protocol:    RFCOMM
Provider:    None
Port:        14
Service id:  None
Name:        OBEX Object Push
Description:
Protocol:    RFCOMM
Provider:    None
Port:        9
Service id:  None
Name:        Dial-Up Networking
Description:
Protocol:    RFCOMM
Provider:    None
Port:        2
Service id:  None
Name:        Imaging
Description:
Protocol:    RFCOMM
Provider:    None
Port:        15
Service id:  None

As you can see it offers a lot of interesting services, but two have caught my eye and call for a bit more investigation later.

Firstly there is the Imaging service on port 15. I wonder what that does? Does it take input from a remote camera?

Secondly there is AVRCP Target the Audio Visual Remote Control Target from Symbian. I wonder if this means I can use a seperate bluetooth device to control my Nokia N93 when it’s playing back video?

A quick google reveals a (confidential?) specification document for AVRCP…

Nokia Lifeblog Posting Protocol Update

Nokia seem to have updated their Atom upload protocol in recent versions of their phones.

I’ve just got hold of a Nokia N93 and tried posting to this blog using Lifeblog and the new Web Upload functionality in the Gallery with use the Atom protocol.

These postings were failing with a bad password error. I know my username and password are correct and they worked using my old phone.

As I had written my own blogging software, and it’s Atom upload functionality I was able to debug the messages being sent from the phone to the server.

It turns out that newer versions of Nokia Lifeblog encrypt their passwords differently to older versions.

They use a method called WSSE. See my previous article on how to use WSSE with Perl for more details on how this works, plus example code.

Very quickly, the password isn’t sent, but a digest of the password and the values to generate the digest yourself are. These are then encoded using base64 for transmission over the internet.

Older versions (pre Lifeblog 2.0) expect a value called the nonce to be used as sent while the password digest is generated. Newer versions (Lifeblog 2.0+) expect the nonce to be decoded from base64 before the digest is generated.

Once I had worked out what was going on, it was simple to modify my code to check against both possible versions of the password digest. This means I can now post using old and new versions of Lifeblog, or the web upload functionality.

Here’s a very quick bit of example Perl code. Assume $my_digest is the digest with the original nonce left in place and $my_alternative_digest is the digest with the nonce decoded first. $digest is the digest sent my incoming Atom request.

## example perl code to check digests)
if (($digest eq $my_digest) || ($digest eq $my_alternative_digest)) {
## one of the digests has validated, so continue here
} else {
## neither digests has validated, so return invalid password responses here.
}

I don’t think this change has been communicated very well amongst the development community. It has probably come about due to changes in the Atom spec, but (admittedly only quickly) checking the documentation I’ve not found it.

I will be updating my WSSE validation example accordingly, and a new version of AtomServer.pm that allows posting from Lifeblog 2.0 to Movable Type shortly.