Uploading Files To A LAMP Server Using .NET

We had to setup and run a webcam for the Q Awards red carpet earlier this week.

The original plan was to have an old Axis ethernet webcam, connected to a wifi network using a Dlink bridge. After a nightmare of not being able to get the Dlink to work, we gave up and went for a USB webcam connected to a laptop approach.

I had to write some software to handle the capture of the image and the upload to the webserver. Because we wanted to do a few custom things on the way we weren’t able to use out of the box software.

I’ll post on how to use a webcam with .NET another time. What I wanted to document was how to upload a file using .NET to a LAMP server.

It turns out to be easier than I thought for .NET, one line of code can achieve this using the My.Computer.Network.UploadFile Method.

For example, to upload test.txt to http://www.robertprice.co.uk/asp_upload.pl we can do the following (in Visual Basic)…

My.Computer.Network.UploadFile("C:test.txt","http://www.robertprice.co.uk/asp_upload.pl")

Now we need some code at the other end to store this upload. As I was building a webcam application, the uploaded file, an image in this case, was always overwritten.

The uploaded .NET file uses an bog standard HTTP upload as described in RFC 1867.

If we use Perl to read this file, we can use the standard CGI.pm module to do most of the hard work for us.

The uploaded file is passed a parameter called file, so we can create a CGI object and get a reference to this using the param method. Once we have this, we can treat it as a filehandle and read the date off it. Then all we have to do is to write it out to a file.

The following code achieves this.

#!/usr/bin/perl -w
use strict;
use CGI.pm;
my $CGI = new CGI;
my $file = $CGI->param('file');
open(FILE, ">/path/to/my/file/test.txt") or die $!;
while(<$file>) {
print FILE $_;
}
close(FILE);
print "Content-type: text/htmlnn";
print "OKn";

Dynamically Setting Domain Names For Webtrends

We use Webtrends at work to track usage of our websites.

We use the on demand version which relies on embedding a JavaScript tag onto the page. Recently, we’ve moved to using 1st party cookies (cookies set by the site) instead of 3rd party cookies (cookies set externally by webtrends). These are set in the JavaScript tag and are hard coded to the site, a real pain if you happen to move domain names as you need to code them up specifically each time. We got caught out recently by using a tag with the wrong domain name set in it, meaning we were unable to track the page impressions. We needed a solution…

As the tag is being set in JavaScript, we can use JavaScript to set the domain dynamically. Not all our sites are served dynamically so using JavaScript should mean it would work for anyone using this tag.

Here’s the offending bit of code…

// Code section for Set the First-Party Cookie domain
var gFpcDom=".kerrang.com";

This works for anything served from a Kerrang.com domain. However, what if we want to use a different domain for some reason?

We’ll JavaScript has a property in the document object called domain, we can use this at runtime to find out the full domain name the page was served from.

However, we don’t need the full domain name, just parent domain. For example, www.kerrang.com should become kerrang.com.

We can fix this by using several approaches, but the easiest is the split, splice, join method.

Firstly we split the domain into it’s components by splitting on the periods.

document.domain.split('.');

This gives us the array (“www”,”kerrang”,”com”).

Secondly we splice this to remove the first element.

document.domain.split('.').splice(1.3);

This is saying to take 3 elements after the 1st element in the array. This should cover us if we want to use something like kerrang.co.uk. So after running this we need to should have an array looking like this, (“kerrang”,”com”).

Finally we join this back together again, making sure a period is between each joined element.

document.domain.split('.').splice(1,3).join('.');

Running this should give us kerrang.com, which is acceptable to use in the Webtrends tag.

So our final code looks like this…

var gFpcDom = document.domain.split('.').splice(1,3).join('.');

Movable Type Comment Problems

Earlier on the guys at Kerrang posted a rather hot topic on the lovely Mr Manson and his thoughts on My Chemical Romance.

This caused a fever of activity on the website, and unfortunately caused it to corrupt with the following messages appearing in the Movable Type activity logs.

Comment save failed with Insertion test failed on SQL error Duplicate entry ‘6692’ for key 1

Now it took a while digging in the code to work out what was wrong as I’ve not seen this before.

Eventually logging into the MySQL database, showed up the problem.

mysql> check table mt_comment;
+———————+——-+———-+———————————————————–+
| Table | Op | Msg_type | Msg_text |
+———————+——-+———-+———————————————————–+
| kerrang2.mt_comment | check | warning | 12 clients are using or haven’t closed the table properly |
| kerrang2.mt_comment | check | warning | Size of datafile is: 2999316 Should be: 2998080 |
| kerrang2.mt_comment | check | error | Found 6561 keys of 6559 |
| kerrang2.mt_comment | check | error | Corrupt |
+———————+——-+———-+———————————————————–+

Ahha! The comments table is screwed. The solution is to run the following…

mysql> repair table mt_comment;
+———————+——–+———-+——————————————+
| Table | Op | Msg_type | Msg_text |
+———————+——–+———-+——————————————+
| kerrang2.mt_comment | repair | warning | Number of rows changed from 6559 to 6561 |
| kerrang2.mt_comment | repair | status | OK |
+———————+——–+———-+——————————————+

Comments on Movable Type now work fine.

Using Nokia Lifeblog On Vodafone UK

Those of you in the UK and using Nokia Lifeblog on Vodafone contract may have noticed it no longer works.

We have problems at the moment with Vodafone’s new mobile optimising technology. This takes a page and tries to fit it onto a mobile phone screen. It does this by pretending to be a more competant browser in it’s HTTP headers.

However, in faking the HTTP headers it also strips out the WSSE authentication that Lifeblog uses for it’s security, meaning that each post will fail as unauthorised.

There is currently no easy work around for this except to change network or to register your site with Bango as they are apparently whitelisting sites to bypass this new proxy.

Using Perl To Send Heartbeat To A Python Server

I’ve been brushing up on my networking and Python skills lately.

One example in the Python Cookbook is recipe 13.11 Detecting Inactive Computers. Here there are two scripts, one that sends a heartbeat, and another that listens for heartbeats. The idea is a simple UDP packet containing the short string “PyHB” is sent every 5 seconds or so by a client. A server then listens for these datagrams, stores the addresses of the clients which it receives messages from, and if it fails to receive a message in a certain time period it flags this up on the console.

I thought I’d just check how easy it would be to create a Perl client to talk to the Python backend. As it’s all through sockets it should be fairly easy.

To start with we define a few constants that cover the address of the server we want to connect to, the port to send the datagram to, how often we want to send it and if we want debugging information displayed or not.

use constant SERVER_IP => '127.0.0.1';
use constant SERVER_PORT => 43278;
use constant BEAT_PERIOD => 5;
use constant DEBUG => 1;

Now we need to create the socket, in this case we’ll create a new IO::Socket object.

my $hbsocket = IO::Socket::INET->new(Proto => 'udp',
PeerPort => SERVER_PORT,
PeerAddr => SERVER_IP)
or die "Creating socket: $!n";

Next we need to send our message, the string “PyHB” over the socket.

$hbsocket->send('PyHB') or die "send: $!";

Finally we need to wrap this up in a loop and sleep BEAT_PERIOD seconds before repeating our message.

Here’s the final script.

#!/usr/bin/perl -w
use strict;
use IO::Socket;
use constant SERVER_IP => '127.0.0.1';
use constant SERVER_PORT => 43278;
use constant BEAT_PERIOD => 5;
use constant DEBUG => 1;
print "Sending heatbeat to IP " . SERVER_IP ." , port " . SERVER_PORT . "n";
print "press Ctrl-C to stopn";
while (1) {
my $hbsocket = IO::Socket::INET->new(Proto => 'udp',
PeerPort => SERVER_PORT,
PeerAddr => SERVER_IP)
or die "Creating socket: $!n";
$hbsocket->send('PyHB') or die "send: $!";
print "Time: " . localtime(time) . "n" if (DEBUG);
sleep(BEAT_PERIOD);
}

Just set the value of DEBUG to 0 if you don’t want to see the time each time we send a message to the server.

To test this you’ll have to be running the server from recipe 13.11 in the Python Cookbook. You can download a zip file of the examples in the Python Cookbook from the O’Reilly website to test this.

Atom WSSE Authentication Using C# .NET

I received an email asking for a hand with a C# .NET implementation of the WSSE atom authentication for Nokia’s Lifeblog.

I took a look at this as I’m brushing up on my C# at the moment for work, and I relish a challenge.

As there are two different methods of authentication depending on the version of Lifeblog being used, the code has to handle this.

The first method is to append the nonce, timestamp and password together, create a SHA1 hash with this and Base64 encode the result. The second way is almost the same, but the nonce has to be Base64 decoded first before being appended with the timestamp and password.

See my previous article on WSSE Authentication For Atom Using Perl for more details on how this works.

The first method can be calculate a digest value using code like this.

string mystring = nonce + created + password;
SHA1Managed SHhash = new SHA1Managed();
string mydigest = Convert.ToBase64String(SHhash.ComputeHash(System.Text.Encoding.ASCII.GetBytes(mystring)));

My first attempt at the second method looked very similar.

string mystring = (System.Text.Encoding.ASCII.GetString(Convert.FromBase64String(nonce))) + created + password;
SHA1Managed SHhash = new SHA1Managed();
string mydigest = Convert.ToBase64String(SHhash.ComputeHash(System.Text.Encoding.ASCII.GetBytes(mystring)));

Looks OK on the surface, but it fails to validate when posting from my Nokia N93 phone.

What’s gone wrong?

Well looking in the debugger, the process of converting the nonce back from the array of bytes Convert.FromBase64String() produces and appending them to form mystring corrupts the data compared with the Perl version in my previous article.

We know the approach is basically right, so we need to cut out this corrupting step.

The best way to do this is to keep the data in byte arrays as we need the data in this format anyway to computer the SHA1.


System.Text.Encoding enc = System.Text.Encoding.UTF8;
byte[] noncebytes = enc.GetBytes(nonce);
byte[] passwordbytes = enc.GetBytes(password);
byte[] createdbytes = enc.GetBytes(created);
byte[] code = new byte[nonce.Length + password.Length + created.Length];
Array.Copy(nonce, code, nonce.Length);
Array.Copy(created, 0, code, nonce.Length, created.Length);
Array.Copy(password, 0, code, nonce.Length + created.Length, password.Length);
System.Security.Cryptography.SHA1Managed SHhash = new System.Security.Cryptography.SHA1Managed();
string digest = Convert.ToBase64String(SHhash.ComputeHash(code));

I’m sure there is a nicer way to achieve the appending of the 3 arrays together in C#, but this works for now.

We can tie this up to form a nice static class to handle our WSSE authentication.

/// <summary>
/// Computes WSSE codes from input for validation.
/// </summary>
public static class WSSE {
/// <summary>
/// Calculate the digest.
/// </summary>
/// <param name="password">A string with your unencrypted password.</param>
/// <param name="nonce">The nonce the digest is to be created with as a string.</param>
/// <param name="created">The timestamp as a string.</param>
/// <returns>A string containing the digest.</returns>
public static string GetDigest(string password, string nonce, string created)
{
return GetDigest(password, nonce, created, System.Text.Encoding.ASCII);
}
/// <summary>
/// Calculate the digest.
/// </summary>
/// <param name="password">A string with your unencrypted password.</param>
/// <param name="nonce">The nonce the digest is to be created with as a string.</param>
/// <param name="created">The timestamp as a string.</param>
/// <param name="enc">The System.Text.Encoding to use.</param>
/// <returns>A string containing the digest.</returns>
public static string GetDigest(string password, string nonce, string created, System.Text.Encoding enc)
{
byte[] noncebytes = enc.GetBytes(nonce);
byte[] passwordbytes = enc.GetBytes(password);
byte[] createdbytes = enc.GetBytes(created);
return generatecode(noncebytes, passwordbytes, createdbytes);
}
/// <summary>
/// Calculate the alternative digest.
/// </summary>
/// <param name="password">A string with your unencrypted password.</param>
/// <param name="nonce">The nonce the digest is to be created with as a string.</param>
/// <param name="created">The timestamp as a string.</param>
/// <returns>A string containing the digest.</returns>
public static string GetDigestAlt(string password, string nonce, string created)
{
return GetDigestAlt(password, nonce, created, System.Text.Encoding.ASCII);
}
/// <summary>
/// Calculate the alternative digest.
/// </summary>
/// <param name="password">A string with your unencrypted password.</param>
/// <param name="nonce">The nonce the digest is to be created with as a string.</param>
/// <param name="created">The timestamp as a string.</param>
/// <param name="enc">The System.Text.Encoding to use.</param>
/// <returns>A string containing the digest.</returns>
public static string GetDigestAlt(string password, string nonce, string created, System.Text.Encoding enc)
{
byte[] noncebytes = Convert.FromBase64String(nonce);
byte[] passwordbytes = enc.GetBytes(password);
byte[] createdbytes = enc.GetBytes(created);
return generatecode(noncebytes, passwordbytes, createdbytes);
}
private static string generatecode(byte[] nonce, byte[] password, byte[] created)
{
byte[] code = new byte[nonce.Length + password.Length + created.Length];
Array.Copy(nonce, code, nonce.Length);
Array.Copy(created, 0, code, nonce.Length, created.Length);
Array.Copy(password, 0, code, nonce.Length + created.Length, password.Length);
System.Security.Cryptography.SHA1Managed SHhash = new System.Security.Cryptography.SHA1Managed();
return Convert.ToBase64String(SHhash.ComputeHash(code));
}
/// <summary>
/// Validates password, nonce and created create the same digest code as digest.
/// </summary>
/// <param name="password">A string with your unencrypted password.</param>
/// <param name="nonce">The nonce the digest is to be created with as a string.</param>
/// <param name="created">The timestamp as a string.</param>
/// <param name="digest">The digest to validate the password, nonce and created strings against as a string.</param>
/// <returns>true or false depending on wether the digest validates</returns>
public static bool IsValid(string password, string nonce, string created, string digest)
{
return digest == GetDigest(password, nonce, created) || digest == GetDigestAlt(password, nonce, created);
}
/// <summary>
/// Validates password, nonce and created create the same digest code as digest.
/// </summary>
/// <param name="password">A string with your unencrypted password.</param>
/// <param name="nonce">The nonce the digest is to be created with as a string.</param>
/// <param name="created">The timestamp as a string.</param>
/// <param name="digest">The digest to validate the password, nonce and created strings against as a string.</param>
/// <param name="enc">A System.Text.Encoding encoding.</param>
/// <returns>true or false depending on wether the digest validates.</returns>
public static bool IsValid(string password, string nonce, string created, string digest, System.Text.Encoding enc)
{
return digest == GetDigest(password, nonce, created, enc) || digest == GetDigestAlt(password, nonce, created, enc);
}
}

We can test the code using the following code snippet.

string digest = "nvvHvNuLb+7wGFrop+cC2tjgQqs=";
string nonce = "bgZ4BHcjmWcg7gVhCxyQOg==";
string timestamp = "2006-04-25T19:40:45Z";
string password = "a";
if (WSSE.IsValid(password, nonce, timestamp, digest, System.Text.Encoding.UTF8))
{
Console.WriteLine("valid");
}
else
{
Console.WriteLine("invalid");
}

This gives us the result “valid” as we’d hope.

To save you time, feel free to download the C# code
WSSE.cs.

Python, XML and iTunes

As I’ve taken the week off work, I thought as well as spending time with my family, I’d brush up my Python skills as they’ve been a bit neglected of late.

I’ve never tried XML parsing with Python so thought I’d cover that. Apple’s iTunes has the ability to export information about your music in XML and I’d been meaning to take a look at that for a while. Why not combine the two, so here’s my take on parsing iTunes export information with Python.

I thought i’d work on a small subset of my library, the ones I’ve actually paid to download from iTunes compared to the ones converted from CD.

Rob's purchased iTunes tracks

The exported XML data is a bit peculiar. I would have assumed it to be values enclosed by sensible tag names e.g <artist>Human League</artist>. However, it’s actually a bunch of neighbouring tags and values like this <key>Artist</key><string>Sheb Wooley</string>

Here’s a snippet from the actual data export I ran…

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Major Version</key><integer>1</integer>
<key>Minor Version</key><integer>1</integer>
<key>Application Version</key><string>6.0.5</string>
<key>Features</key><integer>1</integer>
<key>Music Folder</key><string>file://localhost/D:/Documents%20and%20Settings/Windows%20User/My%20Documents/My%20Music/iTunes/iTunes%20Music/</string>
<key>Library Persistent ID</key><string>C5DD29C89369B278</string>
<key>Tracks</key>
<dict>
<key>312</key>
<dict>
<key>Track ID</key><integer>312</integer>
<key>Name</key><string>The Purple People Eater</string>
<key>Artist</key><string>Sheb Wooley</string>
<key>Album</key><string>20th Century Rocks: 50's Rock 'n Roll - At the Hop</string>
<key>Genre</key><string>Pop</string>
<key>Kind</key><string>Protected AAC audio file</string>
<key>Size</key><integer>2260837</integer>
<key>Total Time</key><integer>135533</integer>
<key>Disc Number</key><integer>1</integer>
<key>Disc Count</key><integer>1</integer>
<key>Track Number</key><integer>5</integer>
<key>Year</key><integer>2001</integer>
<key>Date Modified</key><date>2006-09-28T09:54:23Z</date>
<key>Date Added</key><date>2006-09-28T09:54:10Z</date>
<key>Bit Rate</key><integer>128</integer>
<key>Sample Rate</key><integer>44100</integer>
<key>Play Count</key><integer>11</integer>
<key>Play Date</key><integer>-1042489964</integer>
<key>Play Date UTC</key><date>2007-01-24T08:55:32Z</date>
<key>Normalization</key><integer>7764</integer>
<key>Compilation</key><true/>
<key>Artwork Count</key><integer>1</integer>
<key>Persistent ID</key><string>302B45E87F01479F</string>
<key>Track Type</key><string>File</string>
<key>Protected</key><true/>
<key>Location</key><string>file://localhost/D:/Documents%20and%20Settings/Windows%20User/My%20Documents/My%20Music/iTunes/iTunes%20Music/Compilations/20th%20Century%20Rocks_%2050's%20Rock%20'n%20Roll%20-/05%20The%20Purple%20People%20Eater.m4p</string>
<key>File Folder Count</key><integer>4</integer>
<key>Library Folder Count</key><integer>1</integer>
</dict>
<key>313</key>
<dict>
<key>Track ID</key><integer>313</integer>
<key>Name</key><string>Daisy Daisy</string>
<key>Artist</key><string>Johnny O'Tolle &#38; His Naughty Band</string>
<key>Album</key><string>Gay 90's</string>
<key>Genre</key><string>Vocal</string>
<key>Kind</key><string>Protected AAC audio file</string>
<key>Size</key><integer>2346412</integer>
<key>Total Time</key><integer>125084</integer>
<key>Disc Number</key><integer>1</integer>
<key>Disc Count</key><integer>1</integer>
<key>Track Number</key><integer>2</integer>
<key>Track Count</key><integer>10</integer>
<key>Year</key><integer>2006</integer>
<key>Date Modified</key><date>2006-09-28T09:59:52Z</date>
<key>Date Added</key><date>2006-09-28T09:59:38Z</date>
<key>Bit Rate</key><integer>128</integer>
<key>Sample Rate</key><integer>44100</integer>
<key>Play Count</key><integer>6</integer>
<key>Play Date</key><integer>-1038647848</integer>
<key>Play Date UTC</key><date>2007-03-09T20:10:48Z</date>
<key>Artwork Count</key><integer>1</integer>
<key>Persistent ID</key><string>302B45E87F01490F</string>
<key>Track Type</key><string>File</string>
<key>Protected</key><true/>
<key>Location</key><string>file://localhost/D:/Documents%20and%20Settings/Windows%20User/My%20Documents/My%20Music/iTunes/iTunes%20Music/Johnny%20O'Tolle%20&#38;%20His%20Naughty%20Band/Gay%2090's/02%20Daisy%20Daisy.m4p</string>
<key>File Folder Count</key><integer>4</integer>
<key>Library Folder Count</key><integer>1</integer>
</dict>

This makes parsing the data a bit trickier than I had hoped for. I was hoping to use a nice simple XPath expression, but data like this looks like it’s more a job for a SAX based approach.

I took a look in O’Reilly’s excellent Programming Python, and found a nice SAX parser example to modify.

As it’s just a quick test, I’m making a few assumptions on the XML data that a production system would have to handle. In this case, I’m assume a tag order of Track ID, Name and Artist. Using this order, each time we see one of those tags come past, we can make up a Track object and store the relevant data. In this case, when we see Track ID we need a new Track object to store the data in. When we see Name, we store the track name in the object and when we see Artist we save the artist, push the Track object to our list of Tracks and clear the current Track object.

That’s a bit long winded, so here’s the code.

import xml.sax.handler
class ITunesHandler(xml.sax.handler.ContentHandler):
def __init__(self):
self.parsing_tag = False
self.tag = ''
self.value = ''
self.tracks = []
self.track = None
def startElement(self, name, attributes):
if name == 'key':
self.parsing_tag = True
def characters(self, data):
if self.parsing_tag:
self.tag = data
self.value = ''
else:
# could be multiple lines, so append data.
self.value = self.value + data
def endElement(self,name):
if name == 'key':
self.parsing_tag = False
else:
if self.tag == 'Track ID':
# start of a new track, so a new object
# is needed.
self.track = Track()
elif self.tag == 'Name' and self.track:
self.track.track = self.value
elif self.tag == 'Artist' and self.track:
self.track.artist = self.value
# assume this is all the data we need
# so append the track object to our list
# and reset our track object to None.
self.tracks.append(self.track)
self.track = None
class Track:
def __init__(self):
self.track = ''
self.artist = ''
def __str__(self):
return "Track = %snArtist = %s" % (self.track,self.artist)

In the real world, the Track class would offer a lot more functionality, in this case, it’s just for holding data and providing a pretty printer.

Now we need to parse the XML and display the results, here’s the code…

parser = xml.sax.make_parser()
handler = ITunesHandler()
parser.setContentHandler(handler)
parser.parse('D:\Documents and Settings\Windows User\Desktop\Purchased.xml')
for track in handler.tracks:
print track

Let’s run that code and see what we get…

Track = The Purple People Eater
Artist = Sheb Wooley
Track = Daisy Daisy
Artist = Johnny O'Tolle & His Naughty Band
Track = Don't Dilly Dally
Artist = Kidzone
Track = Jump In My Car
Artist = David Hasselhoff
Track = Puff, the Magic Dragon
Artist = Peter, Paul And Mary
Track = You Give Love a Bad Name
Artist = Bon Jovi
Track = Heart of Glass
Artist = Blondie
Track = Grace Kelly
Artist = Mika
Track = Standing In the Way of Control
Artist = Gossip
Track = Physical
Artist = Olivia Newton-John
Track = Don't You Want Me
Artist = The Human League
Track = Have a Drink On Me
Artist = Lonnie Donegan
Track = My Old Man's a Dustman
Artist = Lonnie Donegan

That’s great! OK, I’m not going to win any awards for my taste in music, but at least I can now think about building music services that use this data.

Using Twitter From Perl

The world and his dog is currently looking at Twitter and eyeing up the possibilities it offers.

I thought i’d jump on the bandwagon, and have a look at the Twitter API.

I wanted to post to a timeline, so the solution is to use one of the update methods. I chose to the XML one, though there is a JSON one also available.

To post to the timeline, Twitter expects an HTTP POST request with a status parameter containing the message you want to post. It associates this to your account by using HTTP’s basic authorization functionality.

It’s simple to throw together a spot of Perl code to post messages to Twitter knowing this. Have a look at this example…


my $message = "A test post from Perl";
my $req = HTTP::Request->new(POST =>'http://twitter.com/statuses/update.xml');
$req->content_type('application/x-www-form-urlencoded');
$req->content('status=' . $message);
$req->authorization_basic($username, $password);
my $resp = $ua->request($req);
my $content = $resp->content;
print $content;

You need to set $username and $password to your username and password, and $message to whatever message you want to appear on your timeline (in this case, “A test post from Perl”).