In April 2010 Ordnance Survey released The Code-Point Open dataset under a licence compatible with the Creative Commons Attribution Licence. The dataset locates, by OSGB grid reference every postcode unit in England, Scotland and Wales. Northern Ireland is missing. You can read more about the limitations of the Code-Point Open dataset in my previous post

All is not lost though. The Northern Ireland Statistics and Research Agency (NISRA) provide the necessary data. As far as I can tell this is freely reusable under the terms of the Click-Use PSI Licence.

NISRA provide the data in MapInfo or ESRI Shapefile format. The coordinates are in Irish Grid Reference format. For ease of use I have re-exported the data as a single CSV file.

Download the CSV

Reproduced under the terms of the Click-Use PSI Licence.

Before I begin, let me just say that decision of Ordnance Survey to release a whole bunch of their data into the public domain is a massive leap in the right direction. The number of freely available datasets is vast, but what I’m most interested in is the Code-Point Open dataset.

Code-Point Open purports to be a dataset containing the geographical location of every postcode unit in the United Kingdom. Released into the public domain under a licence compatible with the Creative Commons Attribution 3.0 Licence, it sounds like exactly what we’ve all been waiting/campaigning for since literally forever. Postcode geodata without the Royal Mail PAF file price tag.

Too Good to be True?

Unfortunately yes. The Code-Point Open dataset is advertised with the bold statement that “Code-Point Open locates every postcode unit in the UK with precision”. Unfortunately this is simply untrue, and as much as it pains me to point this out, they’ve fallen at the first hurdle. The reason? Northern Ireland, The Channel Islands and The Isle of Man are all missing.

They immediately go on to state:

There are approximately 1.7 million postcode units in England, Scotland and Wales. Each postcode unit, such as KY12 8UP or PO14 2RS, contains an average of fifteen adjoining addresses.

Northern Ireland postcodes are not available with Code-Point Open.

The omission of Channel Islands and The Isle of Man is forgiveable. Technically they’re not a part of the UK even though they’re covered by Royal Mail’s postcode system. There is absolutely no way however that Code-Point Open can be deemed a full “UK” dataset with Northern Ireland (BT postcodes) missing.

For those of you for whom geography is not a strong-point, let me clarify (political correctness aside). England + Scotland + Wales == Great Britain. Great Britain + Northern Ireland == The United Kingdom.

Let’s take a look at how the Open offering stacks up against the commercial product

* £5,852.75 is based on the 101+ terminal licence (which would be the licence required for web use and comes without a discount for non-commercial use)

In Conclusion

In my opinion Code-Point Open is little more than a token effort. Code-Point Open can be used to freely and legally locate postcodes in England, Scotland and Wales. Channel Islands (JE and GY), Isle of Man (IM) and Northern Ireland (BT) postcodes are all entirely missing from the dataset, and for this reason It is not a viable option for commercial use or any application where you don’t want to exclude a sizeable portion of the UK population (just over 2 million people according by my Wikipedia based caclulations)

If your year columns are YYYY and your month and day columns are zero padded then your solution is an easy one. Simply concatenate the values together using MySQL’s CONCAT_WS function (concatenate with separator).

SELECT CONCAT_WS('-', '2001', '01','31')
--> '2001-01-31'

If this isn’t the case you could deal with the problem in code, using PHPs famously forgiving strtotime function for example, but my OCD got the better of me and I decided to revisit MySQL’s built in date and time functions

An Elegant Solution

The solution I found is to use the STR_TO_DATE function with the CONCAT_WS function. That way, even if the date parts aren’t correctly formatted as in the following example (2 digit year, 1 digit month), STR_TO_DATE will correctly interpret the string ‘1-1-31′ and return a datetime value of ‘2001-01-31′. This also has the advantage of being an actual DATETIME value rather than a string.

SELECT STR_TO_DATE(CONCAT_WS('-',1,1,31),'%Y-%m-%d') AS date
--> '2001-01-31'

Obviously, this is a solution to a problem that shouldn’t exist, but that’s just how we roll.

Sometimes the penny drops and you realise there’s a childishly simple solution to a problem that’s been bugging you for years. And no, I’m not talking about my wife, I’m talking about Yoda Syntax.

Have you ever spent far too long hunting down an impossible bug in your PHP only to eventually track it down to a typo in an if statement. Chances are it was a misplaced ‘=’.

Named must your fail be before banish it you can

We all do it, and more often than we’d probably like to admit. Unless you don’t, in which case go pat yourself on the back, you’re a PHP rockstar. But as a simplified example say you wanted to write the following rather pointless function:

function is_it_red($colour) {
    if ($colour == 'red') {
         return true;
    }
}

but what you ended up writing was

function is_it_red($colour) {
    if ($colour = 'red') {
         return true;
    }
}

And as you know, whatever you pass into the second function (because you’re accidentally performing a variable assignment rather than testing for equality) it will return true.
Obviously this is an oversimplified example, but typos like this nestled inside large classes can prove hair-removingly difficult to track down.

Help you I can, yes

The solution to this is simply to alter the way you write if statements:

function is_it_red($colour) {
    if ('red' == $colour) {
         return true;
    }
}

It looks odd, yeah, and you might have seen other people writing their if statements backwards in the past and wondered why. But the benefit of this is striking, and so simple that I’m left wondering why I never thought of it before.

Quite simply, if you make a typo and write the following:

function is_it_red($colour) {
    if ('red' = $colour) {
         return true;
    }
}

You will cause a big nasty syntax error that you simply cannot fail to notice.

Parse error: syntax error, unexpected ‘=’ in /var/www/vhosts/example.com/httpdocs/what_colour_is_it.php

It’s handy in that it let’s you directly request a streetview thumbnail with nothing more than a postcode. The format of the request looks like this:

As far as the postcode goes either strip out the space or escape it. The maximum dimensions are 416 x 208 pixels.

Example:
http://geo.jamiethompson.co.uk/streetview/SW1W9TQ_416×208.jpg

As you reduce the thumbnail dimensions the image gets centre cropped accordingy. For the least distortion it’s best to use a very small portion cropped out of the center as Google do with the streetview thumbnails on Google Maps.

I’m interested in possibly taking this further with canvas and JavaScript. I’m fairly sure that if the maths behind it were figured out you could, via cylindrical projection, un-distort the panorama to provide a natural 360 degree panorama.

That’s one for the todo list though.

URL shortening services work by redirecting a users browser from the shortened URL to the actual target URL. Some of these services provide means of reversing the shortening process. Some provide API methods for this.

The following function will take a shortened URL, from any service, and return the original URL. As it works by inspecting the HTTP redirect headers it is 100% service independent, and does away with the need to query the API of services like bit.ly for url expansion.

function expand_url($url) {

	// Use curl to fetch the HTTP Headers
	$ch = curl_init($url);
	curl_setopt($ch, CURLOPT_HEADER, 1); // just the header
	curl_setopt($ch, CURLOPT_NOBODY, 1); // not the body
	curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
	$output = curl_exec($ch);
	preg_match('/Location: (.*)\n/', $output, $matches);

	// if no redirect header then return the original url
	return isset($matches[1]) ? $matches[1] : $url;

}

A Simpler PHP 5 Version

I’ve used curl for portability. PHP 5 has a built in get_headers function which you could use to simplify the function a little. It would look something like this:

function expand_url($url) {

	$h = get_headers($url);

	// if no redirect header then return the original url
	return isset($h['Location']) ? $h['Location'] : $url;

}

Do you want to create short urls from within your own PHP application? Most url shortening services provide an API, it’s how apps like twitter clients provide built-in url shortening. I wrote a small PHP function to provide url shortening from two of the most popular services, is.gd and tinyurl.com. Where’s bit.ly I hear you ask. Well yeah, sure, that’s what all the cool kids are using these days, but bit.ly requires an API key. That’s not a huge issue, it’s free. It just doesn’t sit well with creating a nice portable, drop in a go PHP function. So bit.ly didn’t get an invite to my super mega PHP function party.

If you want to get some bit.ly action on the go then feel free to modify the code below accordingly. I might update this at some point. I keep thinking I should, but meh.

function shorten_url($url,$service='tinyurl.com') {

	// create the request url based on the selected shortening service
	switch ($service) {
	    case 'tinyurl.com':
	       $service_url = 'http://tinyurl.com/api-create.php?url='.urlencode($url);
		   break;
	    case 'is.gd':
	        $service_url = 'http://is.gd/api.php?longurl='.urlencode($url);
	}

	/*
	 * use cURL to fetch the respons
	 * Feel free to swap this out for
	 * $output = file_get_contents($service_url)
	 * if you have fopen wrappers enabled
	 */
	$ch = curl_init($service_url);
	curl_setopt($ch, CURLOPT_HEADER, 0);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
	$output = curl_exec($ch);
	curl_close($ch);

	// return false on error
	return strtolower(substr($output,0,5))=='error' ? false : $output;

}

The function defaults to tinyurl. Yeah. Interesting choice you’re thinking, but I’m old school and that’s how I roll. I’ve used curl to make the HTTP requests for the simple fact that when it comes to hosting curl support is enabled more ofthen than fopen wrappers is on (and you need fopen wrappers on to be able to do a file_get_contents on an url). Either way the result is the same. Use whichever method you like.

Thumbnails (The Easy Bit)

You might have noticed that more often than not, when viewing a location on Google Maps nowadays you get a little Streetview thumbnail linking to the appropriate location on Google Streetview. If you’ve ever wondered if you could get static streetview imaging out Google for your own use then read on.

Inspection of the image source reveals something quite surprising:
http://cbk0.google.com/cbk?output=thumbnail&w=90&h=68&ll=51.494966,-0.146674

http://cbk0.google.com/cbk?output=thumbnail&w=[WIDTH]&h=[HEIGHT]&ll=[LAT,LNG]

Not exactly cryptic is it. We have a width, a height and a lat/lng pair. Playing with the values of width and height you will find the maximum dimensions available are 416×208 pixels, giving you a full, albeit small, 360 degree panorama of your desired location. You can play with the coordinates too but specifying a location without a streetview panorama will result in a 404 error.

I think I’m panoid…

Each Streetview panorama is uniquely identified by an panoid. I’m pretty sure this is a timecode hash but I haven’t looked into it any further. For the purposes of fetching Streetview images the only important thing to consider is that these are unique ids.

If you’re like me, you’ve been looking at the example above and wondering what other output types there are besides ‘thumbnail’. One of the other options is ‘xml’, which returns some handy metadata for a given location. You can use this data to determine if there is a streetview panorama available at your chosen location.

http://cbk0.google.com/cbk?output=xml&ll=[LAT,LNG]

Example:
http://cbk0.google.com/cbk?output=xml&ll=51.494966,-0.146674

<panorama>
<data_properties image_width="13312" image_height="6656" tile_width="512" tile_height="512" pano_id="sLaiF6Jex7mJmNol7tdOoA" num_zoom_levels="3" lat="51.495090" lng="-0.146971" original_lat="51.495098" original_lng="-0.147000">
<copyright>© 2010 Google</copyright>
<text>Eccleston Pl</text>
<region>London, England</region>
<country>United Kingdom</country>
<data_flag>1</data_flag>
</data_properties>
<projection_properties projection_type="spherical" pano_yaw_deg="201.78" tilt_yaw_deg="92.06" tilt_pitch_deg="1.75"/>
<annotation_properties>
<link yaw_deg="204.7" pano_id="CeutxcyB2V74lfN_fJwRww" road_argb="0x80ffffff">
<link_text>Eccleston Pl</link_text>
</link>
<link yaw_deg="24.7" pano_id="t_mnKSugTLrQTEnJplXQ3A" road_argb="0x80ffffff">
<link_text>Eccleston Pl</link_text>
</link>
</annotation_properties>
</panorama>

This returns a lot of data, but for now we’ll simply concentrate on the panoid values. Generally there will be three panoids. One as an attribute of the data_properties element and two as the attributes of the two link elements of the annotation_properties element. As far as I can tell the supplemental two panoids relate to the previous and next panorama on the route.

So in this example we end up with a main panoid of “sLaiF6Jex7mJmNol7tdOoA”

Finally we can plug this value into the thumbnail url as follows:

http://cbk0.google.com/cbk?output=thumbnail&w=[WIDTH]&h=[HEIGHT]&panoid=[PID]

http://cbk0.google.com/cbk?output=thumbnail&w=416&h=208&panoid=sLaiF6Jex7mJmNol7tdOoA

Notice how we end up with exactly the same panorama, but at a different offset.

With a little extra work, tying this into the Open Postcode Geocoding API, I came up with a nice little wrapper around this accepting a UK postcode and directly returning the corresponding panorama thumbnail. I’ve added this to my geo webservice and it is callable as follows:

http://geo.jamiethompson.co.uk/streetview/[POSTCODE]_[WIDTH]x[HEIGHT].jpg

Example
http://geo.jamiethompson.co.uk/streetview/SW1W9TQ_416×208.jpg

The high detail tiles

So, you probably noticed some info relating to tiles in the xml data in the previous example. It’s possible to fetch a set of 512 pixel square tiles at various zoom levels for a given panorama using the following request.

http://cbk0.google.com/cbk?output=tile&panoid=[PID]&zoom=[ZOOM]&x=[X]&y=[Y]

[X] and [Y] relate to the horizontal and vertical tile positions, while [ZOOM] relates to the zoom level. If the XML data is to be believed there are 3 zoom levels, but I have found there to be 5. The number of X and Y positions increases with each zoom level. These are zero based.

Zoom Level 0:

Single Tile – X=0, Y=0

Zoom Level 1:

Tile 1 – X=0, Y=0 Tile 2 – X=1, Y=0

Zoom Level 2

4 X positions (0-3), 2 Y positions (0-1)

Zoom Level 3

6 X positions (0-5), 3 Y positions (0-2)

Zoom Level 4

13 X positions (0-12), 7 Y positions (0-6)

Zoom Level 5

26 X positions (0-25), 13 Y positions (0-12)

Demo

I’ve put together a very quick and dirty demo page that displays the tiles re-assembled for any given UK postcode. Be warned, at zoom level 5 this will download 338 512×512 pixel tiles, roughly 7-8MB.

View the demo here

Because I needed one for various (non-profit) projects of my own, I’ve put together a small Postcode geocoding web service which collates partial data from various sources into what aims to be a full geocoder / reverse geocoder for UK postcodes.

Now, the term ‘geocoding’ can mean several very subtly different things depending on who you talk to. So to clarify, this is what the web service does.

1. It takes a full UK postcode
2. It returns a geographical location (lat/lng and optionally os/landranger grid references)
3. It also attempts to reverse geocode the street address via the Google Maps API (experimental)

Isn’t that Illegal?

Not anymore. Ordnance Survey have open sourced huge chunks of their data. This includes their CodePoint file which contains the coordinates of every single UK postcode.

Usage

There’s nothing complicated involved in calling the service. Simply make an HTTP request in the following format, where SW1A 2AA is your postcode; or more accurately that of 10 Downing Street.

Get the data in XML format
http://geo.jamiethompson.co.uk/SW1A2AA.xml

<result>
  <status>200</status>
  <postcode>SW1A 2AA</postcode>
  <geo>
    <os_x>530047</os_x>
    <os_y>179951</os_y>
    <lat>51.50354</lat>
    <lng>-0.127696</lng>
    <landranger>TQ300799</landranger>
    <accuracy>1</accuracy>
  </geo>
  <address>
    <street>Downing St</street>
    <locality>London</locality>
    <district>Westminster</district>
    <county>Greater London</county>
  </address>
</result>

The data is also available as JSON
http://geo.jamiethompson.co.uk/SW1A2AA.json

{
   "status":200,
   "postcode":"SW1A 2AA",
   "geo":{
      "os_x":"530047",
      "os_y":"179951",
      "lat":"51.50354",
      "lng":"-0.127696",
      "landranger":"TQ300799",
      "accuracy":"1"
   },
   "address":{
      "street":"Downing St",
      "locality":"London",
      "district":"Westminster",
      "county":"Greater London"
   }
}

and additionally as JSONP by specifying a callback
http://geo.jamiethompson.co.uk/SW1A2AA.json?callback=doStuff

Things You Should be Aware of

The data returned should be pretty self documenting but there’s a few things you should be aware of.

Although, more often than not you will receive a full dataset. The only guaranteed values are lat and lng. All other values of geo are optional. Additionally to this, if the accuracy level of a result is 2 then the coordinates relate to the broader postcode area rather than the postcode itself. An accuracy 2 result for the postcode JE3 8FA* will return the centre point of the JE3 8 area.

* All channel island postcodes are returned at accuracy level 2

Northern Ireland (BTXX XXX) postcodes don’t return ordnance survey osgrid coords. Lat/lng values however are accurate to level 1.

A final word

This is an experimental web service. Please don’t hammer it for anything humongous or commercial. The reverse geocoding is experimental and cannot be relied upon for anything super-serious at the moment. If you need this data commercially please do yourself (and me) a favour and check out the reasonably priced services offered by the likes of AFD (which I can personally recommend) or PostcodeAnywhere

<meta name="user_id" content="1337" />
<meta name="user_name" content="Gary Barlow" />

I’m thinking core stuff like user ids which never change and get used again and again in ajax calls. Yes, you could achieve the same thing with cookie data, but if like me you like to keep the amount of data stored in cookies to the bare minimum, then this can be a nice little solution.

You can easily access these values like this:

var user_id = $("meta[name='user_id']").attr('content')

But eww, it’s pretty ugly, and ugly code makes Jamie a sad developer.

The following plugin simply provides a neat shorthand for accessing those variables. And I do like to keep things neat. It also caches by means of lazy-loading each meta contents into an internal object, so that the DOM is only traversed once. This is possibly overkill, and probably doesn’t save a single millisecond in any real-life scenario, but I like to optimise as I go and you can’t stop me.

(function($) {
$.meta = function(name) {
    if (typeof($.meta.cache[name])=='undefined'){
        $.meta.cache[name] = $("meta[name='"+name+"']").attr('content');
    }
    return $.meta.cache[name];
};
$.meta.cache = new Object;
})(jQuery);

You’re then free to access your variables neatly and concisely throughout your application using your new $.meta function.

$.post('path/to/someAjaxController', {method: 'doStuff', id: $.meta('user_id')}, function(r){
    alert('Thanks for doing stuff ' + $.meta('user_name'));
})

This isn’t to be confused with the existing and widely used jQuery Metadata Plugin which extracts meta data from DOM Elements. I should perhaps call this plugin something different, but to be honest I can’t think of a better name.

So to sum up. Ground-breaking? Not really. Useful? Yes.