First of all, the result, you should be able to see a map of the UK below, with coloured dots appearing to represent trains as they arrive at stations in realtime.

This requires a browser that supports SVG and websockets, I’ve only tested it works in Firefox and Chrome.

Click for larger version

So, whats going on here?

It starts with the network rail operational data feeds one of which provides a stream of events for train movements (excellently documented by the Open Rail Data wiki). The interesting fields of data are the timestamp of the event, and the planned timestamp of the event, as well as the STANOX code. These allow me to locate the event, and to calculate the delay between when the event should have happened, and when it actually happened.

To display the events on a map the STANOX code needs to be translated into something more useful, to do that I grabbed one of the reference data files described here. I then use that to convert STANOX codes to TIPLOC codes (bear with me). Once I have the TIPLOC code I can find the Easting and Northing of stations using the NaPTAN dataset. This doesn’t cover the locations of all events as it only has public stations and doesn’t include depot/switch points that trains also report at, but it turns out it locates about 70% of the events received, which is more than enough for a visualisation.

Unfortunately Eastings and Northings are not a very useful co-ordinate system, what I really want is WGS84 latitude and longitude (GPS co-ordinates), thankfully Chris Veness of movable-type.co.uk has written conversion functions in javascript.

At this point I’m connecting to the National Rail data feed in node, looking up the location and coverting to the useful lat/long, so I have a realtime stream of train locations and how delayed they are in my console. The next step is to get this into a browser.

I don’t really have any experience with drawing maps in the browser, but my first guess that d3 will be useful proves correct, and I quickly find this extremely useful tutorial: Let’s Make a Map. I take the end result of this tutorial, slightly zoom and recentre the map (the national rail data doesn’t cover Northern Ireland, and not much goes on in the very north of Scotland). The result is a nice looking SVG UK map that I can draw points on using the d3 projection to convert the lat/long to x/y points.

The last thing to link together is to get the data from node to the browser, I wanted to use websockets given the streaming nature of the data, so I grab ws for node. I currently use lighttpd as a webserver, which can’t handle websocket proxying, so I also setup HAProxy in front of lighttpd to pass the websocket requests directly to node.

At this point it all works, I have dots appearing on my map in the browser as trains arrive at stations. To make things a little more interesting I use d3 to create a colour gradient for on time to very delayed trains, and I use a radial gradient with some transparency to make trains slightly colour an area of the map.

The result is what you (hopefully) see above, an animated map that shows locations of trains as they arrive, colour coded by how delayed the train is. The source for this is all available on GitHub.

After being thoroughly disappointed by a talk from Mozilla labs about Weave (now called Sync) at last years FOSDEM (they appeared to have made very little progress in the last 3 years, and were not even planning to offer any kind of real-time synchronisation). I had decided to attempt to implement the open window/tab synchronisation feature for Google Chrome as an extension. At the time I planned to host the server for this using node.js and allow real-time synchronisation with changes automatically affecting any machines you currently had connected. After quickly writing a solution using websockets it became obvious this was a harder problem than I anticipated – synchronisation is hard, mainly when things go wrong, i.e. how do you deal with conflicts when multiple machines have flakey connections, also the Chrome extension API makes it very difficult to distinguish the difference between closing all a windows tabs, and closing the window – so the project in that form was abandoned until I could find more time for it.

Fast-forward to the summer and I am working at IBM as a technical intern for Extreme Blue, I have a work laptop, home computer and my personal laptop, I am still using Chrome as my main browser and I would really like a way of sharing my browser sessions between my machines. I had recently created LinkPush (something I intend to blog about shortly), which made me think of using Google App Engine to host a simplified session sharing plugin for Chrome using Google Accounts as a means of secure identification. The result – TabCloud – is an extension which allows you to save an open window as a group of tabs, which can be restored on any other computer (or the same computer at a later date) via the extension interface. It also offers some other features such as naming windows, and re-arranging tabs between open windows and saved sessions by dragging and dropping the tab icons. I also released a very simple Android app which allows you to view your saved tabs as a list from your phone and open any individual link.

The project isn’t really anything like my original plans, however I find it extremely useful, hopefully at some point I will find the time to make a truely real-time session synchronisation app (unless Google beat me to it). Firefox 4 beta comes with the current Mozilla Sync (also available as an extension for older Firefoxes), which periodically saves all your open tabs (as well as other data) to their server, which allows you to view open tabs on one machine from another (or mobile device), however (despite this being more what I originally aimed for) this seems a lot less useful in practice than the simpler functionality I have implemented with TabCloud.

If you are interested the whole project is open source and available on GitHub.

After deciding (perhaps wrongly) that Raphaël would be a good library to do this in (or at least one I would like to try out). A quick weekend of implementing and reimplementing various ways of rendering scrolling around a vast 2D block world (in the end using a massive SVG element that is scrolled with blocks being added or removed as they come into and leave the visible area). The results are (not quite 2D MineCraft yet I know) a massive editable grid of “pixels” which are actually rounded rectangles of various colours.

As I wasn’t going to complete 2D MineCraft in a weekend I decided to add in websocket support using Socket.IO and make it a multi-user editable canvas that persists between sessions. Its quite fun to play with and there is a small demo below (works best in Chrome, Safari or IE9, should work slowly in Firefox), for a full page version click here, the last part of the URL defines the “channel” you are in if you wish to create a private page to share with friends.

If you are interested in the source just view the source of that page, I’ll be uploading the server and client to github shortly (once I’ve tided up some of the code a bit). If I have some time in the future I hope to take this further and make some kind of user editable platformer type game in the general theme of MineCraft. I would also like to try implementing this using the HTML5 canvas element (especially with the recent hardware rendering developments on Chrome and IE9).

I have been interested in using mongo as a logging system for Ircster for a while now (it supports large sets of data with full text search and very fast queries), however until recently there was no reasonable driver for node. There are now two drivers available node-mongodb (bindings to the C driver) and node-mongodb-native (a JS implementation of the driver). The native driver especially interests me as I can more easily understand and work with the code.

Before reading this post you should probably know what node.js is.

node_debug allows you to evaluate statements and view their result (much like a REPL) from your favourite browser (assuming your favourite browser isn’t IE). It also allows you to browse objects if the result of your statement is an object, for example the global object in node is process so executing process; will return a browsable tree structure of the global scope.

Additionally it exports a log function that allows you to asynchronously output to any active console, for example setTimeout(function () { debug.log("Hello") }, 5000); will output Hello to any open consoles after 5 seconds, allowing useful debug output when using asynchronous functions.

To get started using node_debug you need to have a server running node, and to grab the latest node_debug from github. There is an example.js included to show how you would include it in your own scripts, the idea being you can simply include this in your own project in order to easily debug issues.

Future features I would like to add include:

• A debug.scope object that allows you to import other objects into the debuggers scope (currently only the global scope is available which can cause confusion due to closures in JavaScript hiding a lot of variables from the debugger).
• debug.error/warn/info functions in line with the browser console object provided by firebug.
• Detect disconnection from node and attempt reconnection.
• Caching of asynchronous messages that can be shown when connecting to the console, in order to log things without having an active browser window all the time.

Node.js is “Evented I/O for V8 javascript” this means you can perform I/O (read files, send across the internet etc) in an extremely efficient way, and write your code in JavaScript (something I find very attractive). I am currently using this for a number of projects including Ircster (more posts on this later).