First of all, the result, you should be able to see a map of the UK below, with coloured dots appearing to represent trains as they arrive at stations in realtime.

This requires a browser that supports SVG and websockets, I’ve only tested it works in Firefox and Chrome.

Click for larger version

So, whats going on here?

It starts with the network rail operational data feeds one of which provides a stream of events for train movements (excellently documented by the Open Rail Data wiki). The interesting fields of data are the timestamp of the event, and the planned timestamp of the event, as well as the STANOX code. These allow me to locate the event, and to calculate the delay between when the event should have happened, and when it actually happened.

To display the events on a map the STANOX code needs to be translated into something more useful, to do that I grabbed one of the reference data files described here. I then use that to convert STANOX codes to TIPLOC codes (bear with me). Once I have the TIPLOC code I can find the Easting and Northing of stations using the NaPTAN dataset. This doesn’t cover the locations of all events as it only has public stations and doesn’t include depot/switch points that trains also report at, but it turns out it locates about 70% of the events received, which is more than enough for a visualisation.

Unfortunately Eastings and Northings are not a very useful co-ordinate system, what I really want is WGS84 latitude and longitude (GPS co-ordinates), thankfully Chris Veness of movable-type.co.uk has written conversion functions in javascript.

At this point I’m connecting to the National Rail data feed in node, looking up the location and coverting to the useful lat/long, so I have a realtime stream of train locations and how delayed they are in my console. The next step is to get this into a browser.

I don’t really have any experience with drawing maps in the browser, but my first guess that d3 will be useful proves correct, and I quickly find this extremely useful tutorial: Let’s Make a Map. I take the end result of this tutorial, slightly zoom and recentre the map (the national rail data doesn’t cover Northern Ireland, and not much goes on in the very north of Scotland). The result is a nice looking SVG UK map that I can draw points on using the d3 projection to convert the lat/long to x/y points.

The last thing to link together is to get the data from node to the browser, I wanted to use websockets given the streaming nature of the data, so I grab ws for node. I currently use lighttpd as a webserver, which can’t handle websocket proxying, so I also setup HAProxy in front of lighttpd to pass the websocket requests directly to node.

At this point it all works, I have dots appearing on my map in the browser as trains arrive at stations. To make things a little more interesting I use d3 to create a colour gradient for on time to very delayed trains, and I use a radial gradient with some transparency to make trains slightly colour an area of the map.

The result is what you (hopefully) see above, an animated map that shows locations of trains as they arrive, colour coded by how delayed the train is. The source for this is all available on GitHub.

Having secure passwords for every site you visit is difficult, in reality many people (including me) use the same password for more than one place. This is bad as it increases the potential problem if your password is compromised, for example, if you share your email account’s password with another website, who then (accidentally or intentionally) give away your password, your email is no longer secure. There are a number of methods of reducing these problems and PassHash uses a technique which generates new passwords based on one secure password which you never share directly with anyone. It is worth noting that PassHash is far from unique, and there are many similar tools, as well as alternatives like LastPass, however, none of the alternatives met my requirements, as I will describe below.

PassHash works by taking your master password (which should be a password which is complicated, long and only known by you), and combines it with a memorable key for each website you want to use it on (using the domain name is the default, i.e. amazon.com or ebay.co.uk). This combination is then transformed in a way that cannot be easily reversed (using SHA-1), and modified to start with a lower case letter, followed by a number, followed by an upper case letter, and finally 9 more characters which can be lowercase, uppercase, or numeric. You can then use this password for the website with the key you gave, and know that you will be able to generate it again in the future, but no one else will.

So, why is PassHash better than any of the available alternatives?

• PassHash has no settings (unlike most of the alternatives), this might seem to be a bad thing, but settings are something else to remember, and something that could be wrong by default. PassHash attempts to have sensible defaults, and generate passwords accepted on as many websites as possible. Passwords generated are 12 characters long, and always include at least one of each of lowercase, uppercase and numeric characters. Longer passwords may not be allowed on all websites, and are harder to type if you need to manually enter them. Symbols are often not allowed in passwords and are also hard to type, especially if being input on a phone or other mobile device.
• PassHash is accessible from anywhere, there is a Chrome extension for use on your own computer, as well as a website you can use anywhere. There is also an Android app which allows you to generate passwords without entering your master password on a computer you may not trust. As nothing is stored PassHash also works even when you have no internet connection.
• The PassHash algorithm is simple and the source code is available, there is no need to trust that it generates passwords in a sensible way, you can check for yourself. As well as this, if I were to withdraw the Chrome extension anyone could release software to generate the same passwords.
• PassHash doesn’t store any of your passwords – including your master password – anywhere, locally or remotely. Services such as LastPass are always at risk of attack and by using them you are trusting the LastPass developers have not made any mistakes which could result in your passwords not being properly encrypted.
• PassHash is completely free, and always will be, with no potential for me to change this, as the source code is publicly available and no part of PassHash is provided as a service.
• The PassHash chrome extension can automatically enter generated passwords into text boxes on websites, this avoids both showing the password on the screen and storing the password in the clipboard at any point.

While this all sounds great, PassHash does have a couple of downsides.

• All of your passwords being based on one master passwords requires you to keep that password completely secret, if you compromise that password, none of your passwords are safe. Unlike services such as LastPass if you compromise your master password you will need to change your password on every site you used PassHash to generate passwords for. This means you must be particularly careful when entering your master password on any computer or device you do not fully trust.
• Remembering passwords generated with PassHash is not really an option, which is not necessarily an issue as long as you have ways of accessing your passwords while not at your computer, which is why the website and Android app exist

Website: http://passhash.connorhd.co.uk/
Source code: GitHub

Having recently acquired a Kindle e-book reader I immediately investigated less obvious ways of using it than just reading books. Amazon themselves offer magazines, newspapers and blogs as well as normal books, however blogs particularly seem like something I shouldn’t need to pay for (especially considering I don’t have the 3G Kindle so won’t be making use of the included Whispernet fee). Of course there are a number of 3rd party services for Kindle out there, calibre is an e-book manager that will also download many popular news/magazines and send them directly to your Kindle. There is also kindlefeeder.com which will send aggregated RSS feeds to your Kindle. Other more generic services such as Instapaper also have some form of Kindle support (Instapaper will send unread saved pages as a digest to your Kindle on a regular basis).

Kindlizer is slightly different to all of these services as it attempts to create an interface to be used directly on the Kindle, rather than sending static files to the Kindle to be read later. Amazon have announced a development kit for the Kindle called the KDK to allow development of applications which run on the Kindle, however it is currently in a closed beta and appears to be quite restrictive in terms of free data usage. This means Kindlizer is restricted to the experimental web browser built into the Kindle.

The Kindle web browser is Webkit based which means it has excellent support for web standards, unfortunately in order for the browser to display all kinds of websites it has a less intuitive interface. The directional pad on the Kindle controls a mouse pointer which jumps around in a grid on the screen and snaps to clickable elements (why spacial navigation wasn’t used I have no idea), as well as this the page back and forward keys on the Kindle become page up/page down in the browser, meaning on each press the page is only scrolled a small portion. On top of this many pages are far too large to fit on the Kindle screen and are first shown zoomed out, requiring the user to first select a region to zoom in on.

Kindlizer attempts to fix some of these problems with the use of some JavaScript to trick the browser into showing the content one page at a time, with the page turn buttons on the Kindle moving forward or backward entire pages at once. On top of this CSS styling makes the documents appear with the same formatting as Kindle books, and large areas around clickable elements allows for far easier movement of the mouse cursor. The JavaScript is written to allow for arbitrary blocks of HTML to be split into pages and displayed on the Kindle, currently this is used in conjunction with the Google Feed API to allow users to view RSS feeds on the Kindle. The results so far are shown in the screenshots below:

Kindlizer homepage as viewed on the Kindle

Viewing this blog's RSS feed on Kindlizer

You can visit the (as of writing unfinished) Kindlizer site on Google App Engine here, alternatively there is a demo which shows this blog’s RSS feed and does not require login here. Neither of which work particularly well on any device other than a Kindle (they may kind of work on your browser, but you won’t be able to change page). The full source code for the App Engine site is open source and available here.

After being thoroughly disappointed by a talk from Mozilla labs about Weave (now called Sync) at last years FOSDEM (they appeared to have made very little progress in the last 3 years, and were not even planning to offer any kind of real-time synchronisation). I had decided to attempt to implement the open window/tab synchronisation feature for Google Chrome as an extension. At the time I planned to host the server for this using node.js and allow real-time synchronisation with changes automatically affecting any machines you currently had connected. After quickly writing a solution using websockets it became obvious this was a harder problem than I anticipated – synchronisation is hard, mainly when things go wrong, i.e. how do you deal with conflicts when multiple machines have flakey connections, also the Chrome extension API makes it very difficult to distinguish the difference between closing all a windows tabs, and closing the window – so the project in that form was abandoned until I could find more time for it.

Fast-forward to the summer and I am working at IBM as a technical intern for Extreme Blue, I have a work laptop, home computer and my personal laptop, I am still using Chrome as my main browser and I would really like a way of sharing my browser sessions between my machines. I had recently created LinkPush (something I intend to blog about shortly), which made me think of using Google App Engine to host a simplified session sharing plugin for Chrome using Google Accounts as a means of secure identification. The result – TabCloud – is an extension which allows you to save an open window as a group of tabs, which can be restored on any other computer (or the same computer at a later date) via the extension interface. It also offers some other features such as naming windows, and re-arranging tabs between open windows and saved sessions by dragging and dropping the tab icons. I also released a very simple Android app which allows you to view your saved tabs as a list from your phone and open any individual link.

The project isn’t really anything like my original plans, however I find it extremely useful, hopefully at some point I will find the time to make a truely real-time session synchronisation app (unless Google beat me to it). Firefox 4 beta comes with the current Mozilla Sync (also available as an extension for older Firefoxes), which periodically saves all your open tabs (as well as other data) to their server, which allows you to view open tabs on one machine from another (or mobile device), however (despite this being more what I originally aimed for) this seems a lot less useful in practice than the simpler functionality I have implemented with TabCloud.

If you are interested the whole project is open source and available on GitHub.

After deciding (perhaps wrongly) that Raphaël would be a good library to do this in (or at least one I would like to try out). A quick weekend of implementing and reimplementing various ways of rendering scrolling around a vast 2D block world (in the end using a massive SVG element that is scrolled with blocks being added or removed as they come into and leave the visible area). The results are (not quite 2D MineCraft yet I know) a massive editable grid of “pixels” which are actually rounded rectangles of various colours.

As I wasn’t going to complete 2D MineCraft in a weekend I decided to add in websocket support using Socket.IO and make it a multi-user editable canvas that persists between sessions. Its quite fun to play with and there is a small demo below (works best in Chrome, Safari or IE9, should work slowly in Firefox), for a full page version click here, the last part of the URL defines the “channel” you are in if you wish to create a private page to share with friends.

If you are interested in the source just view the source of that page, I’ll be uploading the server and client to github shortly (once I’ve tided up some of the code a bit). If I have some time in the future I hope to take this further and make some kind of user editable platformer type game in the general theme of MineCraft. I would also like to try implementing this using the HTML5 canvas element (especially with the recent hardware rendering developments on Chrome and IE9).

In the 2nd year of my computer science degree a module called “Artificial Intelligence” briefly touched on the idea of Genetic Algorithms. After discussion with a housemate as to how these could be used in a 3rd year project I decided to look into making a simple example of such an algorithm to run using JavaScript in the browser.

After a quick Google the best example I could find (that would be visual enough to be interesting to see and simple enough to write in JavaScript quickly) was to generate the string “Hello, World!”. Starting with randomly generated strings and genetically evolving them based on a fitness function of how close they were to the string I wanted. A little bit more detail on how that is achieved shortly, but first the result is shown below (click on “Hello, World!” to replay the demo) a more detailed output can be seen here.

a

So, what is this actually doing? Viewing the detailed output you can see all of the strings generated, how they were combined (the algorithmic equivalent of reproduction) and the mutatations after they reproduced. To start 20 random strings are generated, then the top 2 strings (measured by number of letters that are correct, and then closeness of incorrect letters) are taken. It is worth noting this is perhaps not the best way for a genetic algorithm to work, and strictly all of the strings should have a chance of “reproducing” based on their fitness score, but this works well enough for a simple example. Once the top 2 are taken, they are split at a random point 20 times and combined (shown by the red and green colouring on the detailed output), these resulting strings then have a 75% chance of one of their characters randomly changing (shown with a yellow highlight on the detailed output) and we have 20 (potentially) completely new strings. The scoring process then repeats and more strings are created until we reach the goal string.

The result is we very quickly get a string very close to what we want, which eventually mutates to become exactly what we want, with a nice visual demonstration of what is happening, and hopefully a better understanding of how genetic algorithms work.

To demonstrate here are some screenshots of a sample page using a sample application (if you have an Android phone you can use that link to download it and try it yourself):

The application when you load it:

When clicking go an alert is created:

Viewing the created alert:

The example page is very simple javascript:

function go() {
	window.location = 'android://'+document.getElementById('title').value
		+'/'+document.getElementById('text').value;
}

The source for the app follows and is also fairly simple, most of the code is the rather complex way alerts are generated on Android:

package uk.co.connorhd.android.webapp;
 
import java.net.URLDecoder;
 
import uk.co.connorhd.android.webapp.R;
import uk.co.connorhd.android.webapp.WebApp;
import android.app.Activity;
import android.app.Notification;
import android.app.NotificationManager;
import android.app.PendingIntent;
import android.content.Context;
import android.content.Intent;
import android.os.Bundle;
import android.webkit.WebView;
import android.webkit.WebViewClient;
import android.widget.Toast;
 
public class WebApp extends Activity {
  /** Called when the activity is first created. */
  @Override
  public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    WebView webview = new WebView(this);
 
    // We're testing, clear the cache.
    webview.clearCache(true);
 
    setContentView(webview);
 
    webview.getSettings().setJavaScriptEnabled(true);
 
    final Activity activity = this;
    webview.setWebViewClient(new WebViewClient() {
      int alert = 1;
 
      public void onReceivedError(WebView view, int errorCode,
          String description, String failingUrl) {
        Toast.makeText(activity, "Oh no! " + description,
            Toast.LENGTH_SHORT).show();
      }
 
      public boolean shouldOverrideUrlLoading(WebView view, String url) {
        // Is it a hack?
        if (url.startsWith("android")) {
          String ns = Context.NOTIFICATION_SERVICE;
          NotificationManager mNotificationManager = (NotificationManager) getSystemService(ns);
          int icon = R.drawable.icon;
          CharSequence tickerText = "WebApp Alert!";
          long when = System.currentTimeMillis();
 
          Notification notification = new Notification(icon,
              tickerText, when);
          notification.flags = Notification.FLAG_AUTO_CANCEL;
          Context context = getApplicationContext();
 
          String[] split = url.split("/");
 
          CharSequence contentTitle = URLDecoder.decode(split[2]);
          CharSequence contentText = URLDecoder.decode(split[3]);
 
          Intent notificationIntent = new Intent(activity,
              WebApp.class);
          PendingIntent contentIntent = PendingIntent.getActivity(
              activity, 0, notificationIntent, 0);
 
          notification.setLatestEventInfo(context, contentTitle,
              contentText, contentIntent);
 
          mNotificationManager.notify(alert++, notification);
        } else {
          view.loadUrl(url);
        }
        return true;
      }
    });
 
    webview.loadUrl("http://connorhd.co.uk/files/WebApp.htm");
  }
}

This could easily be extended (and I may try to do this myself for Ircster) to provide all the standard browser functionality (for example the back button doesn’t currently work) and a more complete API to interface with Android.

Looking at the iPhone first, it does allow websites to be directly added as “applications” (applications meaning buttons on the iPhones menu). It also allows some limited customisation of the way the website is displayed when it is accessed as an application using html meta tags (i.e. <meta content="yes" name="apple-mobile-web-app-capable" />). This is a good start, however, the iPhone seems to be completely let down here by not allowing background applications, meaning there is no way to run a website constantly and alert the user on specific events, or receive sensor information (i.e. GPS) while the phone is being used for another purpose.

Android on the other hand supports background applications, however it does not allow (as far as I can tell) for websites to be added as applications, and does not have a way for websites to interact with the phones features. What I would like to see is a JavaScript API allowing phone features such as alerts to be triggered by the website (possibly similar to the message passing feature in Chrome extensions), and also for the phone to reply to the app with sensor information such as GPS location. This seems to fit in very well with the ideas behind Chrome OS and applications being simply websites. (While considering this it occurred to me some of this functionality could be hacked into Android using its current Java app platform, something I have started working on and will demonstrate in another blog post.)

Desktop applications definitely seem to be being replaced by websites, I don’t see why the opposite should be true on phones (websites should not need to write specific phone apps). I’d like to see APIs included with smartphone browsers (Google Gears is a potential start, although this appears to be deprecated on Android) that provide more direct interaction with the phones features (alerts, access to sensors, local storage and local serving of content) and allow complete applications to be developed as websites for multiple platforms.

I have been interested in using mongo as a logging system for Ircster for a while now (it supports large sets of data with full text search and very fast queries), however until recently there was no reasonable driver for node. There are now two drivers available node-mongodb (bindings to the C driver) and node-mongodb-native (a JS implementation of the driver). The native driver especially interests me as I can more easily understand and work with the code.