First of all, the result, you should be able to see a map of the UK below, with coloured dots appearing to represent trains as they arrive at stations in realtime.

This requires a browser that supports SVG and websockets, I’ve only tested it works in Firefox and Chrome.

Click for larger version

So, whats going on here?

It starts with the network rail operational data feeds one of which provides a stream of events for train movements (excellently documented by the Open Rail Data wiki). The interesting fields of data are the timestamp of the event, and the planned timestamp of the event, as well as the STANOX code. These allow me to locate the event, and to calculate the delay between when the event should have happened, and when it actually happened.

To display the events on a map the STANOX code needs to be translated into something more useful, to do that I grabbed one of the reference data files described here. I then use that to convert STANOX codes to TIPLOC codes (bear with me). Once I have the TIPLOC code I can find the Easting and Northing of stations using the NaPTAN dataset. This doesn’t cover the locations of all events as it only has public stations and doesn’t include depot/switch points that trains also report at, but it turns out it locates about 70% of the events received, which is more than enough for a visualisation.

Unfortunately Eastings and Northings are not a very useful co-ordinate system, what I really want is WGS84 latitude and longitude (GPS co-ordinates), thankfully Chris Veness of movable-type.co.uk has written conversion functions in javascript.

At this point I’m connecting to the National Rail data feed in node, looking up the location and coverting to the useful lat/long, so I have a realtime stream of train locations and how delayed they are in my console. The next step is to get this into a browser.

I don’t really have any experience with drawing maps in the browser, but my first guess that d3 will be useful proves correct, and I quickly find this extremely useful tutorial: Let’s Make a Map. I take the end result of this tutorial, slightly zoom and recentre the map (the national rail data doesn’t cover Northern Ireland, and not much goes on in the very north of Scotland). The result is a nice looking SVG UK map that I can draw points on using the d3 projection to convert the lat/long to x/y points.

The last thing to link together is to get the data from node to the browser, I wanted to use websockets given the streaming nature of the data, so I grab ws for node. I currently use lighttpd as a webserver, which can’t handle websocket proxying, so I also setup HAProxy in front of lighttpd to pass the websocket requests directly to node.

At this point it all works, I have dots appearing on my map in the browser as trains arrive at stations. To make things a little more interesting I use d3 to create a colour gradient for on time to very delayed trains, and I use a radial gradient with some transparency to make trains slightly colour an area of the map.

The result is what you (hopefully) see above, an animated map that shows locations of trains as they arrive, colour coded by how delayed the train is. The source for this is all available on GitHub.

Having secure passwords for every site you visit is difficult, in reality many people (including me) use the same password for more than one place. This is bad as it increases the potential problem if your password is compromised, for example, if you share your email account’s password with another website, who then (accidentally or intentionally) give away your password, your email is no longer secure. There are a number of methods of reducing these problems and PassHash uses a technique which generates new passwords based on one secure password which you never share directly with anyone. It is worth noting that PassHash is far from unique, and there are many similar tools, as well as alternatives like LastPass, however, none of the alternatives met my requirements, as I will describe below.

PassHash works by taking your master password (which should be a password which is complicated, long and only known by you), and combines it with a memorable key for each website you want to use it on (using the domain name is the default, i.e. amazon.com or ebay.co.uk). This combination is then transformed in a way that cannot be easily reversed (using SHA-1), and modified to start with a lower case letter, followed by a number, followed by an upper case letter, and finally 9 more characters which can be lowercase, uppercase, or numeric. You can then use this password for the website with the key you gave, and know that you will be able to generate it again in the future, but no one else will.

So, why is PassHash better than any of the available alternatives?

• PassHash has no settings (unlike most of the alternatives), this might seem to be a bad thing, but settings are something else to remember, and something that could be wrong by default. PassHash attempts to have sensible defaults, and generate passwords accepted on as many websites as possible. Passwords generated are 12 characters long, and always include at least one of each of lowercase, uppercase and numeric characters. Longer passwords may not be allowed on all websites, and are harder to type if you need to manually enter them. Symbols are often not allowed in passwords and are also hard to type, especially if being input on a phone or other mobile device.
• PassHash is accessible from anywhere, there is a Chrome extension for use on your own computer, as well as a website you can use anywhere. There is also an Android app which allows you to generate passwords without entering your master password on a computer you may not trust. As nothing is stored PassHash also works even when you have no internet connection.
• The PassHash algorithm is simple and the source code is available, there is no need to trust that it generates passwords in a sensible way, you can check for yourself. As well as this, if I were to withdraw the Chrome extension anyone could release software to generate the same passwords.
• PassHash doesn’t store any of your passwords – including your master password – anywhere, locally or remotely. Services such as LastPass are always at risk of attack and by using them you are trusting the LastPass developers have not made any mistakes which could result in your passwords not being properly encrypted.
• PassHash is completely free, and always will be, with no potential for me to change this, as the source code is publicly available and no part of PassHash is provided as a service.
• The PassHash chrome extension can automatically enter generated passwords into text boxes on websites, this avoids both showing the password on the screen and storing the password in the clipboard at any point.

While this all sounds great, PassHash does have a couple of downsides.

• All of your passwords being based on one master passwords requires you to keep that password completely secret, if you compromise that password, none of your passwords are safe. Unlike services such as LastPass if you compromise your master password you will need to change your password on every site you used PassHash to generate passwords for. This means you must be particularly careful when entering your master password on any computer or device you do not fully trust.
• Remembering passwords generated with PassHash is not really an option, which is not necessarily an issue as long as you have ways of accessing your passwords while not at your computer, which is why the website and Android app exist

Website: http://passhash.connorhd.co.uk/
Source code: GitHub

Having recently acquired a Kindle e-book reader I immediately investigated less obvious ways of using it than just reading books. Amazon themselves offer magazines, newspapers and blogs as well as normal books, however blogs particularly seem like something I shouldn’t need to pay for (especially considering I don’t have the 3G Kindle so won’t be making use of the included Whispernet fee). Of course there are a number of 3rd party services for Kindle out there, calibre is an e-book manager that will also download many popular news/magazines and send them directly to your Kindle. There is also kindlefeeder.com which will send aggregated RSS feeds to your Kindle. Other more generic services such as Instapaper also have some form of Kindle support (Instapaper will send unread saved pages as a digest to your Kindle on a regular basis).

Kindlizer is slightly different to all of these services as it attempts to create an interface to be used directly on the Kindle, rather than sending static files to the Kindle to be read later. Amazon have announced a development kit for the Kindle called the KDK to allow development of applications which run on the Kindle, however it is currently in a closed beta and appears to be quite restrictive in terms of free data usage. This means Kindlizer is restricted to the experimental web browser built into the Kindle.

The Kindle web browser is Webkit based which means it has excellent support for web standards, unfortunately in order for the browser to display all kinds of websites it has a less intuitive interface. The directional pad on the Kindle controls a mouse pointer which jumps around in a grid on the screen and snaps to clickable elements (why spacial navigation wasn’t used I have no idea), as well as this the page back and forward keys on the Kindle become page up/page down in the browser, meaning on each press the page is only scrolled a small portion. On top of this many pages are far too large to fit on the Kindle screen and are first shown zoomed out, requiring the user to first select a region to zoom in on.

Kindlizer attempts to fix some of these problems with the use of some JavaScript to trick the browser into showing the content one page at a time, with the page turn buttons on the Kindle moving forward or backward entire pages at once. On top of this CSS styling makes the documents appear with the same formatting as Kindle books, and large areas around clickable elements allows for far easier movement of the mouse cursor. The JavaScript is written to allow for arbitrary blocks of HTML to be split into pages and displayed on the Kindle, currently this is used in conjunction with the Google Feed API to allow users to view RSS feeds on the Kindle. The results so far are shown in the screenshots below:

Kindlizer homepage as viewed on the Kindle

Viewing this blog's RSS feed on Kindlizer

You can visit the (as of writing unfinished) Kindlizer site on Google App Engine here, alternatively there is a demo which shows this blog’s RSS feed and does not require login here. Neither of which work particularly well on any device other than a Kindle (they may kind of work on your browser, but you won’t be able to change page). The full source code for the App Engine site is open source and available here.

After being thoroughly disappointed by a talk from Mozilla labs about Weave (now called Sync) at last years FOSDEM (they appeared to have made very little progress in the last 3 years, and were not even planning to offer any kind of real-time synchronisation). I had decided to attempt to implement the open window/tab synchronisation feature for Google Chrome as an extension. At the time I planned to host the server for this using node.js and allow real-time synchronisation with changes automatically affecting any machines you currently had connected. After quickly writing a solution using websockets it became obvious this was a harder problem than I anticipated – synchronisation is hard, mainly when things go wrong, i.e. how do you deal with conflicts when multiple machines have flakey connections, also the Chrome extension API makes it very difficult to distinguish the difference between closing all a windows tabs, and closing the window – so the project in that form was abandoned until I could find more time for it.

Fast-forward to the summer and I am working at IBM as a technical intern for Extreme Blue, I have a work laptop, home computer and my personal laptop, I am still using Chrome as my main browser and I would really like a way of sharing my browser sessions between my machines. I had recently created LinkPush (something I intend to blog about shortly), which made me think of using Google App Engine to host a simplified session sharing plugin for Chrome using Google Accounts as a means of secure identification. The result – TabCloud – is an extension which allows you to save an open window as a group of tabs, which can be restored on any other computer (or the same computer at a later date) via the extension interface. It also offers some other features such as naming windows, and re-arranging tabs between open windows and saved sessions by dragging and dropping the tab icons. I also released a very simple Android app which allows you to view your saved tabs as a list from your phone and open any individual link.

The project isn’t really anything like my original plans, however I find it extremely useful, hopefully at some point I will find the time to make a truely real-time session synchronisation app (unless Google beat me to it). Firefox 4 beta comes with the current Mozilla Sync (also available as an extension for older Firefoxes), which periodically saves all your open tabs (as well as other data) to their server, which allows you to view open tabs on one machine from another (or mobile device), however (despite this being more what I originally aimed for) this seems a lot less useful in practice than the simpler functionality I have implemented with TabCloud.

If you are interested the whole project is open source and available on GitHub.

In the 2nd year of my computer science degree a module called “Artificial Intelligence” briefly touched on the idea of Genetic Algorithms. After discussion with a housemate as to how these could be used in a 3rd year project I decided to look into making a simple example of such an algorithm to run using JavaScript in the browser.

After a quick Google the best example I could find (that would be visual enough to be interesting to see and simple enough to write in JavaScript quickly) was to generate the string “Hello, World!”. Starting with randomly generated strings and genetically evolving them based on a fitness function of how close they were to the string I wanted. A little bit more detail on how that is achieved shortly, but first the result is shown below (click on “Hello, World!” to replay the demo) a more detailed output can be seen here.

a

So, what is this actually doing? Viewing the detailed output you can see all of the strings generated, how they were combined (the algorithmic equivalent of reproduction) and the mutatations after they reproduced. To start 20 random strings are generated, then the top 2 strings (measured by number of letters that are correct, and then closeness of incorrect letters) are taken. It is worth noting this is perhaps not the best way for a genetic algorithm to work, and strictly all of the strings should have a chance of “reproducing” based on their fitness score, but this works well enough for a simple example. Once the top 2 are taken, they are split at a random point 20 times and combined (shown by the red and green colouring on the detailed output), these resulting strings then have a 75% chance of one of their characters randomly changing (shown with a yellow highlight on the detailed output) and we have 20 (potentially) completely new strings. The scoring process then repeats and more strings are created until we reach the goal string.

The result is we very quickly get a string very close to what we want, which eventually mutates to become exactly what we want, with a nice visual demonstration of what is happening, and hopefully a better understanding of how genetic algorithms work.

So straight to the code, after (briefly) reading the Android documentation it seemed there was very little info about using the sensors, so I turned to Google, this article looked like what I wanted and linked to this code which is the basis of my app. So I updated the code to use the non-deprecated API, changing its functionality slightly and ended up with:

package uk.co.connorhd.android.tapemeasure;
 
import android.app.Activity;
import android.os.Bundle;
import android.hardware.Sensor;
import android.hardware.SensorEvent;
import android.hardware.SensorEventListener;
import android.hardware.SensorManager;
import android.view.View;
import android.view.View.OnClickListener;
import android.widget.Button;
import android.widget.TextView;
 
public class TapeMeasure extends Activity implements SensorEventListener,
		OnClickListener {
	private SensorManager sensorMgr;
	private Sensor sensorAccel;
	private TextView accuracyLabel;
	private TextView xLabel, yLabel, zLabel;
	private Button calibrateButton;
 
	private float moved = 0;
	private float speed = 0;
	private float accel = 0;
	private float accelDiff = 0;
 
	private float[] a;
 
	private long lastUpdate = 0;
	private float curAccel = 0;
	private int accelUpdates = 0;
 
	@Override
	public void onCreate(Bundle savedInstanceState) {
		super.onCreate(savedInstanceState);
		setContentView(R.layout.main);
		accuracyLabel = (TextView) findViewById(R.id.accuracy_label);
		xLabel = (TextView) findViewById(R.id.x_label);
		yLabel = (TextView) findViewById(R.id.y_label);
		zLabel = (TextView) findViewById(R.id.z_label);
		calibrateButton = (Button) findViewById(R.id.calibrate_button);
		calibrateButton.setOnClickListener(this);
	}
 
	@Override
	protected void onPause() {
		super.onPause();
		sensorMgr.unregisterListener(this, sensorAccel);
		sensorMgr = null;
	}
 
	@Override
	protected void onResume() {
		super.onResume();
 
		sensorMgr = (SensorManager) getSystemService(SENSOR_SERVICE);
		sensorAccel = sensorMgr.getSensorList(Sensor.TYPE_ACCELEROMETER).get(0);
		boolean accelSupported = sensorMgr.registerListener(this, sensorAccel,
				SensorManager.SENSOR_DELAY_FASTEST);
 
		if (!accelSupported) {
			// on accelerometer on this device
			sensorMgr.unregisterListener(this, sensorAccel);
			accuracyLabel.setText(R.string.no_accelerometer);
		}
	}
 
	public void onClick(View v) {
		if (v == calibrateButton) {
			// Clicked button
			moved = 0;
			speed = 0;
			accelDiff = accel+accelDiff;
			accel = 0;
 
		}
	}
 
	public void onAccuracyChanged(Sensor sensor, int accuracy) {
		// this method is called very rarely, so we don't have to
		// limit our updates as we do in onSensorChanged(...)
		if (sensor.getType() == Sensor.TYPE_ACCELEROMETER) {
			switch (accuracy) {
			case SensorManager.SENSOR_STATUS_UNRELIABLE:
				accuracyLabel.setText(R.string.accuracy_unreliable);
				break;
			case SensorManager.SENSOR_STATUS_ACCURACY_LOW:
				accuracyLabel.setText(R.string.accuracy_low);
				break;
			case SensorManager.SENSOR_STATUS_ACCURACY_MEDIUM:
				accuracyLabel.setText(R.string.accuracy_medium);
				break;
			case SensorManager.SENSOR_STATUS_ACCURACY_HIGH:
				accuracyLabel.setText(R.string.accuracy_high);
				break;
			}
		}
	}
 
	public void onSensorChanged(SensorEvent event) {
		int sensorType = event.sensor.getType();
		if (sensorType == Sensor.TYPE_ACCELEROMETER) {
			a = event.values;
			accel = (float) (Math.sqrt((a[0]*a[0]+a[1]*a[1]+a[2]*a[2]))-SensorManager.GRAVITY_EARTH)-accelDiff;
			curAccel += accel;
			accelUpdates++;
 
			long curTime = event.timestamp - lastUpdate;
			if (curTime > 1000000000) {
				curAccel = curAccel/accelUpdates;
				speed += curAccel*curTime*10e-10;
				moved += speed*curTime*10e-10;
 
				lastUpdate = event.timestamp;
 
				xLabel.setText(String.format("Moved: %+2.20f", moved));
				yLabel.setText(String.format("Speed: %+2.20f", speed));
				zLabel.setText(String.format("Accel: %+2.20f Time:  %+2.3f", curAccel, curTime*10e-10));
				curAccel = 0;
				accelUpdates = 0;
			}
		}
	}
}

This calculates the current acceleration of the phone, subtracts gravity (which is added to the raw data), then derives the speed and distance travelled. The calibrate button resets the speed and distance, and assumes the phone is not moving, generating a bias to add to the acceleration in the case of the phones accelerometer being badly calibrated. If you want to try the app it can be downloaded and installed from here. I suggest launching the app, placing the phone on a stable surface and pressing calibrate, if you get any interesting results let me know.

Despite the phone reporting the accelerometer accuracy as “high” (with no description of what high means) and returning values in excess of 10 significant figures,  the acceleration data with the phone sat on a table varied in the range of about 0.5m/s^2 (although this did appear to change depending on the room I was in). This resulted in a massively wrong distance value very quickly increasing to several metres without moving the phone, making any kind of testing almost impossible. It seems the current purpose of accelerometers is to work out if some violent action (such as shaking) has happened to the phone, and not anything much more interesting. All in all a bit of a waste of time, but maybe future phones will have better sensors, and hopefully someone will find the code snippet above useful.

Before reading this post you should probably know what node.js is.

node_debug allows you to evaluate statements and view their result (much like a REPL) from your favourite browser (assuming your favourite browser isn’t IE). It also allows you to browse objects if the result of your statement is an object, for example the global object in node is process so executing process; will return a browsable tree structure of the global scope.

Additionally it exports a log function that allows you to asynchronously output to any active console, for example setTimeout(function () { debug.log("Hello") }, 5000); will output Hello to any open consoles after 5 seconds, allowing useful debug output when using asynchronous functions.

To get started using node_debug you need to have a server running node, and to grab the latest node_debug from github. There is an example.js included to show how you would include it in your own scripts, the idea being you can simply include this in your own project in order to easily debug issues.

Future features I would like to add include:

• A debug.scope object that allows you to import other objects into the debuggers scope (currently only the global scope is available which can cause confusion due to closures in JavaScript hiding a lot of variables from the debugger).
• debug.error/warn/info functions in line with the browser console object provided by firebug.
• Detect disconnection from node and attempt reconnection.
• Caching of asynchronous messages that can be shown when connecting to the console, in order to log things without having an active browser window all the time.