Olly Smith

Test remote javascript APIs with Capybara and puffing-billy

Testing remote APIs is easy in Ruby. Libraries like webmock, vcr and artifice give you all the tools you need to ensure that you’re sending the right requests to the right remote endpoints.

However, it becomes a lot more difficult when it’s your Javascript code that’s making the requests. Using requests specs with Capybara gives you loads of tools to control the browser, but it doesn’t let you stub or mock responses to requests that originate from within the browser.

This is where puffing-billy saves the day. It’s a request stubbing library for browsers. It spawns a HTTP(S) proxy server that it uses to intercept requests from your browser. Using simple webmock-alike syntax, you can configure the proxy to send fake responses to requests for specific URLs.

For example, the following is a simple piece of javascript code that fetches the temperature for Bath, UK from the openweathermap.org service.

<!-- /weather/ -->
<p>Current temperature for Bath, UK: <span id='temp'></span></p>
<script>
  $(function () {
    $.ajax({
      url: 'http://openweathermap.org/data/weather/2656173',
      dataType: 'jsonp',
      success: function (data) { $('#temp').text(data.temp + '°C') },
      error: function () { $('#temp').text('unavailable') }
    });
  });
</script>

And this is a request spec for that snippet. Note how it easily fakes JSONP data and error responses.

# spec/requests/weather_spec.rb
describe 'Weather for Bath, UK', :js => true do

  it 'should fetch the temperature from openweathermap.org' do
    # fake some JSONP data
    proxy.stub('http://openweathermap.org/data/weather/2656173')
      .and_return(:jsonp => {:temp => 12.7})

    visit '/weather/'
    page.should have_content('Current temperature for Bath, UK: 12.7°C')
  end

  it "should fail gracefully when openweathermap.org isn't available" do
    # fake a failure
    proxy.stub('http://openweathermap.org/data/weather/2656173')
      .and_return(:code => 500)

    visit '/weather/'
    page.should have_content('Current temperature for Bath, UK: unavailable')
  end

end

puffing-billy supports HTTP and HTTPS requests and all common HTTP verbs. Go check it out on GitHub now!


Install Ruby Enterprise Edition 1.8.7 in OS X Mountain Lion

Upgrading to Mountain Lion will wipe your old dev environment, so this is what you’ll need to get back to work on ree-1.8.7:

1. Install XCode command-line tools. Available from the Preferences > Download panel in XCode, or as a separate download from the Apple Developer site.

2. Install gcc-4.2. REE doesn’t play well with Apple’s LLVM compiler, so you’ll need to install the old gcc-4.2 compiler. It’s available in the homebrew homebrew/dupes repository.

brew tap homebrew/dupes
brew install apple-gcc42

3. Install xquartz. The OS X upgrade will also remove your old X11.app installation, so go grab xquartz from macosforge and install it (you’ll need v2.7.2 or later for Mountain Lion).

4. Install ree. Remember to add the path to the xquartz X11 includes in CPPFLAGS and the path to gcc-42 in CC. Here I’m using rbenv, but the same environment variables should work for rvm.

CPPFLAGS=-I/opt/X11/include \
  CC=/usr/local/bin/gcc-4.2 \
  rbenv install ree-1.8.7-2012.02

OBDII Wi-Fi Hacking

There are plenty of these OBDII WiFi modules advertised for sale on eBay. I’ve recently bought a RenaultSport Clio which has an OBDII port, so I picked up one to experiment with.

OBDII Wi-Fi Module

This little gadget plugs into the OBDII port in the Clio (which is hidden behind a removable panel just below the ignition card key slot). It creates an ad-hoc WiFi network and listens on a TCP port for client connections.

The chip that does all the hard work of connecting to the car’s onboard computer(s) is an ELM327-compatible, which means it speaks a simple ASCII protocol. Control messages are sent to the chip using old-school AT commands (like old-school modems), and OBD comms use simple hex strings.

Here’s a quick example:

> ATZ
ELM327 v1.4
0100
41 00 9E 3E B8 11
0120
41 20 80 00 00 00
0105
41 05 48

The ATZ command resets the interface chip (and displays its ID). The rest of the commands all request data from the ECU using OBD mode 1.

0100 and 0120 enumerate the OBD PIDS supported by my car. The data returned is a header (41 00 / 41 20) followed by the actual data as a bit-encoded set of flags.

0105 queries the ECU for the current engine temperature (which, in this case, returns a temperature of 32°C).

I’m going to carry on hacking with this module, and I’m hoping to build a natty little iOS telemetry app to use with it. Stay tuned for further blog posts as I learn more!


Graceful server shutdown with node.js / express

howmanyleft.co.uk runs its node.js workers behind supervisord. To avoid dropping requests with 502s when restarting workers, I hook into the SIGTERM signal and call close() on the HTTP server. This stops the server from listening for new connections, and once all the open connections are complete, the worker exits.

Here’s a quick example:

var http = require('http');

var app = http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  setTimeout(function () { res.end('OK\n'); }, 5000);
});

process.on('SIGTERM', function () {
  console.log("Closing");
  app.close();
});

app.listen(3000);

Since I’m using redis on howmanyleft, I need to close my redis connection gracefully too. The close event on the HTTP server fires when all connections have closed, so close my redis connection there. node_redis flushes all active redis commands when you call quit, so I won’t lose any data.

var http = require('http'),
    redis = require('redis').createClient();

var app = http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  setTimeout(function () { res.end('OK\n'); }, 5000);
});

process.on('SIGTERM', function () {
  console.log("Closing");
  app.close();
});

app.on('close', function () {
  console.log("Closed");
  redis.quit();
});

app.listen(3000);

Debian Wheezy Killed My Pogoplug

After months of running a hybrid sarge/wheezy installation on my Pogoplug (the wheezy bits needed for OS X Lion Time Machine support in netatalk), a power cut forced a reboot. Unfortunately the poor thing never came back to life.

A prod with an FTDI cable on the Pogoplug’s serial headers retrieved the following console grumbles:

udevd[45]: unable to receive ctrl connection: Function not implemented
udevd[45]: unable to receive ctrl connection: Function not implemented
udevd[45]: unable to receive ctrl connection: Function not implemented
udevd[45]: unable to receive ctrl connection: Function not implemented
udevd[45]: unable to receive ctrl connection: Function not implemented
udevd[45]: unable to receive ctrl connection: Function not implemented

It seems a package update had introduced a version of udevd that the poor Pogoplug’s kernel isn’t able to support. The good news is that you don’t really need the debian-installed udevd daemon. The initrd image that the Pogoplug boots from has its own, older udevd which is capable enough.

Plugging the pogoplug’s root disk into a Linux laptop and disabling the udevd daemon (insert exit 0 somewhere near the top of /etc/init.d/udev) brought my Pogoplug back to life, and it’ll hopefully mean I don’t have to consider a more expensive NAS for a long time yet. Yay!


Make TimeMachine in OS X Lion work with Debian Squeeze (stable) netatalk servers

All you need to do to get TimeMachine running again is to install the netatalk package from unstable. It’s a really simple process. First add the wheezy (testing) repository to your apt configuration.

# /etc/apt/sources.list
# stable repo
deb http://cdn.debian.net/debian squeeze main

# new testing repo
deb http://cdn.debian.net/debian wheezy main

Then give your squeeze repo a higher priority to your wheezy one (to prevent wheezy packages being accidentally installed when they’re not wanted).

# /etc/apt/preferences.d/priorities
Package: *
Pin: release a=squeeze
Pin-Priority: 700

Package: *
Pin: release a=wheezy
Pin-Priority: 650

Now install netatalk, making sure to tell apt-get to get it from the wheezy repo.

sudo apt-get install netatalk/wheezy

And remember, if you’ve added an afpd service description to your avahi-daemon configuration, remove it and restart netatalk (because netatalk 2.2beta will register itself automatically with avahi).

Happy Lion-ing!


PyOpenSSL in a virtualenv on OS X

If you’re a Python programmer, no doubt you’re now familiar with virtualenv. One of its nicest features is --no-site-packages, which isolates your virtual environment from any packages that are already installed globally.

However, if you’re on OS X, using --no-site-packages means you can’t use the OpenSSL library that’s installed by default. Trying to easy_install or pip install pyopenssl into your virtualenv won’t work, since OS X doesn’t ship with OpenSSL headers.

The solution to this little problem is to symlink the system OpenSSL library into your virtualenv:

ln -sf /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/OpenSSL lib/python2.6/

Running openoffice/libreoffice as a service

So, you’re trying to use JODConverter or unoconverter, you’ve set up openoffice to launch as a service in the background somewhere, but it’s not going anywhere, and all you’re seeing in your logs is

creation of executable memory area failed: Permission denied
creation of executable memory area failed: Permission denied
creation of executable memory area failed: Permission denied
creation of executable memory area failed: Permission denied

The solution, my friend, is to make sure that the openoffice process gets a writable path in its HOME environment variable. In my case, the supervisord config entry looks like this:

[program:openoffice]
command=/usr/lib64/openoffice/program/soffice.bin -headless -nofirststartwizard -accept="socket,host=localhost,port=8100;urp;StarOffice.Service"
environment=HOME="/tmp"
user=www-data

Hopefully this will save someone else the hour of fruitless hackery that I’ve been through to get to this point.


My Bank Holiday Project: Tweet GP

This is my latest project - a simple little web app that plays back twitter feeds from F1 races. It’s just like a Sky+ box, but for F1 tweets. If you miss the race, you can still watch it with all the best commentary from a hand-picked selection of the Twitterati.

Introducing Tweet GP

Events go live on Tweet GP the moment the coverage starts on the BBC, so even if you’re only minutes behind the action, you don’t have to miss a thing.

Where the action's at

It should work on all modern browsers. I’ve tested in on the current versions of Chrome, Firefox and Safari (both on the desktop and on iOS). It also works on IE8 (although it doesn’t quite look as sexy). If anyone can test on IE9 or Opera, I’d be grateful for feedback.

If all you want to do is get time-shifted tweets, go straight to tweetgp.com now.

If you’re of a very geeky persuasion, feel free to read on and get the back story of how it was all built.

A few weeks ago I was sat at home watching the Malaysian Grand Prix. I had woken up at 8am, which was too early for comfort on a Sunday morning and, crucially, two hours too late to watch the coverage on the BBC live.

Thankfully, like the rest of the civilised world, we have a very smart HUMAX set-top box that time-shifts TV for us, so I was able to watch the TV coverage from the start (in HD – thank you, BBC).

However, I had to put my iPhone down and stay away from twitter for the duration of the race in case I found out the result. This meant missing out on the insight and comic talents of several people I follow who usually make the race a much more enjoyable event.

Now, being a hacky kind of guy, I decided I could solve this problem with code (and at the same time learn something new). In time for the Chinese Grand Prix, I’d cobbled up a bit of Node.js code that connected to the Twitter streaming API, and sucked up a live stream from all the interesting F1 people, pushing it into a local Redis instance.

Using that lot, I was able to get out of bed at a sensible time to watch the action in Shanghai, without missing out on the comedy stylings of the likes of @sniffpetrol. Much rejoicing.

Since then, I’ve spent my bank holiday weekends packaging that lot up in a shiny HTML5 box. Everything on the client side is done without any plugins or proprietary extensions. As such, the code should work on all modern browsers (I’ve tested it on Chrome 11, Safari 5, Firefox 4, IE 8 and iOS 4.3).

I’m using express to serve up the front-end pages. All the code’s written in coffeescript, and the styling’s done in sass. Live tweets get pushed straight to the clients using Redis pubsub and socket.io. The server lives in the Rackspace Cloud UK.

And that’s it! Well done if you’ve read this far. I think next I’ll be signing up for a go on Twitter’s Site Streams beta API, and seeing if I can generalise this a bit. Fancy having a recording of all your tweets ready to playback for any TV program? Watch this space!


Using connect-assetmanager with sass and coffee-script

I’m a big fan of using as few curly-brackets as possible in my code, which means I love writing my stylesheets using Sass and my javascripts using CoffeeScript.

Because I love fast websites too, I also mash all my assets together using connect-assetmanager. However, out of the box, connect-assetmanager doesn’t automatically compile Sass or CoffeeScript.

It’s not difficult to get it working though. All you need to do is write a couple of short preManiupulate handlers, like so:

# handlers.coffee
sass = require 'sass'
coffee = require 'coffee'

exports.sassRenderer = (file, path, index, isLast, callback) ->
  if /\.sass/.test path
    console.log "Compiling #{path}"
    callback sass.render(file)
  else
    callback file

exports.coffeeRenderer = (file, path, index, isLast, callback) ->
  if /\.coffee/.test path
    console.log "Compiling #{path}"
    callback coffee.compile(file)
  else
    callback file

Now all you need to do is insert these into your expressjs config and you’re good to go!

# server.coffee
handlers = require 'handlers'

# ...

app.use assetManager
  js:
    route: /\/assets\/js\/[0-9]+\/.*\.js/
    path: './assets/javascripts/'
    dataType: 'javascript'
    files: ['sprintf.js','main.coffee']
    preManipulate:
      '^': [handlers.coffeeRenderer]
  css:
    route: /\/assets\/css\/[0-9]+\/.*\.css/
    path: './assets/stylesheets/'
    dataType: 'css'
    files: ['style.sass']
    preManipulate:
      '^': [handlers.sassRenderer]

Easy!


Embedding fonts in Flex apps

Recently I’ve been developing a Flex application at work using Adobe’s free Flex SDK. If you want to be able to draw transparent text labels in Flex, you need to have the fonts embedded in the application. Embedding a font is as simple as including it in a style block in the mxml file for the application:

<mx:Style>
    @font-face
    {
        font-family: DejaVuSansMonoEmbed;
        src: url("../assets/DejaVuSansMono.ttf");
        unicode-range:
            U+0020-U+0040, /* Punctuation, Numbers */
            U+0041-U+005A, /* Upper-Case A-Z */
            U+005B-U+0060, /* Punctuation and Symbols */
            U+0061-U+007A, /* Lower-Case a-z */
            U+007B-U+007E; /* Punctuation and Symbols */
    }
    .mytextstyle
    {
        font-family: DejaVuSansMonoEmbed;
        font-size: 12pt;
    }
</mx:Style>

However, the next bit gets a bit freaky. The compiler will invariably fail to build the application and give you an error that’s something like:

[mxmlc] app.mxml(14): Error: exception during transcoding: Unexpected exception encountered while reading font file 'assets/DejaVuSansMono.ttf'
[mxmlc] 
[mxmlc]             font-family: DejaVuSansMonoEmbed;
[mxmlc] 
[mxmlc] app.mxml(14): Error: unable to build font 'DejaVuSansMonoEmbed'
[mxmlc] 
[mxmlc]             font-family: DejaVuSansMonoEmbed;
[mxmlc] 
[mxmlc] app.mxml(14): Error: Unable to transcode assets/DejaVuSansMono.ttf.
[mxmlc] 
[mxmlc] 
[mxmlc] 

It turns out that the Flex compiler has a choice of font encoding engines to use and the first one it tries is usually a bit rubbish. In order to avoid the error, you need to force it to use the closed-source Adobe implementation (which is shipped with the free Flex SDK, but not with the open-source Flex SDK). At the command line, you need to add the option -managers flash.fonts.AFEFontManager to your mxlmc command.

If you’re using ant for your builds, you can add a font element to your mxmlc block to do the same.

<target name="main">
    <mxmlc file="src/app.mxml"
           output="${BUILD_DIR}/bin/app.swf"
           incremental="true"
           debug="false">
        <fonts>;
            <manager>flash.fonts.AFEFontManager</manager>
        </fonts>
    </mxmlc>
</target>

(Yes, I know, I’ve not done any ANPR stuff in a while. I’ll get around to it. Promise)


Step 1: complete

The clustering algorithm has been tuned and it’s working reliably—even on all sorts of odd foreign plates!

On top of that, I’ve added a de-skewing and extraction function. These are the results extracted from the previous image.

An extracted plate Another extracted plate

I’m happy with the speed too – these two were extracted in 0.2s. The only downside at the moment is that the clustering algorithm is roughly O(n^2) in terms of the number of contours, so on busier images it can take a second or two to do the clustering stage (and it’ll produce an occasional false positive).


Looking promising...

Had a bit of a brainwave the other night — clustering connected components using size similarity, locality and fit to a line as criteria. The image shows the kind of results I’m getting with a really quick script thrown together late last night.

Plates outlined

Blue boxes outline the individual characters that have been recognised, red boxes outline clusters.

It’s sometimes missing some of the characters at the ends of the plates, and there are still some false positives showing up in some images, but I’m fairly confident that a bit of tuning could fix all that.

Yay!


More results

Since the last update, I’ve been concentrating on stopping character shapes from bleeding into other elements of the image – mostly it’s been the edges of number plates that have been getting in the way. I’ve used a hough transform to find the number plate edges and eliminate them from the thresholded image before component labeling. I’ve also discarded some of the components based on a bit of filtering after labeling.

More results


First results

I’ve been fiddling with OpenCV on-and-off in my spare time for a few weeks now, and I’m starting to get a feel for the challenges involved in number plate recognition. There’s plenty of papers and background material out there to keep me reading for months.

The simple approach as used by a number of researchers is to concentrate on thresholding the image (sometimes many levels of thresholding), and then doing a bit of connected component labelling to find candidate regions of the image to extract and pass to a second-stage character recognition algorithm.

Here’s my first attempt. Source image, then thresholded and labelled image.

Source image

Thresholded and labelled image


Time for a project...

Something to distract me from the television: an open-source project!

I know nothing about the field of computer vision, so this was obviously the most suitable area to base my project in. I’m going to attempt to write an open-source automatic vehicle registration number recognition library.

I’m going to be coding in Python (because I need its wicked rad awesomeness in my life), and relying a heck of a lot on OpenCV for the underlying hardcore bit-pushing shit.

The project is on github now. There’s nothing there to see yet, but a few watchers would be very encouraging.


Netatalk on an Ubuntu virtual machine

I tend to develop on a MacBook running OS X Leopard. In order to keep my main Leopard system clean and tidy, I use Parallels desktop to run my development environments in virtual machines. Parallels is jolly good at sharing data with Windows virtual machines, but a bit lacking when it comes to Linux. Hence I’ve started using netatalk on all my Linux virtual machines to access their disk drives from OS X.

Lots has been written about what it takes to install and tweak netatalk to get it to talk happily with Leopard, so I won’t go into the problems here. All you need to know should be found at the following links.

The process below is an amalgamation of all the instructions from those pages, put together to form the process I go through to set up netatalk on my virtual machines.

Firstly, one needs to download the netatalk source, install all of its dependencies and build it with ssl enabled.

$ mkdir -p ~/src/netatalk
$ cd ~/src/netatalk
$ sudo aptitude install devscripts cracklib2-dev dpkg-dev libssl-dev
$ apt-get source netatalk
$ sudo apt-get build-dep netatalk
$ cd netatalk-2.0.3
$ sudo su
# DEB_BUILD_OPTIONS=ssl dpkg-buildpackage -us -uc
# debi
# echo "netatalk hold" | dpkg --set-selections
# exit

That should have build and installed the netatalk package, and told the debian packaging system to not replace it if a new version appears in the online repositories.

The only netatalk service I’m interested is the file sharing one, so the rest can be turned off in /etc/default/netatalk:

# Set which daemons to run (papd is dependent upon atalkd):
ATALKD_RUN=no
PAPD_RUN=no
CNID_METAD_RUN=no
AFPD_RUN=yes
TIMELORD_RUN=no
A2BOOT_RUN=no

Finally, to enable zeroconf/bonjour discovery of the shares on the system, install avahi-daemon and add a service definition to /etc/avahi/services/afpd.service to advertise the afp service.

<?xml version="1.0" standalone=‘no’?><!–*-nxml-*–>
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
  <name>%u</name>
  <service>
    <type>_afpovertcp._tcp</type>
    <port>548</port>
  </service>
</service-group>

And that’s it! You should now be able to reliably mount your home directory on the Linux virtual machine from OS X.


RESTful Services in Visual C# 2008

Here’s a problem that had me scratching my head for a while this weekend. How do I create a simple REST service using WCF (Windows Communication Foundation)?

MSDN has a great little tutorial that explains how to code the service and get it running in a standalone application context. However, it fails to address getting the service deployed on an IIS server.

The key point that seems to be missing from the documentation is that no matter what you do with the web.config file, you’re never going to get a configuration that works as a plain-old REST web service. Instead, you have to delete the web service configuration completely from the web.config file, and add the declaration Factory="System.ServiceModel.Activation.WebServiceHostFactory" to the ServiceHost element in the .svc file associated with your service.

Here’s a quick and dirty example:

<%@ ServiceHost Factory="System.ServiceModel.Activation.WebServiceHostFactory"
    Language="C#" Service="Service1" %>
using System.ServiceModel;
using System.ServiceModel.Channels;
using System.ServiceModel.Web;
using System.Runtime.Serialization;
[ServiceContract]
public class Service1
{
    [OperationContract]
    [WebGet(UriTemplate="{id}")]
    DataType GetData(string id)
    {
        return new DataType(1,"Foo");
    }
}
[DataContract(Name="data",Namespace="http://example.org")]
public class DataType
{
    public DataType(int id, string name)
    {
        this.id = id;
        this.name = name;
    }
    [DataMember]
    int id;
    [DataMember]
    string name;
}

Calling http://host/Service1.svc/123 returns:

<data xmlns="http://example.org" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
    <id>1</id>
    <name>Foo</name>
</data>

That’s actually quite easy really, isn’t it? If only the MSDN docs told you that..


Subversion and Visual C++

Despite mostly using Java for the work I’m doing at the moment, there’s still some call for a bit of C++. Visual Studio 2005* is still the daddy of all IDEs for developing in C++ as far as I’m concerned, but all the intermediate cruft that it generates doesn’t really need checking in to my Subversion repository.

You can cut down the extraneous crap by excluding all debug and release directories, and all .ncb.suo.vcproj.USERNAME.user and .aps files.

* Yes, I know 2008 is available, and I’ve even got it downloaded from MSDN, but I’ve not got around to installing it yet…