Building software against the meetup API whilst 20,000ft above the North Sea

The Problem

This is a guest post from Mark Coleman of Implicit-Explicit

Every other Wednesday I travel from Amsterdam to London and back. I somehow always resented ‘wasting’ those two hours in the air (not to mention the trains to and from Gatwick but that’s another story entirely) so until recently I would fill the time with reading, even though what I really wanted to be doing was working on one of our pet projects which will be receiving lots of attention in just a few weeks and requires that I get a move on.

The things stopping me were a lack of connectivity and laziness. The projects we’re working on right now talk to the Meetup API and since I was in an aircraft with no wifi this was problematic.

Figure 1 - A rough approximation of my biggest airbourne development hurdle

Secondly, I’m lazy and I didn’t want to write mocks that would only be used for a few hours per month. On a more serious note, the code is going to be changing a lot and mocks would introduce a test management overhead that I don’t want right now.

Figure 2 - Although I’m sure this will create a flame war, I find that mocking slows me unnecessarily at the beginning of a project so normally try to avoid it

The Solution

I figured I would give HoverFly a go, since I’m already working with SpectoLabs on a weekly basis and had heard such good things. I hit a couple of issues on the way which I will explain at the end of the post, but all in all it was smooth sailing.

HoverFly has a few interesting modes but the most basic mode is capture and playback which will be shown here. HoverFly acts as a proxy, waits for outgoing HTTP(S) requests and then captures the response from the remote end point, in this case, the Meetup API.

Then the next time round, we don’t need the Meetup API anymore for that request. HoverFly will recognise that the same request is being made and return the output it previously captured from the Meetup API. Or in other words, that request will now work whilst 20,000ft above the North Sea with no wifi. Hoorah!

Below I will go into some details but if you want to see actual code you should check out the requests.get() calls in the repository. The point is that the application will be totally unaware of what’s happening (which means run the same code everywhere) and I’ll be able to test against the Meetup API whilst flying, or sitting in a Faraday Cage.

Step 1 - Put HoverFly in the request path

Figure 3 - HoverFly acts as a proxy when the request is made

Download your flavour of HoverFly from the downloads page, make sure it is executable and start it. NOTE Make sure you change the wget URL below to your flavour of HoverFly.

[vagrant@vagrant-centos7 tmp]$ wget https://github.com/SpectoLabs/hoverfly/releases/download/v0.5.10/hoverfly_linux_amd64
--- snip ---
[vagrant@vagrant-centos7 tmp]$ chmod +x hoverfly_linux_amd64
[vagrant@vagrant-centos7 tmp]$ ./hoverfly_linux_amd64 --capture
{"databaseName":"requests.db","level":"info","msg":"Initiating database","time":"2016-02-10T13:05:57-05:00"}
{"Destination":".","Mode":"capture","ProxyPort":"8500","level":"info","msg":"Proxy prepared...","time":"2016-02-10T13:05:57-05:00"}
{"AdminPort":"8888","level":"info","msg":"Admin interface is starting...","time":"2016-02-10T13:05:57-05:00"}
[negroni] listening on :8888

Once HoverFly starts successfully you’ll see “ProxyPort”:”8500” in the output.

To get our requests to go through HoverFly we need to set the HTTP_PROXY or HTTPS_PROXY environment variable to point to the now running HoverFly instance. For now we will assume that you started it on the local machine.

export HTTP_PROXY=http://localhost:8500/

…or…

export HTTPS_PROXY=https://localhost:8500/

In our case the Meetup API uses HTTPS so we choose the latter. Please also note that to get HTTPS working correctly there are two approaches, the easy way and the less easy way. I chose the former for this blog which is to add verify = False to the requests.get() call in app.py. The correct way to do this is to add the certificate from the HoverFly repo to your system’s trusted certificates.

Step 2 - Make a request and HoverFly captures the response into its DB on the fly

To make a request with our pet project once HoverFly is running the easiest way is to simply follow the instructions on the README and use the GUI. If all goes well, you’ll notice logging streaming from the running HoverFly process. If you’d like more details about how the actual call is made you can check the Meetup API documentation or have a look in the code.

Figure 4 - HoverFly records any response into its database whilst also passing it back to the application

Step 3 - Make the same request without needing the external API

Now to see the magic of HoverFly in full effect we first need to take HoverFly out of capture mode, which is simple as killing it and restart it without any flags.

[vagrant@vagrant-centos7 tmp]$ ./hoverfly_linux_amd64

Next switch off your wifi, or get in a plane. Now run the same request again and you’ll notice two things:

1. It’s really fast.
2. It works.

The former came as a pleasant surprise actually and I’m now using HoverFly for development even if I have a connection so that I spend less of my life waiting.

Conclusion and next steps

Here I’ve touched on HoverFly’s most basic mode; Capture and Virtualize. It has allowed me to work in the plane, but even with this mode the implications are of course much greater; I can get around speed limitations and rate limits easily during the development process.

Since our work against the Meetup API is soon to be split across multiple remote teams who will be creating individual microservices, there is an interesting new use case on the horizon; testing against APIs that don’t yet, or only partially, exist. Hopefully I’ll find the time soon to write about deploying HoverFly as a ‘fake microservice’ using Docker Compose to solve exactly this problem during testing.

Tricky Bug

HoverFly writes into BoltDB which currently cannot deal with mounted folder in Vagrant or anywhere it seems. Thanks to Karolis for spotting this within minutes.

You have two choices here; 1) run HoverFly outside of a mounted folder or 2) use the environment variable HoverflyDB like so:

HoverflyDB="/somepath/that/not/in/a/shared/folder/in/vagrant/my.db"