Automated Testing for DevOps/Sysadmins (Working in a World of Dependencies)

The SpectoLabs team attended the awesome DevOps Days NYC in September, and very much enjoyed it! This is the first of a two-part blog post that summarises our discussions and learnings from the event in relation to the work we are doing at SpectoLabs HQ.

The event followed the usual DevOps Days format of talks in the morning and unconference sessions in the afternoon, and we joined several interesting focused breakout style sessions. One particular session discussed the difficulty with testing infrastructure/operational work, particularly when many of the platforms we are currently building rely so much on third-party services. 

After we arrived back in the UK we found that the combination of these discussions and the recent publication of Dan Ashby’s excellent blog post about “Continuous Testing in DevOps” made us think more about some of the challenges of quality assurance (QA) within the operational side of the DevOps role…

DevOps: Platform, People (and Dependencies)

Many modern DevOps/sysadmin tasks are based around the creation and maintenance of a ‘platform’ and supporting tooling. The primary purpose of the platform is to support self-service and automation for the development team, which ultimately allows developers to focus on what they do best - creating code to add business value - rather than being overburdened by operational concerns. We heard about this concept many times at the conference: 

DevOps is all about people, and supporting them from the perspective of the (shared) organisational goals, personal (cultural) challenges, and technical delivery (automation/platform)

In a breakout sessions one of our fellow attendees raised that point that several of the components within their platform are now provided by third-parties e.g. payment gateways provided by PayPal, host and application monitoring via Datadog, and platform provisioning by AWS. The person continued by stating that they often found it challenging to test components that depend on these third-party services for a variety of reasons:

• It is often difficult to mock the third-party service
• Using the real service costs money
• The sandbox implementations provided are often slow and unreliable.

Testing with Third-Party Dependencies

At Specto we are strong supporters of using modern Service Virtualistion (SV) tooling to test systems that are coupled to external dependencies, but before we had the chance mention this in the breakout session, one of our other fellow attendees jumped in and discussed how they had used SV tooling to overcome just this problem. We thought we would add our 2 cents to the discussion in this blog…

Modern lightweight SV tooling like our open source Hoverfly application is essentially a proxy that sits in between two services and records requests and responses. Once the appropriate interactions (and traffic) have been captured we are no longer dependent on the third-party service or its sandbox implementation, and we can remove this and have Hoverfly simulate the service:

Because Hoverfly can be deployed as a small binary and managed via CLI commands it’s easy to work across a variety of different languages (or simply the shell). It’s also easy to incorporate within a continuous build pipeline for both infrastructure and applications. We like to use the following tooling for infrastructure testing within the pipeline:

• ServerSpec: Ruby-based server validation
• PuppetLab’s Beaker: Puppet acceptance testing harness
• Goss: Golang-based alternative to ServerSpec

And when testing application functional requirements we’re big fans of using Hoverfly alongside these testing frameworks/harnesses:

• REST-assured: Java-based Behaviour-Driven Development (BDD) testing for APIs
• Geb: Groovy UI testing (via Selenium)
• Gatling: Scala-based load testing
• RSpec: Ruby BDD test framework
• Ginkgo: A Golang BDD framework
• Pyresttest: Python API testing framework

But What About the NFRs?

Automatically verifying the functionality of operational systems that rely on third-party dependencies is possible with service virtualisation (SV) tooling. Our discussion at DevOpsDays NYC highlighted that several operators are already using this technique very effectively, and several others commented that these techniques will become even more important as more and more operational platform components are offered through an API ‘as-a-service’.

Equally as important as verifying functional requirements is verifying non functional (or ‘cross-functional’) properties. As the joke goes, I’ve seen many systems that were not functional due to non functional requirements! We’ll cover this more in part 2 of this blog post…