Testing 3rd party APIs
If you are responsible for shipping software that are built using multiple components, you would want to ensure that all these pieces work as expected when a user is using them in real life. In any such distributed systems context, the complexities and challenges of how you would test your stack grow exponentially with the number of components your entire stack is built of.
There can be multiple databases, multiple services talking to each other, 3rd party APIs that your applications integrate with and so on. If you are writing a monolith, or if your software stack consists of only one component, or if you are in a very, very early stage of your startup, these may not apply to you.
A single server, connecting to a single database, with maybe a single frontend could be tested using the age old wisdom of the Testing Pyramid. But times change, practices evolve, customer specifications get more and more convoluted and users expect richer experiences. If we look at the biggest trends in software development landscape that we have today, it is full of services and apps having elaborate conversations.
Motivation
Very often, your application or service will use a 3rd party API as part of its role. So for example, you might notify customers via SMS and email, using Twilio and Mailgun. Or you might let customers log in via Facebook or Twitter, or GitHub, like we do at CircleCI.
You’ll want to test against these services for two reasons. Firstly, you’ll want to validate that your new code has the expected outputs. Obviously, this is just normal testing: make sure it works.
The second reason is to validate that 3rd party services don’t change in a way that breaks you. Because 3rd party services are out of your control, and maintained by a team that you don’t have access to, they can change their APIs in a way that breaks you, without telling you, possibly intentionally but often not. If they do, you need to know ASAP.
How
There are two ways to deal with this. First, you can test it live! That is, during your tests, make the actual API calls you would in production. You’ll need a test account, of course: in some cases, you can make an account via that company’s API during the test and delete it afterwards. Alternatively you might have a manually set up test account. Or finally, some companies, automatically set up a test API key to use.
The primary reason that live testing sucks is that these tests will be flaky. You’ll get 503s when the 3rd party service has imperfect deployments (which is often, in our experience) or has a minor misconfiguration. You’ll get 500s when they have a bug in their app. And you’ll randomly get 401 errors if you have too many tests running in parallel and you hit their rate-limits.
Mocking
The standard solution is to mock out these interfaces. Mocks are a very common and well-known best practice. In a mock, you record the result of the API call, validate manually that it says what you expect it to, and then during your build you run the test against the saved mock. Thus is very common, and I’m sure most people who have written tests have done this in some form at some point.
The problem with mocks is that you lose the second property: validating that 3rd party services don’t change. You can go on happily running your tests, believing that your Twilio integration works, when actually a subtle, undocumented behaviour that you rely on has changed, and you’re none the wiser.
So the correct way to deal with this is a hybrid approach. First you create the mocks and run your tests against them. This gives you protection from the flakiness of 3rd party services as you’re trying to ship code.
Secondly, you take your mocks and test them continuously.
Further reading