Gap Between Documentation and Actual Implementation

Robustness principle

If you have machine-readable specifications for our message formats, we should be able to verify these properties by analyzing the new specification relative to the old spec. A tough problem arises that we need to address when applying the Robustness Principle, though. There may be a gap between what we say our service accepts and what it really accepts.

JSON payload example

For instance, suppose a service takes JSON payloads with a url field. We discover that the input is not validated as a URL, but just received as a string and stored in the database as a string. We want to add some validation to check that the value is a legitimate URL, maybe with a regular expression. Bad news: the service now rejects requests that it previously accepted. That is a breaking change.

But wait a minute! The documentation said to pass in a URL. Anything else is bad input and the behavior is undefined, meaning it could do absolutely anything. As soon as the service went live, its implementation became the de facto specification.

It’s common to find gaps like these between the documented protocol and what the software actually expects. I like to use generative testing techniques to find these gaps before releasing the software. But once the protocol is live, what should we do? Can we tighten up the implementation to match the documentation? No. The Robustness Principle says we have no choice but to keep accepting the input.

The caller example

A similar situation arises when a caller passes acceptable input but the service does something unexpected with it. Maybe there’s an edge case in our algorithm. Maybe someone passed in an empty collection instead of leaving the collection element out of the input. Whatever the cause, some behavior just happens to work. Again, this isn’t part of the specification but an artifact of the implementation. Once again, we aren’t free to change that behavior, even if it was something you never intended to support. Once the service is public, a new version cannot reject requests that would’ve been accepted before. Anything else is a breaking change.

Inbound testing

Even with these precautions, we should still publish the message formats with something like Swagger or OpenAPI. That allows other services to consume ours by coding to the specification. It also allows us to apply generated tests that will push the boundaries of the specification. That can help us find those two key classes of gaps: between what our spec says and what we think it says, and between what the spec says and what our implementation does. This is inbound testing, as shown in the following figure, where we exercise our API to make sure it does what we think it does.