Who tests the contract tests?

I’ve already written about contract tests here and here, but I haven’t written specifically about the question that contract tests raise. I’ll do that here.

How do I match the collaboration tests to the contract tests? In other words, what stops me from stubbing foo() a certain way, but writing tests for foo() that expect different behavior? What if I change the tests for foo(), but forget to change the corresponding stubs for foo()? Haven’t I just introduced an integration defect without any failing tests?

Yes you have, but you can do something about it. I have vehemently reinforced the notion that collaboration tests and contract tests don’t replace thinking, but rather help make my thinking about tests more systematic. I’ll describe what I mean by that in the next few paragraphs.

Before I started writing contract tests, I would typically write decent collaboration tests, but frequently encounter the case where I found a mistake even though all my tests passed. I would have a case where either I missed an important collaboration test or I stubbed a method in a way that no test checked, or I mocked a method in a way that no test tried to use it. Unfortunately, I didn’t know how to categorize those mistakes at the time, so I had two basic recourses: write collaboration tests with more care or write integration tests. I tried both, and neither alone helped. Both together helped a little, but not much. When I wondered how to solve this problem once and for all, I hit on contract tests. They helped, and considerably, but I still encountered problems.

Good news, though: I could see the problems more carefully. When I made an integration mistake, I could categorize it one of two ways:

I stubbed foo() to return 23 even though foo() would never return 23, or foo() would never return 23 in that situation.
I mocked foo() to expect parameters a and b even though I never checked to see what happens when I invoke foo(a, b).

I could limit this mistake by checking for these two mistakes. I created a simple system. Whenever I stubbed a method to return 23, I’d write a contract test for that method that expected the result 23. If I couldn’t do that, then I didn’t understand the method’s contract well enough yet, and I stopped to figure it out. Whenever I mocked a method to accept parameters a and b, I’d write a contract test for that method taking a and b as parameters. If I couldn’t do that, then I didn’t understand the method’s contract well enough yet, and I stopped to figure it out. This system didn’t solve the problem of mismatched tests, but it gave me a repeatable method for reducing the risk of mismatched tests.

Even better, when I made an integration mistake, I knew what kinds of mistakes to look for:

I missed a collaboration test.
I missed a contract test.
I missed a contract test corresponding to the way I stubbed a method.
I missed a contract test corresponding to the way I mocked a method.

I imagine one could automate these checks, and I think that would make a splendid Ph. D. project for some eager young mind. I don’t believe I’ll do it.

So let me answer the question at least: who tests the contract tests? The collaboration tests and the contract tests check each other, if only you stop to listen to them.

In closing, I refer you to this article in which I describe how this technique could have prevented the Mars rover from prematurely deploying its parachute.