How do I match the collaboration tests to the contract tests? In other words, what stops me from stubbing
foo() a certain way, but writing tests for
foo() that expect different behavior? What if I change the tests for
foo(), but forget to change the corresponding stubs for
foo()? Haven’t I just introduced an integration defect without any failing tests?
Yes you have, but you can do something about it. I have vehemently reinforced the notion that collaboration tests and contract tests don’t replace thinking, but rather help make my thinking about tests more systematic. I’ll describe what I mean by that in the next few paragraphs.
Before I started writing contract tests, I would typically write decent collaboration tests, but frequently encounter the case where I found a mistake even though all my tests passed. I would have a case where either I missed an important collaboration test or I stubbed a method in a way that no test checked, or I mocked a method in a way that no test tried to use it. Unfortunately, I didn’t know how to categorize those mistakes at the time, so I had two basic recourses: write collaboration tests with more care or write integration tests. I tried both, and neither alone helped. Both together helped a little, but not much. When I wondered how to solve this problem once and for all, I hit on contract tests. They helped, and considerably, but I still encountered problems.
Good news, though: I could see the problems more carefully. When I made an integration mistake, I could categorize it one of two ways:
foo()would never return
foo()would never return
23in that situation.
foo()to expect parameters
beven though I never checked to see what happens when I invoke
I could limit this mistake by checking for these two mistakes. I created a simple system. Whenever I stubbed a method to return
23, I’d write a contract test for that method that expected the result
23. If I couldn’t do that, then I didn’t understand the method’s contract well enough yet, and I stopped to figure it out. Whenever I mocked a method to accept parameters
b, I’d write a contract test for that method taking
b as parameters. If I couldn’t do that, then I didn’t understand the method’s contract well enough yet, and I stopped to figure it out. This system didn’t solve the problem of mismatched tests, but it gave me a repeatable method for reducing the risk of mismatched tests.
Even better, when I made an integration mistake, I knew what kinds of mistakes to look for:
- I missed a collaboration test.
- I missed a contract test.
- I missed a contract test corresponding to the way I stubbed a method.
- I missed a contract test corresponding to the way I mocked a method.
I imagine one could automate these checks, and I think that would make a splendid Ph. D. project for some eager young mind. I don’t believe I’ll do it.
So let me answer the question at least: who tests the contract tests? The collaboration tests and the contract tests check each other, if only you stop to listen to them.
In closing, I refer you to this article in which I describe how this technique could have prevented the Mars rover from prematurely deploying its parachute.