The Eternal Struggle Between Business and Programmers
At Øredev 2013, I asked managers to tell me what makes their job difficult. I spoke to one manager who told me an all-too-common story about facing immense pressure from the board of directors to deliver more features more often and similar pressure from the programmers to give them more time to refactor. More important than the details of this situation, I’ve heard dozens of stories like these over the years, and while one single piece of advice can’t possibly address every significant aspect of these situations, I have found that most of the people who tell me these stories identify with what I’d like to share with you.
Warning. You almost certainly have more to deal with in your exact situation than this advice can cover. You might need some more direct help. I can offer that to you, but for now, read on.
I call it the eternal struggle between the Business and Programmers. Specifically programmers, because I almost always hear the specific complaint coming from them. It goes like this.
Business: More features! Now!
Programmers: More refactoring! Now!
It usually involves more colorful language, more intense emotions, and repeats a few times, but in the interest of time, I’ve distilled it to its essence. I notice a few things about this particular exchange.
First, the Business can easily justify their position: they need profit, and profit comes from sales, and sales come from features, and features come from, among other people, the Programmers. Without more profit, nobody keeps a job.1 Clearly, the Business has this one right.
Next, the Programmers can easily justify their position: they need to deliver more features, and they can’t deliver features if they don’t understand the code, and almost nobody writes code clearly the first time, so the Programmers need to edit their work, and they call that editing “refactoring”. Without refactoring, features don’t flow, the Business has nothing to sell, we can’t generate profit, and nobody keeps a job. Clearly, the Programmers have this one right.
Unfortunately, the Business only sees that the Programmers used to deliver features more quickly, and now they want to slow down. This creates obvious tension. This tension slowly pervades every aspect of the relationship between the Business and the Programmers. It obscures something important. I’ll come back to that.
So the Business needs features to sell, but the Programmers want to slow down. The Business needs to feed a market that expects more features sooner, but the Programmers need to feed a design that demands cleanliness as a condition for admitting new features as a steady pace. Thus, the eternal struggle, because it has always been thus.
The Programmers plead: “If only you’d let us refactor! We have a sick design; it actively prevents us from adding features sensibly. It encourages us to do the wrong things. It encourages us to duplicate code, which makes mistakes seem inevitable. It has code that nobody understands any more, and so when we have to change it, we need weeks to read, to research, to gain confidence that changing it won’t anger our existing customers. If you’d let us refactor, then we could spot design improvements that would unblock new features, that could reduce the cost of new features by 20%, 30%, 40%, and more! Why don’t you let us refactor?!” They appear to have a point.
The Business pleads: “If only you’d stop with this refactoring nonsense! The economy has to grow. We have to grow. The board of directors yells at the vice presidents, who yell at the third-line managers, who yell at the second-line managers, who yell at us. More features! More features! They only want more features! Not only can they not afford for you Programmers to slow down, but they need you to speed up! I need you to speed up! What do you go faster?!” I don’t envy their situation.
If you’re a Programmer, then imagine yourself in the position of the Business. The Business relies completely on Programmers (among other people) for its survival. I imagine the Business finds this imbalance uncomfortable, even threatening. Without the Programmers, the Business has nothing to sell. Worse, at the source of this tension we find a haphazard, slow stream of half-working, not-quite-what-we-wanted features. It doesn’t matter that the Programmers can easily justify their behavior, because the Market doesn’t like the results, and as soon as the Market has an alternative, it disappears, and nobody keeps a job.
What can we do?
Refactoring reduces the volatility of the cost of new features! It smooths out the design, distributing the problems more evenly. It helps the Programmers make many small easy-to-clean-when-needed messes, rather than several small-to-large we-have-to-stop-the-line-to-go-any-further messes. It also helps the Programmers identify game-changing structural improvements that they might not otherwise see—the kind that reduce the cost of future features by 20%, 30%, 40% and more. Refactoring even helps eliminate many classes of annoyingly, costly mistakes. (You might call them “bugs”.) This short talk describes how refactoring and the cost of features relate to each other:
(If you prefer to read a transcript of this talk, click here.)
Seven minutes 26 seconds. Fred Brooks, Mythical Man-Month, No Silver Bullet. How many of you read it? Not enough. Accidental complication and essential complication. Not complexity, because complexity implies emergence. And I'm just talking about complication. No such thing as an order of magnitude improvement in performance in part because of accidental complication. Essential complication because the problem is hard. The problem is hard so the system is complicated. You want to audit a tax return? Have you read the tax code in Canada? It's this thick. Essential complication. Accidental complication... we're not so good at our jobs. Accidental complication because we cut corners, we feel pressure, we don't have to worry about it this time, we don't have to re-factor so much, we have to get it out the door. Accidental complication. Essential complication. The cost of a feature is a function of the cost coming from the essential complication because the problem is hard and the cost of accidental complication because we suck at our jobs. Roughly speaking we can add these things together. How do you estimate, if you estimate? Don't estimate. But if you estimate how do you estimate? This thing is kind of hard. This thing is kind of the same kind of hard. This thing took two weeks. This thing, two weeks. Here's the problem: Friday afternoon meeting, "Don't worry, boss. Three days." Monday morning, "Oh shit. Make that three months." Why? Accidental complication. Most people most of the time the cost of a feature is dominated, dominated, dominated by the cost from accidental complication. That means that the cost of your feature has almost nothing to do with how hard it is and almost everything to do with how much your design sucks. What can we do? Test driven development to the rescue. (Of course he was going to say that.) Well, what the hell do you mean? Test driven development step one: think. (Everyone forgets that step.) Step two: write a test. Step three: stand back and ask, "How much does this test suck? Is it too long? Does it have irrelevant details? Can I make the test smaller, simpler?" Congratulations, you're removing accidental complication. Step three: run the test and watch it fail. You'd be surprised how often it doesn't actually fail. Why doesn't it fail? Well, I'm running the wrong tests. No, that's not the problem. I forgot to check something. No, I'm responsible. I write the assertion first. That's not the problem. I wrote too much code last time and it already passes the test. I should stop doing that. Accidental complication. Step four: write just enough code to make it pass. Just enough code to make it pass. Just enough code to make it pass. Not the line of code you know you have to write. Just enough code to make it pass. Return 12. If input is 7, return 12, else return 13. Weird. Good enough. If every product in our shop is either 12 or 13 dollars, we're done. Another if statement: now it's a lookup table. Let's move the lookup table here, then here, then here. Now we have MVC. Accidental complication. We have to figure out how to remove it. And so after we write just enough code to make the test pass the next thing we do is we clean the kitchen. We refactor a bit now. We have made a little mess to add some behavior and now we smooth that little mess out. Because if we don't clean the kitchen then we have to clean the garage. Cleaning the garage is a bigger job. We always put it off. We hate it when we have to do it. The only time we clean the garage is when there's no more room for the car. Clean the kitchen. Cleaning the kitchen, reducing accidental complication. So far: think, write the test, watch it fail, write just enough code to make the test pass... (give your computer caffeine so that you can see the timer, mistype your password because you're trying to go too quickly.) Reduce accidental complication. So write a test, limit the amount of code you're trying to write at once. Reduce accidental complication. Ask yourself, "Does the test suck?" Remove accidental complication. Finally, make the test pass. Refactor, clean the kitchen. Reduce the accidental complication that gets in because you didn't do a good job of avoiding it in the first place. You're getting better. Every few minutes. Every few minutes. Every few minutes. At the top of the circle, avoid accidental complication. At the bottom of the circle, squeeze out the accidental complication that got in while you weren't looking. Why? Because if the cost of a feature is the cost of the essential complication plus the cost of the accidental complication and if we compare estimates based on essential complication - how hard is the problem if it's roughly as hard as that thing that took two weeks it should take two weeks. But if the cost from accidental complication dominates the cost of the feature, then you arrive at the fundamental theorem of agile software development. Not enough of you are leaning in. The fundamental theorem of agile software development says this: If you want to estimate little things you have to refactor because refactoring is how you reduce accidental complication and only by driving accidental complication down as far as you possibly can will your relative estimates have any meaning. Here's the proof: f(essential complication, accidental complication) = g(essential complication) + h(accidental complication). f is supposed to be proportional to g. But you have h. h either has to be a perfect multiple of g (how many times have you worked in a code base where the shit was uniformly distributed throughout the code?) or h has to be - say it with me - zero. Because if h is anything bigger than zero and h is not a multiple of g, then f will not be proportional to g and your relative estimates will be to- tal bull- shit. Therefore, if you're going to estimate, you'd better refactor which means Scrum cannot work without XP. This has been seven minutes and 26 seconds. Questions?
Over time, the Business has coerced the Programmers into stealing future capacity by pressuring them to not cultivate a healthy design. They have probably done this for a long time. Don’t blame the Business for this: the Industry has largely taught them to do it. On the other hand, the Programmers haven’t used a particularly meaningful defence: they’ve appealed to arguments based on some abstraction notion of craftsmanship. Some Businesses want to buy craftsmanship; others need features now!2
Now, we can realise something significant: The Business and the Programmers want the same thing. They want the same thing: to deliver a more predictable stream of important features that will make the market happy—or, at least, knowing the fickle market, that will risk disappointing them the least.
This. Is. Glorious.
The Business and the Programmers needs to agree on two things, each conceding a key point to the other. The Business needs to agree that the Programmers have heretofore gone more quickly than they really can, that they cannot sustain this pace, and that to do so merely hastens the ultimate decline of the entire product line, and perhaps the company. The Programmers need to agree that the Business has a legitimate need to deliver more features, that everyone’s livelihood depends on this, and that trying to ask for more than the Programmers can deliver forms part of the Business’s “job”, as it were. The market compels them to do this. In this equation, the Programmers have the responsibility to do whatever they can to maximise their ongoing delivery speed, and that includes regular, agreessive refactoring.
Finally, both the Business and the Programmers have to agree that they have wasted too much time arguing about this and need to move forward. They want the same thing! They can have it!
Thus, I propose a kind of detente. The Business agrees to remember that the Programmers always try their best to go at a fast-but-sustainable pace. The Programmers agree to remember that the Business also feels threateningly strong pressure to perform, and that they don’t feel comfortable relying so much on the Programmers for their success. Each group needs the other desperately, and together they all want the same thing.
…and let’s not forget the Testers and Operations and other departments. If we simply listen to each other, we’ll realise that we all want the same things, and together, we can make things so much better.
I’ll leave aside the question of “enough” profit for another time.↩
Let me be very, very clear: I support the software craftsmanship movement, because it takes a pragmatic view of craftsmanship. Just because they call it “craft” doesn’t give them licence to take all the time in the world, and a competent, caring software craftsperson will not do this. Even so, a typical Business reaction to the notion of “craft” is “artisan”, a Greek word meaning “slow and expensive”.↩