The Hidden Cost of Estimates

(2023-07-01)

Estimating the effort (and therefore) the cost of building software is difficult at best and impossible at worst. But there is no getting around the need to estimate the cost and effort of building software. If we improve our techniques and communication we can use estimation to partner with our stakeholders instead of creating adversarial situations.

The process of crafting software is full of uncertainties and unknowns even the most experienced developers will get it wrong more often than they get it right. The Planning Fallacy (Kahneman & Tversky) suggests that as humans we tend to be optimistic in our estimates, overweighting positive outcomes and underweighting the probability of negative events. Even though our normal estimates are commonly overly optimistic, as managers and developers we’re often still asked to give optimistic efforts when planning projects or tasks that appear potentially too expensive.

Is the solution maker better estimates?

Given the large cost of software for many organisations, there is continuous research into improving the development and estimation process. Over the last 30+ years, there have been many attempts at industrialising the software process, nearly all of which include an increased focus on planning and decomposition. There is ample evidence that decomposing a task into smaller tasks will make each of those smaller tasks easier to estimate. The more familiar and repeated a task is, the easier it is to estimate. But few software projects are full of well-known tasks. Often there is some form of R&D needed, even if we are just exploring new tools or multiple approaches to solving a problem. When we take the time to fully decompose, spec out, and estimate how long a project will take, we probably spend more time in the planning phase than we do coding the system.

For sure, there are some regimes where this level of effort may be warranted (building aircraft autopilots and the control systems for power plants perhaps), but it is usually not because we produce better estimates. In these domains, the rigorous design process often comes from a disconnect between the availability of the final assembled systems and the need to build software in parallel.
I can’t imagine Boeing shareholders would be happy if the company didn’t start to work on the software for a new jet until after they had assembled it.

Agile was promoted as a response to the Waterfall process model and has been incredibly successful, but even an Agile process needs some form of estimation (feature points, story points etc.). Deciding which features to add to a software project requires some form of cost-benefit analysis, which requires a cost; cost implies estimates.

Estimates are not free. It's a mistake to think they are. If we run any form of sprint-based Agile progress, there will be sprint planning, which needs estimates. The time required to create those estimates is just one way of underestimating the work required in the previous sprint.

Can we do better?

I've seen teams have a once-a-sprint session to produce the estimates for the next sprint planning session. Another meeting just for estimating is one more way the sprint ceremonies can eat into the productivity of the sprint. If everyone has to go to a sprint planning session, an estimating session (probably as long as the planning session), and a sprint retro --- we've probably consumed one workday just for planning. For a two-week sprint, we’ve used up 10% of the time just in the start and end ceremonies. Is 10% overhead too much? For some teams, especially teams just learning Agile, explicit estimating sessions work well and the overhead is worth the effort. As teams mature they may eventually find that explicit planning may only be necessary for larger tasks. When the project backlog is full of well-understood incremental enhancements the team may decide to forego much of the planning. Teams may even be honest with their users that a large enough feature needs to have a design and estimation task in the backlog before they can even produce a good estimate. (Rhetorical question, how accurate do we think our estimate of the work to estimate a task is?)

Other teams use planning poker, where estimates are created off-the-cuff in the sprint planning session. Planning poker can work well if the tasks are small and everyone is familiar with them. However, who ensures the tasks are small enough to estimate?
Some decomposition is required, which probably isn’t free.

One good lesson from planning poker is the use of a Fibonacci series for the estimates. Assuming estimates are off by up to 100%, there is no real difference between an estimate of 3 days and 4 days of work. Using a Fibonacci or exponential sequence builds an increasing error bar into the estimates. Instead, estimate to an ever-increasing scale of effort. My personal preference is a Tee-Shirt size scale — Small, Medium, Large, and Extra-Large. Small is a one-or-two-day task. Medium is a half-week-to-full-week-sized task. Large is a one-to-two-week task. Everything else is extra-large and requires more work before we prioritise.

With one exception, the worst model I've seen is one where "estimates are free" because only the managers or non-developers do them. Assuming that the development team won't need to be consulted on the estimates, is naive. Good estimates will require their input. If the development team isn’t consulted, then they have no stake in the accuracy. When problems arise, the team is less likely to stand behind the estimates if they didn’t have a hand in their construction.

The exception is when estimates are handed down by an outsider as gospel. In this case, we should endeavour to be upfront about the 0% accuracy of the estimate.

Managers and product owners need to remember that estimating is not free. More accurate estimates will require more time and effort. In most cases, everyone is probably better off with very rough estimates and an understanding that work will almost certainly take longer than expected. Like other forms of overhead, the cost of estimates shouldn't be hidden.