It’s not you, it’s your estimates
Or — you’re not a bad developer, you’re a bad estimator; and that’s ok.
Have you ever had it where you’re in the stand-up/sprint planning/by the water cooler (or anywhere really) and a manager or other higher-up are asking you the dreaded question — “how long will that take?”
After your heart starts beating again, you might be able to pluck out some numbers, and let them know that whatever you’re working to is going to take “6–8 weeks”, and before you know that time has passed and you’ve not achieved what you wanted.
So what went wrong?
The first thought we often have, whether due to comments from colleagues or our own internal dialogue, is this:
If I had been a better developer, I would have delivered this before my deadline!
But let’s switch around the perspective. If you could go back in time and give an accurate estimate 6–8 weeks ago, would you have felt as bad? Would you have attributed your success to being a great developer or would you acknowledge your supreme estimation abilities?
Estimation and developer skills go hand in hand
Obviously, it’s not one or the other. But often I find that my estimation skill lags behind my other abilities to a point where I don’t realise it’s my weakest skill.
This lack of perspective can lead to some serious ‘feel-bads’. We should instead, when we miss a deadline, ask ourselves if we have simply misevaluated how long the task will take us. It’s that simple.
Where it all goes wrong
Firstly, before we improve our skills in estimation we have to understand how we’ve actually been developing. Often times we tackle issues we’re not immediately familiar with, which is called Spike Development
Spiking and non-spiking development
Wikipedia defines a spike in a sprint as such:
As a way to familiarize the team with new hardware or software
Think back to how often you’ve worked on something in a sprint, and not quite known what it was you were doing. More than likely you were spiking without knowing about it.
By this point you may have realised the issue with estimating a spike — if you’re unfamiliar with new hardware or software, how are you going to know how long it will take? So how do you estimate these tasks? We are going to need some general, universal, insights to make productive estimates.
The easiest parts take the least time; the hardest the most.
The way I like to apply this to spike development, is to realise that 80% of your gut-feel estimate likely covers only the easiest 20% of the work that you can imagine (the remaining 80% is not yet known to your gut!). Unfortunately, under this model your work likely takes 5 times longer to finish than you first think! Do you feel better for missing targets now?
Let’s break it down with an example. Say you have to add a new feature to your web app, and looking at it you make a quick judgement that it’s about 3 story points or, if you use time estimates, that might mean 1 week of development. Then the full “farm to fork” time taken will be (in the worst case scenario) 5 weeks of development, or the equivalent of 15 story points. This is a quick and dirty way of safeguarding against underestimation.
This pattern of larger easy parts taking longer than the remaining hard portion of work, is sometimes referred to as the law of the vital few, The Pareto Principle, or the 80/20 rule, although my intuition on spike development is not backed up in the same way by statistics.
Inflating the estimate while spiking by 5 times might feel daunting to suggest to someone, but it provides a good safety net. The only remedy to this is to timebox the work when you’re spiking, so that you don’t take longer on a task than you have.
False average equivalency problem
Not all estimates are made equally accurate. It’s most important to bear this in mind when you’re not spiking, and it feels like you might have a good idea of how long you expect work to take.
Some estimates are more accurate than others. However there is a temptation to use the average time to finish a task as the estimate. The name of this section comes from the false equivalency that the average time to complete a task is a good estimate. In fact the average is only accurate 50% of the time. To help illustrate, take a look at the below graph.
If you pin your estimates at the average time (the middle line) a task has taken in the past, the peak of the blue line, you only have 50% confidence. The orange line is the cumulative confidence. This means ‘how confident am I that the task will take less time than my estimate’? As you move to the left or right of the middle line, the estimate is for more (right) or less (left) time than the average.
If you don’t yet see the power of this perspective, imagine rather than being satisfied with 50% confidence (the middle line), you want 95% confidence. To work this out, draw a line from where the y-axis says 95, until it reaches the orange line. Drop it down to the x-axis, and the ‘difference’ (albeit in this example abstractly) would be how much to increase your estimate by.
Will this mean your tasks now take longer? No — refer back to the our initial premise. You’re just communicated a more likely outcome. If your average estimate for a task is for 1 day, or 1 story point (at a 50% confidence), maybe adding a day to it will give you a 95% confidence or taking less time.
And as you get better at estimating, the blue line becomes a sharper peak; you get more confidence in achieving your estimates with only a small increase over your average.
This technique is a way of under committing and over performing — what Scotty achieved perfectly in Star Trek. That’s also why this is also known as the Scotty Factor. However, if you constantly inflate your estimates without telling people why, and how it works this can backfire with people assuming you’ll have things done early.
Note, this is a more nuanced version of our advice on spiking taking ~5 times as long. It requires having a better idea of how long your task takes, but it’s arguably the same idea, just better informed.
Always be finished
This next advice comes from the art world. When working on a piece of art, a common piece of advice is to always be finished. What does this mean? Bear with me, it might feel a little abstract for a bit:
Say you’re drawing a face. One approach is to full draw the eye first. Then draw the nose completely. Next fill in the lips until they’re finished. And so on until you’ve assembled the components of a full face.
The other option is to make an outline of the face. Then sketch in the all the features at the same time, adding more details little by little, until a finished face emerges. At each stage, the whole face is at different stages of finished, that is you can stop at the outline stage and you have a finished outline. Likewise with the sketch stage, inking and colouring etc.
Are you following the two approaches so far?
In one approach each component of the final piece is completed in turn, one after the other. In the other, you have to move between partially finished stages of your task before you’re done. Which one is correct while spiking? Whichever one you you have estimated for.
Sometimes ‘Always Be Finished’ means that you end your task incomplete, but with half of your the components completed and in a state that at last those components can be released (i.e. Finished). Othertimes it means everything is maybe 80% complete, such that even if nothing is done, it’s Finished enough to be released. The last 20% can be moved to another ticket for another sprint/day/task.
And that’s the trick here. The aim is that, even if you’re only at 80% by the time your estimate is up, what you deliver should be considered finished and releasable. It feels much better to release something than nothing, so much so I will be keeping this in mind for all my future work.
It’s also critical to remember having tests is essential to being finished, as well as anything else in you or your team’s definition of done. If you plan to write your tests last, you can’t be finished until you’re 100% done, and as discussed that’s a dangerous place to be if you want to safely release anything on time.
This trick works even when not spiking, but it’s all the more important when you think you might not be done by the time you need to be Finished.
Biting off more than you can chew
There’s a very serious temptation to estimate work at face value, without challenging how it’s presented or how large it is. Failure to break up a piece of work can be the difference between completing the task and not. This simple piece of advice can be one of the most powerful tools in your belt. It also allows you to always be finishing, as we discussed previously.
Not breaking up a task has two main problems: Firstly, if you are constantly chasing to complete a task, you can end up with no work to present, or code to be reviewed for multiple sprints. This can present a moving target if the requirements shift, or much worse, if the code base shifts underneath your code.
Secondly tackling large pieces of work is much harder, much more than just a sum of work if it had been split down. A person’s Working Memory has a limited capacity, and it can be overwhelming to work on a task with too many moving parts — this has concerns for code quality, testability etc.
Finally, both of these are demoralising, and make you less motivated overall, increasing the length of time it takes to complete tasks.
This section naturally follows onto the next one, Rabbit Holes.
Rabbit holes
This pitfall is one of the largest, at least for me: deciding on an approach and mistakenly sticking to it, despite it not initially working out.
Ola Sendecka tells a long story, related via an equally long video, that covers this problem. While you watch it, imagine that you are Alice, and that your manager or supervisor is Alice’s sister, waiting longer and longer while Alice promises she’s ‘almost there’ after your initial estimate:
At 16:24 in the video, Ola summarises what has gone wrong:
- Underestimate the problem
- Fixed on solving only the surface problem
- Continues after a number of failures
- Convinced the final solution is close
- Cannot give up
A large number of these points fall foul of the sunk cost fallacy. You have estimated a fixed task, but on failing, rather than re-evaluating if your original estimate is correct you persist because of the effort you’ve already ‘invested’. In reality, you haven’t made progress to anything finished so nothing is invested, and nothing lost if you re-evaluate.
Alternatively, if you had stopped early and reconsidered your approach, you may only have needed a small extension on your estimate. Often you can make a much better estimate after having started.
Finally, you could also have broken down the task into chunks that can be finished separately, and ended your task on time albeit partly complete.
This investment in this sunkcost fallacy, really plays with your emotions. You’ve put so much of yourself into the task, that psychologically it becomes part of you. It can be very deceptive in this way. When stopping ‘early’ to re-evaluate, there’s nothing wrong about experiencing some very raw feelings of frustration and embarrassment.
Lack of Communication
The last section feeds directly into this one. When one is developing, especially when there is the risk of rabbit holes, bad requirements, spiking or other potential delays, you have to be able to update your estimates as you work.
Stand-ups are a great place to bring this up, especially if you’re following the 3 point checklist of: What did I do yesterday, What am I doing today, What is my blocker. The last point is where you bring this up with the team.
This behaviour will only take you so far, and the level-up moment is when you realise you can talk to your team outside of stand-ups!
Is someone waiting on your work to finish? Are you waiting on someone else? Does your manager or supervisor want an update ahead of time?
Not only does this help the team work efficiently, but it signals others that you’re conscious of how your work fits in with others. Flagging up delays early and often can make sure you look good in the face of potentially having to explain a bad estimation.
Not only that, but revealing the big secret that you’re delayed in being Finished, can be a big relief and a weight off your mind. It can also help you do the re-evaluating of the rabbit hole section, or looking at what can be considered done, and then released or published.
How to improve on your estimates
Most of this article has focused on the emotional effects of underestimating and spotting where estimates go wrong.
Taking the pain points above, where the mistakes happen, into consideration there are a number of behaviours that help you estimate better:
Think before
Are you spiking? Do you need to timebox? Have you considered if your gut estimate only covers 20%? Can you overestimate past your average estimation? Are there potential rabbit holes in your code, and if so can you break up the task?
Holistically consider all the activities required to go from concept to completion. Visualise the ‘farm-to-fork’ that was mentioned earlier. This includes getting requirements, investigating the work involved, the initial attempt, writing tests, the pull request and code review, any other tests that have to be passed.
Think during
Communication is key.
Even if all you say is “I think this might take longer than I thought it might” with no numerical change to the estimate. That will ready someone for when you can give a numerical change to the estimate, and make you feel at ease that this delay is no longer secret.
Make sure you give regular estimates, both in and outside of meetings, when things change. This gives information, but also builds trust that you are on top of your work, and conscious of the time taken.
Think after
Look back at the work you’ve done, think not where did you trip up, but where was your estimate most wrong — and why. What principle above did we not follow on our way to estimating a task or — while were were working on the task — what did we not communicate?
Final Story
I’d like to end this article with a real life story, where I fell foul of these problems, to show how badly things can go if you are not careful.
In this example, a new manager had joined the project to look into why my estimates were always short and I had so many missed targets. My work had stretched over 3 sprints, and I was being asked for an explanation:
Manager: Have you tried refactoring, before tackling this big task?
Me: This work is the refactoring.
Manager: Well, did you try breaking it down into multiple smaller tasks?
Me: Yes, but the code is so coupled I could only do so much.
Have you noticed, until now I had an answer to the questions? As bad as I felt about having missed the deadlines, and been, I had done my work on trying to simplify my problem. But, wait, my mistake becomes obvious next:
Manager: Well if it’s so coupled and hard to work on, why didn’t you tell us before now?
Cue embarrassed non-answer from myself. Exit stage right. End of scene.
Had I explained how complicated the task was early on, I could have gotten help or even had my work re-prioritised and moved on to something else more productive. Instead it took me three weeks to learn the lesson of communicating.
After reflecting, and discussing with colleagues, I was able to write up this guide on estimation. It’s not easy owning up to our mistakes, but it feels better once you have and helps you learn and progress.
