The Problem with Agile Estimation

Many thanks to Satvik Sharma, John Folder, Mat Lawrence, Dave Elkan, Henri Seymour, and Robin D Bailey for contributing their expertise and experience to this article.

Estimation is a common challenge for agile software development teams. The anticipated size and complexity of a task is anything but objective; what is simple for one person may not be for another. Story points have become the go-to measure to estimate the effort involved in completing a task, and are often used to gauge performance. But is there real value in that, and what are the risks of relying too heavily on velocity as a guide?

Agile estimation

As humans, we are generally terrible at accurately measuring big things in units like time, distance, or in this case, complexity. However, we are great at making relative comparisons - we can tell if something is bigger, smaller, or the same size as something else. This is where story points come in. Story points are a way to estimate relative effort for a task. They are not objective and can fluctuate depending on the team's experience and shared reference points. However, the longer a team works together, the more effective they become at relative sizing.

The teams that I coach have all experienced challenges with user story estimation. The historical data tells us that once a story exceeds 5 story-points, the variability in delivery expands. Typically, the more the estimate exceeds 5 points, the more the delivery varies from the estimate.

Robin D Bailey, Agile Coach, GoSourcing

Scale of reference

While story points are useful as an abstraction for planning and estimating, they should not be over-analyzed. In a newly formed team, story points are likely to fluctuate significantly, but there can be more confidence in the reliability of estimations in a long-running team who have completed many releases together. Two different teams, however, will have different scales of reference.

At a company level, the main value I used to seek with story points was to understand any systemic problems. For example, back when Atlassian released to Server quarterly, the sprints before a release would blow out and fail to meet the usual level of story point completion. The root cause turned out to be a massive spike in critical bugs uncovered by quality blitz testing. By performing better testing earlier and more regularly we spread the load and also helped to de-risk the releases. It sounds simple looking back but it was new knowledge for our teams at the time that needed to be uncovered.

Mat Lawrence, COO, Easy Agile

Even with well-established teams, velocity can be affected by factors like heightened complexity with dependencies scheduled together, or even just the average number of story points per ticket. If a team has scheduled a lot of low-complexity tickets, their process might not handle the throughput required. Alternatively having fewer high-complexity tickets could drastically increase the effort required by other team members to review the work. Either situation could affect velocity, but both represent bottlenecks.

Any measured change in velocity could be due to a number of other factors, like capacity shifting through changes in headcount with team members being absent due to illness or planned leave. The reality is that the environment is rarely sterile and controlled.

Relative velocity

Many organizations may feel tempted to report on story points, and velocity reports are readily available in Jira. Still, they should be viewed with caution if they’re being used in a ‘team of teams’ context such as across an Agile Release Train. The different scales of reference across teams can make story points meaningless; what one team considers to be a 8-point task may be a 3-point task for another.

Seeking value

However, story points are still a valuable tool when used appropriately. Reporting story points to the team using them and providing insights into their unique trends could help them gain more self-awareness and avoid common pitfalls. Teams who are seeking to improve how they’re working may wish to monitor their velocity over time as they implement new strategies.

Certainly, teams working together over an extended period will come to a shared understanding of what a 3 story point task feels like to them. And there is value in the discussion and exploration that is needed to get to that point of shared understanding. The case for 8 story points as opposed to 3 may reveal a complexity that had not been considered, or it may reveal a new perspective that helps the work be broken down more effectively. It could also question whether the work is worth pursuing at all, and highlight that a new approach is needed.

The value of story points for me (as a Developer and a Founder) is the conversations where the issue is discussed by people with diverse perspectives. Velocity is only relatively accurate in long-run teams with high retention.

Dave Elkan, Co-CEO, Easy Agile

At a company level, story points can be used to understand systemic problems by monitoring trends over time. While this reporting might not provide an objective measure, it can provide insights into progress across an Agile Release Train. However, using story point completion as a measure of individual or team performance should be viewed with considerable caution.

Story points are a useful estimation tool for comparing relative effort, but they depend on shared points of reference, and different teams will have different scales. Even established teams may notice velocity changes over time. For this reason, and while velocity reporting can provide insights into the team's progress, it must be remembered that story points were designed for an estimation of effort, rather than a measure. And at the end of the day, we’re in the business of producing great software, not great estimates.