Thought Leadership

December 23, 2020

Release Planning - Part 4 Measuring Your Progress (KPI’s)

In the final piece on our release planning series, we cover several important Key Performance Indicators of managing a Software Release Plan on your team.

Executive Summary

This is the final part of our Release Planning articles, visit Part 1, Part 2 and Part 3 for more information.

In Part 4 of Release Planning, we will go into the details for:

  • Measuring the Progress towards your Milestones and Release
  • Warning Signs to look for

Release Key Performance Indicators (KPI’s)

If you want to improve anything, then you must measure and observe where you are at, what is working and what is not. This is true whether you are growing corn in a field, or implementing an enterprise Software Application.

By identifying your Key Performance Indicators (KPI’s), you are better able to tell over time if your progress towards your ultimate goal is on track or not, and make adjustments as needed. Within a Software Development Project or Program there are several KPI’s that we typically use to track our progress and performance.

Release Burn-Down / Burn-Up Chart

After you have Configured the Release and Epic Stories within your Agile Tool of choice and started your Sprints or Progress towards the Release, you will want to be monitoring your Release Burn-Down and Burn-Up Charts. Some Agile Tools combine these into one Report, while others have them separated into two different Reports.  Either way, the Release Report is designed to show you 3 key perspectives:


To give you a very clear idea of what a Release Burn-Down / Burn-Up Charts look like, we are going to use Charts captured from Jira for an actual Client and Agile Development Team. We will cover what is working well and what needs more scrutiny or explanation.

Release Burndown by Milestone

This particular Agile Development Team tracks Epic Stories and Items based on an hourly estimate, rather than using Software Points. Because they do so, there will often be differences between the original estimate and the actual estimate, particularly if an item is shifted from one Team Member to another.

While analyzing these charts there are a couple of other caveats to keep in mind:

  • This Team is frequently working on more than one Release at a time, so you will see wide swings in the amount of Work Completed by Milestone or Sprint, as they shift their focus based on the business needs.
  • Resources did NOT change during the course of these Releases.


Click to enlarge

Observations

  • We can see that right at the start we added a significant amount of New Scope to the Release, easily doubling what the original Plan was.
  • And then based on the Subject Matter Experts working with the Team, another large batch of Scope changes and additions were added at the 3rd Milestone.

Pros:

  • The Team easily adjusted to additional requirements (Scope) discovered by the Business Product Managers.
  • This is why we use Agile Techniques.

Cons:

  • The original projected “End Date” for the Release was obviously way off.
  • This is a MISS of 6 Weeks (3 Sprints).


So, how do we solve this Problem and get better at our Release Estimated End Date? You have to look for Patterns in the Data over Time.

An Approach:

  • As a percentage of prior Projects, what was the Initial Scope in terms of hours or Software Points, compared to the Total Scope?
  • Was there a 10% change, 25%, 50%, 100%, 200%?
  • As you can see we added roughly 288% to the original Scope of this Release.
  • There was simply no way that the original estimate was accurate.
  • Has this happened previously with this Product Management Group representing the Business?  Meaning how accurate are their initial High Level Requirements of what needs to be done?
  • What is their Track Record? If they are always adding 250% to 300% Scope to the Original Requirements, then we need to plan accordingly.
  • Add a Placeholder Epic Story into the Release.
  • Take your initial Scope and set this Epic to be 275% in terms of Estimated Hours or Software Points.
  • This way you will have a more accurate Estimate of the Total Scope, and thus a more realistic Time Estimate for when you will actually complete the Project.
  • Then when the Business starts to add additional Epics or User Stories to the Release, simply reduce your “Placeholder” by the same amount.
  • This way at the end of your Release your Placeholder will be near Zero +/- a small variance.
  • Now take your Actuals and Revise your “Fudge Factor” for the next Release or Project from this specific group. And then Rinse and Repeat.

Click to enlarge

This is another way to show the same information over time (2 Week Sprints).

Regardless of which Chart Style you want to use, what is obvious is that we Underestimated the amount of Scope Change when the project first started.  But now that we know this, we can better plan for the next Project / Release from this Product Management Group. The goal is that over time our Estimates become better and better.


Velocity Chart

Your Velocity Chart is one of the most important KPI’s for every Agile Software Development Team, because it not only lets you know how Productive they are, but also how Predictable the Team is. In addition, if the Team is fairly Predictable, then your Estimates will be more accurate in terms of how long it will take to complete a Release. In order for the Team to be fairly Predictable you want to see a long term Velocity Chart with only a slight variance from Sprint to Sprint. Ideally it should vary no more than 10% from the Moving Average.

However, if the Velocity Chart looks like a roller coaster, then you know this Team needs Help. And typically, it is because they are trying to do too much work, and allowing their Client to add a significant amount of Scope to the individual Sprint, without taking anything out. By doing so, the Team ends up transforming from a Lean / Agile Team, into a “Push” manufacturing model, where everything is being thrown at them and they simply can’t keep up.

Seriously, avoid the Lucile Ball scenario at the Bon-Bon Factory, because it doesn’t end well.

And this is what it looks like in Real Life, when an Agile Team has gone off the rails. Yes, this is a real Team and they are all over the place. What is worse is that this is a fairly large Team, of 10 individuals. And you can see in one Sprint they only completed 5 Software Points worth of work. And if you look at their Sprint by Sprint Burn-Up Charts it becomes obvious, every Sprint they are adding between 35% and 50% more Scope during the course of the Sprint. So, they are allowing themselves to become overwhelmed and taking on way too much work.


Epic Progression

Similar to the Release Burndown Charts, we can also look at individual Epics to see which Epic Stories are either under or over estimated in terms of Scope and thus Time. This is important because it can help tell us which Sub-Teams or Individuals are more accurate in their Estimations; and thus how to plan accordingly.

Human Estimations

Everyone instinctively either tends to underestimate or overestimate things. And the other problem we have is that as the item, process, or project gets larger our “Estimates” become less precise and more inaccurate. If you ask anyone to estimate the weight and dimensions of say an iPhone and they are able to hold it, in general most people will be fairly accurate.

But ask them to estimate the weight and dimensions of a Boeing 737 airliner, and unless they happen to be a pilot or aeronautical engineer, they will be not even close and often by a lot. The key here is that when we are estimating an entire Release, we want to break things down into smaller more manageable Epic Stories. Which will also give us better estimates.

Epic Burndown

The Epic Burndown Chart works very similar to the Release Burndown, as you are tracking progress towards the completion of the Epic Story over multiple Sprints. Likewise, you can see that at times additional Scope is defined or added to this Epic.

The good thing is that after the first couple of Sprints, there is not a large change in the Scope of this Epic Story. The changes in Sprints 4 and 6, once you drill into them further are mainly due to Minor Changes and Bugs that were found by the Subject Matter Experts (SME’s). So, in this case the Team is actually doing a pretty good job of controlling the Scope Changes for this Epic Story.


This next report is just a version of the Epic Burndown Report. But you can easily tell at which points were there major changes or spikes in the Scope of the Epic Story. I’ve circled these in red. Again one of the approaches that you can take is that at the Epic Story Level to create a placeholder User Story that is used to estimate the expected Delta or Change in the overall Scope of the Epic Story. And just like in the Release Plan, you can either do this using Story Points or Hours.

Of course, if you do this at the Epic Level, you can reduce, but not eliminate your Risk Factor at the Release Plan Level. You will still need some Risk Factor at the Release Plan Level to account for Full Integration Testing of the Release and Bugs or Minor Changes that are uncovered once any form of Beta Testing begins.


Control Chart

A Control Chart is used to measure how quickly the Team is able to address or resolve issues over time. This is a measurement of when the Team starts working on an Item and when they complete it. The “Start Time” only begins when it is pulled into a Sprint. Ideally, what you want to see is a relatively flat line or average in Time to Resolution, and also a relatively small Standard Deviation. Naturally, the tighter your Standard Deviation is, it means that the Team is more likely to complete their User Stories or Issues within a predictable period of time.

If you have a very wide Standard Deviation, then that means that:

  • The User Stories are way too large, and need to be broken up into smaller components.
  • The Team is taking on way too much work and trying to accomplish much more than their normal Team Velocity, resulting in Carry Over or unfinished work - thus raising the Time to Completion.
  • The Team is allowing additional work to be added to a Sprint without other work being taken out of equal size.
  • The Team is underestimating the work required.
  • Or the Business SME’s have not “Groomed” the individual User Stories well enough, such that additional requirements or clarification is required during the course of a Sprint. This can easily cause a 3 Point Story to become an 8 or 13 Point Story.

Here is a real world example of a Control Chart. You can see that this Teams long term average for Time to Completion is very good. And their Rolling Average does not deviate by much. In addition, their Standard Deviation is pretty tight with only a few outliers.

Looking at this Control Chart, you can reasonably say that this is a High Performing Team, especially since a Sprint for this team is 2 Weeks or 10 Business Days. This tells us that most of the items that they commit to completing are actually done on time, with only a few exceptions.


Click to enlarge

Software Performance Index Chart

The Software Performance Index Chart was originally developed by Rally Software. What they did was analyze over 19,000 different Agile Software Development Teams using their SaaS product to manage their Projects and Programs.

They selected 4 different criteria for evaluating every Software Team:

  • Productivity
  • Predictability
  • Responsiveness
  • Quality

Each measurement includes a very extensive analysis of your Team’s metrics, and then compares this to 19,000 other Development Teams around the world. Thus you are able to get a benchmark comparison on where your team ranks overall and also in specific areas where your team needs to make specific improvements.

In this example, this team is in the Upper Quartile (top 25%) for both Productivity and Predictability, which is great. However, they are only in the 2nd and 3rd Quartile for Responsiveness and Quality respectively. So, as a leader, if there is one area to focus on - you would probably start with Quality, as it is the lowest rating.

Conclusion

In this blog article, we have covered several important Key Performance Indicators of managing a Software Release and tracking whether the Team(s) are on track or not, as well as what to watch for. This included:

  • Release Burn-Down Charts
  • Release Burn-Up Charts
  • Epic Story Burn-Down Charts
  • Team(s) Velocity Charts
  • Control Charts
  • Software Performance Index Charts

For more info on release planning, visit Part 1, Part 2 and Part 3 of our article series.

We hope that you have enjoyed this Blog Article and found it useful.

Thank you, David Annis.

Case Study from Arkusnexus
David Annis
David is a VP and Agile Coach within ArkusNexus, having served in multiple CIO, VP of Software Development roles. He assists our Sales, Marketing, and Operations Teams on critical initiatives.
dannis@arkusnexus.com
RSS feed
Subscribe to our feed
Back to More ContentMore From this Author
HomeServicesAboutBlogJoin UsContact
Privacy Policy
San Diego:
3065 Beyer Blvd B-2 San Diego, California 92154 +1-619-900-1164

Los Angeles:
530 Technology Drive Suite 100 & 200 Irvine, California CA 92618