Wednesday, July 1, 2009

Efficiency - A Must Have Agile Metric As Important As Velocity

When I first picked up Agile development, I absolutely got hooked on the simple yet powerful way it described how to track your team's progress on projects with REAL data. Before I implemented Agile, I can't remember how many times business stakeholders or execs have tried the Developer-Used-for-My-Benefit approach, only for it to bite the organization hard in the nuts later. I call it the D-U-M-B approach for short.

CEO: "How's the project coming along Pat?"
Pat the Developer: "Uhhh - going well. I can crank this out in no time - easy!"
CEO: "Excellent - I'm going to give you a huge bonus and tell sales and marketing to start pushing it."

Uh-oh... I think we all know the end of this all too well-known scenario... :(

Gathering actual hard data while your project is running is the only practical way to measure the completion of your project - and Agile processes are all about collecting and charting this hard data, making it visible to the entire organization so there is never any question where a project status is at!

One of the core hard data metrics within Agile practices is the team's velocity - the amount of work the team is able to complete in a given amount of time/iteration. Using the team's last velocity, the team can predict what their velocity will be for the next iteration. Charting the team's velocity for the project will allow you to make some smart data-based decisions as well by analyzing their velocity trends. For tips on providing your Agile team the best chance on achieving a high, stable velocity, read this later!

However, even with all the great info velocity charts provide, I found myself wanting something more than just the velocity chart. If resource days are limited in an iteration due to holidays, vacations, or sick days, the team's velocity can drop significantly. It makes complete sense of course - fewer working days = reduced work completed for that iteration. But, how could I prove that the dip in velocity was just due to the limited number of resource days? How could I prove that the team worked just as hard as they would have if they were fully staffed for the entire iteration?

Enter the efficiency metric. The efficiency metric allows you to determine how efficient the team was for each iteration, but in a way that it factors out the number of resource days that were available during that iteration. Thus, even if there was an iteration when the entire team took a few days off, where the Velocity Chart would show an obvious dip in the amount of work accomplished, you can rely on the efficiency chart to show if they were just as efficient working that iteration from the previous ones. Below is an example Team Velocity and Efficiency chart based on data collected for the same iterations.


You can see that even though the team's velocity dropped signficantly during iteration 14 because half of the team was out due to the flu for a few days, the efficiency chart shows that the team remained consistent in its effort to get work completed - which tells you that during that iteration, a dip in velocity was entirely due to the lack of resources that were available that iteration.

COOL BEANS GRANT - I GOT IT!! So how do you calculate a team's efficiency?


E = V/R

E=Efficiency for the iteration
V=Velocity for the iteration
R=Total number of person days worked on the team for the Iteration

Hopefully, you're already tracking velocity (V) for completed iterations. You'll also need to start tracking total actual person days (R) worked for your completed iterations as well. For example, if you have a team of 5 and your iteration is 10 days, then the maximum possible R value for your team would be 5 x 10 = 50 days. The actual R value would thus be the maximum possible R value for your team minus the number of absent days per team member during the iteration (e.g. 50 max - 1 sick day = 49). Tip: Plug all of your numbers in a spreadsheet and setup formulas to do your calculations for you! Efficiency is auto-calculated for me once I've entered velocity and resource days for the completed iteration into my spreadsheet. After you've got your efficiency calculated, producing your Efficiency Chart is as simple as producing your Velocity Chart - just chart it over each iteration!

Below is an example spreadsheet showing calculation of Efficiency using Velocity and number of resource days.



Now that you have your efficiency chart at hand, viewing it in combination with your velocity chart unveils much more information about your iterations than the velocity chart would have provided alone. See a dip in your velocity chart but efficiency was stable? Resource days must have been limited that iteration. Like velocity, the trend for the efficiency chart for teams heading in the right direction should be very similar to that of the velocity chart: new teams will likely have a few beginning iterations where efficiency may be erratic when they start settling into the project and learning to work together. But, eventually, you should see a peak and plateau, representing that the team's current maximum efficiency has been achieved. Note that efficiency ratings are not the same between different teams - so don't get all twisted up if one team seems to have a smaller efficiency than another! To me, a successful efficiency chart is not about how high it reaches, but about if it has reached a peak and consistent plateau.

Having efficiency also allows you to obtain what I'm calling the benchmark velocity for your next iteration. If you're using velocity charts right now, I'm sure you've been told that the best way to determine how much work the team can do in the next iteration is the velocity from the last iteration. I beg to differ!! With the efficiency metric of the last iteration in hand, simply multiply it with the expected number of resource days for the next iteration, and you've got the benchmark velocity - the velocity your team would hit if they work just as efficiently as they did this past iteration.

v = E*r

v = benchmark velocity for next iteration
E = efficiency from just completed iteration
r = Total number of expected resource days available for upcoming Iteration

Below is a spreadsheet showing the simple formula to calculate the benchmark velocity.



Grant - why do you call v the "benchmark" velocity instead of the "expected" velocity? After all, isn't that what it really is? When the team needs to decide what their upcoming velocity commitment should be, there are more things to consider than just the pure numbers from the previous iteration. For example, maybe your office is scheduled to move to another building next week. Even though everyone is still working those days, you shouldn't expect them to be just as efficient as they were last iteration!! Thus, you and the team should expect to have a lower velocity for the upcoming iteration than what the "benchmark" velocity is calculated to be... hence the term "benchmark" instead of "expected." Saves you the trouble in also explaining to project stakeholders, customers, and execs on why the "expected" velocity wasn't reached as well if you had called it that!

Allowing your team to view the benchmark velocity and then talk through any outside factors that could impact the upcoming velocity for better or worse should allow the team to settle upon a velocity number that they can commit to. A fully self-organized, steady team should more often than not accept the calculated team benchmark velocity as their commitment. If your team is consistently committing to numbers that are drastically different from the calculated benchmark, it's a clear signal that they are still learning how to stabilize their development efforts for the project. Don't be surprised if the team decides to commit to a velocity that is higher than the benchmark as well!

The efficiency metric is now a must have for all of my Agile run projects - it's my hope that your team and organization will also benefit from using this metric as well!

7 comments:

  1. Interesting idea. It sounds like the efficiency chart is a great one to have when reviewing progress with stakeholders and execs. I am usually against tracking more than plain old velocity. But this one seems to be a very useful one that does not add too much progess goo.

    ReplyDelete
  2. Grant - why not just use the trailing average of the last three iterations? It smooths out any bumps. In addition I usually just get teams to account for stat holiday's, vacations etc. Seems a little easier than creating another metric.

    ReplyDelete
  3. Mark - taking the average of the last 3 iterations could certainly stand in for the benchmark calculation that I've obtained above, but does the average of the last 3 iterations really give you the latest data to work with? It's my belief that the last iteration often has the best data to project the work that can be accomplished in the next iteration. Either way, I can see both approaches working for obtaining a benchmark velocity. However, I still believe seeing the efficiency chart together with the velocity charts offers more insight into your team's capabilities without adding much overhead - literally takes me 5 minutes a week to obtain appropriate data and calculate efficiency.

    ReplyDelete
  4. Grant

    Interesting idea, but how do you prevent the team from gaming the efficiency measure by subtly inflating the story point estimate for each story over time?

    regards

    Julian

    ReplyDelete
  5. Hi Julian,

    Good question! I also asked that of myself when I was first picking up Agile as well...

    Gaming the efficiency measure would be the same as trying to game the velocity metric each iteration as well.

    Thus, the same methods in preventing the gaming of velocity should also hold true to prevent the gaming of your efficiency metric.

    Some of those methods include making sure your team has a reference point in mind for what a story point represents at the beginning of the project.

    For example, maybe a story in wiring up the login page for an app is 3 points. Your team should think about all stories they estimate from that point forward in relation to that story. It should become blatantly obvious then when a simple text change story suddenly ends up being estimated as a 3 point story!!

    Playing estimation games like planning poker or shooting estimates with their fingers on the count of three are also effective means of ensuring that no ONE team member is responsible for inflating estimates. If estimates widely differ in these types of games, your team should talk about the different estimates until they all agree upon a proper estimate.

    Lastly - if you think your team is inflating their estimates, ask them!! Perhaps there is a good reason why estimates look like they have inflated over time. Maybe the code has incurred a lot of technical debt that needs to be fixed - a good indication that some refactoring could go a long way in helping to reduce future story estimates...

    ReplyDelete
  6. Grant

    All excellent comments in terms of measuring from the perspective of the embedded project manager. However I am looking for ways to apply metrics to teams "from the outside", not least because some of those teams may be in outsourced companies. My thinking is that as soon as we use a velocity-based measure for external comparison then we are building in subtle pressure for story points escalation.

    My current feeling is that for that perspective from the customer perspective we need to measure something like value delivery rate (e.g. customer value points per feature * features delivered / cost).

    ReplyDelete
  7. Julian

    Interesting search you got going here. Let's say you do find another metric to measure something like value delivery rate that you show the client. Wouldn't they be subtly pushing you to improve that metric as well over time? Seems to me that no matter what metric we're putting out there, customers will probably want or demand better from them! Perhaps it's part of our job to inform the client and our teams that it's not always about improving our metrics, but meeting them. For example, perhaps the goal we should be setting for our teams should be about stabilizing velocity, not trying to move it through the roof!

    ReplyDelete