ProjectFlightDeck

Concept: My posts on AgileES have thus far focused on “what” and “why”. That is, I’ve described the concepts behind AgileES and the benefits of using ES on Agile projects. It’s now time to shift the focus to “how to”. The next posts describe how to apply ES to Agile projects. There are several steps. Some of the steps, especially the initial ones, are controversial. In explaining each step, I will identify and address the issues and then describe the actions to be taken.

Practice: AgileES uses schedule performance metrics to assess time utilization on projects. In Sprint reviews, the assessment supports process improvements. In Sprint planning sessions, the assessment supports adjustments in the content, velocity, and team membership of the next Sprint. [1]

5. Assess Performance.

For each metric, there are conditions that ground the AgileES assessment, performance zones against which the assessment is done, and threshold values that demarcate the zones. The following post describes how to read the metrics. The next post describes how to apply them in action planning for Sprints.

The ES Metric

The Earned Schedule metric (ES) offers a quick, easily-understood reading on schedule performance. The ES metric has been used on plan-driven projects for some time. But, it was Agile projects that the most compelling representation was developed: the ES Burndown Chart.

Figure 1

The time required for schedule performance assessment with ES is a small increment on normal Agile practice. In some ways, it even speeds the assessment by accommodating threshold values. They serve as “triggers” that signal the need for action.

Conditions:

• In the ES Burndown Chart, the vertical scale represents the total number of Sprints remaining in the project. The horizontal scale represents the end of each elapsed Sprint.
• The Sprint length equals the number of time periods allocated for a Sprint, e.g., 2 weeks.
• The Sprint length is the same for all Sprints in the project.
• The Baseline ES Burndown line represents the estimated track for time utilization, running straight from the end of the first Sprint to the end of the last planned Sprint.
• The ES Burndown line shows how much time has actually been earned, extending from the first Sprint to the last scheduled Sprint.
• The Baseline ES Burndown line equals the total remaining schedule decremented by the number of elapsed Sprints. [2]
• The ES Burndown equals the total remaining schedule decremented by the amount of schedule earned up to the end of each Sprint.
• The labels in Figure 1 use Sprint end dates, rather than Sprint numbers on the horizontal axis. Stakeholders say that dates provide more context for the chart than durations.

Performance Zones: If the ES Burndown line is above the Baseline Burndown line, schedule performance lags the estimate. If the ES Burndown line is below the Baseline Burndown line, schedule performance is ahead of the estimate. If performance consistently lags the estimate, the total number of Sprints will grow greater than the number of planned Sprints.

Threshold Values: The threshold value is the amount of schedule estimated to be earned by the end of each Sprint. Schedule performance is assessed as early, on-time, or late based on whether the ES Burndown line is below, on, or above the Baseline ES Burndown line.

SPIt

SPIt is a measure of schedule performance efficiency. For starters, it indicates whether or not time is being earned according to plan. Beyond that, it provides a way to gauge how well or how poorly time is being used on the project.

Figure 2

Conditions:

• The SPIt is calculated taking account of Contingency. That is, the SPIt measures performance against the Baseline Schedule.
• Performance zones are calculated as a percentage allowance for Contingency, for Reserve, and beyond (both positive and negative).
• The Baseline Finish Date includes Contingency. When Reserve days are added, the date is referred to as the Committed Finish Date. When both Contingency and Reserve are removed, the date is called the “Deadline”.

Performance Zones: The amount of Contingency implies the range in which the SPIt can vary without significant risk to the timeline. Once the deviation exceeds the Contingency allowance, the timeline is at risk, but it may still be feasible. If the deviation exceeds both Contingency and Reserve, the original timeline is no longer feasible. Deviations can be above or below planned amounts. [3]

Figure 2 illustrates the performance zones. It applies generically to SPIt, EACt, ECDt, and RoD. The zones are demarcated by threshold values. The threshold values vary by metric.

Threshold Values: The efficiency of schedule performance is assessed against specific threshold values for the SPIt. Because the SPIt is a ratio, rather than an absolute amount, the thresholds apply equally to projects of all sizes.

To ease communication, ProjectFlightDeck associates a Red-Yellow-Green label with each performance level. The label helps the project team and stakeholders understand the status of schedule performance and the motivation for action plans related to the status.

Specific threshold values are calculated when the baseline is set. The values are derived from allowances for Contingency and Reserve. For example, suppose that the Contingency allowance is 10% and the Reserve allowance is 5%. The negative thresholds would be as follows:

Figure 3

If the SPIt is below .85, performance efficiency is very poor and is assigned a Red status. When the SPIt is above .85 but below .9, efficiency is poor, and it receives a Yellow label. If the SPIt is between .9 and 1.0, the performance is allocated a Green status. [4]

The Red-Yellow-Green labels are also applied to SPIt greater than 1.0.

Figure 4

If the SPIt is greater than 1.0 but less than 1.1, performance is high, and a Green status is awarded. If the SPIt is greater than 1.1 but less than 1.15, the performance rating is suspiciously high and is therefore deemed Poor. A Yellow label is assigned. For SPIt greater than 1.15, performance is implausibly high, and is considered Very Poor. A Red status is applied.

EACt and ECDt

The EACt and ECDt are estimates for the completion of the project. For simplicity, the following discussion is framed only in terms of the EACt. Analogous comments apply to ECDt because it is simply the Start Date plus EACt.

Figure 5

Conditions:

• As mentioned above, the Baseline Finish Date includes Contingency. When Reserve days added, the date is referred to as the Committed Finish Date. When both Contingency and Reserve are removed, the date is called the “Deadline”.
• Risks that might cause delay are addressed by adding Contingency and Reserve allowances to the Deadline. [5]
• Risks that lead to early delivery are addressed by subtracting Contingency and Reserve allowances from the Deadline.

Performance Zones: Figure 5 illustrates the structure of performance zones based on the Deadline, Contingency, and Reserve.

Threshold Values: Thresholds for EACt are expressed in Sprints. The number of Sprints is calculated from allowances for Contingency and Reserve. For example, suppose that the Contingency allowance is 10% of the time allotted up to and including the Deadline, and the Reserve allowance is an additional 5%. If the Deadline is Sprint 8, Contingency adds (rounded) one Sprint and Reserve one Sprint.

Figure 6 expresses the thresholds for the example.

 
Figure 6

If the EACt is later than the Deadline, but earlier than the Deadline plus Contingency, the outlook for meeting the Baseline Finish Date is good, and it is labelled as Green. If the estimate is earlier than the Deadline but later than the Deadline less Contingency, the outlook for the baseline is also good, and the project is again labelled as Green.

If the EACt exceeds the Deadline plus Contingency but is less than the total including Reserve, the project is not on track to meet the Baseline Finish Date, but it should still finish as committed. So, it is labelled as Yellow.

If the EACt is earlier than the Deadline less early Contingency, the project is again not on track to complete as baselined and is labelled as Yellow. In this case, it is Yellow not because the Baseline Finish Date is likely to be exceeded but because the plan is unsound. After all, an allowance was made for early
delivery, and the estimate makes it appear that something was mistaken. Either the risks relevant to early delivery were incorrectly identified, or the Contingency allowance for early delivery was too small.

Finally, if the EACt exceeds the Reserve Zone, the project will not meet the Committed Date, even if both Contingency and Reserve times are used. The project is labelled as Red. By the same token, if the EACt is earlier than Deadline less Contingency and Reserve, it appears that there are serious flaws in the plan, and it is labelled as Red.

Rate of Discovery

The Rate of Discovery measures the degree of change in the Release Point count. It is the ratio of the current number of Planned Release Points to the original, baselined number of points.

Figure 7

Conditions:

• The RoD does not express time utilization directly.
• It is useful in assessing whether or not the original baseline was sound.
• The Minimum is the Planned Release Point total less Contingency and Reserve.

Performance Zones: If the current number of Planned Release Points (PRP) is within the Baseline total, the plan is considered to be sound. Once the Contingency allowance is exceeded, the plan becomes questionable, and it is labelled as Yellow.

If the PRP count exceeds both the Contingency and Reserve amounts, the plan is in serious trouble, and it is labelled as Red.

Remember, deviations can be positive or negative. That is, the current PRP count can either exceed both Contingency and Reserve, or it can drop below the allowance for both Contingency and Reserve.

Threshold Values: Threshold values for the RoD are represented as either % or as a number of  Release Points using the aforesaid ratio. Suppose that, in the example already cited, the Minimum PRP count (i.e., excluding Contingency and Reserve) is 3500. Say also that Contingency is 10% and Reserve is 5%.

The (rounded) threshold values and labels for the RoD are illustrated in Figure 8.

Figure 8

If the number of PRPs is within Contingency (positive or negative), the outlook for staying within that total is good. The status is labelled as Green. If the number exceeds Contingency but is within Reserve (again, positive or negative), the outlook for staying within the baseline PRP total is poor. The status is labelled as Yellow. Beyond Contingency and Reserve, the outlook for delivering the planned number of Release Points is very poor. The status is abelled as Red.

Advanced Metrics

The metrics cited thus far are sufficient for many Agile projects. In especially demanding situations, additional ES metrics are available. 

The To-Complete Schedule Performance Index (TSPI) indicates the efficiency that will be required to complete the project on time. Studies have shown that if the TSPI exceeds 1.1, the project is unrecoverable. [6]

Statistical analysis of the schedule performance yields probable high and low estimates for project duration. 

Both metrics will be discussed in a future post on advanced AgileES metrics.

[1] ES performance metrics can be used to support process improvement in a current project. They can also be collected into a repository to support improvement across projects. The focus here, however, is to support planning the next Sprint.

[2] The total number of Sprints follows from the total amount of work divided by the project’s estimated velocity.

[3] Reasons for including deviations both above and below planned amounts are mentioned below. Further explanation can be found in earlier posts on SPIt, EACt, and ECDt.

[4] In the Green zone, an SPIt = 1.0 is not often seen in practice, as it indicates efficiency is exactly aligned with expectations