ProjectFlightDeck

Concept: ES Statistical Analysis factors historical variation into the calculation of duration estimates. By analyzing the project’s previous schedule performance, ES stats identify the high and low bounds for the estimate at completion for time. From high/low bounds and target dates, we can determine how well or poorly the project will perform. Thresholds guide the assignment of Green-Yellow-Red status.

Chart 1

Practice: Project B had a 45 week baseline (including 3 weeks of contingency).  There were an additional 2 weeks of reserve. We tracked the estimate at completion each week  and began using ES statistical analysis to steer the project starting at week 30. Chart 1 shows the results through the first 36 periods, stopping at week 36 because we reached a major decision point. This blog posting describes the challenges and our responses up to that point.

On Project B, we used dates rather than durations in our analysis and reporting. Dates and durations are functionally equivalent in this context. To perform the analysis, we used the graphical representation in Chart 1 and the specific date values in Table 1.

 Table 1

At period 30, the project appeared to be in serious trouble. The forecasts were well beyond the end dates reflected in the bare plan, contingency, and reserve. The estimates breached both high and low allowances for uncertainty. The only good news was that the nominal forecast appeared to be within contingency and reserve.

Rather than immediately assigning a Red status, we decided to wait. We wanted to see if a pattern was developing now that there was sufficient information to reliably drive statistical analysis. We assigned the project a Yellow status to indicate that there was concern but not a crisis.

By period 33, we had seen enough. The high forecast was still beyond the bare plan, contingency, and reserve dates. The low forecast was still earlier than Early Contingency, but it was no longer before Early Reserve. Guided by Table 2 and using the most serious status as the default value, we assigned the project a Red status.

We did not, however, initiate a re-baseline. The decision to defer the re-baseline was based on two factors. First, the low forecast showed that there was room to recover. Second, the trend in the high forecast was downward—performance appeared to be improving slowly over time.

Table 2

By period 36, we reached a major decision point: was the project still in Red status, and if so, must we re-baseline the plan?

We decided that the project was no longer in Red status: the high forecast no longer exceeded Reserve, and the low forecast still fell within the Reserve. As indicated by Table 1, the project was in Yellow status.

More important, we decided that we would not re-baseline the plan. High and low estimates were both within Reserve. The high forecast had consistently dropped over 7 successive periods. The low forecast was rising but slowly. The trends were re-assuring. We believed that the bare plan would probably not be met and that contingency would probably be drained, but we also concluded that we would make the committed date. We proceeded with the original bare plan, contingency, and reserve.

Post script. How did it turn out? The project finished beyond the bare plan and contingency but within reserve. The commitment was met, and we avoided the turmoil that is normally associated with a re-baseline.

Concept: ES Statistical Analysis factors historical variation into the calculation of duration estimates. By analyzing the project’s previous schedule performance, ES stats identify the high and low bounds for the estimate at completion for time. From high/low bounds and target dates, we can determine how well or poorly the project will perform. Thresholds guide the assignment of Green-Yellow-Red status.

Diagram 1

Practice: At ProjectFlightDeck, we use statistical analysis to steer the project. The stats help us assess allowances for uncertainty.  How well the project is doing versus uncertainty allowances is a key factor in determining its status. As explained in earlier posts (June 15, 18 and 30, 2015), the allowances for uncertainty consist of Contingency and Reserve. 

• Types of Uncertainty: Contingency is the response to predictable uncertainty and Reserve to uncertainty that cannot be predicted. Historically, the emphasis has been on uncertainties that cause delays. But, there are also uncertainties that result in early delivery.
  
• Late and Early Allowances: Uncertainties that cause delays are addressed by “late” Contingency and Reserve allowances. Uncertainties that lead to early delivery are addressed by “early” Contingency and Reserve allowances.
  
• Illustration: See Diagram 1 for an illustration of Late/Early Contingency and Reserve.

To perform the assessment, we first run statistical analysis on the “bare” plan, i.e., one stripped of contingency and reserve. Among the resulting stats, we focus on the high and low Estimates at Completion for time (EACt).

Early in the project, when the amount of performance data is minimal, the natural variation in performance produces wide gaps between the high and low EACt. As more data is collected, the statistical performance stabilizes, as is evidenced by a narrowing gap between high and low EACt. Once we are satisfied that performance has stabilized, we take the next step.

We compare the high and low EACt estimates to target values. The target values reflect allowances for uncertainty, specifically:

1. Contingency, and

2. Contingency plus Reserve.

Based on the comparisons, we assess the status of the project. If the high forecast is greater than target values, or the low forecast is less than target values, the project is in trouble. Otherwise, the project is on track (with the exception of two rare cases).

ProjectFlightDeck uses thresholds to set the project status.  Similar to other ES metrics, we assign Red-Yellow-Green labels based on threshold values for the high and low forecasts. The labels help the project team and stakeholders understand the implications of the stats and the motivation for related action plans.

For common cases, we use Table 1 to determine project status. The two exceptional cases are dealt with separately.

Table 1

In practice, the high forecast is generally later than the end date of the bare plan, and the low forecast is generally earlier than the end date of the bare plan. When the forecasts are within contingency, the project is assigned a Green status.

If the high forecast is greater than the Late Contingency, the project is likely to miss the plan, but the deadline might be saved by reserve. The status is set to Yellow. If the high forecast is greater than both the Late Contingency and the Late Reserve, the project might be in serious trouble: the allowances for Late Contingency and Late Reserve appear to be inadequate. The status is set to Red, and a re-baseline should be considered.

If the low forecast is less than Early Contingency, the project again might be in trouble. The problem is not because the project is late but because the plan might be unsound. After all, allowances were made for early delivery, and the forecast makes it appear that something was mistaken. Either the relevant uncertainties were incorrectly identified, or the allowances were too small. The project is placed in Yellow status.

If the low forecast is earlier than both Early Contingency and Early Reserve, there is even more reason to suspect that the plan is unsound. The project is placed in Red status, and a re-baseline should  be considered.

Now, let’s address the exceptions and then proceed to an example.

Exception 1: It is possible for the high forecast to be less than Early Contingency. For instance, if the project is consistently running well ahead of schedule, the high forecast might dip below the allowance for Early Contingency. In such a case, the plan appears to be unrealistic, as it seems that there is no need for any Contingency or Reserve. In such a case, the status is immediately set to Red, and a re-baseline should be considered.

Exception 2: It is also possible for the low forecast to be greater than Late Contingency and Late Reserve. For instance, if the project is consistently running well behind schedule, the low forecast may rise above the allowance for Late Contingency and Late Reserve. So, even in the best case, the target date would be at risk, and the plan must be called into question. The status would be set to Red, and a re-baseline should be considered.

In the next post, a real-life example illustrates how the stats are used to steer a project.

Concept: ES Statistical Analysis factors historical variation into the calculation of duration estimates. By analyzing the project’s previous schedule performance, ES stats identify the high and low bounds for the estimate at completion for time. From high/low bounds and target dates, we can determine how well or poorly the project will perform. Thresholds guide the assignment of Green-Yellow-Red status.

Diagram 1

Practice: ES Statistical Analysis looks at schedule performance over time. When Walt Lipke examined ten years of EVM data, he found that, surprisingly, the natural variation in SPIt was not a good candidate for statistical analysis. The values did not fall into the kind of pattern that supports normal statistical analysis. The term “normal” here is not used in its colloquial sense. It is used to indicate that the SPIt data did not have a normal distribution.

But, Walt then found that the SPIt could be transformed mathematically into a form that was amenable to normal statistical analysis. I promised not to go into details of the theory. So, Diagram 1 must suffice to illustrate the difference between the two data sets. If you want details, check Lipke (2011).

Analysis of the transformed SPIt yields high and low bounds for the Estimate at Completion for time (EACt). At ProjectFlightDeck, we use the EACt (High) and EACt (Low) to steer the project. Before describing how to do so, we need to identify three conditions that limit the use of ES statistical analysis.

• Finite Data: there is an assumption in statistical analysis that the amount of data is infinite. But, by definition, projects are finite: they have a beginning and an end. For finite amounts of data, statistical analysis can still be done, but the calculations have to be adjusted. The impact of the adjustment is that the high and low estimates converge as the project progresses. For those of us who use ES statistical analysis, that is a feature rather than a limitation.

• Number of Time Periods: the normal calculation requires enough data to produce credible results. The rule-of-thumb is a minimum of 30 observations. At ProjectFlightDeck, we have found statistical analysis most useful when the project has been underway for a while. So, the large number of observations has not been a concern. For short duration projects (or for the periods before the minimum 30 is reached), other statistical methods are available.*

• Calculation Mechanism: you need a mechanism to produce the metrics. ES statistical calculations are more complex than the ones for basic metrics. To do the calculations by hand, you would need an understanding of the math, lots of patience, and cybernetic skill at data entry. Alternatively, you could build the calculations into a spreadsheet, but again you would need to understand the math and also spreadsheet statistical functions. The easiest route is to use a tool developed by somebody who has already done the ground work. If you are interested in receiving a free trial copy of ProjectFlightDeck’s ES Statistical Analysis, email Robert.VanDeVelde@ProjectFlightDeck.com.

The next post describes how to use ES Statistical Analysis to steer a project.

Notes:

 *See Lipke (2009, p 143) for more information on other techniques. On long-duration projects, we have not found the analysis particularly useful early in the project, primarily because schedule performance has not yet stabilized, and the gap between high and low estimates is wide, regardless of the calculation used. On short duration projects, some of our customers use daily observations to quickly reach the recommended number of observations.

References