ProjectFlightDeck

Concept: The general reliability of ES metrics has been repeatedly demonstrated by empirical studies and research simulations. There is a concern, however, that ES might not be reliable in specific situations. The concerns arise from confusion between two distinct views of schedule performance assessment. Once the confusion is cleared up, doubts about reliability vanish, and an opportunity for improved practice emerges.

Practice: Criticism of ES reliability has its roots in an article by Jacob and Kane (2004). Jacob (2006) followed up with more specific objections, and he referenced a similar criticism by Book (2006). The underlying assumption behind the critiques is that the Critical Path (CP) has a privileged place in the assessment of schedule performance. It is the primary determinant of whether a project is on, behind, or ahead of schedule.

The primacy of the CP was clearly stated by Jacob and Kane (2004):

Activities on the critical path should be scrutinized first. p 21

Why? As Book (2006) points out, there are cases in which:

…the combined dollar value of all work not on the critical path…outweighs the dollar value of the work that is on the critical path. p 25

Vanhoucke and Vandevoorde (2009) spelled out the implications:

…small delays in critical activities and huge accelerations in non-critical activities might mask potential problems since the schedule performance indicator might report good project performance while the opposite is true. p 27

All of the researchers state that, if the CP is late, but ES performance indicators show that the overall project is ahead of schedule, the indicators are “false”, “misleading”, “invalid”, or “incorrect”. If the CP is early and the ES schedule performance indicators show that the overall project is behind schedule, the indicators are again regarded as wrong.

Jacob (2006), however, makes a surprising admission. He says that understanding schedule performance goes beyond what the current CP indicates:

If the [Schedule Performance Indicator] of a critical path activity is equal to or greater than 1.0, it could mean an on-schedule or ahead of schedule condition exists for the project. However, one cannot be assured of this unless late activities with float are interrogated to determine how much of their respective float they have used or forecasted [sic] to use [italics added]. p 21

According to Jacob, late activities might force a change in the CP. If so, the current CP is not the sole determinant of schedule performance--non-CP activities must be taken into account. But, if non-CP activities must be taken into account, it is difficult to understand why ES project indicators are regarded as wrong.

After all, such indicators incorporate the effect of non-CP activities in performance assessment. They do so differently from the bottom-up network analysis that Jacob has in mind, but they take account of the same data, namely, the non-CP activities.

A difference between CP and overall indicators functions as a signal that non-CP activities must be interrogated. Far from being false or misleading, overall indicators give early warning that further analysis is required. Thus, they complement CP analysis, rather than clash with it.

References

Book, S. (2006). “Earned Schedule” and Its Possible Unreliability as an Indicator. The Measurable News, Spring, 24-30.

Jacob, D. (2006). Is “Earned Schedule” an Unreliable Indicator? The Measurable News, Fall, 15-21.

Jacob, D. and Kane, M. (2004). Forecasting Schedule Completion Using Earned Value Metrics…Revisited. The Measurable News, Summer, 1 and 11-17.

Vanhoucke, M., & Vandevoorde, S. (2008). Earned Value Forecast Accuracy and Activity Criticality. The Measurable News, Summer, 13-16.

Concept: The general reliability of ES metrics has been repeatedly demonstrated by empirical studies and research simulations. There is a concern, however, that ES might not be reliable in specific situations. The concerns arise from confusion between two distinct views of schedule performance assessment. Once the confusion is cleared up, doubts about reliability vanish, and an opportunity for improved practice emerges.

Diagram 1

Practice: Do you see the duck? How about the rabbit? This famous image, which dates back to 1892, has inspired investigation by philosophers and psychologists for generations. Surprisingly, it also has a lesson for the ES reliability controversy.  Before describing the lesson, I need to set the stage with an example from a recent project.

The project depicted in Chart 1 was a highly parallel, long-term initiative. Only the first 12 weeks (out of a total of 87) are shown—the relevant situation occurred while the project was underway.

Chart 1

At first, the project proceeded on plan, but, then, it began to slow. Around week 9, the Project Manager decided to get things back on track by building delivery momentum. She encouraged team members to “harvest low-hanging fruit”, i.e., to complete quick, easy, stand-alone tasks. It seemed to work—performance stabilized, as shown by the flat-lining of overall performance in weeks 10-12.

An analysis of the Critical Path told a different story. The estimated end date continued to stretch out, making on-time completion increasingly difficult.

The case might seem to support criticism of ES reliability. After all, the overall SPIt (and by implication the EACt) seemed to stabilize, but the Critical Path end date got worse, indeed much worse. A bit more context reveals the weakness in the criticism of ES reliability.

On the project, analysis of the Critical Path was quantitative, not just an observation about duration. An ES analysis was run on Critical Path tasks. It showed explicitly that the Critical Path SPIt and EACt were both suffering. We knew not only that the end date was at risk but also the size of the gap. At the same time, analysis of the overall schedule showed a stronger SPIt and an earlier end date.

Here’s where the duck-and-rabbit image becomes relevant. Like the image, schedule performance can be seen from divergent perspectives. It is wrong to conclude that one way is right, and other ways are false or misleading. Instead, multiple views can be right, conditional upon their context.

The example illustrates that there are two distinct views of schedule performance at play. In one case, the view is based on Critical Path activities. Such a view is especially useful in large projects with lots of parallel tasks. It is a way to isolate and track key items amid the mass of activities underway.

At the same time, it carries a risk of “tunnel vision”, of getting too focused on a small sub-set of tasks and missing the overall picture. (I address this kind of risk in my article, “Managing the Unexpected with Earned Schedule”. See References.) The overall view is an antidote to the risk, providing a broader perspective on schedule performance.

In the example, the overall view reveals that non-Critical Path tasks are being done at a higher rate than expected. That often means short-term gain but long-term pain. Not only can it distract attention from critical items, but it easily leads to re-work.

Conversely, the overall view can identify when a significant number of non-Critical Path tasks are behind schedule, even though the CP is on track. That is an early warning about activities that might move onto the Critical Path—a potential problem that warrants equal attention.

So, while Vanhoucke and Vandevoorde have correctly identified scenarios in which performance indicators for the Critical Path can differ from those for the overall project, there is no reason to therefore infer that ES metrics yield false or misleading results. That would be so only if the two views purported to give us the same information, but they do not.

On one hand, ES metrics measure schedule performance on a sub-set of key items—ones that will affect the end date. On the other hand, ES metrics measure schedule performance overall, including items that might affect the end date.

Neither view is the sole determinant of schedule performance. Together, the two perspectives provide a balanced assessment. Using them in concert has become a standard practice at ProjectFlightDeck—one that we recommend others adopt.

References

Lipke, W. (2014a). Examining Project Duration Forecasting Reliability. PM World Journal, III (III).

Lipke, W. (2014b). Testing Earned Schedule Forecasting Reliability. PM World Journal, III (VII).

Van De Velde, R. (2010). Managing the Unexpected with Earned Schedule. PM World Today, XII(X).

Vanhoucke, M., & Vandevoorde, S. (2008). Earned Value Forecast Accuracy and Activity Criticality. The Measurable News, Summer, 13-16.

Concept: The general reliability of ES metrics has been repeatedly demonstrated by empirical studies and research simulations. There is a concern, however, that ES might not be reliable in specific situations. The concerns arise from confusion between two distinct views of schedule performance assessment. Once the confusion is cleared up, doubts about reliability vanish, and an opportunity for improved practice emerges.

Chart 1

Practice: A recent study by Vanhoucke and Vandevoorde challenges the reliability of ES metrics. Walt Lipke has responded to the challenge. (See the previous post for the critique, and see the References below for the sources of both the critique and the response.)

In Lipke’s view, the simulations are unrealistic. They fail to reflect the inter-dependency between Critical Path (CP) and non-CP tasks. Furthermore, they ignore the fact that, at the end of a project, the Schedule Performance Index for time (SPIt) converges to a value that ensures the EACt equals the Real Duration (RD), eliminating any discrepancy.

Lipke starts by pointing out that the simulations are constructed with the following conditions:

1. Real Duration (RD) always matches CP performance
2. SPIt correlates with non-CP performance.

One way to ensure that condition 1 holds is to assume that the project completes on the CP of the planned schedule. In practice, projects rarely complete as planned.

Condition 2, however, appears to be realistic. If non-CP tasks are delayed in sufficient volume, they can suppress the SPIt. But, Lipke suggests that, while it can occur, such a distribution of on-time and delayed tasks is not necessary. He claims that building it into the simulations “perturbs” the results.

Lipke then points out there is often an interaction between non-CP tasks and CP items. If a non-CP predecessor of a CP task is delayed, it can, and often does, delay the CP task. The simulations appear to force an unrealistic degree of separation between CP and non-CP tasks.

Lipke’s final criticism of the simulation study is that at the end of a project, any discrepancy between SPIt, EACt, and RD must disappear. ES math ensures that the SPIt converges on a value that accurately represents the time in which the planned value is actually earned. That convergence ensures that the EACt equals the RD. Ultimately, there can be no discrepancy between the SPIt and the RD. To turn out otherwise, the simulations have to be unrealistic.

Although Lipke has pointed out weaknesses in the ES reliability study, it must be admitted that, prior to project completion, there can be real cases that fit the conditions imposed on the study. Whether or not simulations would produce the same results without forcing the conditions on the scenarios is left as an area for further study. Lipke outlines a testing scheme for such research.

Apart from the outcome of further research, there are reasons to acknowledge the reliability of ES metrics. Lipke used notional data to make a positive case for ES reliability. He then followed up with an  empirical study that showed similar results.

In the empirical study, Lipke analyzed data from 16 projects. The results are displayed in Chart 1. The chart clearly shows that the cases regarded as false or misleading by Vanhoucke and Vandevoorde decrease over time, ending at 0%.

Assuming a 10% margin of error, the chart shows the following reliability of ES forecasts:

• at 5% complete, forecasting is 60% reliable;
• at 50% complete, forecasting is 80% reliable.
• At 100% complete, forecasting is 100% reliable, i.e., the forecast converges to the RD.

In summary, ES forecasting is good to excellent for 95% of project duration and is therefore considered reliable.

Intuitively, it seems likely that ES forecasting is more reliable than that of other methods, but that conclusion needs to be supported by additional research.

References

Lipke, W. (2014a). Examining Project Duration Forecasting Reliability. PM World Journal, III (III).

Lipke, W. (2014b). Testing Earned Schedule Forecasting Reliability. PM World Journal, III (VII).

Vanhoucke, M., & Vandevoorde, S. (2008). Earned Value Forecast Accuracy and Activity Criticality. The Measurable News, Summer, 13-16.

Concept: The general reliability of Earned Schedule (ES) metrics has been repeatedly demonstrated by empirical studies and research simulations. There is a concern, however, that ES might not be reliable in specific situations. The concerns arise from confusion between two distinct views of schedule performance assessment. Once the confusion is cleared up, doubts about reliability vanish, and an opportunity for improved practice emerges.

Diagram 1

Practice: Pioneering research by Mario Vanhoucke and Stephan Vandevoorde demonstrated that ES metrics outperform, on average, more traditional Earned Value-based metrics. The researchers used innovative simulation techniques to establish the superiority of Earned Schedule for predictions of final duration.

In a follow-up study (see References below), the researchers extended their simulation scenarios to test ES reliability for critical and non-critical activities. The study showed that some results from ES are apparently inferior to those from other techniques. Diagram 1 summarizes the findings. Rather than explaining each cell in detail, I’ll focus on the results that appear to be problematic for ES.

Problems for ES emerge from a discrepancy between what indicators such as SPIt and EACt tell us about schedule performance and what appears to be the Real Duration (RD) of the project.

The discrepancies can be summarized as follows:

1. For scenarios 4 and 7 in Diagram 1, the ES performance indicator shows that the project is ahead of schedule (SPIt>1), but the Real Duration appears to be on or behind schedule (RD=PD or RD>PD).

2. For scenarios 3 and 6 in Diagram 1, the ES performance indicator shows that the project is behind schedule (SPIt<1), but the Real Duration appears to be on or ahead of schedule (RD=PD or RD<pd).< span="">

The discrepancies were tested in simulations constructed by the researchers as follows:

(i)  For (1), non-CP tasks were set ahead of plan, and Critical Path tasks were set on or behind plan. By setting non-CP tasks ahead, the SPIt went over 1.0. By setting CP tasks either on or behind schedule, the RD appeared to be on or greater than plan. Given that the EACt is generated from the SPIt, the estimate is too low.

(ii) For (2), non-Critical Path tasks were set on or behind plan, and Critical Path tasks were set ahead of plan. By setting non-CP tasks behind, the SPIt dropped below 1.0. By setting CP tasks either on or behind schedule, the RD appeared to be on or better than plan. Given that the EACt is generated from the SPIt, the estimate is too high.

As shown in Diagram 2, the simulation studies indicated that the discrepancies produced inferior results for ES metrics compared to other techniques.

Diagram 2

Vanhoucke and Vandevoorde concluded that the inferior results undermined the reliability of ES metrics. The claim spawned a flurry of debate within the ES community.

References

Vanhoucke, M., & Vandevoorde, S. (2008). Earned Value Forecast Accuracy and Activity Criticality. The Measurable News, Summer, 13-16.