Concept: The general reliability of ES metrics has been repeatedly demonstrated by empirical studies and research simulations. There is a concern, however, that ES might not be reliable in specific situations. The concerns arise from confusion between two distinct views of schedule performance assessment. Once the confusion is cleared up, doubts about reliability vanish, and an opportunity for improved practice emerges.
Practice: Criticism of ES reliability has its roots in an article by Jacob and Kane (2004). Jacob (2006) followed up with more specific objections, and he referenced a similar criticism by Book (2006). The underlying assumption behind the critiques is that the Critical Path (CP) has a privileged place in the assessment of schedule performance. It is the primary determinant of whether a project is on, behind, or ahead of schedule.
The Critiques
The primacy of the CP was clearly stated by Jacob and Kane (2004):
Activities on the critical path should be scrutinized first. p 21
Why is it necessary to point this out? Otherwise, as Book (2006) says, attention might be directed elsewhere. For, there are cases in which:
…the combined dollar value of all work not on the critical path…outweighs the dollar value of the work that is on the critical path. p 25
V&V (Vanhoucke and Vandevoorde, 2009) spelled out the implications of overlooking the CP:
…small delays in critical activities and huge accelerations in non-critical activities might mask potential problems since the schedule performance indicator might report good project performance while the opposite is true. p 27
All of the researchers state that, if the CP is late, but ES performance indicators show that the project is ahead of schedule, the ES indicators are “false”, “misleading”, “invalid”, or “incorrect”. If the CP is early and the ES schedule performance indicators show that the project is behind schedule, the indicators are again regarded as wrong.
Lipke’s Response
Lipke’s reply to the criticisms is summed up in his following statement:
“…the reality of inter-dependency between critical and non-critical activities will not allow performance on the planned critical path, solely, to dictate the final duration”, Lipke (2014a), p 8 of 10.
Lipke supports the claim with statistical analysis of data from real projects (Lipke, 2014b).
A corollary of Lipke’s response is that the simulations used by V&V are unrealistic. They force non-CP activities to be completed before CP activities. Ignoring inter-dependencies artificially depreciates ES metrics.
Evaluating the Arguments
Assume that Jacob, et al. are correct, and the CP dictates the final duration. That holds if and only if the current CP survives intact from the Actual Time to the end of the project. And, how do we know that that will be the case?
We don’t know for sure. In fact, we don’t even know its likelihood. No doubt, the probability the CP will remain unchanged increases toward the end of the project. But, up to the last, there’s always a possibility it will change.
That seems to imbue the CP with uncertainty and thereby bring into question its reliability.
Next, what about the charge that V&V’s simulation is unrealistic?
Lipke offers strong evidence that naturally occurring inter-dependencies affect final duration. But, that does not rule out the possibility that good performance on non-CP tasks can outweigh poor performance on CP tasks. If that happens, performance indications from ES can differ from those of CP.
Here’s a case in which that actually happened.
With ES schedule performance lagging, the PM decided to “build delivery momentum”. So, the team began completing deliverables that were easy to finish but far from critical. The result: a significant number of activities were done out-of-sequence, early.
Here’s what the Earned Schedule burndown looked like at week 16 of 87.
The decision to “build delivery momentum” came in period 7. A gap quickly opened between CP and non-CP items. Availability of quick wins then declined, and the initial boost died out. The project was left with an on-going performance gap.
Oddly, the case simultaneously supports both V&V and Lipke.
The structure of activities in the case was similar to that in V&V simulations: non-CP completions outweighed CP completions. It actually happened. So, it can happen. That is, it’s possible.
But, the case equally conforms to Lipke’s argument: the discrepancy occurred only by ignoring inter-dependencies and performing work out of sequence.
What wasn’t anticipated in either argument was that, “in the wild”, idiosyncratic behaviour that can and does occur.
Where does this leave us?
If both positions can be viewed as “true”, there’s something wrong with the dichotomy itself.
The Third Way Forward
Here’s where a third alternative comes into play. Rather than it’s either one or the other, CP or ES, why not allow both, but with a condition.
Here's an analogy.
Have a look at these old ads.
They tout a car based on fuel performance. In the ads, not one, but two figures represent performance: city and highway.
Nobody says that one or the other must be false.
That’s because there’s a difference in the behaviour being measured.
City use comprises frequent stopping and starting, accelerating and braking, and idling.
Highway use comprises constant speed and few turns.
There's no conflict because the behavior being measured differs. For city measurements, it's “stop and go”, “fast and slow”, but not a constant speed. For highway measurements, it's a constant speed, but not “stop and go”, “fast and slow”.
The same can be said for schedule performance.
Critical Path comprises a specific sequence of deliverables. CP is conditioned on what is complete.
Earned Schedule comprises the volume of periods earned. ES is conditioned on how much is complete.
There’s no conflict because the behaviour being measured differs. For volume measurements, it's how much is complete, but not if the right parts are complete. For sequence measurements, it's if the right parts are complete, but not how much is complete.
In fact, although they're distinct, they are usually aligned. But, because distinct, they can differ. And, if they do, it's useful to find out why. The difference might indicate schedule problems.
In any case, difference does not entail conflict.
References:
Book, S. (2006). “Earned Schedule” and Its Possible Unreliability as an Indicator. The Measurable News, Spring, 24-30.
Jacob, D. (2006). Is “Earned Schedule” an Unreliable Indicator? The Measurable News, Fall, 15-21.
Jacob, D. and Kane, M. (2004). Forecasting Schedule Completion Using Earned Value Metrics…Revisited. The Measurable News, Summer, 1 and 11-17.
Lipke, W. (2014a). Examining Project Duration Forecasting Reliability. PM World Journal, III (III).
Lipke, W. (2014b). Testing Earned Schedule Forecasting Reliability. PM World Journal, III (VII).
Vanhoucke, M., & Vandevoorde, S. (2008). Earned Value Forecast Accuracy and Activity Criticality. The Measurable News, Summer, 13-16.
|
