Concept: The To-complete Schedule Performance Index (TSPI) is a prescription for future schedule efficiency. It prescribes the level of efficiency required to complete the project within a target duration. The target duration may be either the Planned Duration (TSPIpd) or the Estimated Duration (TSPIed).
Practice: The U.S. National Defense Industry Association (NDIA) uses TSPIed to assess confidence in the forecasted duration. The assessment is based on the difference between the SPIt and the TSPIed. The regime is illustrated in Diagram 1. (See the previous post for details on the NDIA technique.)
Diagram 1
The NDIA approach differs in several ways from the one used by ProjectFlightDeck. (For the ProjectFlightDeck approach, see posts for July 23 and August 04, 2015). Most of the differences are minor. There is, however, one difference that entails an important limitation on NDIA TSPI.
Before addressing that limitation, it is important to commend the NDIA for including Earned Schedule in its list of Predictive Measures. Such leadership promotes an understanding of ES and encourages adoption of the technique. It is also important to acknowledge the NDIA’s innovative regime for applying TSPI. Such efforts increase the reach of ES techniques.
Now, for the limitation: if you use the IEACt* as the Estimated Duration in the NDIA's TSPIed calculation, the NDIA thresholds do not return meaningful results. So, for a key duration estimate, the NDIA approach does not work. The rationale follows.
ProjectFlightDeck tested the NDIA approach on 6 project schedules. The projects were all actual initiatives. They were highly varied in size, duration, domain, and status. The schedules comprised 126 time periods. Most used weekly time increments, but some used monthly increments. All of the projects used Earned Schedule metrics to manage schedule performance.
The test applied the NDIA equation for TSPIed, namely, (Planned Duration – Earned Schedule) / (Estimated Duration – Actual Duration). The Planned Duration and Actual Duration were given quantities. All remaining terms used standard ES equations, with IEACt substituted for Estimated Duration.
To evaluate the results, the test used the NDIA thresholds shown in Table 1. (SPIt = Schedule Performance Index for time.)
Table 1
**Test Results**
For the 126 time periods in the 6 test projects:
- In none of the time periods was (SPIt - TSPIed) > 0.10.
- In none of the time periods was (SPIt - TSPIed) > 0.10.
- In all time periods, (SPIt - TSPIed) = 0.000.
**Mathematical Confirmation**
Concerned that the test data was anomalous, we then researched the equations involved. We were able to prove algebraically that, when ED = IEACt, TSPIed = SPIt. (To see the proof, click here.) The math confirmed the validity of the empirical results.
**Conclusions**
If (SPIt – TSPIed) is always 0, the threshold always indicates that the forecast warrants confidence. Clearly, that is not always the case. So, we concluded that, when ED = IEACt, the thresholds do not provide meaningful guidance on confidence in the forecast.
As the problem follows specifically from the relationship between IEACt and SPIt, estimating techniques that use performance measures other than SPIt do not necessarily entail the same issue. Clearly, the technique need not be abandoned. Still, its use must be limited—the IEACt cannot be used for the Estimated Duration. That is especially unfortunate because the IEACt is demonstrably superior to other duration estimation methods (see References).
**Additional Observations**
**TCPIeac:** There is a similar limitation on the NDIA’s approach to TCPIeac. We proved algebraically that CPI = TCPIeac, when CPI = EV / AC, TCPI = (BAC – EV) / (EAC – AC), and EAC = BAC / CPI or EAC = AC + (BAC – EV / CPI). So, the thresholds for (CPI – TCPIeac) will not return meaningful guidance on cost forecasts when EAC is either BAC / CPI or AC + (BAC – EV / CPI. (To see one of the proofs, click here.)
**Test Warning:** If you test the NDIA TSPIed equations, it is important to use precise values for the terms. In our testing, we found that we could artificially trigger NDIA threshold breaches with imprecise values for the terms. For instance, in one case, we rounded terms and produced significant differences in (SPIt – TSPIed). When we removed the rounding, the differences shrank. When we increased the precision of the values, the differences disappeared. One reason we pursued the algebraic proof was to confirm mathematically that SPIt = TSPIed when ED = IEACt. Thus, we ensured any residual differences were due to imprecision in terms or calculations.
Notes
*There are several formulas for the IEACt. The most widely used are the "long form": IEACt = Actual Time + (Planned Duration - Earned Schedule) / Performance Factor for time, or AT + (PD - ES) / PFt, and the "short form": IEACt = PD / SPIt. When the Performance Factor for time is the SPIt, the long form reduces to the short form. My comments apply only to cases where the PFt = SPIt.
References
Vanhouche, M. and Vandevoorde, S. “A Simulation and Evaluation of Earned Value Metrics to Forecast the Project Duration”, Journal of the Operational Research Society, vol 58: 1361-1374 (October 2007). |