ProjectFlightDeck

Concept: The To-complete Schedule Performance Index (TSPI) is a prescription for future schedule efficiency. It prescribes the level of efficiency required to complete the project within a target duration. The target duration may be either the Planned Duration (TSPIpd) or the Estimated Duration (TSPIed).

Practice: The U.S. National Defense Industry Association (NDIA) uses TSPIed to assess confidence in the forecasted duration. The assessment is based on the difference between the SPIt and the TSPIed. The regime is illustrated in Diagram 1. (See the previous post for details on the NDIA technique.)

Diagram 1

The NDIA approach differs in several ways from the one used by ProjectFlightDeck. (For the ProjectFlightDeck approach, see posts for July 23 and August 04, 2015). Most of the differences are minor. There is, however, one difference that entails an important limitation on NDIA TSPI.

Before addressing that limitation, it is important to commend the NDIA for including Earned Schedule in its list of Predictive Measures. Such leadership promotes an understanding of ES and encourages adoption of the technique. It is also important to acknowledge the NDIA’s innovative regime for applying TSPI. Such efforts increase the reach of ES techniques.

Now, for the limitation: if you use the IEACt* as the Estimated Duration in the NDIA's TSPIed calculation, the NDIA thresholds do not return meaningful results. So, for a key duration estimate, the NDIA approach does not work. The rationale follows.

ProjectFlightDeck tested the NDIA approach on 6 project schedules. The projects were all actual initiatives. They were highly varied in size, duration, domain, and status. The schedules comprised 126 time periods. Most used weekly time increments, but some used monthly increments. All of the projects used Earned Schedule metrics to manage schedule performance.

The test applied the NDIA equation for TSPIed, namely, (Planned Duration – Earned Schedule) / (Estimated Duration – Actual Duration).  The Planned Duration and Actual Duration were given quantities. All remaining terms used standard ES equations, with IEACt substituted for Estimated Duration.

To evaluate the results, the test used the NDIA thresholds shown in Table 1. (SPIt = Schedule Performance Index for time.)

Table 1

Test Results

For the 126 time periods in the 6 test projects:

• In none of the time periods was (SPIt - TSPIed) > 0.10.
• In none of the time periods was (SPIt - TSPIed) > 0.10.
• In all time periods, (SPIt - TSPIed) = 0.000.

Mathematical Confirmation

Concerned that the test data was anomalous, we then researched the equations involved. We were able to prove algebraically that, when ED = IEACt, TSPIed = SPIt. (To see the proof, click here.) The math confirmed the validity of the empirical results.

Conclusions

If (SPIt – TSPIed) is always 0, the threshold always indicates that the forecast warrants confidence. Clearly, that is not always the case. So, we concluded that, when ED = IEACt, the thresholds do not provide meaningful guidance on confidence in the forecast.

As the problem follows specifically from the relationship between IEACt and SPIt, estimating techniques that use performance measures other than SPIt do not necessarily entail the same issue. Clearly, the technique need not be abandoned. Still, its use must be limited—the IEACt cannot be used for the Estimated Duration. That is especially unfortunate because the IEACt is demonstrably superior to other duration estimation methods (see References).

Additional Observations

TCPIeac: There is a similar limitation on the NDIA’s approach to TCPIeac. We proved algebraically that CPI = TCPIeac, when CPI = EV / AC, TCPI = (BAC – EV) / (EAC – AC), and EAC = BAC / CPI or EAC = AC + (BAC – EV / CPI). So, the thresholds for (CPI – TCPIeac) will not return meaningful guidance on cost forecasts when EAC is either BAC / CPI or AC + (BAC – EV / CPI. (To see one of the proofs, click here.)

Test Warning: If you test the NDIA TSPIed equations, it is important to use precise values for the terms. In our testing, we found that we could artificially trigger NDIA threshold breaches with imprecise values for the terms. For instance, in one case, we rounded terms and produced significant differences in (SPIt – TSPIed). When we removed the rounding, the differences shrank. When we increased the precision of the values, the differences disappeared. One reason we pursued the algebraic proof was to confirm mathematically that SPIt = TSPIed when ED = IEACt. Thus, we ensured any residual differences were due to imprecision in terms or calculations.

Notes

*There are several formulas for the IEACt. The most widely used are the "long form": IEACt = Actual Time + (Planned Duration - Earned Schedule) / Performance Factor for time, or AT + (PD - ES) / PFt, and the "short form": IEACt = PD / SPIt. When the Performance Factor for time is the SPIt, the long form reduces to the short form. My comments apply only to cases where the PFt = SPIt. 

References

Vanhouche, M. and Vandevoorde, S. “A Simulation and Evaluation of Earned Value Metrics to Forecast the Project Duration”, Journal of the Operational Research Society, vol 58: 1361-1374 (October 2007).

Concept: The To-complete Schedule Performance Index (TSPI) is a prescription for future schedule efficiency. It prescribes the level of efficiency required to complete the project within a specified duration.

Practice: The U.S. National Defense Industry Association (NDIA) offers a different approach to TSPI than the one described in the previous post.

The NDIA focuses exclusively on the TSPI for Estimated Duration, rather than Planned Duration. The TSPI for Estimated Duration (TSPIed) is the ratio between the remaining Planned Duration (PDWR in Diagram 1) and the Remaining Estimated Duration (RD in Diagram 1). In other words, it is the ratio between the two sections of the chart highlighted in yellow. See NDIA (2014).

Diagram 1

The PDWR is the result of subtracting Earned Schedule (ES) from Planned Duration (PD), and the RD is the result of subtracting Actual Duration (AD) from Estimated Duration (ED). Thus, we have: (PD - ES) / (ED – AD). This is similar to Walt Lipke’s equation for TSPI_E: (PD – ES) / (IEACt – AT). The specific term, IEACt, is replaced with the generic term, Estimated Duration, and the term Actual Duration is used instead of Actual Time, but the same actual periods are covered.

The NDIA then proposes an innovative use of the TSPI. The TSPIed is used to assess confidence in the forecasted duration based on the difference between the SPIt and the TSPIed. The thresholds used by the NDIA and their interpretation are summarized in Table 1 (based on NDIA, A Guide to Managing Programs Using Predictive Measures, September 17, 2014, p. 33).

Table 1

The following diagram illustrates the regime.

Diagram 2

The NDIA interprets the impact on confidence level as follows. If the difference between the SPIt and the TSPI is between -0.1 and+0.1, the forecast appears to be sound. (Note: the NDIA guide does not specify the interpretation when the difference exactly equals 0.1 or -0.1.)  Any difference beyond these boundaries means that the forecast deserves less confidence.  The reasons for doubt vary with the sign of the breach.

• If the difference is greater than .1, future performance must be worse than previous performance. If performance declines, the timeline extends. But, performance may rise to its previous level (or higher), and thus shorten the timeline. So, the forecast may be considered overly pessimistic because it assumes future poor performance.
• By contrast, if the difference is less than -.1, future performance must be better than previous performance. If performance improves, the timeline shrinks. But, performance may return to its previous level (or lower), and thus lengthen the timeline. So, the forecast may be considered overly optimistic because it assumes future performance will improve.

The next post offers a critique of the NDIA approach to TSPI. 
 

References:

NDIA, A Guide to Managing Programs Using Predictive Measures, September 17, 2014, p. 32.

Concept: The To-complete Schedule Performance Index (TSPI) is a prescription for future schedule efficiency. It prescribes the level of efficiency required to complete the project within a specified duration.

Practice: At ProjectFlightDeck, we use the TSPI to assess how well the schedule must perform to achieve the Planned Duration.

We assume that the nominal future performance level is 1.00—that value indicates that the project will earn schedule at the planned rate. If the TSPI is less than or equal to 1.00, the Planned Duration is achievable because the required future performance will be at or less than the nominal rate. (Note: if it is less than the nominal rate, that means future performance can be less than nominal and the target date will still be intact.) We assign a Green status to projects with TSPI less than or equal to 1.00.

On the other hand, if the TSPI is greater than or equal to[1] 1.10, it is generally conceded that the Planned Duration is not achievable. Beyond that threshold, the level of performance required to meet the Planned Duration rises rapidly to unreasonable heights. As Walt Lipke puts it, “the project is very rapidly becoming “out of control” (Lipke, 2009). On a recent project, we witnessed that happening.

Diagram 1

At period 5, the project breached the 1.10 threshold. By period 7, it was clearly out of control. The old plan had to be scrapped, a new one created, and the contract re-negotiated.

At ProjectFlightDeck, we assign a Red status to projects with TSPI greater than 1.10.

To complete the picture, we assign a Yellow status to projects where TSPI is greater than 1.00 and less than or equal to 1.10.

The TSPI is also useful as a cross-check on other ES metrics. A recent project illustrates the approach.

Diagram 2

In this project, the SPIt was, for a long time, above the .9-threshold used to denote Green status. Toward the end of the time frame shown, the SPIt began to slip. In January, 2015, it dropped below the threshold for Green status and fell to Yellow territory.

Given that the project had been in Green status for a long time, had only recently dipped into the Yellow, and had a long way to go (Diagram 2 only shows the timeline to the end of 2015--in fact, the plan went another 18 months), you might think that situation warranted concern but had not reached a critical point.

When we cross-checked the SPIt with the TSPI, however, a different picture emerged.

Diagram 3

The TSPI had been gradually climbing toward the 1.10 threshold. In February, 2015, it breached the threshold—from that point, we risked it spinning out of control. We had, indeed, reached a critical stage. Dramatic corrective action was required.

Even though ES metrics are all rooted in the same starting point, they provide different perspectives on schedule performance. The different views support a balanced assessment within the ES framework.

Notes:

[1] Edited to remove the clause "or greater than". The threshold value is 1.10. If that threshold is exceeded, the project becomes unrecoverable. (Ed. 08 Sep 2017)

References:

Lipke, W. The To Complete Performance Index…An Expanded View. The Measurable News, 2009 Issue 2, pp. 18-22.

Lipke,  Walt. Examination of the Threshold for the To Complete Indexes. The Measurable News, 2016 Issue 1, pp. 9-14 [Reference added 08 Sep 2017].