ProjectFlightDeck

Give Probability a Chance: ES Stats in Action

Statistical variance is not just an academic exercise. You can apply high and low bounds to actually manage schedule performance.

Here's how.

Start with the nominal Estimate at Complete for time (ES/SPIt). Find the high and low bounds of the estimate.

(The math behind the calculations is daunting. Save time and headache by using an app. Both freeware and commercial software are available to do the calculations. For info on ProjectFlightDeck’s solution, click here.)

Now, look for trends. What's a trend? It's three or more consecutive measurements headed in the same direction.

What does a trend tell you?

For Walt Lipke, the direction of the trend tells you the likely duration, given historical performance on the project.*

Here’s a trend we like to see: high and low bounds converge symmetrically on the nominal. The nominal, in turn has a flat trajectory. The interpretation: estimates will be fairly close to the actual outcome. No further action is required.

By contrast, here's a pattern we don't like to see: high and low bounds suddenly diverge. That means performance is erratic. The estimates are not reliable indicators of final duration. As soon as the trend emerges, take action to recover the project.

If there is a synchronous downward trend, that means performance is improving over time. The low bound will be slightly lower than the actual outcome. That's often viewed positively, but it has a downside. It might indicate padding in the schedule. Maybe, it's time for action.

Finally, see if there is a coordinated upward trend in the estimates. That means performance is worsening over time. The high bound will likely end up slightly higher than the actual outcome. Again, maybe it's time for action.

When performance is stable, trend analysis signals the likely outcome. But when does the signal call for action? Too early, you lose credibility. Too late, you lose recoverability.

For an answer, you need two things.*

First, the performance baseline. That's the Planned Duration. And, second, performance thresholds—limits of acceptable performance.

One form of threshold is a fixed percentage of the Planned Duration. It's usually set by governance documents, project or program plans, or contracts. In our practice, we have used ±10% as the fixed rate.

Another form of threshold is a variable percentage. In our practice, it's derived from uncertainty allowance.

Uncertainty leads to risk, and the risk needs to be mitigated. Some uncertainty arises from a lack of knowledge. It creates reducible risk. You buy down the risk with scheduled tasks that fill in knowledge gaps. Contingency covers the buy down.

Uncertainty also arises from the randomness that occurs naturally on projects. You can't buy down that kind of risk. It's irreducible. You can only buffer it through Reserve.

Add or subtract the uncertainty allowance to or from the Planned Duration. That sets performance thresholds.

Next, assess the variance of estimates from the baseline (or the total commitment). If the variance trends towards a threshold, prepare to take action. If the variance breaches a threshold, it's time to act.

Finally, use high and low limits to assess the reliability of the nominal estimate. A wide range means you should be cautious about depending on the nominal. A small range, especially later in a project, warrants more confidence in the nominal.

Watch for our video, Marie Gives Probability a Chance. It drops next week.

*References:

Videos: Making (Common) Sense of ES Statistical Analysis
            Building Blocks 
            Build the Bounds 
            Apply the Bounds 

Blog:
            Statistical Analysis Revisited (2021) 
            Statistical Analysis Deep Dives
                        Background
                        How it works
                        Steering the Project
                        Example
                        Pro/Con 

After introducing himself, Yancy set his challenge: which is better Earned Schedule’s Estimated Completion Date (ES) or an Independent Estimated Completion Date (IECD) derived from BCWS* and BCWP?

Yancy noted an asymmetry between reactions to cost and schedule forecasts. If in years 1 and 2, the BCWP=200K and ACWP=246K, the cost variance is -46K. If in years 3 and 4, the BCWR=200K and estimate for completion ETC=153K, the cost variance is +46K. Credible? No! Nobody would believe that you’ would close the gap and that the project would finish on budget.

On the other hand, if the Critical Path showed a current lag of 57 periods from the baseline but most of the schedule remained available, the planned finish would likely remain intact. Many would accept that the time could be “made up” before the planned finish.

Optimism bias makes actual data from projects unreliable. We cannot trust claims that the estimated finish is still in place.

Instead, Yancy proposed to determine which duration estimates were reliable. That presented another challenge.

EAC and ETC metrics are mature, offering several formulas. We can identify the formula that best fits. By contrast, IECD calculations are not as mature, and we don’t know which has the best fit.

So, Yancy built a simulator that performed several types of IECD calculations simultaneously. That enabled him to compare the results to those from ES’ EACt.

Yancy’s IECD calculations*:

•           BCWS-based: IECD = Time Now + BCWR / BCWS(current or average)
•           BCWP-based: IECD = Time Now + BCWR / BCWP(average)
•           SPIt-based IECD = Time Now + PDWR / SPIt(current or average)
•           SVt-based IECD = Time Now + BL Complete + SVt
•           CP-based IECD = CP finish = BL Finish 

To make the simulation more realistic, Yancy added variations in start and end dates, ramp up and down, slower and faster rates, smoother and rougher curves. He also varied project length, took data from different points in the project timeline, and varied the size of the window.

 Yancy demonstrated the simulator, looping through several scenarios. In each run, the generated estimate that came closest to the given end date was measured and stored.

 What were the results of his investigation?

 Yancy reported that SPIt was “almost always” the best, and SVt was the next best. The worst performer was the BCWS-based estimate. The most stable was the SVt.

 Yancy’s conclusion was that schedulers should use and report ES and the EACt (which is based on SPIt).

 * Acronyms used in this report: BCWS=Budgeted Cost of Work Scheduled, aka PV; BCWP=Budgeted Cost of Work Performed, aka EV; BCWR=Budgeted Cost of Work Remaining; PDWR=Planned Duration Work Remaining; BL=Baseline; CP=Critical Path; EACt=Estimate at Complete for time; SPIt=Schedule Performance Index for time; SVt=Schedule Variance for time.

Paulo became aware of Earned Schedule in 2007 through course work on his master’s degree. With expertise in technical translation, he translated some of Walt Lipke’s articles and both of his books into Portuguese.

Paulo published articles and gave presentations on Earned Schedule. Some Brazilian companies began using ES, but adoption has been slow. Once use of EVM increases, ES adoption will increase.

Paulo developed Abacus, an Excel spreadsheet, to serve as a demo and instructional tool for ES. It’s available for download from Walt’s Earned Schedule website.

Paulo then developed an executive project reporting service through his company Techisa Abacus. It features an Excel/VBA application that performs ES analysis and presents results in tabular and graphical form.

Paulo conducts research on ES. He’s pursuing a Ph.D. with Prof. Mario Vanhoucke at Ghent University. Paulo’s research focuses on determining the reliability of a project’s forecasted completion. He’s exploring whether or not the shape of the Performance Measurement Baseline (PMB) affects estimate reliability.

Paulo proposed a reliability categorizer based on the PMB shape. First, he created a measure of schedule topology: the degree to which the schedule is serial (S) or parallel (P).

Next, he introduced Batselier and Vanchoucke’s Regularity Indicator (RI). Projects with higher RI yield better forecast reliability than other projects. And, they are more accurate than the SP indicator.

Paulo combined SP and RI to produce a third categorizer, the PMB Limits Categorizer (PLC). Rather than taking a theoretical approach, he pursued an empirical one. He used empirical data and statistical analysis techniques to identify limits for the PMB curves.

Regular (R) PMB curves were contained within Inner Limits. Irregular (I) PMB curves fell outside the Outer Limits. All other curves were called Medium-regular (M). The position of the PMB curve within the limits was determined by its deviation from a central line.

Paulo used the Batselier and Vanhoucke repository of real project data for his research. He extracted 100 projects out of the 133 projects in the data base, using selection criteria developed by Vanhoucke and Martens. He normalized the data and used statistical analysis to identify the central line. From this, he constructed the limits and moved to categorize projects.

Paulo used real data to test the approach. The average curve for the data had an unexpected shape. Rather than an S-curve, it approximated a straight line (with some irregularities). Construction projects constituted 75% of the sample, and they are known to be mostly serial. That accounted for the shape of the curve. And, they became the focus of his additional research.

Paulo applied statistical techniques to smooth out irregularities in the construction data. The smoothed line was the basis for the limits. They are placed at a fixed distance from the central line. The distance varies depending on the data, and a single parameter is used to control variation in the calculations.

Given the limits, Paulo partitioned the sample projects into R and I categories. Further statistical analysis produced a measure of “forecasting goodness”.

Applied to the sample, the categorizers were ranked. PLC showed the best results in categories balance, clustering quality, and correlation. Also, for the R category, the PLC showed the best mean absolute percentage error.

Conclusions: PLC is superior to SP and RI in the study. It best fits construction projects and small projects rather than large ones.

For future research: validate PLC for megaprojects, sectorize the central line and limits (once more data is available), and create guidelines for using PLC in project planning.