Abstract
Purpose
There appears to be no empirical-based method in the literature for estimating if an engineering change proposal (ECP) will occur or the dollar amount incurred. This paper aims to present an empirically based approach to address this shortfall.
Design/methodology/approach
Using the cost assessment data enterprise database, 533 contracts were randomly selected via a stratified sampling plan to build two regression models: one to predict the likelihood of a contract experiencing an ECP and the other to determine the expected median per cent increase in baseline contract cost if an ECP was likely. Both models adopted a stepwise approach. A validation set was placed aside prior to any model building.
Findings
Not every contract incurs an ECP; approximately 80 per cent of the contracts in the database did not have an ECP. The likelihood of an ECP and the additional amount incurred appears to be statistically independent of acquisition phase, branch of service, commodity, contract type or any other factor except for the basic contract amount and the number of contract line item numbers; both of these later variables equally affected the contract percentage increase because of an ECP. The combined model overall bested current anecdotal approaches to ECP withhold.
Originality/value
This paper both serves as a published reference point for ECP withholds in the archival forum and presents an empirically based method for determining per cent ECP withhold to use.
Keywords
Citation
Ellis, J.C., White, E., Ritschel, J.D., Valentine, S.M., Lucas, B. and Cordell, I.S. (2018), "Likelihood and cost impact of engineering change requirements for DoD contracts", Journal of Defense Analytics and Logistics, Vol. 2 No. 1, pp. 22-37. https://doi.org/10.1108/JDAL-02-2018-0002
Publisher
:Emerald Publishing Limited
Copyright © 2018, In accordance with section 105 of the US Copyright Act, this work has been produced by a US government employee and shall be considered a public domain work, as copyright protection is not available.
Introduction and background
History suggests that, by and large, the Department of Defense (DoD) and the military departments have underestimated the cost of buying new weapon systems. Arena et al. (2006) analyzed major DoD programs and discovered these experienced nearly 46 per cent cost growth before the end of Milestone B and another 16 per cent growth by Milestone C. As a recent example, this cost growth trend continued with the Joint Strike Fighter (JSF) program. As of the 2009 Selected Acquisition Report (SAR), the JSF per-unit estimate has grown 57 per cent from its initial October 2001 estimated value.
To combat and possibly militate against cost growth, Congress enacted the Weapon System Acquisition Reform Act of 2009, often called WSARA as public law 111-23. The act created a Pentagon office – Office of Cost Assessment and Program Evaluation (CAPE) – to analyze the cost of defense programs. One particular factor related to cost growth are technical changes. These engineering change proposals (ECPs) can occur for many reasons. An ECP necessitates a scope change to a contract. They can be initiated by the government, the contractor or even feedback from the users.
The Government Accountability Office (GAO) found that 63 per cent of major defense programs had requirement changes after system development began [Government Accountability Office (GAO), 2008]. Additionally, those programs with requirement changes encountered, on average, cost growth of 72 per cent, while costs grew by only 11 per cent among those programs that did not change requirements. The fundamental purpose of an ECP is to change the requirements of a contract [Engineering Change Proposal (ECP), 2017]. To build in flexibility, the acquisition practice is to estimate a dollar value to hold in reserve after the contract is awarded. This amount has several names, for the purpose of this article, we call it ECP withhold, as it is the amount of money the Government withholds for ECPs.
There are three major cost estimating guides commonly used by Air Force cost analysts today: the air force cost analysis handbook (AFCAH), the GAO cost estimating and assessment guide and the USA Air Force – cost risk and uncertainty analysis handbook (AFCRUH). Each provides overlapping material and views to best estimate cost and risk; however, none of the guides provide an empirical-based method for estimating ECP withhold. Given this lack of guidance, practitioners use common rules-of-thumb for ECP withhold.
A relatively common one among the acquisition community is that estimates may vary by 10 per cent. This is seen in several separate fields and disciplines. The starting amount for estimating management reserve is 5-10 per cent (Project Management Institute, 2017). The amount over cost for an acquisition program baseline breach is 10 per cent [Department of Defense (DoD), 2015]. Currently, the Air Force Life Cycle Management Center allots the following percentages. For development cost estimates, a 10 per cent cost is added above the total estimate, while for procurement estimates, that percentage lowers to 6 per cent (S. Valentine, personal communication, multiple dates, 2015-2017). Finally, the automated cost estimating integrated tools software package uses a range between 6-10 per cent.
The purpose of this article is to present empirically based models via historical data that can be used to not only estimate the likelihood of a contract experiencing an ECP but also determine the amount of ECP withhold as a percentage of the total contract cost. In addition, the study compares these models to three alternative rules-of-thumb. The analysis presented in this paper is at the contract level and not a program level.
Database and methodology
CAPE’s mission is to provide independent program analyses and insights as requested by the Under Secretary of Defense for Acquisition, Technology and Logistics and Congress. Additionally, CAPE reviews programs that may be, or already are, struggling in the acquisition process. To facilitate their mission, CAPE initiated the development of the cost assessment data enterprise (CADE), the Department’s initiative to identify and integrate data from disparate databases and systems for better decision-making, management of, and oversight of the Department’s acquisition portfolio.
As of 11 April 2017, CADE hosted a contract level database consisting of 7,343 unique contracts with details extracted from the electronic document access (EDA) system. EDA is a Web-based system that provides for storage and retrieval of not only DoD contracts but also contract modifications [Electronic Document Access (EDA), 2017]. This study uses EDA data extracted from CADE as the primary data source. This database in turn forms the basis from which to develop empirical regression models to predict not only the likelihood of a contract having an ECP but also the additional per cent increase from the ECP.
Both the large number of available contracts and the requisite time to manually check contract details necessitated using a stratified sampling plan. The preliminary intent was to randomly sample 10 per cent of the population; but from the time constraints, the resultant percentage ended closer to 9 per cent (as documented in the next section). Initial stratification included four main criteria consisting of the following contract elements: phase, schedule, size, and type. Phase is divided into development versus other (production/operations and support). Schedule is divided into contracts that have an initial schedule longer than one year or those equal to or less than a year. Size separates contracts that have a baseline cost greater than $5m ($5,000,000) versus contracts that have a baseline cost less than or equal to $5m. Contract type splits contracts by those with greater than 90 per cent fixed firm price (FFP). An additional stratum takes into consideration very large contracts, specifically those exceeding $400m. This was because of the preliminary findings from Cordell (2017). [Note: all dollar amounts are in Base Year (BY) 2016 dollars.] Table I displays the initial strata.
This arrangement presents 16 possible strata from which to sample in addition to the one accounting for very large ECP contracts. As shown in the Analysis and Results section, these 17 total strata ultimately collapse to 7 for sample collection purposes from the population. Sample percentages are statistically matched to that of the population percentages to ensure these bins correspond accordingly and are checked via a paired t-test. In addition, a second paired t-test is conducted to show that the sample matches that of the population with respect to percentile distributions by type of contract, branch of service, and commodity. These inferential tests are conducted at a significance level of 0.05.
Once the stratification plan is finalized, errant contracts are filtered out prior to populating the study’s database. These errors may include, for example, missing contract dates, missing contract amounts or even a negative contract award since a contract cannot possess a negative value. Other errors may also include modifications being incorrectly classified as a cost modification despite adding scope, which is an ECP by definition. In the next section, we list the main errors detected in building the modeling database and the number affected by each exclusion criteria. As mentioned previously, all dollar amounts are converted to BY 2016 dollars to account for the effects of inflation. Because of the unavailability of military appropriation category for the contracts in the database and the length of some aircraft contracts, we used the total manufacturing producer price index as reported by the Bureau of Labor Statistics for the conversions. All analysis in this article used JMP12 Pro, Excel or R.
The models presented in this article predict two response variables. The first is a binary (dichotomous) variable for the logistic regression model. If a contract has any technical ECPs, the response is a 1. If the contract has no technical ECPs, the response is a 0. The second is a continuous variable for the ordinary least squares (OLS) model. This variable is the natural logarithm of the per cent cost growth strictly relating to changing requirements. [Note: the highly skewed percentage growth, in addition to other skewed predictors variables, necessitated a log-log modeling approach. This is mentioned and demonstrated via a graph in the next section.] The per cent ECP growth is the sum of all modifications that are listed as a change in requirements, divided by the contract’s baseline cost. The end result of the OLS model is the predicted median per cent of ECP withhold.
With respect to identifying possible explanatory variables either associated with the likelihood of an ECP occurring or the median per cent of ECP withhold, we turn to the literature. Trudelle et al. (2017); Bolten et al. (2008) and Arena et al. (2006) document several potential variables to be predictive factors for determining if a program will experience cost growth. Additionally, Harmon and Arnold (2013) performed an assessment on contracts types, attempting to understand the impact of overall contract price. They determined that for a series of production contracts in which the system design is mature and stable, the best choice of contract type is FFP. This finding played a key role in determining one of the strata in the sampling plan. Table II lists the possible explanatory variables considered in the analysis to predict the likelihood of a contract experiencing an ECP as well as the expected median percentage increase. As noted in the next section, given the large number of F/A-18E/F contracts in the population, a dichotomous predictor variable, labeled F18, has been added to statistically test if that program overly influences the modeling database either with the likelihood of a contract experiencing an ECP or the expected median percentage increase.
Prior to any model building, the study’s database is randomly partitioned into two components: the modeling data set and the validation data set. This is accomplished by generating a random uniform number for each contract, sorting from smallest to largest random number, and then pulling the requisite percentage as needed. For the logistic model, because of the large sample size, approximately 50 per cent of the contracts are set aside for model validation. For the OLS model, approximately 20 per cent of the contracts that experienced an ECP are set aside for validation since the sample is greatly reduced when modeling just those contracts with an ECP. None of the contracts in the validation set were used to create the respective statistical models.
Before validating the OLS model, the customary residual assumptions of normality and constant variance are tested by using the Shapiro–Wilk test and the Breusch–Pagan test, respectively. Both are conducted at the 0.05 level of significance. Additionally, multicollinearity, outliers and influential data points are investigated to prevent model bias. Variance inflation factors (VIF) highlight the linear relationship between independent variables and a VIF score higher than 5 suggests multicollinearity. Regarding outliers, any studentized residual greater than three standard deviations is categorized as an outlier and a possible source of concern. Finally, Cook’s Distance detects overly influential data points possibly skewing the results. Any value greater than 0.5 is investigated closely.
To assess validity of the developed logistic regression model, the confusion matrix is used. This matrix assesses the number of true positives, true negatives, false positives and false negatives, respectively. A cutoff criterion of 0.5 is set as the prediction threshold to separate a contract into “ECP likely” vs “ECP not likely”. The validity of the OLS model is assessed in multiple criteria: mean absolute per cent error (MAPE), median absolute per cent error (MdAPE), coefficient of determination (R2) and adjusted R2. Each absolute per cent error is calculated as the absolute value of the difference between a predicted response minus the actual response divided by the actual response.
For both the OLS model and the logistic model, a mixed stepwise procedure is adopted to arrive at the models presented in the next section. A level of significance is set to 0.01 to determine initial predictive ability of an explanatory variable. From there, the preliminary selected variables are investigated to determine their practical effect on the respective model. If a particular explanatory variable is determined to have less than a 1 per cent relative effect on a particular model’s response, then that variable is excluded from being in the final model that is presented for practitioners’ use. This is done to minimize a variable being statistically significant but having little practical effect.
To finalize the presented results, four different methods and their recommended ECP withholds are compared descriptively. The first method uses the application of the presented regression models in this article. The second method involves having no ECP withhold (essentially assuming no additional costs for a contract). The third adopts the per cent found anecdotally in the literature: 6 per cent for development and 10 per cent for procurement contracts. The last method simply uses a flat average. This average is the average per cent ECP growth for all contracts with no discrimination between life-cycle phases and applied indiscriminately to all contracts.
Analysis and results
From an initial population of 7,343 contracts housed within the CADE database on 11, April 2017, 1,416 were excluded because of missing or erroneous data, resulting in an effective population size of 5,927 or approximately 81 per cent of the total starting number. Table III highlights the main reasons that attributed to approximately 84 per cent of the exclusions. Of the 1,416 contracts removed prior to creating a stratified random sample for the study’s analysis, missing contract type associated with a contractual amount was the dominant exclusionary reason, accounting for approximately 43 per cent of the 1,416 contracts removed for consideration. The next highest reason was missing an end date to a modification of an initial contract. That reason accounted for 313 contracts or approximately 22 per cent of the excluded contracts.
With respect to the bin characteristics as discussed in the previous section, Table IV shows the population percentages, while Table V highlights the final selected seven strata percentages for both the population and sample, respectively. The paired t-test comparing the percentages of the population strata to that of the sample strata results in a p-value of 0.96, which concludes in failing to reject the null hypothesis. Therefore, at a 0.05 level of significance, the paired t-test strongly suggests that the modeling database appears statistically equivalent to the population.
Regarding acquisition phase, branch of service and commodity type, Table VI lists the percentages by both population and sample. Conducting another paired t-test comparing the percentages of the population strata to that of the sample strata results in a p-value of 0.99, which again supports the preceding results that the modeling database appears statistically equivalent to the population. One other noticeable conclusion from Table VI highlights a high number of Navy or aircraft contracts in the population.
Delving further, a large per cent of the population contracts stem from the F/A-18E/F (Super Hornet) program only. Approximately 42 per cent of all the contracts originated from the F/A-18E/F with the second highest being approximately 4 per cent for the AWACS (Airborne Warning and Control System) program. However, in terms of total cost (sum of all contracts for a given program), the F-22 (Raptor) leads all programs and account for approximately 7.6 per cent of the total population contract cost, while the F/A-18E/F accounts for approximately 5.9 per cent of the total amount. In total, the population database contains 132 unique programs, while the random sample contains 68 unique programs. Appendix lists all the programs considered in the sample database.
Logistic model
Out of the 541 contracts in the sample database, 271 were randomly set aside for the validation set. The remaining 270 were used as the modeling set for the logistic model. Customarily, an 80/20 ratio is used; however, because of the relatively large size of the sample database, a 50/50 split was preferable to allow for greater generalization testing. A relatively equal number of ECPs were in each set, which was commensurate with the overall ECP percentage of approximately 17 per cent.
Prior to commencing any analysis, a histogram reflecting the baseline cost for all 541 contracts highlighted 8 contracts (approximately 1.5 per cent of the sample database) that noticeably appeared as outliers, whereas the remaining 533 contracts (which were equal to or less than $164m in Fiscal Year 2016 dollars) had a relatively smooth lognormal distribution (p-value of approximately 0.04 for the Kolmogorov–Smirnov goodness of fit statistic, which is reasonable given the large sample size). The smallest baseline cost of these eight outliers was $260m, while the largest was $2.6bn. Given the sparsity and 10-fold difference between the lowest and largest baseline contract cost of these eight, we chose at this point to remove from consideration these contracts, which in turns removes from consideration the explanatory variable of contract size extra-large (programs exceeding $400m) and limits the paper’s inferential results to contracts $160m or less.
With this change in mind, the model building set reduced to 265 and the validation set lowered to 267. Table VII highlights two possible models for predicting the likelihood of a contract experiencing an ECP as selected by stepwise regression. The explanatory variable Ln (Baseline cost) [the natural logarithm of a contract’s baseline cost] and the explanatory variables contract size large and contract size small are complementary in nature given both small and large contracts are included in the entire gambit of baseline contract costs. Therefore, stepwise only flagged one model at a time as being significant, but we chose both going forward into validation to determine which of the two might be ultimately better in predicting the likelihood of an ECP. No other explanatory variable proved statistically significant given the cost of the contract was already in the model, including the F18 variable. The overall takeaway is that the cost of the contract appears to be the overwhelmingly dominant factor in determining the likelihood of an ECP.
With respect to how well both models predict a contract experiencing an ECP, Table VIII displays the confusion matrix for the model building and validation datasets. Overall, both models reflect a high predictive ability for detecting contracts that do not experience an ECP; however, both reflect quite poorly at predicting ECPs, with Model 1 displaying some ability to predict the true likelihood of an ECP. Model 2 (as evident from Table VII) reflects that a breakpoint potentially occurs somewhere in the lower cost spectrum for contracts equal to or less than $100,000 have a lower chance of incurring an ECP compared to contracts greater than $5m. In fact, of the 533 contracts in the sample database whose baseline cost is less than or equal to $100,000 (171 in total), only one contract had an ECP. We use this information later in terms of overall model application.
OLS model
The linear (OLS) model is designed to predict the amount of cost growth solely attributable to ECPs. This response is in the form of the natural logarithm of the percentile increase. To revert to the actual expected percentage, one would take the natural exponent of the predicted value, which results in the expected median percentile increase due to incurring an ECP. As mentioned in the previous section, contracts that had a net negative ECP growth are not considered. This occurs whenever a contract de-scopes effort and cannot be used to obtain an accurate ECP withhold before contract award.
Of the 541 contracts in the original sample database, 99 experienced a contract increase because of an ECP. Of these 99, 20 were initially randomly set aside for the validation set, while the remaining 79 were used to develop the OLS model. After accounting for the eight contracts excluded because of a very large baseline contract cost (exceeding $164m), these numbers changed to 71 for the modeling dataset with 20 remaining in the validation set, respectively.
Given the highly skewed pattern of ECP percentage increase, basic contract cost, baseline contract cost and the schedule (in days) of the contract, all of these variables were transformed via the natural logarithm function. As an example of this, Figure 1 displays the typical skewed right pattern of the basic contract cost in addition to its distribution after the transformation.
Table IX highlights the preliminary model for predicting the expected natural log percentage increase in contract cost because of an ECP. The model has an R2 of 0.37 and an adjusted R2 of 0.35, respectively. No other explanatory variable proved statistically significant after accounting for the cost of the basic contract along with the number of CLINs truncated at five (that is five or more CLINs are grouped into the five CLIN group). In no iteration did the F18 variable prove statistically significant, similar to the findings of the logistic model. The candidate model in Table IX passed all model diagnostics with no issues with multicollinearity (largest VIF score of 1.13), outliers (largest studentized residual value of 2.5), influential datapoints (largest Cook’s D value of 0.13), normality (Shapiro–Wilk test p-value of 0.47) or constant variance (Breusch–Pagan test p-value of 0.50).
For the OLS model, for both the modeling and validation datasets, the MAPE and MdAPE are 86 per cent and 35 per cent and 130 per cent and 36 per cent, respectively. The higher MAPEs in comparison to the lower MdAPEs reflects moderate outliers in both datasets, while the relative comparable MdAPEs are a better measure of the model’s consistency and generalization. Table X highlights the final empirical model after updating with the validation dataset and used in conjunction with the results of the previously presented logistic models. Equation (1) represents the user model in mathematical form, taking into consideration back-transformation. Note: a user would only use equation (1) for any contract whose baseline cost exceeds $100,000 but is less than or equal to $164m (both in BY $16). Otherwise, the expected median percentage would be zero given a very, very low chance of lower cost contracts (equal to or less than $100,000) incurring an ECP:
Table XI compares using equation (1) to three other ECP withhold methods. For all methods, we use the entire sample database of 533 contracts for the comparisons. The first method assumes no ECP withhold, essentially assuming the contract never experiences an ECP. The second method, referenced in the literature, uses 6 per cent for development and 10 per cent for procurement contracts. The last and third method uses a simple flat average applied indiscriminately regardless of phase. The average ECP withhold of the 533 contracts in the sample database is approximately 5.9 per cent. This is the value used for the flat average method. Based on the average ranking of best (1) to worst (4), equation (1) appears to be the best method of the four presented with an average rank of 1.8. All methods used the true final cost of a contract to determine the percentages and amounts. Final cost equaled total baseline cost in addition to any ECP amounts documented.
Discussion and conclusion
To the best of our knowledge, no peer-reviewed source could be found that documents the amount of ECP withhold that should be set aside for DoD contracts. Only anecdotal amounts were present in the literature. The aim of this paper served dual purposes: one, as a published reference point for ECP withholds in the archival forum; and two, derive an empirically-based method for determining percent ECP withhold.
Based on the analysis presented, several points became evident. One, not every contract incurs an ECP; however, ECPs do occur and not budgeting accordingly results in a serious shortfall as shown in Table XI. Two, both the likelihood of an ECP and the additional amount incurred appears to be statistically independent of acquisition phase, branch of service, commodity, contract type or any other factor except for the basic contract amount and the number of contract line item numbers (CLINs). Both of these variables equally affected the contract percentage increase due to an ECP. Finally, the logistic regression approach proved a poor predictor of determining the likelihood of a DoD contract incurring an ECP. However, it did provide invaluable insight that lower cost contracts appeared statistically less likely to incur an ECP. Preliminary analysis suggests that this breakpoint might be around $100,000; however, future research is encouraged to further delve into this lower boundary.
As with any research, limitations do exist for the results in this paper. Quality statistical analysis depends on quality data. Therefore, any errors within CAPE’s database pulled from EDA will pass down to the sample database that formed the conclusions stated in this paper. Additionally, the encouraged use of equation (1) requires a portfolio managed approach to contracts in an organization. That is, an agency or manager overseeing a multitude of contracts is able to move ECP withhold amounts from contract to contract as needed. In that context, the OLS model as shown in Table XI represents an almost balanced approach. Finally, using equation (1) for contracts exceeding $164m in BY 16 dollars would be model extrapolation, and we caution against such use.
The field of changing requirements and their impact at the contract level is full of opportunity. One major recommendation we provide is to use or add a different source of data. Adding data from the SARs might provide details on program elements that might increase the chance of all contracts within that program experiencing an ECP. Another source of information is EVM reports. Not only could the EVM metric provide a snapshot of contract health but also research has shown that sentiment analysis of EVM/Status Reports of programs might provide insightful information and prediction capability (Freeman, 2013).
Finally, we suggest simultaneous analysis at the contract and the program level. From our experience, many programs will start a new contract rather than adding requirements on an existing contract. This practice, while valid and legal, may skew the analysis if performed solely at the contract level. That is, a program might experience cost growth by adding new contracts, while existing contracts show no increased cost. Overall, a broader and more holistic view is needed to accurately assess the impact of changing requirements for the final cost of a DoD program and elements that affect its bottom-line.
Figures
Initial population stratum characteristics
| Population stratum elements (name) | Characteristic present |
|---|---|
| 1 (DEV) | Development contracts |
| 2 (Non-DEV) | Production or operations and support contracts |
| 3 (Short) | Initial contract duration equal to or less than a year |
| 4 (Long) | Initial contract duration longer than a year |
| 5 (Small) | Baseline contract cost equal to or less than $5,000,000 |
| 6 (Large) | Baseline contract cost exceeds $5,000,000 but less than $400,000,000 |
| 7 (FFP) | Total per cent of initial contract type and modification contract types greater than 90% FFP |
| 8 (Non-FFP) | Total per cent of initial contract type and modification contract types is equal to or less than 90% FFP |
| Special | Baseline contract cost equals or exceeds $400,000,000 |
Strata pairs 1/2, 3/4, 5/6, and 7/8 are complementary events. All dollars presented in base year 2016 values. Baseline contract cost equals initial contract cost plus all priced options
Explanatory variables considered in the development of the logistic regression model to predict the likelihood of a contract having an ECP and the expected median percentage increase caused by the ECP
| Variable name | Description of subcategories |
|---|---|
| Phase | Acquisition phase of the contract: operations and support, production, or development |
| Service | Branch in the Government that let the contract: Navy, Air Force, Marine Corps, Army or Department of Defense (Joint) |
| Commodity type | Majority of product on contract. For example, Aircraft, Missile, Ground Vehicle, AIS (Automated Information System), Munitions, Ordnance, Electronics, etc. A total of 17 types were considered |
| Contract type | Funding outlay as defined in Federal Acquisition Report (FAR) Part 16 Procurement. For example, Fixed-Price Contracts, Cost-Reimbursement Contracts, or Incentive Contracts. A total of 9 types were considered |
| Mod category | Assigned classification of the contract modification (technical, baseline, or schedule for example). A total of 8 categories were considered |
| Baseline cost | Cost in Fiscal Year (FY) 2016 dollars of the initial contract plus priced options |
| Basic cost | Cost in Fiscal Year (FY) 2016 dollars of the initial contract |
| Schedule | Length of the contract in terms of days |
| Contract Year Start/End | Date the initial contract started or ended. Investigated years 2010, 2011, and 2012 to determine if the Weapon Systems Acquisition Reform Act (WSARA) of 2009 or Better Buying Power (BBP) initiatives launched in 2010 had an effect on a contract |
| F18 | This dichotomous variable assumes a value of ‘1’ if the contract is a part of the F/A-18E/F program |
| Few CLINs | This dichotomous variable assumes a value of ‘1’ if the number of CLINs equaled 5 or less. This number was chosen since it represented the 90th percentile of CLINs in the database. Only 10 % of contracts have more than 5 CLINs on their basic contract |
| CLINs on Basic Truncated at 5 | This continuous variable assumes the value equal to the number of CLINs if the number of CLINs equaled 5 or less, else it assumes the value of 5 |
| CLINs on Basic | This continuous variable assumes the value equal to the number of CLINs regardless of the number |
Primary contract exclusion reasons
| Criteria | No. | (%) |
|---|---|---|
| No contract type associated with the dollar value given | 603 | 43 |
| Missing end date of a modification to an initial contract | 313 | 22 |
| Missing end date to the initial contract | 183 | 13 |
| Total contract amount summed (initial plus any modifications) to a negative dollar value | 79 | 6 |
Percentages rounded to two decimal places
Population breakdown of the 5,927 contracts and associated percentages
| Bin no. | Contract phase | Contract length | Contract cost | Contract type | No. | Population (%) |
|---|---|---|---|---|---|---|
| 1 | Non-DEV | Short | Small | FFP | 1,360 | 23 |
| 2 | Non-DEV | Short | Small | Non-FFP | 1,147 | 19 |
| 3 | Non-DEV | Long | Small | FFP | 966 | 16 |
| 4 | Non-DEV | Long | Small | Non-FFP | 767 | 13 |
| 5 | Non-DEV | Long | Large | FFP | 367 | 6 |
| 6 | DEV | Short | Small | Non-FFP | 264 | 4 |
| 7 | Non-DEV | Long | Large | Non-FFP | 227 | 4 |
| 8 | Non-DEV | Short | Large | FFP | 174 | 3 |
| 9 | DEV | Long | Large | Non-FFP | 153 | 2 |
| 10 | Non-DEV | Short | Large | Non-FFP | 135 | 2 |
| 11 | DEV | Long | Small | Non-FFP | 99 | 2 |
| 12 | DEV | Short | Small | FFP | 82 | 1 |
| 13 | DEV | Long | Small | FFP | 52 | 1 |
| 14 | DEV | Short | Large | Non-FFP | 50 | 1 |
| 15 | DEV | Long | Large | FFP | 25 | 0* |
| 16 | DEV | Short | Large | FFP | 7 | 0* |
| 17 | N/A | N/A | N/A | N/A | 52 | 1 |
Percentages rounded to the nearest whole number; the 0%*denotes a percentage less than 1%
Consolidated strata for study sample along with population and matching sample characteristics
| Stratum no. | Bin Grouping from Table III | Population no. | Population % (Out of 5,927) | Sample no. | Sample % (Out of 541) | Paired percentile difference |
|---|---|---|---|---|---|---|
| 1 | 1 | 1,360 | 22.9 | 118 | 21.8 | 1.1 |
| 2 | 2 | 1,147 | 19.4 | 101 | 18.7 | 0.7 |
| 3 | 3 | 966 | 16.3 | 92 | 17.0 | −0.7 |
| 4 | 5,7,8, and 10 | 903 | 15.2 | 84 | 15.5 | −0.3 |
| 5 | 4 | 767 | 12.9 | 71 | 13.1 | −0.2 |
| 6 | 6,9,11-16 | 732 | 12.4 | 70 | 12.9 | −0.5 |
| 7 | 17 | 52 | 0.9 | 5 | 0.9 | 0 |
Percentages rounded to the nearest tenth
Population and sample characteristics by acquisition phase, service, and commodity type
| Variable | Subcategory | Population/Sample no. | Population/Sample percentage |
|---|---|---|---|
| Acquisition phase | Operations and Support | 2,822/252 | 47.6/46.6 |
| Production | 2,364/218 | 39.9/40.3 | |
| Development | 741/71 | 12.5/13.1 | |
| Service | Navy | 3,286/300 | 55.4/55.5 |
| Air Force | 1,375/133 | 23.2/24.6 | |
| Marine Corps | 726/63 | 12.2/11.6 | |
| DoD | 329/21 | 5.6/3.9 | |
| Army | 211/24 | 3.6/4.4 | |
| Commodity type | Aircraft | 3,856/358 | 65.1/66.2 |
| Missiles | 565/62 | 9.5/11.5 | |
| Ground Vehicle | 483/43 | 8.2/8.0 | |
| AIS | 453/30 | 7.6/5.5 | |
| M&O | 235/20 | 4.0/3.7 | |
| Electronics | 221/17 | 3.7/3.1 | |
| Other | 114/11 | 1.9/2.0 |
Commodity Other group consists of contracts for unmanned aerial vehicles, decoys, engines, guns, lasers, non-lethal systems, radar, ships, space, or targets/drones, AIS stands for Automated Information Systems contracts, while M&O stands for Munitions and Ordnance contracts. Percentages rounded to the nearest tenths
Candidate models for predicting the likelihood of a contract experiencing an ECP
| Model no. | Variable | Estimate | p-value |
|---|---|---|---|
| 1 | Intercept | −8.41 | <0.0001 |
| Ln (Baseline cost): natural logarithm of the contract baseline cost | 0.50 | <0.0001 | |
| 2 | Intercept | −1.52 | <0.0001 |
| Contract size large | 1.44 | 0.0001 | |
| Contract size small | −2.96 | 0.0041 |
All values rounded to two significant digits
Confusion matrices for the two logistic regression candidate models for predicting the likelihood of a contract experiencing an ECP
| Model no. | True positives | True negatives | False positives | False negatives |
|---|---|---|---|---|
| 1 Model | 13 (28%) | 208 (95%) | 10 (5%) | 34 (72%) |
| 1 Test | 13 (28%) | 199 (90%) | 23 (10%) | 33 (72%) |
| 2 Model | 0 (0%) | 218 (100%) | 0 (0%) | 47 (100%) |
| 2 Test | 0 (0%) | 222 (100%) | 0 (0%) | 46 (100%) |
Parenthetical percentages reflect the accuracy rate of the chosen metric. Information is reflective of both the modeling dataset (Model; 265 contracts) and validation set (Test; 268 contracts). Percentages rounded to two decimal places
Candidate linear model for predicting the natural logarithm of the expected percentage contract increase due to experiencing an ECP
| Variable | Estimate | t-ratio | p-value |
|---|---|---|---|
| Intercept | 3.34 | 2.42 | 0.0184 |
| Ln (Basic Cost) | −0.30 | −3.13 | 0.0025 |
| CLINs on Basic Truncated at 5 | −0.41 | −4.14 | <0.0001 |
Numbers truncated to two decimal places
Final linear model for predicting the natural logarithm of the expected percentage contract increase due to experiencing an ECP
| Variable | Estimate | t-ratio | p-value |
|---|---|---|---|
| Intercept | 3.32 | 2.93 | 0.0043 |
| Ln (Basic Cost) | −0.30 | −3.87 | 0.0002 |
| CLINs on Basic Truncated at 5 | −0.33 | −3.76 | 0.0003 |
Numbers truncated to two decimal places
Comparison of the method presented in this paper [equation (1)] to having no ECP withhold, engaging the apparent current practice of 6 per cent ECP withhold for development contracts and 10 per cent ECP withhold for non-development contracts, and applying a straight average ECP percent to all contracts (5.9 per cent for the sample database)
| Model Method | Underfund (%) | Overfund (%) | Underfund (Dollars) | Overfund (Dollars) | Portfolio Total | Overall Rank |
|---|---|---|---|---|---|---|
| Equation (1) | 9.38 | 58.90 | (178M) | 197M | 19M | 1 |
| No Withhold | 17.40 | 0 | (262M) | 0M | (262M) | 3 |
| Current Practice | 9.57 | 89.30 | (169M) | 284M | 116M | 3 |
| Flat Average | 11.30 | 88.70 | (175M) | 217M | 42M | 2 |
| Rank | Rank | Rank | Rank | Rank | Average Rank | |
| Equation (1) | 1 | 2 | 3 | 2 | 1 | 1.8 |
| No Withhold | 4 | 1 | 4 | 1 | 4 | 2.8 |
| Current Practice | 2 | 4 | 1 | 4 | 3 | 2.8 |
| Flat Average | 3 | 3 | 2 | 3 | 2 | 2.6 |
All dollar amounts rounded to the nearest dollar (BY 16)
Appendix. Department of defense programs used in the modeling database
F/A-18E/F (Super Hornet).
AWACS (Airborne Warning and Control System).
LVSR (Logistics Vehicle System).
C-5 (Galaxy).
MTVR (Medium Tactical Vehicle Replacement).
B-52 CONECT (B-52 Stratofortress Combat Network Communications Technology).
SH-60/HH-60H/MH-60 (Seahawk).
AHLTA (Armed Forces Health Longitudinal Technology Application).
V-22 (Osprey).
F-15 RMP (F-15 Radar Modernization Program (RMP)).
B-2 (Spirit).
GCSS-MC (Global Combat Support Systems - Marine Corps).
BGM-109 (Tomahawk Block IV).
UH-60L/UH-60M/HH-60M (Black Hawk).
MRAP (Mine Resistant Ambush Protected Vehicle).
MIM-104F (PAC-3) Patriot Missile upgrade.
MIM-104A/B/C/D (Patriot Missile).
F/A-18A-D (Hornet).
P-8A (Poseidon).
H-1 (H-1 Upgrades).
AGM-84/SLAM E (Harpoon Missile).
AGM-154C (Joint Standoff Weapon).
JDAM (Joint Direct Attack Munition).
JASSM (Joint Air-to-Surface Standoff Missile).
AIM-9 (Sidewinder Missile).
F-35 (Lightning II).
Essentris (Documentation and electronic medical record system).
MQ-9 (Reaper, unmanned aerial vehicle).
GBU-53/B (Small Diameter Bomb II).
GBU-24 (Paveway III, laser-guided missile).
AGM-158 (Joint Air-to Surface Standoff Missile).
LAIRCM (Large Aircraft Infrared Counter-Measure system).
SLAM ER (Standoff Land Attack Missile Expanded Response, cruise missile).
F-22 (Raptor).
GBU-12 (Paveway II).
AIM-120 (Advanced Medium-Range Air-to-Air Missile (AMRAAM)).
CHCS (Composite Health Care System).
GBU-39 (Small Diameter Bomb).
DCGS-N (Distributed Common Ground Station-Navy).
AGM-88 HARM (HARM (High-speed Antiradiation Missile)).
RIM-66 (SM-2 BLK IVA, Missile Program).
E-3A (Sentry).
BQM-34 (Firebee, target drone).
C-40 (Clipper).
ADM-160 MALD (Miniature Air-Launched Decoy).
MHS (Military Health System).
AV-8B (Harrier II).
F-16 (Fighting Falcon).
CH-47 (Chinook).
CH-53E (Sea Stallion).
Standard Missile Program (Shipborne guided missiles).
ASIP (Advanced Special Improvement Models, radio
WCMD (Wind Corrected Munitions Dispenser system).
AGM-65 (Maverick, air-to-ground tactical missile).
EA-18G (Growler).
CBU-97 (SFW, Sensor Fuzed Weapon).
C-130 (Hercules).
JLTV (Joint Light Tactical Vehicle).
B-2 DMS (B-2 DMS Modernization).
AEHF (Advanced Extremely High Frequency communication satellites).
AWS (Affordable Weapon System, cruise missile program).
ASLS (Aircraft Structural Life Surveillance program).
MGM-140 (ATACMS, Army Tactical Missile system).
AH-64 (Apache).
F-136 Engine.
E-2D (Advanced Hawkeye).
GQM-163 (Coyote, supersonic sea skimming target).
MQ-4C (Triton, unmanned aircraft system).
References
Arena, M.V., Leonard, R.S., Murray, S.E. and Younossi, O. (2006), Historical Cost Growth of Completed Weapon System Programs (TR-343), RAND Corporation, Santa Monica, CA.
Bolten, J.G., Leonard, R.S., Arena, M.V., Younossi, O. and Sollinger, J.M. (2008), Sources of Weapon System Cost Growth: Analysis of 35 Major Defense Acquisition Programs (MG 670), RAND Corporation, Santa Monica, CA.
Cordell, I.S. (2017), “Engineering change orders and their impact on DoD acquisition contracts”, Unpublished masters thesis, Air Force Institute of Technology, Dayton, OH.
Department of Defense (DoD) (2015), Operation of the Defense Acquisition System (DoDI 5000.02), Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, Washington, DC.
Electronic Document Access (EDA) (2017), AcqNotes, available at: http://acqnotes.com/acqnote/careerfields/electronic-document-access-eda (accessed 12 January 2018).
Engineering Change Proposal (ECP) (2017), AcqNotes, available at: http://acqnotes.com/acqnote/careerfields/engineering-change-proposal-ecp (accessed 12 January 2018).
Freeman, C.E. (2013), Multivariate and Naïve Bayes Text Classification Approach to Cost Growth Risk in. Department of Defense Acquisition Programs, Defense Technical Information Center, Fort Belvoir, VA, available at: www.dtic.mil/dtic/tr/fulltext/u2/a583708.pdf (accessed 12 January 2018).
Government Accountability Office (GAO) (2008), Defense Acquisitions: Assessments of Selected Weapon Programs (08-467SP), US Government Printing Office, Washington, DC.
Harmon, B.R. and Arnold, S.A. (2013), Choice of Contract Type and Other Policy Initiatives for Reducing Contract Prices (D-5002), Institute for Defense Analyses, Alexandria, VA.
Project Management Institute (2017), A Guide to the Project Management Body of Knowledge (PMBOK Guide), 6th ed., Project Management Institute, Newtown Square, PA.
Trudelle, R., White, E.D., Koschnick, C., Ritschel, J.D. and Lucas, B. (2017), “Estimating an acquisition program’s likelihood of staying within cost and schedule bounds”, Defense Acquisition Research Journal, Vol. 24 No. 4, pp. 600-625, available at: https://doi.org/10.22594/dau.17-775.24.04 (accessed 12 January 2018).
Corresponding author
About the authors
Captain James C. Ellis is a Cost Analyst at Eglin Air Force Base, Florida. Captain Ellis graduated with a Bachelor of Arts (BA) in Business Finance and in Business Management from Troy University in 2012, and a Master of Science (MS) in Cost Analysis from the Air Force Institute of Technology (AFIT) in 2018.
Dr Edward White is a Professor of Statistics in the Department of Mathematics and Statistics at AFIT. He received his BS in Mathematics from the University of Tampa, MAS from The Ohio State University, and PhD in Statistics from Texas A&M University. His primary research interests include statistical modeling, simulation and data analytics.
Jonathan D. Ritschel, PhD, is an Assistant Professor of Cost Analysis, in the Department of Systems Engineering and Management at AFIT. He received his BBA in Accountancy from the University of Notre Dame, his MS in Cost Analysis from AFIT and his PhD in Economics from George Mason University. Dr Ritschel’s research interests include public choice, the effects of acquisition reforms on cost growth in DoD weapon systems and economic institutional analysis.
Mr Shawn M. Valentine is an Operations Research Analyst at AFLCMC/FZC at Wright Patterson AFB, Ohio. Mr Valentine graduated from Ohio University with a Bachelor of Science (BS) in Actuarial Science and a Master of Science (MS) in Financial Economics.
Brandon Lucas Lieutenant Colonel Lucas currently serves as an Assistant Professor and Director of the Graduate Cost Analysis Program at AFIT, Wright-Patterson AFB, Ohio. He has completed a MS in cost analysis and a PhD in economics. He has served in the budget, cost and finance communities at base, center and Air Staff levels.
Captain Ian S. Cordell is a Cost Analyst for AFLCMC/HNI at Hanscom AFB, MA. Captain Cordell graduated with a Bachelor of Arts (BA) from the University of Central Florida in Political Science, and a Master of Science (MS) in cost analysis from the Air Force Institute of Technology (AFIT) in 2017.