Developing an Efficient and Repeatable Life Cycle Cost Estimation Process

Overview

PRICE Systems, Logapps LLC and ALON Inc. have developed an integrated Life Cycle Cost Estimation (LCCE) capability for the Department of Homeland Security (DHS) Office of the Chief Information Officer (OCIO). One of the major challenges that federal IT program offices encounter is to develop accurate and defendable life cycle cost estimates, given shrinking budgets and limited availability of key resources. Additionally, program offices frequently have neither “embedded” cost estimation resources nor tools required to deliver LCCEs. To respond to this situation, we have leveraged a “best of breed” parametric model (PRICE System’s TruePlanning), cost estimation best practices such as requirements-based software sizing (function point analysis), involvement of trained and experienced cost estimators, and early and effective communication with the program manager and program SMEs. This recipe has allowed our team to deliver defendable cost estimates in a timely manner while achieving client buy-in of the tools, methods and work products.

The make up of our team involves a Program Management Office (PMO) support contractor, a parametric model developer, and an independent cost estimation organization. By combining three unique disciplines, our team was able to leverage the various strengths of the team. ALON provided insight into the PMO as well as access to key decision-makers. PRICE Systems provided its well-respected parametric estimation model, TruePlanning, insight into the embedded cost estimating relationships, and product demonstrations. Logapps provided independent cost estimation services, requirements-based software sizing capability, trained TruePlanning estimators, and use of cost estimation best practices.

RAMP Program

Our team collaborated to deliver an LCCE for the Risk Assessment and Management Program (RAMP) system, which is designed to support aspects of the Federal Protective Service (FPS) mission focused upon improving security at federal facilities. RAMP has four major objectives: increasing Inspector operational efficiency, enforcing a standardized assessment methodology, improving customer service, and maintaining a database for tracking risks, analyzing trends, maintaining oversight on reporting, and managing the workforce.

RAMP will provide specialized hand-held computers to approximately 700 Inspectors in 11 FPS regions supporting some 9000 facilities. Specifically, the RAMP system will assist the Inspector in periodically producing the required Facility Security Assessment (FSA) report and in continually assessing and risk, recommending countermeasures, measuring countermeasure effectiveness, and conducting guard-post inspections.

The Project Manager’s Challenge

Federal IT Program Managers face dwindling budgets, tight schedules and System Integrators that require significant oversight to remain on schedule and within budget. We have observed that many PMOs in civilian agencies, like DHS, do not have access to an organic cost estimation capability. Unlike large DoD organizations, like Naval Air Systems Command (NAVAIR), DHS agencies do not have a pool of cost estimators that can be provided to their organization for the purposes of delivering cost estimation deliverables. Thus, Civilian PMOs need to outsource its cost estimation services. Program Managers also need estimates to be developed with low impact to their organizations. Another key issue is the Program Managers may be incentivized to keep estimates underneath certain thresholds. As is the case with most software development projects, civilian IT Program Managers frequently operate with undefined requirements, which places a burden on the Program Manager to allocate appropriate risk management funds.

The Solution

To operate in this environment, our team focused on a few key issues. By developing requirements-based software estimates using function point methodology, we were able to develop independent estimates without minimal influence from the client. We developed a three-tiered deliverable construct, including a Ground Rules and Assumptions Document (GRAD), Interim Estimate, and Basis of Estimate. The importance of the GRAD is that it forces the PMO and sponsor to agree upon basic assumptions and methodologies, and when the estimate is briefed later, the team was not in the position of defending the ‘number’, but rather the process.
Because our team included the PMO support team (ALON), we were able to collect programmatic and technical data up front. Data collection is typically one of the major challenges faced by estimators, but our team did not have any issues with data availability.

Another key ingredient in our solution was the use of a well-respected parametric cost model (TruePlanning). TruePlanning provided a framework for estimation that ensure all activities of an IT project were included, as well as embedded cost estimation relationships developed thru the analysis of thousands of IT projects. Our team provided informal model training and product demonstrations to the PMO, which aided in ensuring comprehension and support of the LCCE.

Software Sizing

Federal organizations are relying more and more on commercial applications to supplement, enhance, or replace older systems. In recent years, software has become the most expensive component of Information Technology (IT) projects. Accurate software cost estimates are critical to both developers and customers as software size is an essential element in the calculation of estimated project costs and schedules. Not only does size impact the technical solution; it also impacts the project management solution. Failure to estimate size can result in schedule and cost overruns, or even project cancellation.

Software sizing is difficult for number of reasons. First it is performed in a variety of contexts, some requiring a great deal of information about the system and others much less information. Second, software design documents vary in levels of detail, which affects the type sizing methodology . Third, software projects are often a combination of new, reused, and modified components. A sizing methodology must be able to incorporate all three modes.

Function point analysis is extremely useful for the transaction processing systems/data processing applications that make up the majority of federal IT projects.

Function Points can generate size estimates from data available early in a project’s life cycle. Analysts need only detailed specifications. In our case, the Systems Requirements Document (SRD) was available. Other characteristics of function point analysis are that they are independent of the computer language, development methodology, technology or capability of the project team used to develop the application. Function points are well-suited for size estimation from Use Cases, User’s Guide, or List of Requirements.

Our team followed PRICE Systems’ ‘fast function point’ process, which followed International Function Point User’s Group (IFPUG) standards, and involved the following steps:

• Determine Type of Count
• Identify Counting Scope and Application Boundary
• Count Data Functions
• Count Transactional Functions
• Determine Unadjusted Function Point Count
• Determine Value Adjustment Factor
• Calculate Adjusted Function Point Count

Cost Estimation Best Practices

While there is no one “right” process for ensuring quality cost and schedule estimates of software projects, there are several sources that provide guidance on developing high quality cost analyses. Using industry best practices is an excellent method of gaining client support of estimation processes. Two main sources used in our processes are:

• Carnegie Mellon University Software Engineering Institute (SEI) Checklists
• Government Accountability Office (GAO) Cost Assessment Guide

Each source will be briefly discussed to provide a basic understanding of its relevance to the cost estimation process.

SEI Checklists: The SEI released two relevant special reports in 1995 on software cost estimation: A Manager’s Checklist for Validating Software Cost and Schedule Estimates [SEI-95-SR-004] and Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations [SEI-95-SR-005].

The first document, [SEI-95-SR-004], provides seven key questions, and corresponding evidence characteristics, that should be explored when developing a software cost estimate. The seven questions are listed below.

• Are the objectives of the estimate clear and correct?
• Has the task been appropriately sized?
• Are the estimated cost and schedule consistent with demonstrated accomplishments on other projects?
• Have the factors that affect the estimate been identified and explained?
• Have steps been taken to ensure the integrity of the estimating process?
• Is the organization’s historical evidence capable of supporting a reliable estimate?
• Has the situation changed since the estimate was prepared?

The second document, [SEI-95-SR-005], provides two basic checklists: one for the requisites for a reliable estimating process; one for the indicators of an estimating capability.. The requisites for a reliable estimate process are listed below.

• A corporate memory (historical database)
• Structured processes for estimating product size and reuse
• Mechanisms for extrapolating from demonstrated accomplishments on past projects
• Audit trails (Values for the cost model parameters used to produce each estimate are recorded and explained.)
• Integrity in dealing with dictated costs and schedules (Imposed answers are acceptable only when legitimate design-to-cost or plan-to-cost processes are followed.)
• Data collection and feedback processes that foster capturing and correctly interpreting data from work performed

GAO Cost Assessment Guide: The purpose of the Cost Assessment Guide [GAO-07-1134SP] is to share generally accepted best practices for cost estimation and to illustrate the important link between the cost estimate and the earned value management method (used later in the lifecycle).The Cost Guide speaks specifically to validation of a cost estimate by an outside organization, often in the form of an Independent Cost Estimate (ICE). However, the recommended approach toward estimation validation can be used in virtually any estimate.

The Cost Guide is a compilation of experiences from both industry and government representatives. Figure 3 illustrates the estimation process presented in the Cost Guide, which maps well to our three-tiered deliverable process.

Four characteristics of high quality cost estimates are identified, which are listed below.

• Well Documented: An estimate is thoroughly documented, including source data and significance, clearly detailed calculations and results, and explanation of why particular methods and references were selected. Data can be traced back to source documents.
• Comprehensive: An estimate has enough detail to ensure that cost elements are neither omitted nor double-counted. All cost-influencing ground rules and assumptions are detailed in the estimate’s documentation.
• Accurate: An estimate in unbiased, not overly conservative or optimistic and is based on assessment of most-likely costs.
• Credible: Any limitations of the analysis because of uncertainty or bias surrounding the data or assumptions are discussed.

The GAO Cost Assessment Guide also provides specific questions to be considered when reviewing software estimates. We use the checklist in our internal QA process. Some of the keys are listed below.

• Was the software estimate broken into unique categories-new development, reuse, commercial-off-the-shelf, modified, glue code, integration?
• What input parameters-programmer skills, application experience, development language, environment and process-were used for commercial software cost models (such as SEER), and how were they justified?
• How was the software effort sized?
• How were productivity factors determined?
• What are the assumptions regarding code reuse, and are they justified?
• Was software maintenance costs adequately identified and are they reasonable?

The GAO Cost Assessment Guide specifically addresses use of commercial software models. The best practices with parametric models are to calibrate to the user’s environment, provide extensive training, develop formal estimation procedures, and provide frequent updates to the models. It is noted that our team adheres to all of the commercial software model best practices.

Lessons Learned

Our team observed several lessons while delivering the RAMP LCCE. Parametric models do not take the place of mature estimation processes, but are an extremely useful tool in the hands of a trained estimator. Some Program Managers may be distrustful of any cost estimation model, but these concerns can be mitigated through providing high-level training as part of the estimation process. We also observed that following industry best practices, as documented by organizations such as GAO and CMU-SEI, increases client confidence in deliverables.

Conclusion

Our team has developed an integrated cost estimation capability for DHS OCIO by combining a parametric model developer, independent cost estimators, and PMO support contractors. We have provided a mature cost estimation capability on a low-impact basis to a PMO that does not have “embedded” cost estimation resources or tools. Our team provided access and exposure to a parametric model, identified and followed used cost estimation best practices, , and leveraged an on-site PMO contractor to provide early and effective communication with the program manager and program SMEs. We were able to deliver cost estimates in a timely manner while achieving client buy-in of the tools, methods and work products.

References

[CMMI 2007] CMMI: Guidelines for Process Integration and Product Improvement, Second Edition. Boston Massachusetts: Addison Wesley. 2007.
[GAO-07-1134SP] Cost Assessment Guide: Best Practices for Estimating and Managing Program Costs (Exposure Draft). US Government Accountability Office. GAO-07-1134SP. Washington, DC. 2007.
[SEI-95-SR-004] Park, Robert E., A Manager’s Checklist for Validating Software Cost and Schedule Estimates. Software Engineering Institute. Pittsburgh, PA. 1995.
[SEI-95-SR-005] Park, Robert E., Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of Software Organizations. Software Engineering Institute. Pittsburgh, PA. 1995.