Measurement Automation Using MARINE
by Logapps LLC
November 9, 2018
Introduction
Advances in technology have allowed time-consuming, manual and repetitive tasks to be automated through artificial intelligence. Law firm associates are utilizing E-Discovery software using syntactic analysis and keyword recognition. The combination of machine algorithms and Big Data has automated financial analytics that were once the domain of financial advisors and equity analysts. Thus, it stands to reason that software measurement can also be automated. This white paper discusses a tool to improve and automate the requirements analysis and software sizing process. A cutting-edge application developed by Logapps, Machine Assisted Requirements Inspection and Evaluation (MARINE), automates the review of software requirements for quality and consistency, and develops high-level software size estimates through Function Point automation. Software measurement professionals will not be replaced; some mundane tasks can be automated through artificial intelligence. Such innovations will help analysts understand requirements, identify duplication in both language and meaning and dramatically reduce the time and effort necessary to accurately analyze projects.
The Key to Successful Software Planning
The first step in most software projects is for Business Analysts to elicit requirements from stakeholders and document within a requirements management tool. Requirements come in the form of requirement statements, user stories, use cases, and other formats. Well-written requirements specify system behavior and determine what should be implemented. Poorly written requirements often result in lower quality, ineffective design, additional rework and unnecessary test runs. Once functional requirements have been documented, they can be sized through function point analysis, though the accuracy of the functional size measurement is largely dependent on the quality of the requirements. Although MARINE is not a requirement management system, it can identify duplicate and similar requirements, non-functional requirements, and improve requirement quality.
“The hardest single part of building a software system is deciding precisely what to build. No other part of the conceptual work is as difficult as establishing the detailed technical requirements . . . No other part of the work so cripples the resulting system if done wrong. No other part is as difficult to rectify later.”
– Fred Brooks, American computer architect, computer scientist, and software engineer [1].
Poor requirements are a problem that has plagued the software industry since its earliest years. Similarly, software estimates in the early stages of a project are frequently inaccurate due to the cone of uncertainty, as changes in requirements and technology can have a huge impact on software development costs. Part of the reason is the difficulty in finding resources with the talents and background needed to parse through hundreds and sometimes thousands of requirements in an effort to define system design needs, determine the expected software size and accurately estimate the software development cost. This detail-oriented and demanding process requires analysts, who are often functional experts, but not necessarily software developers, to have a tremendous amount of tolerance for repetitive requirements parsing—and despite even the most meticulous analysis, mistakes are still inevitable.
From Good to Great: Machine Assisted Analysis and Automation
The capability to automate functional sizing has evolved due to changes in technology, such as Natural Language Processing (NLP) and Artificial Intelligence. In 2013, Object Management Group (OMG) adopted the Automated Function Point (AFP) specification using the International Function Point User Group (IFPUG) counting guidelines. This adoption was met very positively from the functional sizing community. Capers Jones, noted software industry Subject Matter Expert, stated, “the arrival of automated high-speed function point counting…will elevate the importance of function point analysis from being a tool for mid-range applications to becoming a powerful tool for executive analysis of the largest and costliest software applications. Both software productivity and software quality data based on function points will expand rapidly, as will reliable software benchmarks” [2]. To date, automated functional sizing has been performed on developed code through static code analysis. The next breakthrough is automating Functional Size Measurement (FSM) through evaluation of project requirement statements.
Automation with the MARINE Tool
The MARINE desktop tool automates both requirements analysis and function point estimation processes with NLP and a robust rules engine. MARINE provides the Requirements Analyst with immediate feedback on the clarity of requirement statements and also removes duplicate requirements. The tool assists the Cost Estimator in producing software size, cost and schedule estimates from a given set of requirements. MARINE also allows the Project Manager to summarize system capabilities and verify that business needs align with the project requirements as well as provide rough order of magnitude (ROM) costs by requirement to support trade-off decisions.
MARINE is built around a core NLP capability that processes each requirement. It separates the statements that make up each individual requirement into parts of speech that are important to the analyst: action word, action phrase, object, prepositional phrase and word pairings. Figure 1 below shows the main dashboard of the MARINE application after the requirements set has been uploaded into the application.
Figure 1: MARINE Dashboard
Automating Software Measurement
FSM is based upon IFPUG rules and counting practices, which is now an ISO standard (ISO/IEC 20926:2009). FSM has evolved since Alan Albrecht defined function points in Measuring Application Development Productivity in 1979 [3], but it remains a labor-intensive activity.
Trained, and frequently certified, practitioners go through artifacts such as user stories, use cases, logical data models, user guides and design specifications to develop robust software size estimates, and then iterate through discussions with the development organization’s engineers or project managers. FSM practitioners apply and document counting rules at the requirement level. There are different methods of FSM, some of which require less rigor, such as Fast Function Points. The challenge with manual FSM is the time involved in extracting, reviewing and evaluating each requirement. Many project managers have at best an elementary understanding of FSM, and there is a limited supply of trained Certified Function Point Specialists (CFPS). The opportunity with FSM automation is to expand the capability to create rough order of magnitude FSM and reach a broader audience.
MARINE automation involves FSM at the proposal or requirements phase. It should be noted that within MARINE, the analyst will need to refine the FSM, but the tasks of identifying transactions and data objects are simplified. Some of the intricacies of FSM within the IFPUG context, such as identifying File Types Referenced (FTRs), Data Element Types (DETs), and Record Element Types (RETs), are currently beyond the scope of MARINE. In many cases, not enough information is available early in the life-cycle to identify DETs in the requirements phase. However, fast function point counting at the proposal stage can be automated, in which case the analyst identifies data and transactional functions, and then makes assumptions regarding complexity. A key aspect of automation is simply exporting requirements into a manageable format, which in itself can be a cumbersome process. The core of the automation involves cycling through functional requirements and identifying transactions and their associated objects (data) through keyword analysis. There is still a human factor, as the user can review MARINE’s initial size evaluation and make adjustments at the requirements level, or export to a CSV or Microsoft Excel format for further evaluation. MARINE is also designed as a learning tool that will aid analysts with tips and automated suggestions that can be used as a training resource. Reports can be exported from MARINE in three formats: HTML, PDF and MS Word. Figure 2 below displays a sample metrics summary report.
Figure 2: MARINE Metrics Summary Report
Alignment with IFPUG
MARINE has been designed based upon IFPUG CPM 4.3 rules but is not IFPUG compliant in the sense that it follows the Fast Function Point counting process at the requirements phase, and thus does not identify FTRs, DETs and RETs. While many within the IFPUG community may view this as a limitation, it may also be viewed as the evolving role of the analyst. It is reasonable to expect MARINE’s automation capability to expand into ingesting images (such as entity relationship models) and design files (such as wireframes) and be able to automate in a manner that is closely aligned with IFPUG rules.
Community Feedback and the High-Level Roadmap
A 2017 study commissioned by Logapps identified variances between 2% and 60%, with an average variance of 20% between automated and manual size estimates. The variance can be reduced through analysis of requirements, and review of redundant data functions and duplicate requirements. As with function point analysis in general, MARINE is well suited for transactional systems.
Logapps has used a crowd-sourcing model to identify desired future capabilities. The feedback has been positive with many users seeing great value in the efficiency of automated FSM. Many reviewers have requested integration with requirements and estimation tools, the ability to ingest images and access to a database of project requirements and associated size metrics.
Conclusion: The Next Generation Role of Software Management
The manners in which software is developed, driven by Agile and Dev Ops, is rapidly evolving. Thus, it stands to reason that software analysis will also change. In the not-so-distant future, functional size measurement will rely more on automation. Delivered systems will be sized from static code analysis tools, and early design sizing will rely on tools like MARINE. The shift in the automation of functional software measurement and requirements analysis will present opportunities for labor efficiency and change the manner in which software applications are analyzed. The role of CFPS and other functional sizing subject matter experts will likely focus on adjusting the dials of the automated models, and then identifying DETs, RETs and FTRs for more precise functional sizing. MARINE brings this technology to the analyst’s desktop.
While it is a human tendency to resist change, those who see value in evolving technology stand to benefit the most. In the words of the famed statistician who revolutionized quality control methods, W. Edwards Deming, “It is not necessary to change. Survival is not mandatory” [4].
References
[1] F. Brooks, “No Silver Bullet – Essence and Accident in Software Engineering,” p. 16, 1986.
[2] “OMG Adopts Automated Function Point Specification,” Object Management Group, 17-Jan-2013. [Online]. Available: http://www.omg.org/news/releases/pr2013/01-17-13.htm. [Accessed: 01-Jun-2018].
[3] A. Albrecht, “Measuring Application Development Productivity,” 1979.
[4] F. Voehl, Deming The Way We Knew Him. CRC Press, p. 125, 1995.