Studies and Publications

Agile Myth Busters: Story Points are Useless

In this recording we address the Myth “Story Points are Useless." Story Points are the most prominent way Agile teams apply estimation methods and is fundamentally built on Relative Estimation. We will share where the myth came from and the reasons why many groups have struggled with leveraging Story Points. We then review the research behind relative estimation and reasoning Story Points are used for Agile.

Agile Myth Busters: It was better with Waterfall

The NDIA SED Agile Delivery for Agencies, Programs & Teams (ADAPT) Committee is tackling myths associated with the adoption of Agile practices. Here we are addressing the myth "Product Development was better with Waterfall than it was with Agile." During this short video, we review the foundations of this myth and what challenges created the myth. The focus then turns to the comparison of Waterfall to Agile along with improvements that can be realized when applying Agile practices.

Moving from Predictive Planning to Empirical Planning for Systems Engineering

Systems Engineering coupled with Agile practices is a key enabler to successful execution in an environment where focus is on incrementally delivering product while providing ongoing management of the technical baseline and incorporation of new information. This approach provides an understanding of the current and target state of the system development from early planning through deployment of the program. This paper provides a perspective on the responsibilities of Systems Engineering functions in a modern engineering environment coupled with techniques used to plan and execute the work which must allow for agility through continuous learning cycles and feedback from users. This supports the reality of program execution where comprehensive specification up front may no longer be desired - or even possible.

Accelerating Value Delivery in Highly Complex Domains

This article provides an example, based on a missile system, on how to map out an operational value stream, to the nested development value streams that build and deliver the solutions within the operational value stream, to an organizational design and lean-agile operating system that enables optimal execution of those value streams. If you are experiencing challenges in value stream identification, decomposition, and organizational design, use this information to help guide your reasoning process. By sharing this information, we hope to deepen the understanding in our community on how to leverage the power of value streams in the delivery of large solutions in complex domains.

The Intersection of MOSA and Agile to Build Better Systems in DoD

By: NDIA Systems Engineering Division Agile in Defense/ADAPT

This presentation explores how the intersection of MOSA with Agile and DevOps practices help to increase adaptability, shorter delivery schedules, reduce costs through improved quality, and improve transparency of the systems current state. This presentation addresses practices and digital capabilities for delivering value in the shortest sustainable lead time.

MOSA Implementation Considerations, Information Needs and Metrics

By: NDIA Systems Engineering Division Architecture Committee

Find Associated Appendix Documents Here

Modular Open Systems Approach (MOSA) is one of the key elements of the Department of Defense Systems Engineering Modernization effort. The SE Modernization project has three primary goals: 1) build an integrating framework that incorporates key activities across these domains and focus areas; 2) align and integrate these systems engineering practices to specific acquisition pathways; and 3) develop a set of artifacts and associated meta-data for a categorization and information framework that captures policy, guidance, and lessons learned into a body of knowledge.

MOSA Implementation Challenges and Opportunities 2023

By: NDIA Systems Engineering Division Architecture Committee

As a follow-on to its July 2020 paper “MOSA: Considerations Impacting Both Acquirer and Supplier Adoption,” the National Defense Industrial Association (NDIA) Systems Engineering(SE) Architecture Committee offers this report to convey additional information regarding challenges and opportunities in employing MOSA, primarily from the perspective of the defenseindustrial base. This paper incorporates real-world use cases in advising what acquirers must beaware of to increase the likelihood of successful implementations by contractors and other supplychain participants in the delivery of MOSA-involved solutions.

NDIA SED CTO Brief Architecture Committee MOSA 2021 (Click to view the brief)

On September 15th, 2021, The NDIA Systems Engineering Division (SED) Architecture Committee was invited to brief the A&D CTO forum on Modular Open Systems Architecture. The A&D CTO Forum are technical leaders across the U.S. Aerospace and Defense (A&D) industry responsible for the development, enhancement, and dissemination of technology within their respective organizations. During the bi-annual meeting between A&D CTO Forum leaders and guest speakers, the NDIA SED Architecture Committee discussed current, disruptive, and emerging technologies within Aerospace & Defense.

Principal Attendees and Members of the A&D CTO Forum are:

Jim Carlini, Leidos
Naveed Hussain, Boeing Research & Technology
Ross Niebergall, VP and CTO, L3Harris Technologies
Cheryl Paradis, VP and GM, FAST Labs, BAE Systems
Mark Russell, CTO, Raytheon Technologies
Scott Stapp, Corporate VP and CTO Northrop Grumman
Steven Walker, VP and CTO, Lockheed Martin

National Defense September 2021 Issue Systems Engineering Division Published Article

Practical Software and Systems Measurement Continuous Iterative Development Measurement Framework

The collaborative PSM, NDIA, and INCOSE report provides recommendations for the measurement of continuous iterative developments (CID). The report includes a Practical Software and Systems Measurement (PSM) CID measurement framework detailing common information needs and measures that are effective for evaluating CID approaches. The information needs address the team, product, and enterprise perspectives to provide insight and drive decision-making. The framework also identifies and specifies an initial set of measures that have been identified as being practical measures to address these information needs. This guidance is intended to be used by team, program, and enterprise personnel who are implementing CID approaches, as a reference for common, practical measures that can be utilized.

Parts: The report has now been separated into three parts. Part 1 of this report includes a series of diagrams and an ontology to describe the development approaches and terminology used. It also includes an “Information Category-Measurable Concept-Measures” (ICM) Table detailing potential information needs and measures for CID developments. For the highest priority measures, sample measurement specifications have been developed that detail the identified measures. These are included in a separate Part 2 of the paper, along with a discussion of how to use these measures for enterprise decision making. Part 3 of the paper separately extends the main CID paper with detailed information and guidance on Software Assurance and Technical Debt.

Version 1.05 detailed potential information needs and measures that are common to CID approaches, and an initial set of ten measurement specifications that were prioritized by user surveys as highest value. This Version 2.1 includes added material that has been researched and developed by the CID working group. The new materials include information on measuring:

Continuous Iterative Development (Agile) Measurement Framework

We are currently looking for pilots in each of the services, to verify usage in the community, especially for the product value measure. Please contact Cheryl Jones or Geoff Draper if you are interested in participating. Phase 3 of the project will begin soon, focused on the areas identified in part 1, section 6, including size and estimation, additional measures, and enhancements based on user pilots.

Modular Open Systems Approach - Considerations Impacting Both Acquirer and Supplier Adoption

Systems Engineering Architecture Committee
July 1, 2020

The Systems Architecture Committee provided this white paper containing recommendations to government and industry program managers and acquisition professionals regarding the implementation of the Modular Open Systems Approach (MOSA). The Architecture Committee consisting of a broad representation of industry, government, and the Services, studied the various facets of MOSA (past and present) in the context of their professional experience in order to provide practical guidance regarding the approach.

NDIA MOSA White Paper Final Release NDIA Architecture Committee 2020

Implementing Continuous Iterative Development and Acquisition
Defense Science Board (DSB) released a report in Feb-2018 containing seven recommendations regarding software design and acquisition. Section 868 of NDAA 2019 mandates implementation of these recommendations within 18 months. The Defense Innovation Board (DIB) Software Acquisition and Practices (SWAP) study group has also provided many insightful and largely compatible recommendations.

NDIA, INCOSE and PSM support the DSB and DIB concepts and the opportunities they offer to DoD and the defense industry. In 2018 the NDIA Systems Engineering Division commissioned a working group to study the implementation of the DIB and SWAP recommendations. NDIA offers the working group's recommendations below to OUSD(A&S) and OUSD(R&E) representing a broad “industry perspective” on a path forward.

Executive Summary Brief (pdf)

Executive Summary Brief (ppt with links) view in presentation mode

Full Study Brief

Transmittal Letter to Dr. Boleng, OUSD (A&S)

Top Five SE Issues
The NDIA Systems Engineering Division determined that an update to the Top Issues in Systems Engineering (SE) report that was issued initially in 2003, updated in 2006 and again in 2010 was needed. The issues related to our defense industry are complex, affecting both the industry organizations as well as the government and military organizations.

A Working Group was formed, inputs were solicited in advance, and a reconciliation meeting with 18 members was held on August 17th and 18th, 2016. Although numerous separate issues were identified, the group found that the bulk of these actually fell into several major issue categories. The detailed results, including the status of the previous 2006 and 2010 SE issues, are described in this report.

A Path Towards Cyber Resilient and Secure Systems, April 2016
Taking a holistic approach to system security requires bringing together multiple communities with rich histories introduces varying perspectives, terminologies, taxonomies and methodologies. This diversity provides opportunities and challenges for evaluating the security quality system attributes. In this paper, we discuss a path towards system security metrics and measures. Ultimately, we are committed to providing systems that are resistant to attack and resilient when under attack.

System Security Engineering Committee Chairs:
Holly Dunlap, Raytheon & Beth Wilson, Raytheon

Developmental Test & Evaluation Committee Chair:
Joe Manas, Raytheon

In collaboration with:
INCOSE SSE Committee, Trusted Supplier Steering Group, and Mitre AF Cyber Integration

Final Report on the Digital System Model Workshop, August 2015

Guidance for Utilizing Systems Engineering Standards (IEEE 15288.1 and IEEE 15288.2) on Contracts for Defense Projects, July 2015
Recommended guidance on tailoring, acquisition language, conformance, and evaluating compliance.

Mapping Standards for Integrated Development, v0.7. October 4, 2015
Overview of the NDIA standards mapping tasks, strategy, and approach. Initial draft (v0.7), intended to be an industry asset improved upon with use and extended to include additional standards.

Mappings between CMMI-DEV v1.3 and IEEE 15288 (Excel spreadsheet)
Initial mappings developed by consensus of a team of industry subject matter experts

The Business Case for Systems Engineering: Results of the Systems Engineering Effectiveness Survey
This report summarizes the results of a survey that had the goal of quantifying the connection between the application of systems engineering (SE) best practices to projects and the performance of those projects. The survey population consisted of projects and programs executed by system developers reached through the NDIA, the IEEE Aerospace and Electronic Systems Society, and INCOSE. Analysis of survey responses revealed strong statistical relationships between project performance and several categories of specific SE best practices. The survey results show notable differences in the relationship between SE best practices and performance between more challenging and less challenging projects.

The Business Case for Systems Engineering: Detailed Response Data
This report contains detailed response data from The Effectiveness of Systems Engineering: A Survey. The survey had the goal of quantifying the connection between the application of systems engineering (SE) best practices to projects and programs and the performance of those projects and programs. The survey population consisted of projects and programs executed by system developers reached through the NDIA, the IEEE Aerospace and Electronic Systems Society, and INCOSE. Analysis of survey responses revealed strong statistical relationships between project performance and several categories of SE best practices. The survey results show notable differences in the relationship between SE best practices and performance between more challenging and less challenging projects. The statistical relationship with project performance is quite strong for survey data of this kind when both SE capability and project challenge are considered together.

Report on Systemic Root Cause Analysis of Program Failures.
National Defense Industrial Association Systems Engineering Division in conjunction with Office of Under Secretary of Defense Acquisition, Technology & Logistics, Systems & Software Engineering, Arlington, VA, December 2008.