oversight

Defense Intelligence: Comprehensive Plan Needed to Improve Stakeholder Engagement in the Development of New Military Intelligence System

Published by the Government Accountability Office on 2020-11-19.

This report is unreleased, or missing. Visit its landing page for more detail.

                United States Government Accountability Office
                Report to Congressional Committees




                DEFENSE
November 2020




                INTELLIGENCE

                Comprehensive Plan
                Needed to Improve
                Stakeholder
                Engagement in the
                Development of New
                Military Intelligence
                System




GAO-21-57
                                             November 2020

                                             DEFENSE INTELLIGENCE
                                             Comprehensive Plan Needed to Improve Stakeholder
                                             Engagement in the Development of New Military
Highlights of GAO-21-57, a report to
                                             Intelligence System
congressional committees




Why GAO Did This Study                       What GAO Found
Foundational military intelligence—all-      The Defense Intelligence Agency (DIA) intends for its new foundational military
source intelligence collected by the         intelligence system, the Machine-Assisted Analytic Rapid-Repository System
Intelligence Community (IC) on other         (MARS), to have capabilities to collect and synthesize data that significantly
countries’ militaries—is a critical          exceed those of its legacy system. DIA began developing MARS in 2018. DIA
element in planning for military             and stakeholders have identified policy, technical, and operational risks facing
operations. The DIA legacy system for        the program. For example, MARS needs to ingest a significant variety of data
processing such intelligence is unable       types (e.g., signals, text, imagery), which cannot be done using any one data
to meet current needs, and DIA intends       model. DIA’s use of Agile development processes helps to manage risk to some
to replace it with MARS. MARS is
                                             degree, because it emphasizes continuously delivering software that addresses
expected to transform the way the IC
                                             users’ priority needs and incorporating frequent user feedback to inform
approaches and generates
foundational military intelligence.
                                             development. However, these processes do not address all MARS risks, such as
However, agencies can face a wide            those related to implementing new technologies. To address these and other
array of issues in developing a new          risks, DIA has begun establishing a risk management process. It will be important
system of this magnitude, including          for DIA to continue to manage risk throughout MARS’s life cycle.
incorporating feedback from a large
number of stakeholders.                      DIA has taken a number of actions to identify the needs of Department of
                                             Defense (DOD) and intelligence community organizations that will use MARS—
A committee report accompanying the          including combatant commands, services, and intelligence agencies—but it lacks
Intelligence Authorization Act for Fiscal    a comprehensive plan for engaging with these stakeholders (see fig.).
Years 2018, 2019, and 2020 includes a
provision for GAO to review MARS             Machine-Assisted Analytic Rapid-Repository System (MARS) Stakeholder Roles
development. This report (1) describes
the initial risks DIA and stakeholders
have identified in the development of
MARS and the actions DIA has taken
to manage risk and (2) assesses how
DIA is engaging potential stakeholders
in the development of capabilities for
the MARS program.
GAO reviewed related documentation;
interviewed DOD and intelligence
community officials about the risks to
MARS development and how DIA is
engaging stakeholders; and attended a
stakeholder engagement meeting.

What GAO Recommends
                                             DIA has conducted workshops to identify initial requirements and town halls to
GAO recommends that the Office of            disseminate MARS updates. However, stakeholders expressed mixed views on
the Director of National Intelligence        the quality and extent of the program’s continual engagement with them, and
(ODNI), in coordination with the             stakeholders were unsure about how their feedback and input are helping DIA
Secretary of Defense, ensure that DIA        prioritize and select MARS features. Further, DIA’s initial stakeholder
develops a comprehensive plan for            engagement and test plan lacks key details called for in leading practices for
engaging MARS stakeholders. ODNI             system development and GAO’s Agile Assessment Guide, such as how
concurred with GAO’s                         stakeholders will be engaged at different times and how their feedback will be
recommendation.                              compiled to support the prioritization of features. Without a comprehensive plan
View GAO-21-57. For more information,        to guide stakeholder engagement, MARS capabilities could fall short of
contact Brian M. Mazanec at (202) 512-5130   stakeholder expectations, and DIA could face greater challenges in mitigating
or mazanecb@gao.gov.                         risks to MARS.
                                                                                       United States Government Accountability Office
Contents


Letter                                                                                            1
                       Background                                                                 3
                       DIA and Stakeholders Have Identified Multiple Risks to MARS
                         Development, and DIA Is Establishing a Risk Management
                         Process                                                                  8
                       DIA Is Taking Actions to Provide Information to Stakeholders, but
                         the MARS Program Lacks a Comprehensive Stakeholder
                         Engagement Plan                                                         14
                       Conclusions                                                               21
                       Recommendation for Executive Action                                       21
                       Agency Comments and Our Evaluation                                        21

Appendix I             Scope and Methodology                                                     25



Appendix II            Comments from the Office of the Director of National Intelligence         30



Appendix III           GAO Contact and Staff Acknowledgments                                     31



Related GAO Products                                                                             32


Tables
                       Table 1: Selected Policy Risks to MARS Program Development
                               Identified by DIA and MARS Stakeholders                            9
                       Table 2: Selected Technical Risks to MARS Program
                               Development Identified by DIA and Stakeholders                    10
                       Table 3: Selected Operational Risks to MARS Program
                               Development Identified by DIA and Stakeholders                    12

Figures
                       Figure 1: MARS Stakeholder Roles                                           5
                       Figure 2: MARS Acquisition Timeline                                        6
                       Figure 3: Agile Development Process                                        7




                       Page i                                          GAO-21-57 Defense Intelligence
Abbreviations

CCMD                       Combatant Command
DIA                        Defense Intelligence Agency
DOD                        Department of Defense
IC                         Intelligence Community
IT                         Information Technology
MARS                       Machine-Assisted Analytic Rapid-Repository
                           System
MIDB                       Modernized Integrated Database
ODNI                       Office of the Director of National Intelligence
OUSD(I&S)                  Office of the Under Secretary of Defense for
                           Intelligence and Security




This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page ii                                                     GAO-21-57 Defense Intelligence
                       Letter




441 G St. N.W.
Washington, DC 20548




                       November 19, 2020

                       Congressional Committees

                       Foundational military intelligence—all-source intelligence collected by the
                       Intelligence Community (IC) on other countries’ militaries and
                       infrastructure—is a critical element in the planning for military operations.
                       The Defense Intelligence Agency’s (DIA) legacy system, the Modernized
                       Integrated Database (MIDB), which captures such intelligence, is unable
                       to meet current needs, and DIA intends to replace it over the long term
                       with a new system—the Machine-Assisted Analytic Rapid-Repository
                       System (MARS). MARS is expected to transform the way the IC
                       approaches and generates foundational military intelligence. However,
                       agencies can face an array of planning and technical issues in developing
                       new information technology (IT) systems of this magnitude, including
                       maintaining a legacy system while developing a new one and
                       incorporating feedback from a large number of stakeholders; in the case
                       of DIA these stakeholders include combatant commands (CCMDs), the
                       military services, and the services’ intelligence centers. 1 Given the
                       importance of foundational military intelligence to the warfighter, it is
                       critical for DIA to meet stakeholder expectations and have sound plans to
                       successfully develop and employ MARS.

                       The House Permanent Select Committee on Intelligence’s report
                       accompanying a bill for the Intelligence Authorization Act for Fiscal Years
                       2018, 2019, and 2020 includes a provision for us to review MARS
                       development. 2 Our report (1) describes the initial risks DIA and
                       stakeholders have identified in the development of MARS and the actions
                       DIA has taken to manage them and (2) assesses how DIA is engaging
                       potential stakeholders in the development of capabilities for the MARS
                       program. 3 As a result of limitations on government operations in response
                       to the Coronavirus Disease 2019, we were not able to analyze classified

                       1According   to Department of Defense (DOD) officials, the sustainment of MIDB until
                       MARS can effectively replace it is crucial for the warfighter, particularly MIDB’s provision
                       of intelligence to DOD’s targeting capabilities.
                       2H.R.    Rep. No. 116-151, at 88-89 (2019).
                       3MARS  stakeholders are defined by the roles they play related to MARS, which are
                       discussed later in this report and in appendix I. These roles include consumers,
                       producers, and contributors.



                       Page 1                                                        GAO-21-57 Defense Intelligence
data related to the cost and schedule of the MARS program. We plan to
address these aspects of the mandate at a later date.

For our first objective, we reviewed DIA documentation and spoke with
DIA officials about their efforts to identify and document risks. We also
interviewed officials from other Department of Defense (DOD) and
selected IC elements and reviewed documentation to identify the risks
facing MARS development from their perspective. 4 We then spoke with
DIA and IC officials and reviewed associated documentation to identify
what actions they are taking to address and manage program risks.

For our second objective, we collected documentation and interviewed
officials from DIA and other IC elements to assess how well DIA’s
processes and plans for MARS development adhere to key
characteristics of effective user engagement as collectively laid out in the
National Defense Authorization Act for Fiscal Year 2010, DOD guidance, 5
and GAO’s Agile Assessment Guide 6: (1) early engagement, (2) continual
engagement, (3) feedback based on working software, and (4) feedback
incorporated into subsequent development. For “early engagement,” we
reviewed DIA’s process for identifying initial user needs, such as whether
DIA leveraged workshops or surveys to identify these needs. For
“continual engagement,” we assessed DIA’s ongoing mechanisms to
engage stakeholders and any plans for stakeholder communications. For
“feedback based on working software” and “feedback incorporated into
subsequent development,” we spoke with DIA officials to determine
whether DIA has established a discrete process to collect user feedback
and suggestions as users interact with the system. As part of this effort,
we assessed DIA’s stakeholder engagement and testing plan against
leading practices for system development, including whether it contained

4Officialsand documents used varying terms to describe these risks, including challenges,
dependencies, and risks; for clarity and consistency, we refer to them throughout the
report only as risks.
5Section  804 of the National Defense Authorization Act for Fiscal Year 2010, Pub. L. No.
111-84 (2009), directed the Secretary of Defense to develop and implement a new
acquisition process for information technology systems that, to the extent determined
appropriate by the Secretary, would include early and continual involvement of the user,
among other things. This statute, in addition to DOD’s 2010 report to Congress in
response to the statute and DOD Instruction 5000.02, Operation of the Adaptive
Acquisition Framework, (Jan. 23, 2020) identifies the characteristics of effective user
engagement for DOD acquisitions.
6GAO,Agile Assessment Guide: Best Practices for Agile Adoption and Implementation,
GAO-20-590G (Washington, D.C.: September 28, 2020).



Page 2                                                     GAO-21-57 Defense Intelligence
                          a specific feedback process and engagement-related metrics. 7 For a
                          detailed description of our scope and methodology, see appendix I.

                          We conducted this performance audit from October 2019 to November
                          2020 in accordance with generally accepted government auditing
                          standards. Those standards require that we plan and perform the audit to
                          obtain sufficient, appropriate evidence to provide a reasonable basis for
                          our findings and conclusions based on our audit objectives. We believe
                          that the evidence obtained provides a reasonable basis for our findings
                          and conclusions based on our audit objectives.


Background
Transition from MIDB to   MARS is intended to eventually replace MIDB, a legacy system that has
MARS                      limited capability to leverage emerging technologies and available
                          intelligence. MIDB is one of the primary systems DIA currently uses to
                          fulfill its core mission of providing foundational military intelligence to
                          warfighters and decision makers. This intelligence consists of information
                          across a wide spectrum, including military capabilities of adversaries and
                          infrastructure in all domains. However, MIDB is over 20 years old and can
                          no longer meet the IC’s current demands for information. For example, it
                          does not include information in areas such as cyberspace and space, and
                          it does not have enough capacity to store all the information needed by
                          warfighters. In addition, MIDB relies heavily on manual operation—that is,
                          human involvement to update records—and cannot use new technology
                          such as machine learning and automation. During the multi-year transition
                          period, MARS is intended to work in parallel with MIDB before replacing
                          it. During this time, MIDB is expected to receive only limited software
                          enhancements to sustain it, but DIA is not planning to provide resources
                          for any significant upgrades or modernization to MIDB.

                          MIDB currently connects to hundreds of DOD intelligence, planning, and
                          operations-based tools and systems that will eventually need to connect
                          directly to MARS, according to DOD officials. For example, the Joint
                          Targeting Toolbox includes a number of tools and programs that support

                          7For system development, the Capability Maturity Model® Integration for Development
                          provides a comprehensive integrated set of guidelines and leading practices for
                          developing products and services, including new software, to meet the needs of
                          customers and end users. Models are developed by product teams with members from
                          industry, government, and the Software Engineering Institute. See the Software
                          Engineering Institute’s CMMI® for Development, Version 1.3 (November 2010).


                          Page 3                                                  GAO-21-57 Defense Intelligence
                    targeting for the joint force and will need to interoperate with critical
                    MARS data. Based on information from DOD officials, a number of these
                    systems are expected to require significant modernization before they are
                    able to connect to MARS and take full advantage of the increased scale
                    and scope of the data that MARS will offer, which may be costly. For
                    instance, the Distributed Common Ground System, a system with a wide
                    user base that includes the services, is expected to be streamlined to
                    enable it to use MARS data and features. Other stakeholders told us that
                    MARS development may provide an opportunity to evaluate which
                    systems can effectively be replaced by MARS or other new systems.

MARS Stakeholders   According to DIA officials, MARS stakeholders fall into three different
                    roles. DOD components and personnel can perform more than one role.

                    •   Consumers. DOD components and individuals who use the
                        foundational military intelligence that will be stored in MARS. Many of
                        these would consume this intelligence through systems connected to
                        MARS. These include the military services, the service intelligence
                        centers, and other components such as the CCMDs.
                    •   Producers. DOD components that create authoritative foundational
                        military intelligence records in MIDB and will create and update such
                        records in MARS. These include defense intelligence agencies,
                        service intelligence centers, and CCMDs.
                    •   Contributors. DOD or IC components that provide source information
                        or data to support the development of foundational military intelligence
                        and related records that will be stored in MARS. These include the
                        National Geospatial-Intelligence Agency, the National Security
                        Agency, and the CCMDs.
                    Figure 1 shows the relationships between these stakeholder roles and
                    MARS.




                    Page 4                                            GAO-21-57 Defense Intelligence
                   Figure 1: MARS Stakeholder Roles




MARS Acquisition   DIA began developing MARS in 2018, and MARS is currently an
                   experimental research demonstration. The National Intelligence
                   Acquisition Board is planning to meet in the late fall of 2020 to determine
                   whether MARS is mature enough to be designated as a program of
                   record. 8 The board is expected to evaluate MARS’s progress to date,
                   including the MARS infrastructure module that was released in May
                   2020. 9 The infrastructure module is the first of five modules that DIA plans

                   8The   board initially planned to meet in September 2020 but was delayed because of the
                   Coronavirus Disease 2019, according to officials in ODNI. The principal forum for
                   executing ODNI’s acquisition authorities is the National Intelligence Acquisition Board.
                   Among other things, the board will assess the program’s concept of operations, estimated
                   life cycle costs, and its technology development strategy to determine whether MARS is
                   ready to move from the initial concept phase of the acquisition to Concept Refinement and
                   Technological Maturity, or Phase A.
                   9According   to DOD officials, the infrastructure module released at that time provides basic
                   functionality and is considered to be a minimally viable product. We received a
                   demonstration of this product in August 2020.


                   Page 5                                                       GAO-21-57 Defense Intelligence
                                      to develop. Each module will focus on a specific mission area and will
                                      develop analytic functions and capabilities—such as searching for and
                                      displaying information and detecting anomalies—for that area. The other
                                      four modules are Order of Battle, Intelligence Mission Data, Cyberspace,
                                      and Space and Counterspace. As part of MARS development, targeting
                                      tools and systems will also link to data provided in all five modules,
                                      according to DOD officials. Additionally, DOD officials stated that the
                                      Office of the Director of National Intelligence (ODNI) is expected to
                                      prepare an independent cost estimate before the board meets, which will
                                      estimate the programmatic costs for MARS’ lifetime. Figure 2 depicts the
                                      general timeline for MARS acquisition.

Figure 2: MARS Acquisition Timeline




                                      MARS is expected to remedy MIDB’s long-standing deficiencies and
                                      provide considerably enhanced capabilities. For instance, MIDB requires
                                      a significant amount of manual operation to perform tasks, whereas
                                      MARS is intended to use automation for many processes, such as data
                                      ingestion. MARS is also expected to provide a dynamic way of tracking
                                      order of battle information—that is, the organization, command structure,
                                      strength, disposition of personnel, and equipment of units and formations
                                      of an armed force. Currently, MIDB only offers the ability to track order of
                                      battle in a static manner. MARS is intended to track order of battle
                                      dynamically—such as tracking the positions of forces as they are on the
                                      move—in part by incorporating larger volumes of data that provide real-
                                      time updates and higher levels of automation to make these data usable
                                      for analysts. MARS officials also plan for MARS to include new types of
                                      data, such as emergent sources and publicly available information, which
                                      are not contained in MIDB.


                                      Page 6                                            GAO-21-57 Defense Intelligence
Agile System                          As part of MARS development, DIA is planning to use an approach
Development Approach                  known as Agile, which enables continuous product development and
                                      delivery. The Agile approach stresses the delivery of software in short,
                                      incremental segments, which allows for greater flexibility and adaptation
                                      to meet changing customer needs and requirements. The product owner
                                      must routinely interact with stakeholders in order to fully understand how
                                      they value each feature and direct developers in the creation of a product
                                      that matches stakeholder needs and priorities. During Agile development,
                                      the priority is providing a product that maximizes value to the
                                      stakeholders. Constant feedback and communication between developer
                                      and stakeholder are crucial to achieving this priority. Figure 3 depicts the
                                      Agile development process and its emphasis on user feedback.

Figure 3: Agile Development Process




                                      Because Agile teams are self-organizing and Agile’s iterative process is
                                      viewed as a way to mitigate the risk inherent in developing complex
                                      software programs, a perception can develop that explicit risk
                                      management practices are unnecessary. All programs face risk and
                                      uncertainty, whose likelihood and potential impact should be examined.
                                      While Agile emphasizes that teams will uncover risk via early and
                                      frequent delivery of software, the potential impact of some issues, such
                                      as technical debt or team size, should be considered earlier rather than
                                      later.

                                      The Agile approach fits within the larger context of system development. 10
                                      System development models offer leading practices for both risk
                                      management and stakeholder involvement in the development of IT
                                      10Software Engineering Institute, CMMI® for Development, Version 1.3 (November 2010).
                                      For system development, the Capability Maturity Model® Integration for Development
                                      provides a comprehensive integrated set of guidelines and leading practices for
                                      developing products and services, including new software.


                                      Page 7                                                  GAO-21-57 Defense Intelligence
                          systems like MARS. For example, according to leading practices for
                          system development, risk management is a continuous, forward-looking
                          process intended to address issues that could endanger achievement of
                          critical objectives before they occur. A continuous risk management
                          approach effectively anticipates and mitigates risks that can have a
                          critical impact on a project. Early and aggressive detection of risk is
                          important because it is typically easier, less costly, and less disruptive to
                          make changes and correct work efforts during the earlier, rather than the
                          later, phases of the project. The risk management process should
                          describe how the program is to identify, analyze, mitigate, and monitor
                          risks. Agile development also manages risk throughout development,
                          however, it does not focus on high level risks—such as programmatic
                          concerns related to affordability or policy constraints that may affect how
                          users are able to operate the system—that face the program as a whole.
                          Additionally, system development models offer practices on how to
                          engage stakeholders, including guidance related to the timing,
                          prioritization, and monitoring of stakeholder input.

                          DIA and stakeholders have identified multiple risks that MARS will face in
DIA and Stakeholders      achieving its intended capabilities. DIA has taken some initial actions in
Have Identified           response, including establishing a process to manage risk.
Multiple Risks to
MARS Development,
and DIA Is
Establishing a Risk
Management Process
DIA and Stakeholders      DIA intends for MARS to have capabilities that significantly exceed the
Have Identified Initial   current capabilities of MIDB. These include the capability to ingest large
                          volumes of data, support machine-assisted technologies, and include new
Risks to MARS
                          intelligence sources such as publicly available information. To achieve
Development and Begun     these capabilities, DIA and stakeholders have identified many initial risks
Exploring Options to
Mitigate Some Risks




                          Page 8                                             GAO-21-57 Defense Intelligence
                                                                 that must be addressed, including policy, technical, and operational
                                                                 risks. 11

Policy Risks                                                     There are many DOD and IC policies in place that govern how
                                                                 foundational military intelligence can be created, modified, and shared. In
                                                                 some cases these existing policies could limit the ways in which MARS
                                                                 can operate, according to DOD officials. DIA will need to ensure that such
                                                                 policies support its vision for a significant increase in the amount of data
                                                                 MARS will ingest and the number of new data sources from which it will
                                                                 draw. Table 1 describes in more detail some of the policy risks that DIA
                                                                 and MARS stakeholders have identified.

Table 1: Selected Policy Risks to MARS Program Development Identified by DIA and MARS Stakeholders

 Policy Risks                                                          Description of Risks
 Classification Policies                                               Current classification policies make it challenging to share new data streams among
                                                                       Intelligence Community (IC) components and with U.S. allied partners. These
                                                                       underlying policies do not belong to the Defense Intelligence Agency (DIA) but to
                                                                       other IC elements. The Machine-Assisted Analytic Rapid-Repository System (MARS)
                                                                       is expected to include and share significantly more data than the Modernized
                                                                       Integrated Database, which will exacerbate existing challenges.
 Quality Assurance Process                                             Existing policies for the Modernized Integrated Database allow only certain defense
                                                                       components to update and produce records in specific intelligence areas, but DIA’s
                                                                       intent is to allow anyone in the defense IC to add or update system records in MARS.
                                                                       It is unclear how DIA will ensure that these records are checked for quality and fully
                                                                       validated in a timely manner.
 Data Standardsa                                                       There is some disagreement among stakeholders on what data standards are
                                                                       needed. Although some stakeholders have noted the need to specify new data
                                                                       standards, DIA has not yet disseminated a data dictionary or detailed information on
                                                                       the data standards it will leverage and does not plan to do so. MARS program
                                                                       officials have highlighted their intent to use a data model that minimizes the need for
                                                                       new data standards, but it is unclear how this model will be implemented.
Source: GAO analysis of Department of Defense information and interviews with Department of Defense officials. | GAO-21-57
                                                                 a
                                                                  According to Department of Defense officials, data standards refer to specific data types, such as
                                                                 geospatial data which will be used in MARS, and how they are to be formatted within systems or for
                                                                 system ingest.




                                                                 11Based  on information from stakeholders and our own understanding of the types of risks
                                                                 facing MARS, we divided risks into three categories—policy, operational, and technical.
                                                                 Risks were categorized based on the content of the information provided by DOD
                                                                 stakeholders and in coordination with GAO IT experts. We communicated our
                                                                 categorization to MARS officials, who agreed with both the categorization and the
                                                                 description we compiled for each risk. We describe these risks in the report as initial risks,
                                                                 because as the MARS program progresses, DIA and stakeholders may identify new risks.


                                                                 Page 9                                                            GAO-21-57 Defense Intelligence
                                                                 According to DOD officials, in some cases these policies—such as the
                                                                 classification policies or policies focused on data standards—are not
                                                                 owned by DIA. DIA officials told us that, in these instances, they will need
                                                                 to work with stakeholders outside of DIA to modify these policies if MARS
                                                                 is to be able to ingest data and disseminate records as planned. DIA
                                                                 officials stated that for some of these risks, such as classification policies
                                                                 and data standards, they are actively engaging in community forums or
                                                                 otherwise monitoring updates and advocating for policy changes. DIA
                                                                 officials have also stated that they will rely heavily on automation to
                                                                 manage MARS data under existing policies to ensure that the ingestion of
                                                                 data and the creation of new records occur seamlessly and align with
                                                                 existing policies, rather than trying to change policies themselves.

Technical Risks                                                  The projected capabilities for MARS are expected to transform the current
                                                                 data environment, leading to many technical risks as the MARS program
                                                                 attempts to implement new technologies, according to DIA officials. Table
                                                                 2 describes in more detail some of the technical risks that DIA and MARS
                                                                 stakeholders have identified.

Table 2: Selected Technical Risks to MARS Program Development Identified by DIA and Stakeholders

 Technical Risks                                                       Description of Risks
 Cross-Domain Solutions                                                Machine-Assisted Analytic Rapid-Repository System (MARS) data need to be able to
                                                                       move up and down networks at higher and lower classification levels. The
                                                                       Modernized Integrated Database currently has this capability, but it functions in a
                                                                       limited manner, and MARS needs to have an improved capability in part because of
                                                                       the significant increase in data that MARS is expected to ingest.
 Integration of Various Data Types                                     MARS needs to ingest a significant variety of data types (e.g., audio, text, imagery,
                                                                       signals), and this cannot be done using any one data model. MARS will have to use
                                                                       a multi-model data environment that private industry has little experience in
                                                                       managing or integrating.
 Technical Interoperability with Other Systems                         MARS needs to be interoperable with hundreds of systems used by stakeholders.
                                                                       The significant number of systems and the long lead time that may be needed to
                                                                       modify certain systems could challenge users’ ability to fully utilize MARS data.
 Automation                                                            Automation of data ingest and data services is intended to be an important feature of
                                                                       MARS. However, MARS may be challenged to accomplish the level of automation
                                                                       DIA intends and to validate the algorithms supporting the automated services in a
                                                                       way that is clear and understandable to the analysts using the data, because the
                                                                       technologies to do this are immature.
Source: GAO analysis of Department of Defense information and interviews with Department of Defense officials. | GAO-21-57




                                                                 DIA is working to mitigate these technical risks through a number of
                                                                 technology pilots. According to a DIA document, these pilots allow DIA to
                                                                 learn what technology is available, what its limitations are, and how

                                                                 Page 10                                                       GAO-21-57 Defense Intelligence
                    private industry is handling challenges similar to those facing MARS.
                    DOD officials reported that they have completed all of their initially
                    planned technology pilots and that they may repeat some of them in the
                    future to gain more information, because the pilots they have conducted
                    have yielded mixed results. For instance, summaries of the pilots show
                    that one pilot identified a potential cross-domain solution, while another
                    pilot highlighted that attempts to use automation or machine learning for
                    data ingestion may require third party developer support rather than being
                    supportable by DIA developers. 12 Further, according to officials in the
                    Office of the Under Secretary of Defense for Intelligence and Security
                    (OUSD(I&S)), they have identified cross-cutting DOD intelligence systems
                    to test whether MARS can connect and achieve interoperability with key
                    systems. These officials added that this effort is referred to as
                    “pathfinders” and will be used to better understand the interoperability
                    requirements of the systems that need to interact with MARS. MARS
                    program officials have also reported that they are using a thin data model
                    as well as developing general application programming interfaces to help
                    mitigate the challenge of technical interoperability for some systems. 13

Operational Risks   According to DOD officials, DIA also faces operational risks with MARS,
                    because the program will have to develop procedures to assure that
                    MARS is easily usable by and accessible to stakeholders. Table 3
                    describes in more detail some of the operational risks that DIA and MARS
                    stakeholders have identified.




                    12DIA requested that the Army Research Laboratory conduct informal technical
                    assessments of the technology demonstrations. These assessments reviewed the
                    demonstrations and provided some technical feedback on each one. We reviewed four of
                    these assessments.
                    13An application programming interface enables machine-to-machine communication,
                    which can allow users to connect directly with the data set and obtain real-time data
                    updates. This is particularly useful for large, frequently updated, or highly complex
                    datasets. As part of its thin data model, MARS is intended to provide a simplified data
                    model to stakeholders to incentivize ease-of-use and data acceptance that will require
                    roughly 10 data tags for data sets that are shared between MARS and other systems.


                    Page 11                                                     GAO-21-57 Defense Intelligence
Table 3: Selected Operational Risks to MARS Program Development Identified by DIA and Stakeholders

 Operational Risks                                                     Description of Risks
 Interoperability Costs to Stakeholders                                Systems that currently access the Modernized Integrated Database, the legacy
                                                                       system, may have to be modified at some point in order to interact with the Machine-
                                                                       Assisted Analytic Rapid-Repository System (MARS), although the extent of such
                                                                       modifications is currently unknown.a One initial cost estimate from Joint Staff cites a
                                                                       cost of $155 million to modernize three targeting systems to take full advantage of all
                                                                       projected MARS capabilities. However, DOD officials said that these modernization
                                                                       costs are beyond what would be required to access the current data available within
                                                                       the Modernized Integrated Database, are not imposed by the MARS program, and
                                                                       are separate from MARS program costs. In addition, there could be costs to
                                                                       stakeholders associated with modifying legacy data for interoperability, which means
                                                                       ensuring that legacy data are formatted in a way that allows MARS to ingest, store,
                                                                       and disseminate the data.
 Bandwidth Concerns                                                    MARS is intended to function as a web-based data environment that stores and
                                                                       processes exponentially larger amounts of data than its predecessor. This will result
                                                                       in higher bandwidth requirements for the systems that connect to MARS, which some
                                                                       stakeholders—especially at the tactical level—may not be able to accommodate.
                                                                       Some stakeholders stated that they do not have the capacity to build additional
                                                                       servers, even if it is necessary to utilize MARS. Furthermore, in environments with
                                                                       limited bandwidth capabilities, especially at the tactical level, it may be difficult to
                                                                       utilize a web-based platform such as MARS.
 User Adoption                                                         Key users may not transition to MARS if it differs too much from the Modernized
                                                                       Integrated Database or does not meet stakeholder requirements. If MARS and
                                                                       stakeholders do not ensure that users are aware of new MARS functions and know
                                                                       how to use them, they may go unused.
Source: GAO analysis of Department of Defense information and interviews with Department of Defense officials | GAO-21-57
                                                                 a
                                                                  We talked to numerous DOD stakeholders, including combatant commands and services, who stated
                                                                 they have not completed cost estimates to achieve the interoperability of their systems with MARS yet
                                                                 because they have not yet received enough detail from DIA about the technical specifications of
                                                                 MARS to develop an estimate.




                                                                 DIA officials stated that engagement with stakeholders will mitigate these
                                                                 operational risks. For instance, DIA officials reported that to mitigate the
                                                                 risks of user adoption they are proactively reaching out to stakeholders to
                                                                 demonstrate MARS features and familiarize stakeholders with them.
                                                                 Similarly, for the risk related to interoperability costs, DIA officials told us
                                                                 they will rely on communication with stakeholders to ensure that DIA’s
                                                                 proposed technical interoperability solutions effectively solve stakeholder
                                                                 concerns and mitigate expenses as much as possible. Officials in
                                                                 OUSD(I&S) have also stated that they are actively encouraging DIA’s
                                                                 communication with key stakeholders through DIA’s pathfinder efforts to
                                                                 ensure they will be aware of future costs related to interoperability. Lastly,
                                                                 DIA officials stated that they do not yet have a mitigation strategy in place
                                                                 to resolve bandwidth concerns but that they are actively pursuing a
                                                                 solution.

                                                                 Page 12                                                           GAO-21-57 Defense Intelligence
DIA Is Establishing a Risk   DIA’s use of the Agile approach to develop MARS builds in some risk
Management Process           management activities. The emphasis on early and continuous delivery of
                             working software that addresses the users’ highest priority needs, the
                             breakdown of development work into small iterations, and the
                             incorporation of frequent user feedback into future development help
                             reduce risks. However, there can also be risks that would not be fully
                             addressed by the Agile development process. The Defense Agile
                             Acquisition Guide acknowledges that an Agile environment can result in
                             increased complexity that must be mitigated. 14

                             A DIA official stated that, accordingly, DIA has begun taking actions and
                             developing key documents to manage risk as the program moves
                             forward. An official reported that, to guide its efforts, DIA is establishing a
                             risk management process as laid out in the Department of Defense Risk,
                             Issue, and Opportunity Management Guide for Defense Acquisition
                             Programs. 15 The guide outlines processes for identifying and
                             documenting risks; analyzing and prioritizing those risks; and developing
                             risk mitigation plans, risk monitoring, and risk reporting. A DIA official
                             stated that, based on the process laid out in this guide, DIA is also
                             developing standard operating procedures on risk management for
                             MARS. This official stated that the team working on this effort intends for
                             this risk management process and related procedures to be used to
                             address programmatic risks related to cost, technical, scheduling, and
                             external risks that the Agile development process will not address. 16 An
                             official stated that DIA is unsure about the exact timeline for finalizing the
                             risk management process, but noted that DIA will be required by ODNI to
                             brief the National Intelligence Acquisition Board on the status of its risk
                             management efforts before MARS can become a program of record. This
                             briefing is planned for late fall of 2020. Some of these efforts are detailed
                             in a key capabilities document that DIA provided to ODNI in late June,
                             2020.



                             14The   MITRE Corporation, Defense Agile Acquisition Guide (March 2014).
                             15DOD, Risk, Issue, and Opportunity Management Guide for Defense Acquisition
                             Programs (January 2017).
                             16We inquired on multiple occasions about DIA’s preparation to address risk with DIA,
                             ODNI, and OUSD(I&S). During the early stages of our review, DIA had not established a
                             risk management process. However, in July 2020, DIA provided us with documentation
                             identifying a risk management process. We are encouraged by DIA’s progress during our
                             review.



                             Page 13                                                   GAO-21-57 Defense Intelligence
                         OUSD(I&S) officials stated that, in addition to DIA’s actions, the Defense
                         Digital Service is conducting an independent assessment of MARS that is
                         expected to be completed in the fall of 2020. 17 Although the exact scope
                         and focus of the review was not yet fully defined as of August 2020,
                         officials from OUSD(I&S) anticipated the review focusing on issues such
                         as the capability of the MARS Program Management Office to develop
                         the product and manage contractor support. They also anticipate a high
                         level examination of whether the program office has a clear vision of what
                         needs to be delivered. The Defense Digital Service may also make
                         recommendations on any efforts needed to improve MARS’s chances for
                         success. This assessment may help DIA in its overall evaluation and
                         management of risks.

                         It is too soon in MARS development to know whether DIA’s efforts will
                         sufficiently mitigate the risks that the program faces, but DIA’s initial risk
                         management actions are positive. It will be important for DIA to continue
                         to actively manage these risks with mitigations that are integrated into the
                         acquisition and program processes throughout MARS’s life cycle.

                         DIA has taken a number of actions to identify the needs of stakeholders
DIA Is Taking Actions    and engage them on the development of MARS. For example, DIA has
to Provide Information   conducted a series of workshops to identify initial requirements and town
                         halls to disseminate MARS updates. However, stakeholders have
to Stakeholders, but     expressed mixed views on the quality and extent of the MARS program’s
the MARS Program         engagement with them, and stakeholders we spoke with were generally
                         unsure about how their feedback and input are helping DIA prioritize and
Lacks a                  select MARS features. The MARS program is leveraging Agile
Comprehensive            development processes, which stress frequent stakeholder feedback, but
                         the program does not have a comprehensive plan for continual
Stakeholder              engagement with and collection of feedback from stakeholders.
Engagement Plan
                         In the National Defense Authorization Act for Fiscal Year 2010, Congress
                         directed the Secretary of Defense to develop and implement a new
                         acquisition process for IT systems that, as he determined appropriate,




                         17The Defense Digital Service is a technology organization within the Department of
                         Defense that reports directly to the Secretary of Defense. The organization includes
                         technology experts who conduct in-depth assessments of DOD’s high-impact projects to
                         evaluate each project’s ability to achieve objectives and provide suggestions to improve
                         program processes and efforts, according to DOD officials.



                         Page 14                                                    GAO-21-57 Defense Intelligence
                           includes early and continual involvement of the user. 18 This statute, in
                           addition to DOD’s 2010 report to Congress in response to the statute,
                           DOD Instruction 5000.02, and GAO’s Agile Assessment Guide, identifies
                           characteristics of effective user engagement for DOD acquisitions,
                           including the following: 19

                           •   Early engagement. Users are involved early during development to
                               ensure that efforts are aligned with user priorities.
                           •   Continual engagement. Users are involved on a regular, recurring
                               basis throughout development to stay informed about the system’s
                               technical possibilities, limitations, and development challenges. 20
                           •   Feedback based on actual working software. User feedback during
                               development is based on usable software increments to provide early
                               insight into the implementation of the solution and to test whether the
                               design works as intended.
                           •   Feedback incorporated into subsequent development. User
                               feedback is incorporated into the next build or increment.


DIA Has Taken Actions to   DIA has conducted a series of workshops to identify initial requirements
Engage Stakeholders, but   and town halls and other forums to disseminate MARS updates. First,
                           prior to beginning system development, DIA held 12 multi-day MARS
Stakeholders Expressed
                           workshops—including ones on infrastructure and order of battle—with
Mixed Views on the         CCMD, service, and Joint Staff participation, to identify stakeholder
Adequacy of Engagement     requirements for MARS, according to a DIA document. For example, the
                           MARS office sponsored a 3-day facilitated order of battle workshop in
                           June 2019 to identify opportunities and learn more about the mission
                           needs of those responsible for characterizing and tracking foreign military

                           18National Defense Authorization Act for Fiscal Year 2010, Pub. L. No. 111-84, § 804
                           (2009).
                           19DOD, A New Approach for Delivering Information Technology Capabilities in the
                           Department of Defense (November 2010) was DOD’s 2010 report to Congress.
                           20Section 804’s requirement for early and continual involvement of the user is consistent
                           with the recommendations in Chapter 6 of the March 2009 report of the Defense Science
                           Board Task Force, which noted that the success of the proposed model acquisition
                           process depended on early and continual involvement of the user. The proposed model
                           also calls for enhanced stakeholder engagement and analytical rigor throughout the
                           acquisition life cycle. In earlier phases of the acquisition, the program reviews should be
                           quarterly, calendar-based events, while later phases should link such reviews with
                           iterations or delivery of multiple, rapidly-executed increments/releases of capability. See
                           Defense Science Board Task Force, Department of Defense Policies and Procedures for
                           the Acquisition of Information Technology (March 2009).


                           Page 15                                                      GAO-21-57 Defense Intelligence
forces, according to a workshop summary. The workshop brought
together a coalition of stakeholders from three CCMDs, a Service
Component Command, four DIA regional centers, and a service
intelligence center. From this series of workshops, DIA was able to
identify 91 capabilities and develop over 900 functional requirements
known as “user stories”—the building blocks of software development.
According to a list compiled by MARS program officials, some of the user
stories related to the infrastructure mission area, including a requirement
to see operating patterns at facilities by periods of time to enable users to
understand normal operations. As the MARS program initially engaged
stakeholders, including officials from the Joint Staff, six of the nine
CCMDs and multiple service intelligence centers, these stakeholders
stated that they participated in one or more of DIA’s MARS workshops
and helped to identify key stakeholder needs and MARS requirements.

Second, DIA has taken a number of actions to continually engage its
MARS stakeholders. DIA, CCMD, and service officials reported different
forums and tools that MARS program officials use to update and provide
information on MARS to its stakeholders, including the following:

•   Town halls. DIA has held a number of IC-wide town halls to provide
    high level information on MARS, according to CCMD and service
    officials.
•   Quarterly Program Reviews. DIA holds quarterly update meetings
    on MARS progress for senior leadership in the CCMDs, intelligence
    agencies, joint staff, services, ODNI, and OUSD(I&S).
•   MARS Program Management Office meetings and site visits.
    MARS program officials held individual meetings and visits with a
    significant number of DOD stakeholders—including CCMDs, services,
    intelligence agencies, and Joint Staff from January 2018 through
    January 2020.
•   MARS working groups. DIA has hosted working groups to develop
    intelligence workflows related to operational readiness and production
    for the MARS system, according to CCMD officials.
•   MARS websites. DIA has established websites with MARS
    information on the Joint Worldwide Intelligence Communications
    System and on the Non-Secure Internet Protocol Router Network,
    according to DOD officials.

However, CCMD stakeholders expressed mixed views on the quality and
extent of the MARS program’s continual engagement with them. On one

Page 16                                            GAO-21-57 Defense Intelligence
hand, officials in six of the nine CCMDs we spoke with reported that they
were satisfied with the level of engagement with DIA on MARS
development or indicated that they had the opportunity to engage
substantively with DIA, including in working group meetings. For example,
Central Command officials told us that DIA is very open to communication
and engagement on MARS and is actively addressing and
communicating with Central Command on any issues that arise; these
CCMD officials described their relationship with DIA as very collaborative.
On the other hand, officials from three other CCMDs reported
dissatisfaction with how the MARS program was engaging with them. For
example, officials from Africa Command said they want MARS program
officials to give them more input on the use cases they had developed
and do a better job of disseminating information on MARS developments.

Service stakeholders also expressed mixed views on the quality and
extent of the MARS program’s continual engagement with them. Officials
from Air Force Air Combat Command highlighted that they were very
pleased with the considerable degree of engagement and exchange with
DIA on MARS across a broad range of their command’s personnel.
Officials from the National Air and Space Intelligence Center similarly
reported that the MARS functional requirements team is working closely
with them. However, officials from the Navy and Marine Corps expressed
concern about their limited involvement in MARS development potentially
leading to a lack of alignment between MARS and their services’ IT
modernization efforts. Air Force Headquarters officials also stated that
DIA needed to develop a more systematic method for stakeholder
engagement. MARS program officials stated that they always attempt to
reach out to all stakeholders affected during key phases of MARS and
engage with those who respond. The lead MARS program official for
engaging the services explained that many of the services’ personnel
have heard a limited amount about the MARS program, in part because
the services are not large producers of infrastructure analysis. As MARS
develops follow-on modules, including ones on the order of battle and
targeting, the services are expected to become more involved, because
they are larger producers of intelligence in these areas, according to this
MARS official.

Further, MARS stakeholders we spoke with generally said it was not clear
to them how their feedback and input on the system were helping to
prioritize or select the specific features for the initial and subsequent
releases of MARS. Such input and prioritization are key to the Agile
development methodology the MARS program is using, because
shortcomings in user feedback can lead to the delivery of systems that do

Page 17                                          GAO-21-57 Defense Intelligence
                         not meet user needs. For instance, Joint Staff officials said that they had
                         the opportunity to affect the features included in the initial MARS release.
                         However, five CCMDs stated that they either had not provided key input
                         on MARS features or were unsure how the feedback they provided had
                         been incorporated into system development. For example, officials from
                         Central Command, Transportation Command, and Special Operations
                         Command reported that they did not provide any input on the features
                         being developed for the infrastructure module—the initial MARS release.
                         DIA officials highlighted that—in order to get a working product out on
                         deadline—they focused mainly on developing the initial MARS release
                         instead of communicating how stakeholder input affected system
                         development. A senior DIA representative explained that substantive
                         stakeholder engagement to date has focused mainly on those who
                         perform infrastructure analytic work—primarily DIA analysts within DIA
                         and the CCMDs—to inform the first MARS release. 21 Although we found
                         this explanation to be generally reasonable given the status of the
                         program, it was unclear how DIA planned to fully engage other
                         stakeholders moving forward, as we discuss further below.

MARS’s Use of Agile      The MARS program’s use of Agile development processes allows for
Development Enables      frequent and iterative user engagement and feedback, based on actual
                         working software, which is then incorporated into subsequent
Frequent User
                         development. For example, Agile development calls for a designated
Engagement and           product owner who represents the stakeholder community and has the
Feedback from            authority to establish priorities based on stakeholder needs and approve
Stakeholders, but MARS   whether the completed software meets their needs. The product owner
Lacks a Comprehensive    also is to work daily with the development team to help clarify
                         requirements, make decisions, and maintain the backlog—the primary
Stakeholder Engagement   source for all requirements that is continually updated to reflect changes
Plan                     and stakeholder priorities and is used to select the stakeholders’ highest
                         priorities for which capabilities that should be developed next. Agile also
                         calls for actual working software to be demonstrated and released to
                         users on a frequent and iterative basis and for user feedback to be
                         incorporated into subsequent development.




                         21According to MARS program officials, for the incremental software updates leading up to
                         the release of the infrastructure module, the MARS program focused on test groups in
                         Central Command, Air Force’s Air Combat Command, and Joint Staff as their main
                         providers of feedback on the system, because these components are very significant
                         stakeholders in infrastructure-related intelligence.



                         Page 18                                                   GAO-21-57 Defense Intelligence
These Agile processes necessitate robust stakeholder involvement, but
the MARS program lacks a comprehensive stakeholder engagement plan
to guide its interactions with all stakeholders and provide a common
understanding to all stakeholders on what to expect. MARS program
officials shared a high-level stakeholder engagement and test plan, which
we reviewed. 22 The plan included some general information on selecting
stakeholders, capturing feedback, and forming test groups. However, we
found that the plan lacked key details that are called for in leading
practices for system development and reinforced in GAO’s Agile
Assessment Guide, such as how

•   stakeholders would be engaged at different times during system
    development, including when specific stakeholders for each MARS
    module would be engaged with the program and the extent that
    stakeholders in later modules would have early input to requirements
    and design decisions that affect them,
•   all stakeholder feedback would be compiled to support the
    prioritization of features,
•   stakeholder engagement and feedback would be monitored, and
•   user satisfaction would be measured. 23

First, regarding the timing of stakeholder engagement, the senior MARS
program official responsible for stakeholder engagement highlighted the
difficulty of trying to engage with such a large number of stakeholders in
the system. Officials in OUSD(I&S) who are responsible for MARS
oversight also told us that the MARS program office could have a better
stakeholder communications plan, including better communicating the
MARS program’s intent with stakeholders—especially how the pathfinder
programs are intended to address stakeholder concerns about system
interoperability—and how DIA plans to sequentially engage stakeholders
during system development.

Secondly, regarding compiling feedback and linking this feedback to the
prioritization of features, MARS program officials explained that, following
the release of the infrastructure module, they leveraged a software
application—Version One—to capture emails with stakeholder feedback

22DIA officials provided us with a brief document entitled “MARS Stakeholder
Engagement/Testing Plan” that provides general information on stakeholder engagement
and testing.
23Software   Engineering Institute, CMMI® for Development, Version 1.3 (November 2010).


Page 19                                                   GAO-21-57 Defense Intelligence
on the MARS release and track the dates of these emails and of the
MARS program’s responses to them. MARS program officials
acknowledged that for the next MARS software release, they will need to
communicate better with stakeholders about stakeholder feedback and
input related to the MARS features they were developing at DIA and how
these features would address system requirements—key elements of
Agile development. The stakeholder plan they shared with us did not lay
out a clear process moving forward for responding to feedback received
after stakeholders interact with the system or how such feedback would
help prioritize work on system features.

Lastly, regarding monitoring stakeholder engagement and measuring
user satisfaction with software, a senior MARS official acknowledged that
the MARS program did not have engagement metrics, which could help
ensure that stakeholders stay informed and satisfied, particularly in a
program with a large number of stakeholders. According to MARS
officials, the program used a test survey on the initial release to collect
data from stakeholders on user satisfaction, and the survey served as a
primary feedback mechanism. However, MARS program officials stated
that they needed to develop more specific engagement and user
satisfaction metrics moving forward.

A comprehensive stakeholder engagement plan is especially important
for MARS, given the large number of stakeholders and DIA’s phased
approach for delivering modules to different stakeholders during the
program life cycle. Involving stakeholders in early stages and throughout
software development allows an organization to detect deficiencies early.
Industry studies have shown that the later flaws are found, the more
expensive it becomes to correct them. 24 Additionally, according to our
previous work, shortcomings in user involvement and feedback in
programs can lead to the delivery of systems that do not meet user
needs. 25 Without a comprehensive plan to guide engagement with all
MARS stakeholders—CCMDs, services, and intelligence agencies—DIA

24Software Engineering Institute, Results of SEI Independent Research and Development
Projects and Report on Emerging Technologies and Technology Trends—Technical
Report CMU/SEI-2004-TR-018, (Pittsburgh: Oct. 2004); Iosif Alvertis and Sotiris
Koussouris, et al, “User Involvement in Software Development Processes,” Procedia
Computer Science, vol. 97 (2016): 73-83; JC Westland, “The Cost Of Errors In Software
Development: Evidence from Industry,” The Journal of Systems and Software 62, (2002)
p.1-9; and Deloitte, Agile in Government (2017).
25GAO, DOD Space Acquisitions: Including Users Early and Often in Software
Development Could Benefit Programs, GAO-19-136 (Washington, D.C.: Mar. 18, 2019).


Page 20                                                 GAO-21-57 Defense Intelligence
                     risks that MARS capabilities will not fully meet stakeholder expectations
                     and needs. Further, as previously noted, DIA is relying on engagement
                     with stakeholders to help it mitigate certain risks to MARS development,
                     but absent a more systematic approach to doing so, its efforts could fall
                     short.

                     Foundational military intelligence is crucial in planning military operations,
Conclusions          but current systems for managing this intelligence are too slow, outdated,
                     and limited in capacity to adequately support military needs. The
                     successful development of MARS could improve the way DOD plans for
                     and responds to military threats, improving the security of the United
                     States. To achieve this promise of MARS development, DIA will have to
                     address many risks to MARS development. DIA’s initial risk management
                     actions are encouraging, but it is still too early in the program’s
                     development to know whether its actions will sufficiently mitigate these
                     risks. Further, DIA has not planned adequately for ongoing, robust
                     engagement with all stakeholders. As a result, DIA may not be able to
                     identify and rectify deficiencies in MARS early on in its development cycle
                     and risks deploying a system that does not meet user needs or is
                     underutilized. Developing a comprehensive stakeholder engagement plan
                     could enhance DIA’s communications with stakeholders, improve the
                     likelihood that MARS will meet stakeholder needs, and strengthen DIA’s
                     efforts to mitigate program risks.

                     The Director of National Intelligence, in coordination with the Secretary of
Recommendation for   Defense, should ensure that the Defense Intelligence Agency develops a
Executive Action     comprehensive plan that details how it will engage all MARS
                     stakeholders, to include how specific stakeholders will be engaged at
                     different times, how stakeholder feedback will support the prioritization of
                     features, how stakeholder engagement and feedback will be monitored,
                     and how user satisfaction will be measured.

                     We provided a draft of this report to ODNI and DOD. ODNI provided
Agency Comments      written comments, in which it concurred with our recommendation.
and Our Evaluation   ODNI’s written comments are reprinted in their entirety in appendix II.
                     DOD also provided technical comments, which we incorporated into the
                     report where appropriate, and concurred with our recommendation.




                     Page 21                                            GAO-21-57 Defense Intelligence
We are sending copies of this report to the appropriate congressional
committees, the Director of National Intelligence, the Secretary of
Defense, the Under Secretary of Defense for Intelligence and Security,
and the Director of the Defense Intelligence Agency. In addition, the
report is available at no charge on the GAO website http://www.gao.gov.

If you or members of your staff have any questions regarding this report,
please contact me at (202) 512-5130 or mazanecb@gao.gov. Contact
points for our Offices of Congressional Relations and Public Affairs may
be found on the last page of this report. Key contributors to this report are
listed in appendix III.




Brian M. Mazanec
Director, Defense Capabilities and Management




Page 22                                            GAO-21-57 Defense Intelligence
List of Committees

The Honorable James M. Inhofe
Chairman
The Honorable Jack Reed
Ranking Member
Committee on Armed Services
United States Senate

The Honorable Marco Rubio
Acting Chairman
The Honorable Mark Warner
Vice Chairman
Select Committee on Intelligence
United States Senate

The Honorable Richard C. Shelby
Chairman
The Honorable Dick Durbin
Ranking Member
Subcommittee on Defense
Committee on Appropriations
United States Senate

The Honorable Adam Smith
Chairman
The Honorable Mac Thornberry
Ranking Member
Committee on Armed Services
House of Representatives

The Honorable Adam Schiff
Chairman
The Honorable Devin Nunes
Ranking Member
Permanent Select Committee on Intelligence
House of Representatives




Page 23                                      GAO-21-57 Defense Intelligence
The Honorable Pete Visclosky
Chairman
The Honorable Ken Calvert
Ranking Member
Subcommittee on Defense
Committee on Appropriations
House of Representatives




Page 24                        GAO-21-57 Defense Intelligence
Appendix I: Scope and Methodology
             Appendix I: Scope and Methodology




             To identify key risks facing the Machine-Assisted Analytic Rapid-
             Repository System (MARS) and Defense Intelligence Agency’s (DIA)
             efforts to mitigate these risks to date, we reviewed documents such as
             DIA’s Statement of Capabilities-Interim, technology demonstration
             assessments, and a technical risk summary. The risk assessment
             component of internal control—including the organization’s identification
             of, analysis of, and response to risk—was significant to this objective. We
             assessed DOD’s approach to the identification of, analysis of, and
             response to risks through MARS documentation and interviews with
             MARS stakeholders. We interviewed stakeholders to understand
             stakeholder perspectives on initial risks. 1 (See discussion of stakeholders
             later in this section.) We summarized selected risks identified by program
             documentation, DIA officials, and stakeholders and categorized each as a
             policy, operational, or technical risk. Risks included in our summary table
             were mentioned in multiple DIA documents or referenced by numerous
             stakeholders. We categorized risks based on the content of the
             information provided by DOD stakeholders and in coordination with GAO
             IT experts. We presented this summary document to DIA for validation;
             DIA officials validated our summary of key risks and described their
             mitigation efforts for each risk. To describe DIA’s risk management
             efforts, we interviewed officials from the DIA MARS Program
             Management Office and a variety of stakeholder organizations and
             reviewed key documents, such as DOD’s Technical Risks summary
             document and Army Research Laboratory technology pilot reviews. We
             spoke with oversight officials from the Office of the Director of National
             Intelligence (ODNI) and the Office of the Under Secretary of Defense for
             Intelligence and Security (OUSD(I&S)) to understand what risk
             management efforts DIA is expected to make and the expected time line
             for these efforts. We reviewed documents such as the DOD’s Risk, Issue,
             and Opportunity Management Guide for Defense Acquisition Programs to
             understand DIA’s intended risk framework.

             To assess how DIA is engaging potential stakeholders in the
             development of capabilities for the MARS program, we collected
             documentation and interviewed officials from DIA and other DOD and IC
             elements to assess how DIA’s processes and plans for MARS
             development adhere to key characteristics of effective user engagement

             1Officialsand documents used varying terms to describe these risks, including challenges,
             dependencies, and risks; for clarity and consistency, we refer to them throughout the
             report only as risks. We describe these risks in the report as initial risks, because as the
             MARS program progresses, DIA and stakeholders may identify new risks.



             Page 25                                                     GAO-21-57 Defense Intelligence
Appendix I: Scope and Methodology




as laid out in the National Defense Authorization Act for Fiscal Year 2010,
DOD guidance, 2 and GAO’s Agile Assessment Guide 3: (1) early
engagement, (2) continual engagement, (3) feedback based on working
software, and (4) feedback incorporated into subsequent development.

•   For “early engagement,” we reviewed DIA’s process for identifying
    initial user needs, such as whether DIA leveraged workshops or
    surveys to identify these needs. The GAO Agile Assessment Guide
    notes that the process for collecting customer needs and expectations
    relies in part on surveys and forums, which could include workshops.
•   For “continual engagement,” we identified DIA’s ongoing mechanisms
    to engage stakeholders and any plans for stakeholder
    communications. We spoke with DIA officials from the MARS Program
    Management Office, as well as officials from a wide variety of
    stakeholder organizations, to identify the forums and tools DIA was
    using to communicate MARS progress and updates, such as town
    halls and quarterly program reviews. We interviewed MARS
    stakeholders—including 9 of 11 Combatant Command (CCMDs) and
    service headquarters and intelligence center officials—to understand
    the relative quality of the engagement they were receiving on MARS.
    For quality of engagement, we asked whether stakeholders were
    receiving enough technical details on MARS to meet their needs and
    were generally satisfied with the level of communications from DIA.
•   For “feedback based on working software” and “feedback
    incorporated into subsequent development,” we interviewed DIA
    officials to determine whether DIA has established a discrete process
    to collect user feedback and suggestions as users interact with the
    system. We interviewed them also to understand whether their
    stakeholder feedback and input was being used by the MARS
    development team to determine and prioritize the MARS features in

2Section  804 of the National Defense Authorization Act for Fiscal Year 2010, Pub. L. No.
111-84 (2009), directed the Secretary of Defense to develop and implement a new
acquisition process for information technology systems that, to the extent determined
appropriate by the Secretary, would include early and continual involvement of the user.
This statute, in addition to DOD’s 2010 report to Congress in response to the statute and
DOD Instruction 5000.02, Operation of the Adaptive Acquisition Framework, (Jan. 23,
2020), identifies the characteristics of effective user engagement for DOD acquisitions.
3GAO,   Agile Assessment Guide: Best Practices for Agile Adoption and Implementation,
GAO-20-590G (Washington, D.C.: September 28, 2020). The guide presents best
practices to assess Agile adoption, execution, and program monitoring and control that
can be used across the federal government for agencies’ IT investments that rely on Agile
methods.



Page 26                                                    GAO-21-57 Defense Intelligence
                           Appendix I: Scope and Methodology




                                  the software releases and affect MARS development. As part of this
                                  effort, we assessed DIA’s stakeholder engagement and testing plan—
                                  a document DIA provided to us—against leading practices in system
                                  development, such as whether it included a specific feedback process
                                  and engagement-related metrics. 4

                           We also attended in person a MARS town hall meeting to observe DIA’s
                           stakeholder engagement and observed a demonstration of the MARS
                           infrastructure module to gain an understanding of DIA’s progress on the
                           development of MARS.

DOD and Other              In support of our work, we interviewed officials from the DOD and IC
Organizations with Whom    organizations listed here. We selected these organizations based on their
                           involvement in development, testing, outreach, and usage of both MIDB
GAO Conducted
                           and MARS. In particular, we selected stakeholders based on the roles
Interviews                 they will play in MARS, as identified by DOD officials. These roles include
                           the following: (1) consumers, which are DOD components that will use
                           the foundational military intelligence that will be stored in MARS, (2)
                           producers, which are DOD components that will create and update
                           authoritative foundational military intelligence records in MARS, and (3)
                           contributors, which are DOD or IC components that will provide source
                           information or data to support the development of foundational military
                           intelligence and related records that will be stored in MARS. We
                           interviewed the stakeholder organization one time, but the interview
                           included numerous officials from within the organization who will use
                           MARS. The full list of organizations that we interviewed follows:

United States Government   •      Defense Intelligence Agency
Organizations                     •   Program Management Office
                                      •   Technical Operations
                                      •   Program Manager
                                      •   Strategic Engagement Chief
                                  •   Chief Intelligence Officer
                           •      Office of the Director of National Intelligence


                           4For  system development, the Capability Maturity Model® Integration for Development
                           provides a comprehensive integrated set of guidelines and leading practices for
                           developing products and services, including new software. See Software Engineering
                           Institute, CMMI® for Development, Version 1.3 (November 2010).


                           Page 27                                                  GAO-21-57 Defense Intelligence
Appendix I: Scope and Methodology




•   Office of the Under Secretary of Defense for Intelligence and Security
•   Office of the Principal Cyber Advisor
•   Joint Staff
•   Combatant Commands
    •     Central Command
    •     Indo Pacific Command
    •     Strategic Command
    •     Special Operations Command
    •     Africa Command
    •     Transportation Command
    •     Cyber Command
    •     Southern Command
    •     European Command
•   National Geospatial-Intelligence Agency
•   National Reconnaissance Office
•   National Security Agency
•   U.S. Air Force
    •     Headquarters
    •     Air Combat Command
    •     National Air and Space Intelligence Center
•   U.S. Army
    •     Headquarters
    •     National Ground Intelligence Center
    •     Army Intelligence and Security Command
•   U.S. Navy
    •     Naval Information Warfare Systems Command
    •     Naval Intelligence Activity Chief Information Officer
•   U.S. Marine Corps
    •     Headquarters



Page 28                                              GAO-21-57 Defense Intelligence
        Appendix I: Scope and Methodology




            •     Marine Corps Intelligence Activity

Other   •   United Kingdom Defence Intelligence



        We conducted this performance audit from October 2019 to November
        2020 in accordance with generally accepted government auditing
        standards. Those standards require that we plan and perform the audit to
        obtain sufficient, appropriate evidence to provide a reasonable basis for
        our findings and conclusions based on our audit objectives. We believe
        that the evidence obtained provides a reasonable basis for our findings
        and conclusions based on our audit objectives.




        Page 29                                         GAO-21-57 Defense Intelligence
Appendix II: Comments from the Office of the
              Appendix II: Comments from the Office of the
              Director of National Intelligence



Director of National Intelligence




              Page 30                                        GAO-21-57 Defense Intelligence
Appendix III: GAO Contact and Staff
                  Appendix III: GAO Contact and Staff
                  Acknowledgments


Acknowledgments

                  Brian M. Mazanec, (202) 512-5130 or mazanecb@gao.gov.
GAO Contact

                  In addition to the contact named above, key contributors to this report
Staff             were Kasea Hamar, Assistant Director; Tracy Barnes; Robert Breitbeil;
Acknowledgments   Carolyn Demaree; Emile Ettedgui; Joanne Landesman; Jennifer Leotta;
                  and Jeanne Sung.




                  Page 31                                         GAO-21-57 Defense Intelligence
Related GAO Products
             Related GAO Products




             Agile Assessment Guide: Best Practices for Agile Adoption and
             Implementation. GAO-20-590G. Washington, D.C.: September 28, 2020.

             Intelligence Community: Actions Needed to Enhance Oversight of
             Business Functions and Systems. GAO-18-584SU. Washington, D.C.:
             Sep. 21, 2018.

             DOD Space Acquisitions: Including Users Early and Often in Software
             Development Could Benefit Programs. GAO-19-136. Washington, D.C.:
             Mar. 18, 2019.

             Space Command and Control: Comprehensive Planning and Oversight
             Could Help DOD Acquire Critical Capabilities and Address Challenges.
             GAO-20-146. Washington, D.C.: Oct. 30, 2019




             Page 32                                       GAO-21-57 Defense Intelligence
(103873)
                         The Government Accountability Office, the audit, evaluation, and investigative
GAO’s Mission            arm of Congress, exists to support Congress in meeting its constitutional
                         responsibilities and to help improve the performance and accountability of the
                         federal government for the American people. GAO examines the use of public
                         funds; evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.

                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through our website. Each weekday afternoon, GAO posts on its website newly
GAO Reports and          released reports, testimony, and correspondence. You can also subscribe to
                         GAO’s email updates to receive notification of newly posted products.
Testimony
Order by Phone           The price of each GAO publication reflects GAO’s actual cost of production and
                         distribution and depends on the number of pages in the publication and whether
                         the publication is printed in color or black and white. Pricing and ordering
                         information is posted on GAO’s website, https://www.gao.gov/ordering.htm.
                         Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                         TDD (202) 512-2537.
                         Orders may be paid for using American Express, Discover Card, MasterCard,
                         Visa, check, or money order. Call for additional information.

                         Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO         Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts.
                         Visit GAO on the web at https://www.gao.gov.

                         Contact FraudNet:
To Report Fraud,
                         Website: https://www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in
                         Automated answering system: (800) 424-5454 or (202) 512-7700
Federal Programs
                         Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400,
Congressional            U.S. Government Accountability Office, 441 G Street NW, Room 7125,
Relations                Washington, DC 20548

                         Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs           U.S. Government Accountability Office, 441 G Street NW, Room 7149
                         Washington, DC 20548

                         Stephen J. Sanford, Acting Managing Director, spel@gao.gov, (202) 512-4707
Strategic Planning and   U.S. Government Accountability Office, 441 G Street NW, Room 7814,
External Liaison         Washington, DC 20548




                            Please Print on Recycled Paper.