oversight

Homeland Security: DHS Requires More Disciplined Investment Management to Help Meet Mission Needs

Published by the Government Accountability Office on 2012-09-18.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States Government Accountability Office

GAO              Report to Congressional Requesters




September 2012
                 HOMELAND
                 SECURITY
                 DHS Requires More
                 Disciplined
                 Investment
                 Management to Help
                 Meet Mission Needs




GAO-12-833
                                           September 2012

                                           HOMELAND SECURITY
                                           DHS Requires More Disciplined Investment
                                           Management to Help Meet Mission Needs
Highlights of GAO-12-833, a report to
congressional requesters




Why GAO Did This Study                     What GAO Found
DHS invests extensively in major           Nearly all of the Department of Homeland Security (DHS) program managers
acquisition programs to develop new        GAO surveyed reported their programs had experienced significant challenges.
systems that help the department           Sixty-eight of the 71 respondents reported they experienced funding instability,
execute its many critical missions. In     faced workforce shortfalls, or their planned capabilities changed after initiation,
2011, DHS reported to Congress that it     and most survey respondents reported a combination of these challenges. DHS
planned to invest $167 billion in these    lacks the data needed to accurately measure program performance, but GAO
major acquisition programs. We             was able to use survey results, information DHS provided to Congress, and an
previously found that DHS had not          internal DHS review from March 2012 to identify 42 programs that experienced
managed its investments effectively,
                                           cost growth, schedule slips, or both. GAO gained insight into the magnitude of
and its acquisition management
                                           the cost growth for 16 of the 42 programs, which increased from $19.7 billion in
activities have been on GAO’s High
Risk List since 2005. This report
                                           2008 to $52.2 billion in 2011, an aggregate increase of 166 percent.
addresses the extent to which (1)          DHS acquisition policy reflects many key program management practices that
major DHS acquisition programs face        could help mitigate program risks. It requires programs to develop documents
key challenges; (2) DHS has policies       demonstrating critical knowledge that would help leaders make better informed
and processes to effectively manage        investment decisions when managing individual programs. However, DHS has
individual acquisition programs; (3)       not consistently met these requirements. The department has only verified that
DHS has policies and processes to          four programs documented all of the critical knowledge the policy requires to
effectively manage its portfolio of
                                           proceed with acquisition activities. Officials explained that DHS’s culture has
acquisition programs as a whole; and
                                           emphasized the need to rapidly execute missions more than sound acquisition
(4) DHS has taken actions to address
the high-risk acquisition management       management practices. Most major programs lack reliable cost estimates,
issues GAO has identified in previous      realistic schedules, and agreed-upon baseline objectives, limiting DHS
reports. GAO surveyed all 77 major         leadership’s ability to effectively manage those programs and provide information
program offices DHS identified in 2011     to Congress. DHS recognizes the need to implement its acquisition policy more
(92 percent response rate), reviewed       consistently, but significant work remains.
available documentation of acquisition     DHS acquisition policy does not fully reflect several key portfolio management
decisions from November 2008 to April
                                           practices, such as allocating resources strategically, and DHS has not yet re-
2012, and interviewed officials at DHS
                                           established an oversight board to manage its investment portfolio across the
headquarters and components.
                                           department. As a result, DHS has largely made investment decisions on a
What GAO Recommends                        program-by-program and component-by-component basis. The widespread risk
                                           of poorly understood cost growth, coupled with the fiscal challenges facing the
GAO recommends that DHS modify its         federal government, makes it essential that DHS allocate resources to its major
policy to better reflect key program and   programs in a deliberate manner. DHS plans to develop stronger portfolio-
portfolio management practices,
                                           management policies and processes, but until it does so, DHS programs are
ensure acquisition programs fully
                                           more likely to experience additional funding instability, which will increase the risk
comply with DHS acquisition policy,
prioritize major acquisition programs
                                           of further cost growth and schedule slips. These outcomes, combined with a
departmentwide and account for             tighter budget, could prevent DHS from developing needed capabilities.
anticipated resource constraints, and      DHS has introduced seven initiatives that could improve acquisition management
document prerequisites for delegating      by addressing longstanding challenges GAO and DHS survey respondents have
major milestone decision authority.        identified, such as funding instability and acquisition workforce shortfalls.
DHS concurred with all of GAO’s            Implementation plans are still being developed, and DHS is still working to
recommendations, and noted its             address critical issues. Because of this, it is too early to determine whether the
progress on a number of fronts, which
                                           DHS initiatives will be effective, as GAO has previously established that agencies
is accounted for in the report.
                                           must sustain progress over time to address management challenges. DHS is
View GAO-12-833. For more information,     also pursuing a tiered-governance structure, but it must reduce risks and improve
contact John Hutton at (202) 512-4841 or   program outcomes before regularly delegating major milestone decision
huttonj@gao.gov.
                                           authority.


                                                                                     United States Government Accountability Office
Contents


Letter                                                                                         1
                       Background                                                              3
                       Nearly All Program Managers Surveyed Reported Significant
                         Challenges                                                          10
                       Programs Proceed without Meeting Sound DHS Acquisition Policy         21
                       DHS Needs Policy and Process Enhancements to Effectively
                         Manage Its Portfolio of Investments                                 33
                       DHS Acquisition Management Initiatives Target Longstanding
                         Challenges, but Key Implementation Issues Remain                    39
                       Conclusions                                                           45
                       Recommendations for Executive Action                                  46
                       Agency Comments and Our Evaluation                                    47

Appendix I             Objectives, Scope, and Methodology                                     51



Appendix II            Key Acquisition Management Practices                                   54



Appendix III           Program Office Survey Results                                          60



Appendix IV            Major DHS Acquisition Programs and their Key Acquisition
                       Documents                                                              94



Appendix V             Comments from the Department of Homeland Security                      99



Appendix VI            GAO Contact and Staff Acknowledgments                                103



Related GAO Products                                                                        104




                       Page i                                        GAO-12-833 Homeland Security
Tables
          Table 1: DHS Acquisition Levels for Major Acquisition Programs             4
          Table 2: Key DHS Acquisition Documents Requiring Department-
                   level Approval                                                    7
          Table 3: GAO Assessment of DHS’s Acquisition Policy Compared to
                   Key Program-management Practices                                23
          Table 4: GAO Assessment of DHS’s Acquisition Policy Compared to
                   Key Portfolio-management Practices                              34
          Table 5: GAO Assessment of DHS Acquisition Management
                   Initiatives                                                     43
          Table 6: Programs that responded to our survey                           94
          Table 7: Programs that did not respond to our survey                     98
          Table 8: Programs cancelled in 2011                                      98


Figures
          Figure 1: DHS Acquisition Life Cycle and Document Requirements
                   for Major Acquisition Programs                                    6
          Figure 2: DHS Acquisition Managers                                         9
          Figure 3: Breakdown of 68 Major Acquisition Programs
                   Experiencing One or More Significant Management
                   Challenges                                                      11
          Figure 4: Number of Major Acquisition Programs Experiencing
                   Cost Growth or Schedule Slips                                   12
          Figure 5: Programs that Changed Planned Capabilities and
                   Experienced Cost Growth or Schedule Slips                       14
          Figure 6: Reasons Programs Changed Their Planned Capabilities            15
          Figure 7: Effects of Funding Instability on Programs, as Identified
                   by Program Managers                                             17
          Figure 8: Reasons Programs Experienced Funding Instability               18
          Figure 9: Functional Areas Contributing to Workforce Shortfalls          20
          Figure 10: DHS Acquisition Documents Requiring Department-level
                   Approval                                                        25
          Figure 11: Programs That Have Key Acquisition Documents
                   Approved in Accordance with AD 102                              27
          Figure 12: Programs Reviewed by DHS’s Executive Review Board             28
          Figure 13: Programs with Department-approved APBs, Reliable
                   Cost Estimates, and Realistic Schedules                         30
          Figure 14: Councils and Offices in DHS’s Proposed IILCM                  36
          Figure 15: Acquisition Management Challenges and Corresponding
                   Initiatives                                                     40



          Page ii                                          GAO-12-833 Homeland Security
Figure 16: Initiatives that Slipped from DHS’s Original January 2011
         Schedule to the June 2012 Update                                                 42
Figure 17: DHS’s Proposed IRB/ESC Governance Structure                                    44




Abbreviations

AD 102            Acquisition Management Directive 102-01
AOA               Analysis of alternatives
APB               Acquisition Program Baseline
CAE               Component Acquisition Executive
CASR              Comprehensive Acquisition Status Report
CRC               Capabilities and Requirements Council
DHS               Department of Homeland Security
ESC               Executive Steering Committee
FYHSP             Future Years Homeland Security Program
IILCM             Integrated Investment Life Cycle Model
IT                Information Technology
IRB               Investment Review Board
JRC               Joint Requirements Council
MAOL              Major Acquisition Oversight List
MNS               Mission Need Statement
OCFO              Office of the Chief Financial Officer
PA&E              Office of Program Analysis and Evaluation
PARM              Office of Program Accountability and Risk Management
QPAR              Quarterly Program Accountability Report
USM               Under Secretary for Management




This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page iii                                                    GAO-12-833 Homeland Security
United States Government Accountability Office
Washington, DC 20548




                                   September 18, 2012

                                   Congressional Requesters

                                   The Department of Homeland Security (DHS) invests extensively in
                                   acquisition programs to develop new systems that help the department
                                   execute its many critical missions. DHS is acquiring systems to help
                                   secure the border, facilitate trade, screen travelers, enhance cyber
                                   security, improve disaster response, and execute a wide variety of other
                                   operations. Several of DHS’s major acquisition programs—the
                                   department’s most expensive and critical investments—existed prior to
                                   the creation of DHS and were managed by one of the 22 separate
                                   agencies that merged to form the department. These acquisition
                                   programs are now managed by senior officials at DHS headquarters and
                                   12 component agencies. In 2011, DHS reported to Congress that it
                                   planned to ultimately invest $167 billion in its major acquisition programs.
                                   In fiscal year 2012, DHS reported it was investing more than $18 billion in
                                   the department’s acquisition programs.

                                   DHS acquisition management issues have been highlighted in our High-
                                   Risk List since 2005. 1 Over the past several years, our work has identified
                                   significant shortcomings in the department’s ability to manage an
                                   expanding portfolio of complex acquisitions. 2 In 2008, DHS revised its
                                   acquisition review process to include more detailed guidance for key
                                   acquisition decision events, documentation requirements, and the roles
                                   and responsibilities of DHS decision makers. In January 2010, DHS again
                                   updated its acquisition policy, but later that year, we found that the
                                   department still was not effectively carrying out its acquisition


                                   1
                                    GAO, High-Risk Series: An Update, GAO-05-207 (Washington, D.C.: January 2005)
                                   2
                                    For examples, see GAO, Department of Homeland Security: Billions Invested in Major
                                   Programs Lack Appropriate Oversight, GAO-09-29 (Washington, D.C.: November 18,
                                   2008); Department of Homeland Security: Assessments of Selected Complex
                                   Acquisitions, GAO-10-588SP (Washington, D.C.: June 30, 2010); Secure Border Initiative:
                                   Controls over Contractor Payment for the Technology Component Need Improvement,
                                   GAO-11-68 (Washington, D.C.: May 25, 2011); High-Risk Series: An Update,
                                   GAO-11-278 (Washington, D.C.: February 2011); Opportunities to Reduce Potential
                                   Duplication in Government Programs, Save Tax Dollars, and Enhance Revenue,
                                   GAO-11-318SP (Washington, D.C.: March 1, 2011); Coast Guard: Action Needed As
                                   Approved Deepwater Program Remains Unachievable, GAO-11-743 (Washington, D.C.:
                                   July 28, 2011)




                                   Page 1                                                   GAO-12-833 Homeland Security
management responsibilities. 3 We have previously established that a
program must have a sound business case that includes firm
requirements, a knowledge-based acquisition strategy, and realistic cost
estimates in order to reduce program challenges. 4 These conditions
provide a program a reasonable chance of overcoming challenges yet
delivering on time and within budget.

Because DHS invests significant resources developing capabilities to
support the department’s mission, you asked us to assess the extent to
which (1) DHS’s major acquisition programs face challenges that increase
the risk of poor outcomes; (2) DHS has policies and processes in place to
effectively manage individual acquisition programs; (3) DHS has policies
and processes in place to effectively manage its portfolio of acquisition
programs as a whole; and (4) DHS has taken actions to address the high-
risk acquisition management issues we have identified in previous
reports. In addition to this report, we are also issuing a report focused on
the performance of DHS’s major information technology (IT)
investments. 5

To determine the extent to which major acquisition programs identified by
DHS in 2011 face challenges, we surveyed all 77 major program offices
from January to March 2012, and achieved a 92 percent response rate. 6
Through our survey, we collected information regarding major programs’
performance, program managers’ understanding of acquisition guidance,
challenges developing requirements, and funding issues. We also
reviewed all available documentation of department-level acquisition
decisions from November 2008 to April 2012 and interviewed acquisition
officials at DHS headquarters and components to understand program


3
GAO-10-588SP.
4
 GAO, Defense Acquisitions: Managing Risk to Achieve Better Outcomes, GAO-10-374T
(Washington, D.C.: Jan. 20, 2010).
5
 This forthcoming report assesses the extent to which subsidiary projects within DHS IT
investments are meeting their cost and schedule commitments, and the primary causes of
any commitment shortfalls. The report focuses on investments that had one or more
subsidiary projects exceeding 10 percent of their cost and schedule commitments. A
project is a temporary effort (e.g. 9 months) to accomplish a unique product or service,
such as adding enhancements to a system.
6
 DHS originally identified 82 major acquisition programs in the 2011 major acquisition
oversight list, but five of those programs were subsequently cancelled in 2011. Seventy-
one program managers responded to the survey. See appendix IV.




Page 2                                                     GAO-12-833 Homeland Security
             risks, management challenges, and data limitations—particularly data
             limitations regarding program performance. Further, we reviewed
             resource plans and DHS performance reports to establish the extent to
             which major acquisition programs are achieving their cost, schedule and
             capability objectives. To determine the extent to which DHS has policies
             in place to effectively manage individual acquisition programs, as well as
             the department’s acquisition portfolio as a whole, we compared our key
             acquisition management practices to DHS acquisition policy, and
             identified the extent to which DHS has implemented its policy. 7 We also
             met with DHS officials to discuss our analysis, identify relevant sections of
             the policy that we had not yet accounted for, and solicit their thoughts on
             those key practices that were not reflected in the policy. To determine the
             extent to which DHS has taken actions to address the high-risk
             acquisition management issues we have identified in previous reports, we
             analyzed the department’s recently proposed efforts to address high-risk
             acquisition management challenges, including the department’s progress
             in implementing new initiatives and any challenges DHS must overcome
             moving forward.

             We conducted this performance audit from August 2011 to September
             2012 in accordance with generally accepted government auditing
             standards. Those standards require that we plan and perform the audit to
             obtain sufficient, appropriate evidence to provide a reasonable basis for
             our findings and conclusions based on our audit objectives. We believe
             that the evidence obtained provides a reasonable basis for our findings
             and conclusions based on our audit objectives.


             DHS invests in major acquisition programs to develop capabilities
Background   intended to improve its ability to execute its mission. DHS generally
             defines major programs as those expected to cost at least $300 million
             over their respective life cycles, and many are expected to cost more than
             $1 billion. DHS Acquisition Management Directive 102-01 (AD 102) and
             DHS Instruction Manual 102-01-001 (Guidebook), which includes 12
             appendixes, establish the department’s policies and processes for
             managing these major acquisition programs. DHS issued the initial
             version of AD 102 in 2008 in an effort to establish an acquisition


             7
              Appendix II identifies key acquisition management practices established in our previous
             reports examining DHS, the Department of Defense, National Aeronautics and Space
             Administration, and private sector organizations.




             Page 3                                                     GAO-12-833 Homeland Security
                                            management system that effectively provides required capability to
                                            operators in support of the department’s missions. 8 AD 102 establishes
                                            that DHS’s Chief Acquisition Officer—currently the Under Secretary for
                                            Management (USM)—is responsible for the management and oversight
                                            of the department’s acquisition policies and procedures. 9 The USM,
                                            Deputy Secretary, and Component Acquisition Executives (CAE) are the
                                            Acquisition Decision Authorities for DHS’s major acquisition programs.
                                            Table 1 identifies how DHS categorizes the 77 major acquisition
                                            programs it identified in 2011.

Table 1: DHS Acquisition Levels for Major Acquisition Programs

                                                                                                                                               Number of
                                                                                                                                              programs in
Level     Life-cycle cost                          Acquisition Decision Authority                                                           fiscal year 2011
1         Greater than or equal to $1 billion      Deputy Secretary, Chief Acquisition Officer, or the Under
                                                   Secretary for Management                                                                        43
2         $300 million or more, but less than      Chief Acquisition Officer, Under Secretary for Management, or
          $1 billion                               the Component Acquisition Executive                                                             34
                                            Source: GAO analysis of AD 102 and DHS’s fiscal year 2011 Major Acquisition Oversight List.

                                            Notes: Currently, the USM is designated DHS’s Chief Acquisition Officer, but another official could
                                            serve in that role in the future.
                                            Non-major acquisition programs expected to cost less than $300 million are designated Level 3. An
                                            acquisition may be raised to a higher acquisition level if (a) its importance to DHS’s strategic and
                                            performance plans is disproportionate to its size, (b) it has high executive visibility, (c) it impacts more
                                            than one component, (d) it has significant program or policy implications, or (e) the Deputy Secretary,
                                            Chief Acquisition Officer, or Acquisition Decision Authority recommends an increase to a higher level.
                                            Level 1 and 2 acquisitions may be delegated to components through formal letters of delegation from
                                            the Acquisition Decision Authority.


The Acquisition Life Cycle and              The Acquisition Decision Authority is responsible for reviewing and
Key Documents                               approving the movement of DHS’s major acquisition programs through
                                            four phases of the acquisition life cycle at a series of five predetermined
                                            Acquisition Decision Events. These five Acquisition Decision Events



                                            8
                                             The interim version of AD 102 replaced Management Directive 1400, which had
                                            governed major acquisition programs since 2006. DHS originally established an
                                            investment review process in 2003 to provide departmental oversight of major investments
                                            throughout their life cycles, and to help ensure that funds allocated for investments
                                            through the budget process are well spent. DHS issued an updated version of AD 102 in
                                            January 2010 and subsequently updated the guidebook and appendices.
                                            9
                                             The Secretary of DHS designated the USM the department’s Chief Acquisition Officer in
                                            April, 2011.




                                            Page 4                                                                                GAO-12-833 Homeland Security
provide the Acquisition Decision Authority an opportunity to assess
whether a major program is ready to proceed through the life-cycle
phases. The four phases of the acquisition life cycle, as established in AD
102, are:

1. Need phase: Department officials identify that there is a need,
   consistent with DHS’s strategic plan, justifying an investment in a new
   capability and the establishment of an acquisition program to produce
   that capability;
2. Analyze/Select phase: the Acquisition Decision Authority designates a
   qualified official to manage the program, and this program manager
   subsequently reviews alternative approaches to meeting the need,
   and recommends a best option to the Acquisition Decision Authority;
3. Obtain phase: The program manager develops, tests, and evaluates
   the selected option; during this phase, programs may proceed through
   ADE 2B, which focuses on the cost, schedule, and performance
   parameters for each of the program’s projects; and ADE 2C, which
   focuses on low rate initial production issues; and
4. Produce/Deploy/Support phase: DHS delivers the new capability to its
   operators, and maintains the capability until it is retired; this phase
   includes sustainment, which begins when a capability has been
   fielded for operational use; sustainment involves the supportability of
   fielded systems through disposal, including maintenance and the
   identification of cost reduction opportunities; this phase tends to
   account for up to 70 percent of life-cycle costs.

Figure 1 depicts the acquisition life cycle.




Page 5                                           GAO-12-833 Homeland Security
Figure 1: DHS Acquisition Life Cycle and Document Requirements for Major Acquisition Programs




                                       a
                                        Programs with multiple increments are categorized based on their most mature increment. We were
                                       unable to identify the phase for one program.
                                       b
                                        Documents identified for ADE 2B are required for capital assets. Programs providing services only
                                       require an APB and AP at ADE 2B.
                                       c
                                        Analyses of Alternatives and Concepts of Operations are approved at the component level at ADE
                                       2A.
                                       d
                                           Level 2 programs’ life-cycle cost estimates do not require department-level approval.


                                       An important aspect of the Acquisition Decision Events is the review and
                                       approval of key acquisition documents critical to establishing the need for
                                       a major program, its operational requirements, an acquisition baseline,
                                       and testing and support plans. AD 102—and the associated DHS
                                       Instruction Manual 102-01-001 and appendixes—provide more detailed
                                       guidance for preparing these documents than DHS’s predecessor policy.
                                       See table 2 for descriptions of the key acquisition documents requiring
                                       department-level approval before a program moves to the next acquisition
                                       phase.




                                       Page 6                                                                GAO-12-833 Homeland Security
Table 2: Key DHS Acquisition Documents Requiring Department-level Approval

Document                            Description
Mission Need Statement              Provides a high-level description of the mission need, whether from a current or impending
                                    gap. Outlines only the concept of the solution to fill the gap and does not provide information
                                    on specific types of acquisitions that could provide that capability.
Capability Development Plan         Serves as the agreement between the component head, program manager and the
                                    Acquisition Decision Authority on the activities, cost, and schedule for the work to be
                                    performed in the Analyze/Select phase.
Operational Requirements Document   Provides a number of performance parameters that must be met by a program to provide
                                    useful capability to the operator by closing the capability gaps identified in the Mission Need
                                    Statement.
Acquisition Plan                    Provides a top-level plan for the overall acquisition approach. Describes why the solution is
                                    in the government’s best interest and why it is the most likely to succeed in delivering
                                    capabilities to operators.
Integrated Logistics Support Plan   Defines the strategy for ensuring the supportability and sustainment of a future capability.
                                    Provides critical insight into the approach, schedule, and funding requirements for integrating
                                    supportability requirements into the systems engineering process.
Life-Cycle Cost Estimate 10         Provides an exhaustive and structured accounting of all resources and associated cost
                                    elements required to develop, produce, deploy, and sustain a particular program.
Acquisition Program Baseline        Establishes a program’s critical baseline cost, schedule, and performance parameters.
                                    Expresses the parameters in measurable, quantitative terms, which must be met in order to
                                    accomplish the investment’s goals.
Test and Evaluation Master Plan     Documents the overarching test and evaluation approach for the acquisition program.
                                    Describes the Developmental and Operational Test and Evaluation needed to determine a
                                    system’s technical performance, operational effectiveness/suitability, and limitations.
                                       Source: DHS acquisition policy.




Acquisition Management                 DHS acquisition policy requires that the DHS Investment Review Board
Officials                              (IRB) support the Acquisition Decision Authority by reviewing major
                                       acquisition programs for proper management, oversight, accountability,
                                       and alignment to the department’s strategic functions at Acquisition
                                       Decision Events and other meetings as needed. DHS acquisition policy
                                       establishes that the IRB shall be chaired by the Acquisition Decision
                                       Authority and consist of individuals that manage DHS’s mission
                                       objectives, resources, and contracts, including:

                                       •     Under Secretary for Science and Technology,
                                       •     Assistant Secretary for Policy,
                                       •     General Counsel,


                                       10
                                           Level 2 programs’ life cycle cost estimates do not require department-level approval.




                                       Page 7                                                        GAO-12-833 Homeland Security
•   Chief Financial Officer,
•   Chief Procurement Officer,
•   Chief Information Officer,
•   Chief Human Capital Officer,
•   Chief Administrative Services Officer,
•   Chief Security Officer,
•   CAE responsible for the program being reviewed, and
•   User representatives from component(s) sponsoring the capability.

The Office of Program Accountability and Risk Management (PARM) is
responsible for DHS’s overall acquisition governance process, supports
the IRB, and reports directly to the USM. PARM, which is led by an
executive director, develops and updates program management policies
and practices, oversees the acquisition workforce, provides support to
program managers, and collects program performance data. In March
2012, PARM issued its first Quarterly Program Accountability Report,
which provided an independent evaluation of major programs’ health and
risks.

The department’s program management offices are responsible for
planning and executing DHS’s individual programs within cost, schedule,
and performance goals. The program managers provide the IRB key
information by preparing required acquisition documents that contain
critical knowledge about their respective programs, facilitating the
governance process. Nearly all of DHS’s program management offices
are located within 12 of the department’s component agencies, such as
the Transportation Security Administration, or U.S. Customs and Border
Protection. 11 Within these components, CAEs are responsible for
establishing acquisition processes and overseeing the execution of their
respective portfolios. Additionally, under AD 102, the USM can delegate
Acquisition Decision Authority to CAEs for programs with life-cycle cost
estimates between $300 million and $1 billion. Figure 2 depicts the
relationship between acquisition managers at the department,
component, and program level.




11
  According to DHS’s fiscal year 2011 Major Acquisition Oversight List, the department’s
major acquisition programs were sponsored by 12 different component agencies, as
identified in appendix IV.




Page 8                                                     GAO-12-833 Homeland Security
Figure 2: DHS Acquisition Managers




The Office of Program Analysis and Evaluation (PA&E), within the Office
of the Chief Financial Officer (OCFO), is responsible for advising the
USM, among others, on resource allocation issues. PA&E coordinates
with DHS’s Office of Policy on the department’s long-term strategic
planning efforts, analyzing budget submissions, cost estimates, and
resource constraints. PA&E also oversees the development of the Future
Years Homeland Security Program (FYHSP). DHS is required to submit
the FYHSP to Congress annually with each budget request. The FYHSP
is DHS’s 5-year funding plan for programs approved by the Secretary that
are to support the DHS strategic plan. The FYHSP provides a detailed
account of time-phased resource requirements for each component, as



Page 9                                         GAO-12-833 Homeland Security
                       well as programs’ cost estimates, milestones, and performance
                       measures. 12


                       Nearly all of the program managers we surveyed reported their programs
Nearly All Program     had experienced significant challenges increasing the risk of poor
Managers Surveyed      outcomes, particularly cost growth and schedule slips. Sixty-eight of the
                       71 programs that responded to our survey reported that they experienced
Reported Significant   funding instability, faced workforce shortfalls, or their planned capabilities
Challenges             changed after initiation. Most program managers reported a combination
                       of these challenges, as illustrated in figure 3.




                       12
                        6 U.S.C. § 454.




                       Page 10                                            GAO-12-833 Homeland Security
Figure 3: Breakdown of 68 Major Acquisition Programs Experiencing One or More
Significant Management Challenges




Note: Fifty-nine survey respondents reported whether their program’s planned capabilities changed
since design and development activities began; 71 survey respondents reported whether their
programs experienced funding instability; 62 survey respondents reported whether their program’s
experienced workforce shortfalls.


We have previously reported that these challenges increase the likelihood
acquisition programs will cost more and take longer to deliver capabilities
than expected. 13 Although DHS lacks the reliable cost estimates and
realistic schedules needed to accurately measure program performance,



13
  GAO, Defense Acquisitions: Assessments of Selected Weapon Programs,
GAO-10-388SP (Washington, D.C.: March 30, 2010); Best Practices: Better Matching of
Needs and Resources Will Lead to Better Weapon System Outcomes, GAO-01-288
(Washington, D.C.: March 8, 2011); GAO-10-588SP; GAO-11-743.




Page 11                                                          GAO-12-833 Homeland Security
it has submitted some cost information to Congress, and PARM
conducted an internal review of its major acquisition programs in March
2012. We used this information and our survey results to identify 42
programs that experienced cost growth, schedule slips, or both. 14 Cost
information DHS submitted to Congress provides insight into the
magnitude of the cost growth for 16 of the 42 programs. Using this
information, we found total project costs increased from $19.7 billion in
2008 to $52.2 billion in 2011, an aggregate increase of 166 percent. See
figure 4.

Figure 4: Number of Major Acquisition Programs Experiencing Cost Growth or
Schedule Slips




Note: We calculated the magnitude of the total-project cost growth using DHS FYHSPs issued in
2008 and 2011. DHS did not submit FYHSPs in 2009 or 2010. FYHSP cost estimates are presented
in then-year dollars. Additionally, 40 survey respondents reported having a DHS approved Acquisition
Program Baseline and whether their program had experienced cost growth or schedule slips.


We have previously reported that cost growth and schedule slips can lead
to reduced capabilities, decreasing the value provided to the operator—as
well as the value of the resources invested in the programs. 15 This poor


14
  The cost estimates DHS reported to Congress were presented in then-year dollars.
15
  GAO, Best Practices: Better Support of Weapon System Program Managers Needed to
Improve Outcomes, GAO-06-110 (Washington, D.C.: November 30, 2005);
GAO-10-588SP.




Page 12                                                           GAO-12-833 Homeland Security
                             performance threatens the department’s ability to successfully field the
                             capabilities it is pursuing.


Most Major Programs          Prior to entering the Obtain phase, programs are to establish the specific
Reported Their Planned       capabilities they plan to develop to improve DHS’s ability to execute its
Capabilities Changed after   mission. Forty-three survey respondents reported that their programs
                             changed planned capabilities after the initiation of design and
Initiation                   development activities, which occurs between ADE 2B and testing. We
                             have previously found that both increases and decreases in planned
                             capabilities are associated with cost growth and schedule slips. 16 We
                             have found that increasing planned capabilities can lead to cost growth or
                             schedule slips because programs are more costly to change after they
                             begin development activities. Alternatively, we have stated that programs
                             may choose to decrease their planned capabilities in response to cost
                             growth or schedule slips in an effort to maintain affordability or deliver
                             certain capabilities when needed. At DHS, we found that more than half
                             of the 43 programs that reported changing their capabilities had
                             experienced cost growth or schedule slips, regardless of whether their
                             planned capabilities increased, decreased, or both. See figure 5.




                             16
                              GAO-10-388SP.




                             Page 13                                          GAO-12-833 Homeland Security
Figure 5: Programs that Changed Planned Capabilities and Experienced Cost
Growth or Schedule Slips




Note: Fifty-nine survey respondents replied to whether their program had changed planned
capabilities. Forty respondents reported having a DHS approved Acquisition Program Baseline and
whether their program had experienced cost growth or schedule slips.


The 43 survey respondents that reported their planned capabilities
changed identified five key reasons for the changes. Nineteen of the 43
survey respondents reported more than one reason. See figure 6.




Page 14                                                         GAO-12-833 Homeland Security
Figure 6: Reasons Programs Changed Their Planned Capabilities




                                       Note: Fifty-nine survey respondents replied to whether their programs’ planned capabilities changed;
                                       43 reported reasons their planned capabilities changed; 19 of the 43 reported multiple reasons.


                                       Survey respondents identified operator input as the most common reason
                                       for increasing planned capabilities after the initiation of development
                                       efforts, even though officials at the department, component, and program
                                       levels all said operator input at the initiation of design and development is
                                       very useful. For example, in 2011, we reported that the U.S. Citizenship
                                       and Immigration Services’s Transformation program did not fully define its
                                       planned capabilities before it awarded a contract to develop a new
                                       system to enhance the adjudication of applications. After the contract was
                                       awarded, the program office worked with those officials most familiar with
                                       adjudication operations and discovered that the functions were more
                                       complex than expected. As a result, the program office revised the
                                       requirements, and the deployment date for key capabilities slipped from
                                       April 2011 to October 2012. 17




                                       17
                                         GAO, Immigration Benefits: Consistent Adherence to DHS’s Acquisition Policy Could
                                       Help Improve Transformation Program Outcomes, GAO-12-66 (Washington, D.C.:
                                       November 22, 2011).




                                       Page 15                                                           GAO-12-833 Homeland Security
                      Alternatively, DHS program managers identified funding availability as the
                      most common reason for decreasing planned capabilities after the
                      initiation of development efforts. In the past, we have stated that agencies
                      may reduce planned capabilities in this manner when their programs
                      experience cost growth. 18 Decreasing planned capabilities in response to
                      affordability concerns may be fiscally responsible, but as a result,
                      operators may not receive the capability originally agreed upon to
                      address existing capability gaps.


Most Major Programs   DHS is required to establish out-year funding levels for programs annually
Reported They         in the FYHSP. Changes to planned out-year funding levels create funding
Experienced Funding   instability, which we have previously found increases the risk of cost
                      growth, schedule slips, and capability shortfalls. 19 Sixty-one survey
Instability
                      respondents reported that their programs have experienced funding
                      instability, and we found that 44 of the 61 programs had also realized cost
                      growth, schedule slips, or capability reductions. 20 Additionally, 29 survey
                      respondents reported that their programs had to resequence the delivery
                      of certain capabilities. 21 For example, Coast Guard officials told us they
                      deferred some of the HH-60 helicopter’s capabilities because of funding
                      constraints across their portfolio of programs. The Coast Guard delayed
                      delivery of dedicated radar to search the surface of the water in order to
                      replace critical components, such as main rotor blades, as planned.
                      Figure 7 identifies how program managers reported funding instability has
                      affected their programs.




                      18
                       GAO, Defense Acquisitions: Assessments of Selected Weapon Programs,
                      GAO-11-233SP (Washington, D.C.: March 29, 2011).
                      19
                       GAO-01-288.
                      20
                        Seventy-one survey respondents replied to whether their program experienced funding
                      instability.
                      21
                       Forty survey respondents reported at least one effect of funding instability.




                      Page 16                                                     GAO-12-833 Homeland Security
Figure 7: Effects of Funding Instability on Programs, as Identified by Program
Managers




Note: Seventy-one survey respondents replied to whether their program experienced funding
instability; 61 survey respondents reported whether they experienced an effect of funding instability;
40 reported at least one effect; 30 of the 40 reported multiple effects.


Forty-five of the 61 survey respondents that reported their programs
experienced funding instability also reported reasons for the funding
instability. Twenty-two survey respondents reported more than one
reason. See figure 8.




Page 17                                                             GAO-12-833 Homeland Security
Figure 8: Reasons Programs Experienced Funding Instability




                                       Note: Seventy-one survey respondents replied to whether their program experienced funding
                                       instability; 45 reported reasons for their funding instability; 22 of the 45 reported multiple reasons for
                                       funding instability.


                                       Eighteen survey respondents reported that their program experienced a
                                       funding decrease because of another program’s funding needs. We have
                                       previously reported that agencies often change funding levels in this
                                       manner when they commit to more programs than they can afford. 22 A
                                       PA&E official told us that DHS’s resource requirements exceed the
                                       department’s funding levels, and that the department has allowed major
                                       acquisition programs to advance through the acquisition life cycle without
                                       identifying how they will be funded. Furthermore, a PA&E official stated
                                       that DHS has not been able to determine the magnitude of its forthcoming
                                       funding gap because cost estimates are unreliable. The director of the
                                       department’s cost analysis division determined that only 12 major
                                       acquisition programs met most of DHS’s criteria for reliable cost



                                       22
                                         GAO, Defense Acquisitions: A Knowledge-Based Funding Approach Could Improve
                                       Major Weapon System Program Outcomes, GAO-08-619 (Washington, D.C.: July 2,
                                       2008).




                                       Page 18                                                                GAO-12-833 Homeland Security
                        estimates when it reviewed the components’ fiscal year 2013 budget
                        submissions. 23 In 2010, we reported that DHS officials had difficulty
                        managing major programs because they lacked accurate cost
                        estimates. 24

                        Given the fiscal challenges facing the federal government, funding
                        shortfalls may become an increasingly common challenge at DHS,
                        leading to further cost growth that widens the gap between resource
                        requirements and available funding.


Most Major Programs     DHS acquisition policy establishes that each program office should be
Reported They           staffed with personnel who have appropriate qualifications and
Experienced Workforce   experience in key disciplines, such as systems engineering, logistics, and
                        financial management. 25 Fifty-one survey respondents reported that their
Shortfalls
                        programs had experienced workforce shortfalls—specifically a lack of
                        government personnel—increasing the likelihood their programs will
                        perform poorly in the future. We have previously reported that a lack of
                        adequate staff in DHS program offices—both in terms of skill and staffing
                        levels—increased the risk of insufficient program planning and contractor
                        oversight, which is often associated with cost growth and schedule slips. 26
                        Figure 9 below identifies the functional areas where DHS acquisition
                        programs reported workforce shortfalls.




                        23
                          The department’s cost analysis division was combined with the acquisition program
                        management division to create PARM in 2011. The Cost Estimating and Analysis center
                        of excellence is now responsible for supporting cost analyses within DHS.
                        24
                         GAO-10-588SP
                        25
                          DHS Instruction Manual 102-01-001, Appendix E, Acquisition Program Office and
                        Component Acquisition Executive Core Staffing Requirements (Oct. 1, 2011).
                        26
                         GAO-10-588SP




                        Page 19                                                 GAO-12-833 Homeland Security
Figure 9: Functional Areas Contributing to Workforce Shortfalls




Note: Sixty-two survey respondents reported whether their program experienced workforce shortfalls
in government full-time-equivalents in three functional areas; 46 of the 51 reported shortfalls in
multiple functional areas.
a
 Includes auditing, business, cost estimating, financial management, property management, and
purchasing
b
 Includes systems planning, research, development, and engineering; life-cycle logistics; test and
evaluation; production; quality and manufacturing; and facilities engineering.


We found that 29 of the 51 DHS programs that identified workforce
shortfalls had also experienced cost growth or schedule slips. 27 The
workforce shortfalls have led to insufficient program planning, hindering
the development of key acquisition documents intended to inform senior-
level decision making. For example, CAEs and program managers said
that workforce shortfalls limited program management offices’ interaction
with stakeholders and operators, and delayed or degraded test plans and
cost estimates. In addition, a PARM official explained that DHS has had
to rely on contractors to produce cost estimates because of workforce
shortfalls, and the quality of these cost estimates has varied.



27
  Sixty-two survey respondents reported whether their program experienced a workforce
shortfall in government full-time-equivalents in three functional areas; 40 reported that
their program had a DHS-approved Acquisition Program Baseline and whether the
program had experienced cost growth and schedule slips.




Page 20                                                            GAO-12-833 Homeland Security
                     The USM has stated that properly staffing programs is one of DHS’s
                     biggest challenges, and we have previously reported that the capacity of
                     the federal government’s acquisition workforce has not kept pace with
                     increased spending for increasingly complex purchases. 28 PARM officials
                     told us that the IRB’s program reviews include assessments of the
                     program office workforce, but that the IRB considers staffing issues a
                     relatively low priority, and we found the IRB has formally documented
                     workforce-related challenges for only 11 programs.


                     DHS acquisition policy reflects many key program management
Programs Proceed     practices. 29 It requires programs to develop documents demonstrating
without Meeting      critical knowledge that would help leaders make better informed
Sound DHS            investment decisions when managing individual programs. This
                     knowledge would help DHS mitigate the risks of cost growth and
Acquisition Policy   schedule slips resulting from funding instability, workforce shortfalls, and
                     planned-capability changes. However, as of April 2012, the department
                     had only verified that four programs documented all of the critical
                     knowledge required to progress through the acquisition life cycle. In most
                     instances, DHS leadership has allowed programs it has reviewed to
                     proceed with acquisition activities without meeting these requirements.
                     Officials explained that DHS’s culture has emphasized the need to rapidly
                     execute missions more than sound acquisition management practices,
                     and we have found that most of the department’s major programs are at
                     risk of cost growth and schedule slips as a result. In addition, they lack
                     the reliable cost estimates, realistic schedules, and agreed-upon baseline
                     objectives that DHS acknowledges are needed to accurately track
                     program performance, limiting DHS leadership’s ability to effectively
                     manage those programs and provide information to Congress. DHS
                     recognizes the need to implement its acquisition policy more consistently,
                     but significant work remains.




                     28
                       GAO, The Office of Management and Budget’s Acquisition Workforce Development
                     Strategic Plan for Civilian Agencies, GAO-10-459R (Washington, D.C.: April 23, 2010).
                     29
                       DHS acquisition policy consists of AD 102-01, an associated guidebook, and 12
                     appendices.




                     Page 21                                                   GAO-12-833 Homeland Security
DHS Acquisition Policy   In 2005, we reported that DHS established an investment review process
Generally Reflects Key   that adopted many practices to reduce risk and increase the chances for
Program Management       successful outcomes. 30 In 2010, we reported that AD 102 provided more
                         detailed guidance for preparing key acquisition documents than the
Practices
                         department’s predecessor policy. In October 2011, DHS updated the
                         Guidebook and its appendixes, and we have found that it establishes a
                         knowledge-based acquisition policy for program management that is
                         largely consistent with key practices.

                         A knowledge-based approach to capability development allows
                         developers to be reasonably certain, at critical points in the acquisition life
                         cycle, that their products are likely to meet established cost, schedule,
                         and performance objectives. 31 This knowledge provides them with
                         information needed to make sound investment decisions, and it would
                         help DHS address the significant challenges we identified across its
                         acquisition programs: funding instability, workforce shortfalls, and
                         planned-capability changes. Over the past several years, our work has
                         emphasized the importance of obtaining key knowledge at critical points
                         in major system acquisitions and, based on this work, we have identified
                         eight key practice areas for program management. These key practice
                         areas are summarized in table 3, along with our assessment of DHS’s
                         acquisition policy.




                         30
                           GAO, Homeland Security: Successes and Challenges in DHS’s Efforts to Create an
                         Effective Acquisition Organization, GAO-05-179 (Washington, D.C.: March 29, 2005).
                         31
                           In our past work examining weapon acquisition issues and best practices for product
                         development, we have found that leading commercial firms pursue an acquisition
                         approach that is anchored in knowledge, whereby high levels of product knowledge are
                         demonstrated by critical points in the acquisition process. See GAO-11-233SP.




                         Page 22                                                   GAO-12-833 Homeland Security
Table 3: GAO Assessment of DHS’s Acquisition Policy Compared to Key Program-management Practices

                                                                                                                    GAO assessment of
GAO key practice area            Summary of key practices                                                          DHS acquisition policy
Identify and validate needs      Current capabilities should be identified to determine if there is a
                                 gap between the current and needed capabilities. A need statement
                                 should be informed by a comprehensive assessment that considers
                                                                                                                                
                                 the organization’s overall mission.
Assess alternatives to select    Analyses of Alternatives should be conducted early in the
most appropriate solution        acquisition process to compare key elements of competing
                                 solutions, including performance, costs, and risks. Moreover, these                            
                                 analyses should assess many alternatives across multiple
                                 concepts.
Clearly establish well-defined   Requirements should be well defined and include input from
requirements                     operators and stakeholders. Programs should be grounded in well-
                                 understood concepts of how systems would be used and likely
                                                                                                                                
                                 requirements costs.
Develop realistic cost           A cost estimate should be well documented, comprehensive,
estimates and schedules          accurate, and credible. A schedule should identify resources
                                 needed to do the work and account for how long all activities will                             
                                 take. Additionally, a schedule should identify relationships between
                                 sequenced activities.
Secure stable funding that       Programs should make trade-offs as necessary when working in a
matches resources to             constrained budget environment.                                                                
requirements
Demonstrate technology,          Capabilities should be demonstrated and tested prior to system
design, and manufacturing        development, making a production decision, and formal operator                                 ◑
maturity                         acceptance.
Utilize milestones and exit      Milestones and exit criteria—specific accomplishments that
criteria                         demonstrate progress—should be used to determine that a
                                 program has developed required and appropriate knowledge prior
                                                                                                                                ◕
                                 to a program moving forward to the next acquisition phase.
Establish an adequate            Acquisition personnel should have appropriate qualifications and
workforce                        experience. Program managers should stay on until the end of an
                                 acquisition life-cycle phase to assure accountability. Government
                                                                                                                                ◑
                                 and contractor staff should also remain consistent.
                                             Source: GAO analysis of DHS acquisition policy.

                                             Note: Appendixes I and II present a more detailed description of key program-management practices
                                             and how we assessed them.


                                             Legend:     DHS policy reflects key practices; ◕   DHS policy substantially reflects key practices;
                                             ◑ DHS policy partially reflects key practices.
                                             We found that DHS’s acquisition policy generally reflects key program-
                                             management practices, including some intended to help develop
                                             knowledge at critical points in the acquisition life cycle. Furthermore, the
                                             revised policy the department issued in October 2011 better reflects two
                                             key practice areas by bolstering exit criteria and taking steps to establish


                                             Page 23                                                              GAO-12-833 Homeland Security
an adequate acquisition workforce. Specifically, the revised Guidebook
and its appendixes require that refined cost estimates be reviewed at
major milestones after the program baseline has been established, and
used to determine whether a program has developed appropriate
knowledge to move forward in the acquisition life cycle. These reviews
can help reduce risk and the potential for unexpected cost and schedule
growth. Additionally, the revised policy establishes that major program
offices should be staffed with personnel with appropriate qualifications
and experience in key acquisition disciplines. We have previously
identified that the magnitude and complexity of the DHS acquisition
portfolio demands a capable and properly trained workforce and that
workforce shortfalls increase the risk of poor acquisition outcomes. The
policy revisions could help mitigate this risk.

However, there are three areas where DHS could further enhance
acquisition oversight:

•   The policy requires that DHS test technologies and manufacturing
    processes, but it does not require that 1) programs demonstrate
    technologies in a realistic environment prior to initiating development
    activities at the outset of the Obtain phase, or 2) manufacturing
    processes be tested prior to production. These practices decrease the
    risk that rework will be required, which can lead to additional cost
    growth and schedule slips.
•   The policy requires that DHS establish exit criteria for programs
    moving to the next acquisition phase, and standardizes document
    requirements across all major programs, but it does not require that 1)
    exit criteria be quantifiable to the extent possible, or 2) consistent
    information be used across programs when approving progress within
    the Obtain phase, specifically at ADE 2B and 2C. These practices
    decrease the risk that a program will make an avoidable error
    because management lacks information needed to leverage lessons
    learned across multiple program reviews.
•   The policy requires that program managers be certified at an
    appropriate level, but it does not state that they should remain with
    their programs until the next major milestone when possible. This
    practice decreases the risk that program managers will not be held
    accountable for their decisions, such as proceeding without reliable
    cost estimates or realistic schedules.

PARM officials generally acknowledged DHS has opportunities to
strengthen its program-management guidance. Officials reported that
they are currently in the process of updating AD 102, which they plan to



Page 24                                          GAO-12-833 Homeland Security
                            complete by the end of fiscal year 2012. They also plan to issue revisions
                            to the associated guidebook and appendixes in phases. PARM officials
                            told us that they plan to structure the revised acquisition policy by
                            function, consolidating guidance for financial management, systems
                            engineering, reporting requirements, and so forth. PARM officials
                            anticipate that this organization will make it easier for users to identify
                            relevant information as well as streamline the internal review process for
                            future updates.


DHS Has Approved Few        DHS acquisition policy establishes several key program-management
Programs’ Key Acquisition   practices through document requirements. AD 102 requires that major
Documents                   acquisition programs provide the IRB documents demonstrating the
                            critical knowledge needed to support effective decision making before
                            progressing through the acquisition life cycle. For example, programs
                            must document that they have assessed alternatives to select the most
                            appropriate solution through a formal Analysis of Alternatives report,
                            which must be approved by component-level leadership. Figure10
                            identifies acquisition documents that must be approved at the department
                            level and their corresponding key practice areas.

                            Figure 10: DHS Acquisition Documents Requiring Department-level Approval




                            DHS acquisition policy requires these documents, but the department
                            generally has not implemented its acquisition policy as intended, and in
                            practice the department has not adhered to key program management


                            Page 25                                            GAO-12-833 Homeland Security
practices. DHS’s efforts to implement the department’s acquisition policy
have been complicated by the large number of legacy programs initiated
before the department was created, including 11 programs that PARM
officials told us were in sustainment when AD 102 was signed. 32 We
found that the department has only approved four programs’ required
documents in accordance with DHS policy: the National Cybersecurity
and Protection System, the Next Generation Network, the Offshore Patrol
Cutter, and the Passenger Screening Program. Additionally, we found
that 32 programs had none of the required documents approved by the
department. See figure 11.




32
  Sustainment begins when a capability has been fielded for operational use, and it
involves the supportability of fielded systems through disposal, including maintenance and
the identification of cost reduction opportunities. System operations, support, and
sustainment costs tend to approach up to 70 percent of life-cycle costs.




Page 26                                                    GAO-12-833 Homeland Security
Figure 11: Programs That Have Key Acquisition Documents Approved in
Accordance with AD 102




Note: Appendix IV identifies which documents have been approved for each of the 71 programs that
responded to our survey. Five programs are not accounted for in this figure because their documents
did not require department-level approval. See appendix IV for additional information.
According to PARM officials, 10 of the 32 programs with no documents approved were in sustainment
at the time AD 102 was signed, as was 1 of the 30 programs with some documents approved – the
Application Support Center. DHS approved the Application Support Center’s Mission Need Statement
and Acquisition Program Baseline in March 2011.


Since 2008, DHS leadership—through the IRB or its predecessor body
the Acquisition Review Board—has formally reviewed 49 of the 71 major
programs that responded to our survey. It permitted 43 of those programs
to proceed with acquisition activities without verifying the programs had
developed the knowledge required for AD 102’s key acquisition
documents. See figure 12.




Page 27                                                          GAO-12-833 Homeland Security
Figure 12: Programs Reviewed by DHS’s Executive Review Board




Officials from half of the CAE offices we spoke to reported that DHS’s
culture has emphasized the need to rapidly execute missions more than
sound acquisition management practices. PARM officials agreed,
explaining that DHS has permitted programs to advance without
department-approved acquisition documents because DHS had an
operational need for the promised capabilities, but the department could
not approve the documents in a timely manner. PARM officials explained
that, in certain instances, programs were not capable of documenting
knowledge, while in others, PARM lacked the capacity to validate that the
documented knowledge was adequate. In 2008 and 2010, we reported
that several programs were permitted to proceed with acquisition




Page 28                                           GAO-12-833 Homeland Security
activities on the condition they complete key action items in the future. 33
However, PARM officials told us that many of these action items were not
addressed in a timely manner. Additionally, program managers reported
that there has been miscommunication between DHS headquarters and
program offices regarding implementation of the acquisition policy, and
we found that DHS headquarters and program managers often had a
different understanding of whether their programs were in compliance
with AD 102. For example, DHS headquarters officials told us that 19 of
the 40 programs that reported through our survey they had department-
approved acquisition program baselines (APB) in fact did not.

Because DHS has not generally implemented its acquisition policy, senior
leaders lack the critical knowledge needed to accurately track program
performance: (1) department-approved APBs, (2) reliable cost estimates,
and (3) realistic schedules. Specifically, at the beginning of 2012, DHS
leadership had approved APBs for less than one-third of the 63 programs
we reviewed that are required to have one based on their progression
through the acquisition life cycle. Additionally, we found that none of the
programs with a department-approved APB also met DHS’s criteria for
both reliable cost estimates and realistic schedules, which are key
components of the APB. This raises questions about the quality of those
APBs that have been approved, as well as the value of the DHS review
process in practice. Figure 13 identifies how many programs currently
have department-approved APBs, reliable cost estimates, and realistic
schedules.




33
 GAO-09-29, GAO-10-588SP.




Page 29                                           GAO-12-833 Homeland Security
                                Figure 13: Programs with Department-approved APBs, Reliable Cost Estimates, and
                                Realistic Schedules




                                Note: We analyzed DHS acquisition decision memorandums to determine whether a program had an
                                approved APB, and we reviewed an internal DHS assessment to determine the reliability of programs’
                                cost estimates. Sixty-eight survey respondents replied to our scheduling questions.


Acquisition Program Baselines   The APB is a critical tool for managing an acquisition program. According
                                to DHS’s acquisition Guidebook, the program baseline is the agreement
                                between program, component, and department level officials, establishing
                                how systems will perform, when they will be delivered, and what they will
                                cost. 34 In practice, when the Acquisition Decision Authority approves a
                                program’s APB, among other things, it is concurring that the proposed
                                capability is worth the estimated cost. However, we found that DHS plans
                                to spend more than $105 billion on programs lacking current, department-
                                approved APBs. Specifically, when DHS submitted the FYHSP to



                                34
                                 DHS Acquisition Instruction/Guidebook 102-01-001: Appendix K, Acquisition Program
                                Baseline; October 1, 2011.




                                Page 30                                                         GAO-12-833 Homeland Security
                               Congress in 2011, it reported that 34 of the 43 programs lacking
                               department-approved APBs were expected to cost $108.8 billion over
                               their acquisition life cycles. DHS did not provide cost estimates for the
                               other 9 programs because the data were unreliable.

                               In addition to overall cost, schedule, and performance goals, the APB
                               also contains intermediate metrics to measure a program’s progress in
                               achieving those goals. These intermediate metrics allow managers to
                               take corrective actions earlier in the acquisition life cycle. DHS’s lack of
                               APBs, PARM officials explained, makes it more difficult to manage
                               program performance. In March 2012, PARM reported that 32 programs
                               had experienced significant cost growth or schedule slips in its internal
                               Quarterly Program Accountability Report. However, DHS has only
                               formally established that 8 of its programs have fallen short of their cost,
                               schedule, or performance goals, because approximately three-quarters of
                               the programs PARM identified lack the current, department-approved
                               APBs needed to authoritatively measure performance.

Cost Estimates and Schedules   To accurately assess a program’s performance, managers need accurate
                               cost and schedule information. However, DHS acquisition programs
                               generally do not have reliable cost estimates and realistic schedules, as
                               required by DHS policy. 35 In June 2012, the department reported to GAO
                               that its senior leaders lacked confidence in the performance data they
                               receive, hindering their efforts to manage risk and allocate resources.

                               PARM and PA&E officials said that cost estimates provided by program
                               management offices often understate likely costs. PA&E officials
                               explained that many programs have not included operations and
                               maintenance activities in their cost estimates, which we have previously
                               reported can account for 60 percent or more of a program’s total costs. 36
                               The director of the department’s cost analysis division determined that
                               only 12 major acquisition programs met most of DHS’s criteria for reliable
                               cost estimates, and 6 of these programs lacked current, department-




                               35
                                DHS’s criteria for assessing the cost estimates are based on GAO Cost Estimating and
                               Assessment Guide: Best Practices for Developing and Managing Capital Program Costs,
                               GAO-09-3SP (Washington, D.C.: March 2009).
                               36
                                GAO-10-588SP.




                               Page 31                                                  GAO-12-833 Homeland Security
                           approved APBs. 37 Additionally, only 12 program offices reported that they
                           fully adhered to DHS’s scheduling guidance, which requires that
                           programs sequence all activities, examine the effects of any delays,
                           update schedules to ensure validity, and so forth. Eight of these programs
                           lacked department-approved APBs. 38

                           DHS’s lack of reliable performance data not only hinders its internal
                           acquisition management efforts, but also limits Congressional oversight.
                           Congress mandated the department submit the Comprehensive
                           Acquisition Status Report (CASR) to the Senate and House Committees
                           on Appropriations as part of the President’s fiscal year 2013 budget,
                           which was submitted in February 2012. However, DHS told us that it did
                           not do so until August 2012. Congress mandated DHS produce the CASR
                           in order to obtain information necessary for in-depth congressional
                           oversight, including life-cycle cost estimates, schedules, risk ratings, and
                           out-year funding levels for all major programs. The CASR has the
                           potential to greatly enhance oversight efforts by establishing a common
                           understanding of the status of all major programs.


DHS Recognizes the Need    In April 2012, PARM officials told us that DHS had begun to implement its
to Implement Its           acquisition policy in a more disciplined manner. They told us that they had
Acquisition Policy More    adequate capacity to review programs, and would no longer advance
                           programs through the acquisition life cycle until DHS leadership verified
Consistently, but          the programs had developed critical knowledge. For example, in February
Significant Work Remains   2012, the IRB denied a request from the BioWatch Gen 3 program—
                           which is developing a capability to detect airborne biological agents—to
                           solicit proposals from contractors because its draft APB was not valid.
                           PARM officials said they are using a risk-based approach to prioritize the
                           approval of the department’s APBs. Specifically, they explained that one
                           of their fiscal year 2011 initiatives was to attain department-level approval
                           of APBs for all Level 1 programs in the Obtain phase of the acquisition life
                           cycle. However, we found only 8 of the 19 programs PARM said fell into




                           37
                             The department’s cost analysis division was combined with the acquisition program
                           management division to create PARM in 2011. The Cost Estimating and Analysis center
                           of excellence is now responsible for supporting cost analyses within DHS.
                           38
                             One of these eight programs—the Medium Endurance Cutter Sustainment program—is
                           not required to have a department-approved APB.




                           Page 32                                                 GAO-12-833 Homeland Security
                         this category had current, department-approved APBs as of September
                         2012.

                         In an effort to improve the consistency of performance data reported by
                         program managers, PARM officials stated that they are establishing
                         scorecards to assess cost estimates and standard work breakdown
                         structures for IT programs. The PARM officials also explained that CAE’s
                         performance evaluations now include an assessment of the
                         completeness and accuracy of performance data reported for their
                         respective programs. However, DHS must overcome significant
                         challenges in order to improve the reliability of performance data and
                         meet key requirements in the department’s acquisition policy. For
                         example, department and component-level officials told us that program
                         managers do not report on their programs in a consistent manner.
                         Additionally, DHS officials told us that they lack cost estimating capacity
                         throughout the department and that they must rely heavily on contractors,
                         which do not consistently provide high-quality deliverables. In August
                         2012, a PARM official stated that DHS was currently in the process of
                         hiring eight additional government cost estimators to support programs.


                         DHS acquisition policy does not fully reflect several key portfolio-
DHS Needs Policy and     management practices, such as allocating resources strategically, and
Process                  DHS has not yet reestablished an oversight board to manage its
                         investment portfolio across the department. As a result, DHS has largely
Enhancements to          made investment decisions on a program-by-program and component-by-
Effectively Manage Its   component basis. The widespread risk of poorly understood cost growth,
Portfolio of             coupled with the fiscal challenges facing the federal government, makes it
                         essential that DHS allocate resources to its major programs in a
Investments              deliberate manner. DHS plans to develop stronger portfolio-management
                         policies and processes, but until it does so, DHS programs are more likely
                         to experience additional funding instability in the future, which will
                         increase the risk of further cost growth and schedule slips. These
                         outcomes, combined with a tighter budget, could prevent DHS from
                         developing needed capabilities.




                         Page 33                                          GAO-12-833 Homeland Security
DHS Acquisition Policy                    In our past work, we have found that successful commercial companies
Shortfalls Hinder Portfolio               use a disciplined and integrated approach to prioritize needs and allocate
Management Efforts                        resources. 39 As a result, they can avoid pursuing more projects than their
                                          resources can support, and better optimize the return on their investment.
                                          This approach, known as portfolio management, requires companies to
                                          view each of their investments as contributing to a collective whole, rather
                                          than as independent and unrelated. With this enterprise perspective,
                                          companies can effectively (1) identify and prioritize opportunities, and (2)
                                          allocate available resources to support the highest priority—or most
                                          promising—opportunities. Over the past several years, we have
                                          examined the practices that private and public sector entities use to
                                          achieve a balanced mix of new projects, and based on this work, we have
                                          identified four key practice areas for portfolio management, summarized
                                          in table 4, along with our assessment of DHS acquisition policy.

Table 4: GAO Assessment of DHS’s Acquisition Policy Compared to Key Portfolio-management Practices

                                                                                                               GAO assessment of DHS
GAO key practice area           Summary of key practices                                                         acquisition policy
Clearly define and empower      Portfolio managers, with the support of cross-functional teams,
leadership                      should be empowered to make investment decisions, and held                                   ◑
                                accountable for outcomes.
Establish standard assessment   Investments should be ranked and selected using a disciplined
criteria, and demonstrate
comprehensive knowledge of
                                process to assess the costs, benefits, and risks of alternative
                                products to ensure transparency and comparability across
                                                                                                                             ◑
the portfolio                   alternatives.
Prioritize investments by       Organizations should use long-range planning and an integrated
integrating the requirements,
acquisition, and budget
                                approach to prioritize needs and allocate resources in accordance
                                with strategic goals, so they can avoid pursuing more products
                                                                                                                             ◑
processes                       than they can afford and optimize return on investment.
Continually make go/no-go       Reviews should be scheduled (1) annually to consider proposed
decisions to rebalance the      changes, (2) as new opportunities are identified, (3) whenever a
portfolio                       program breaches its objectives, and (4) after investments are
                                completed. Information gathered during these reviews should be
                                                                                                                             ◔
                                used to adjust and balance the portfolio to achieve strategic
                                outcomes.
                                          Source: GAO analysis of DHS acquisition policy.

                                          Note: Appendixes I and II present a more detailed description of key portfolio management practices
                                          and how we assessed them.
                                          Legend:    ◑ DHS policy partially reflects key practices; ◔ DHS policy minimally reflects key practices

                                          39
                                            GAO, Best Practices: An Integrated Portfolio Management Approach to Weapon System
                                          Investments Could Improve DOD’s Acquisition Outcomes, GAO-07-388 (Washington,
                                          D.C.: March 30, 2007).




                                          Page 34                                                             GAO-12-833 Homeland Security
We found that DHS’s acquisition policy reflects some key portfolio-
management practices. DHS has not designated individual portfolio
managers, but it requires that the department’s Chief Acquisition Officer—
currently the USM—be supported by the IRB, which includes officials
representing key functional areas, such as budget, procurement, IT, and
human capital. DHS’s acquisition policy also establishes that
requirements, acquisition, and budget processes should be connected to
promote stability.

However, as acknowledged by DHS officials, the policy does not reflect
several other key portfolio-management practices:

•   The policy does not empower portfolio managers to decide how best
    to invest resources. This practice increases the likelihood resources
    will be invested effectively, and that portfolio managers will be held
    accountable for outcomes.
•   The policy does not establish that investments should be ranked and
    selected using a disciplined process. This practice increases the
    likelihood the portfolio will be balanced with risk spread across
    products.
•   The policy does not establish that (1) resource allocations should
    align with strategic goals, or (2) the investment review policy should
    use long-range planning. These practices increase the likelihood that
    the right amount of funds will be delivered to the right projects,
    maximizing return on investments.
•   The policy does not require portfolio reviews (1) annually to consider
    proposed changes, (2) as new opportunities are identified, or (3)
    whenever a program breaches its objectives. These practices provide
    opportunities for leaders to increase the value of investments,
    determine whether or not the investments are still relevant and
    affordable, and help keep programs within cost and schedule targets.

PARM officials acknowledge that the department does not currently have
a policy that addresses these key portfolio-management practices.
Further, they told us that there has been less focus on portfolio
management than program management to date because the acquisition
process is still relatively immature. As a result, DHS largely makes
investment decisions on a program-by-program and component-by-
component basis. In our work at the Department of Defense, we have
found this approach hinders efforts to achieve a balanced mix of




Page 35                                          GAO-12-833 Homeland Security
                                        programs that are affordable and feasible and that provide the greatest
                                        return on investment. 40

                                        PARM officials anticipate that DHS will improve its portfolio-management
                                        guidance in the future by formalizing its proposed Integrated Investment
                                        Life Cycle Model (IILCM). In January 2011, DHS presented a vision of the
                                        IILCM as a means to better integrate investment management functions,
                                        including requirements development, resource allocation, and program
                                        governance. DHS explained that the IILCM would ensure mission needs
                                        drive investment decisions and establish a common framework for
                                        monitoring and assessing the department’s investments. The IILCM
                                        would be implemented through the creation of several new department-
                                        level councils, as illustrated in figure 14, which would identify priorities
                                        and capability gaps.

Figure 14: Councils and Offices in DHS’s Proposed IILCM




                                        40
                                         GAO-07-388.




                                        Page 36                                           GAO-12-833 Homeland Security
DHS Investment Process        In 2003, DHS established the Joint Requirements Council (JRC) to
Shortfalls Hinder Portfolio   identify crosscutting opportunities and common requirements among DHS
Management Efforts            components, and help determine how DHS should use its resources.
                              However, as we have previously reported, the JRC stopped meeting in
                              2006. 41 In 2008, we recommended that the JRC be reinstated, or that
                              DHS establish another joint requirements oversight board. At that time,
                              DHS officials recognized that strengthening the JRC was a top priority.
                              The department has proposed the creation of a Capabilities and
                              Requirements Council (CRC) to serve in a similar role as the JRC, but the
                              CRC is not yet established.

                              In the absence of a JRC, or the proposed CRC, DHS budget officials
                              explained it is difficult to develop a unified strategy to guide trade-offs
                              between programs because of the diversity of the department’s missions.
                              Poor program outcomes, coupled with a tighter budget, could prevent
                              DHS from developing needed capabilities. In our work at the Department
                              of Defense, we have found that agencies must prioritize investments, or
                              programs will continually compete for funding by promising more
                              capabilities than they can deliver while underestimating costs. 42 We also
                              found that success was measured in terms of keeping a program alive
                              rather than efficiently delivering the capabilities needed. 43 It appears the
                              lack of prioritization is affecting DHS in the same way. As discussed
                              earlier in our assessment of program challenges, 18 of the department’s
                              programs reported DHS decreased their out-year funding levels because
                              of another program’s funding needs, and 61 programs reported they
                              experienced some form of funding instability.

                              Until recently, the responsibility for balancing portfolios has fallen on
                              components. However, DHS policy officials noted that component-level
                              officials have a relatively limited perspective focused on those programs
                              under their authority, making it more difficult to ensure the alignment of
                              mission needs to department-level goals. Additionally, component-level
                              officials can only make trade-offs across the portion of the DHS portfolio
                              that falls under their purview, limiting opportunities to increase the
                              department’s return on its investments.



                              41
                               GAO-09-29.
                              42
                               GAO-08-619.
                              43
                               GAO-06-110.




                              Page 37                                            GAO-12-833 Homeland Security
The USM and PARM officials have stated they recognize the value of
portfolio management, and they have taken some steps to fill the gap left
without a functioning JRC or CRC. A PARM official stated that, starting in
2012, PARM is collaborating with the Offices of the Chief Information,
Financial, and Procurement Officers, as well as the Office of Policy, to
conduct portfolio reviews from a functional, cross-component perspective.
In the past, PARM’s portfolio reviews focused on each component
individually. This new functional approach is establishing portfolios based
on departmentwide missions, such as domain awareness or screening,
and PARM officials intend to produce trade-off recommendations for
prioritizing funding across different components. They also intend to use
functional portfolio reviews to provide greater insight into the effects of
funding instability, and the USM has stated that the portfolio reviews will
inform the department’s fiscal year 2014 budget. DHS intends for the
proposed CRC to make trade-offs across the functional portfolios.

PARM’s Quarterly Program Accountability Report (QPAR), issued in
March 2012, also has the potential to inform DHS’s portfolio management
efforts. In developing the QPAR, PARM used a standardized set of five
criteria to measure the value of each program: mission alignment,
architectural maturity, capability gap, mission criticality, and DHS benefit.
This allowed PARM to identify 48 high-value and 13 low-value programs.
However, the QPAR does not recommend using the information to
prioritize resource allocations, which would address a key portfolio
management practice. Further, DHS’s widespread lack of department-
approved Mission Need Statements (MNS) undermines efforts to improve
portfolio management and prioritize investments. The MNS links
capability gaps to the acquisitions that will fill those gaps, making it a
critical tool for prioritizing programs. The MNS also provides formal
executive-level acknowledgment that there is a mission need justifying
the allocation of DHS’s limited resources. However, only about 40 percent
of DHS’s major acquisition programs have a department-approved MNS.




Page 38                                           GAO-12-833 Homeland Security
                           DHS has introduced seven initiatives that could improve acquisition
DHS Acquisition            management by addressing longstanding challenges we have identified—
Management                 such as funding instability and acquisition workforce shortfalls—which
                           DHS survey respondents also identified in 2012. Implementation plans
Initiatives Target         are still being developed for all of these initiatives, and DHS is still
Longstanding               working to address critical issues, particularly capacity questions.
Challenges, but Key        Because of this, it is too early to determine whether the DHS initiatives
                           will be effective, as we have previously established that agencies must
Implementation             sustain progress over time to address management challenges. 44 DHS is
Issues Remain              also pursuing a tiered-governance structure that it has begun to
                           implement for IT acquisitions. Before the department can regularly
                           delegate ADE decision authority through this tiered-governance structure,
                           DHS must successfully implement its seven acquisition management
                           initiatives and apply its knowledge-based acquisition policy on a more
                           consistent basis to reduce risks and improve program outcomes.


DHS Initiatives Are        In 2005, we identified acquisition management as a high-risk area at
Intended to Address        DHS. 45 Since then, we have issued multiple reports identifying acquisition
Longstanding Acquisition   management challenges. 46 In 2008, we made several recommendations
Management Challenges      intended to help DHS address those challenges, and in September 2010,
                           we provided DHS a list of specific acquisition management outcomes the
                           department must achieve to help address the high-risk designation. This
                           list largely drew from our past recommendations, and stressed that the
                           department must implement its knowledge-based acquisition policy
                           consistently. DHS has generally concurred with our recommendations,
                           but still faces many of the same challenges we have previously identified.

                           In 2011, DHS began to develop initiatives to address these challenges,
                           and DHS has continued to evolve these plans in 2012. In January 2011,
                           DHS produced the initial iteration of its Integrated Strategy for High Risk
                           Management in order to measure progress in addressing acquisition
                           management challenges we had identified, as well as financial
                           management, human capital, IT, and management integration issues.
                           The department subsequently produced updates in June 2011, December


                           44
                            GAO-11-278.
                           45
                            GAO-05-207.
                           46
                            For examples, see GAO-09-29, GAO-10-588SP, GAO-11-68, GAO-11-278,
                           GAO-11-318SP, GAO-11-743.




                           Page 39                                           GAO-12-833 Homeland Security
                                       2011, and June 2012. These updates present the department’s progress
                                       in developing and implementing its initiatives. Additionally, in December
                                       2011, DHS issued the Program Management and Execution Playbook
                                       (Playbook), which expounded on some of those initiatives, and introduced
                                       a vision for a “more mature, agile, and effective process for program
                                       governance and execution.” Figure 15 identifies seven key DHS initiatives
                                       and how they correspond to acquisition management challenges we have
                                       identified.

Figure 15: Acquisition Management Challenges and Corresponding Initiatives




                                       As envisioned, the DHS initiatives would better position the department to
                                       implement its knowledge-based acquisition policy on a more consistent
                                       basis to reduce risks and ultimately improve individual program
                                       outcomes. The initiatives would also help address challenges identified by
                                       survey respondents in 2012, particularly funding instability and acquisition




                                       Page 40                                           GAO-12-833 Homeland Security
                            workforce shortfalls. Additionally, the IILCM would enhance DHS’s ability
                            to effectively manage its acquisition portfolio as a whole.


Some Initiatives Are        DHS has made progress implementing some of the initiatives intended to
Taking Longer to            address the challenges we have identified. In June 2012, DHS reported
Implement Than Originally   that all of its components had an approved CAE in place and the
                            Procurement Staffing Model had been completed. In August 2012, DHS
Envisioned, and Capacity    told us that eight Centers of Excellence had been chartered. However,
Issues Could Create         from January 2011 to June 2012, the schedules for four of the seven
Further Challenges          initiatives slipped by at least 6 months, including the schedule for the
                            IILCM, which slipped by a year. In March 2012, an official responsible for
                            the IILCM initiative stated that many acquisition officials throughout the
                            department do not yet understand the intended benefits of the IILCM.
                            Thirty-two survey respondents reported that they were not at all familiar
                            with the initiative, as opposed to nine that reported they were very familiar
                            with the IILCM. 47 Additionally, officials from three CAE offices, including
                            two CAEs, told us that they were not familiar with the IILCM. Previously,
                            we have reported that it is important to involve employees and obtain their
                            ownership when transforming organizations. 48 Figure 16 identifies the
                            schedule slips and their causes.




                            47
                             Sixty-seven survey respondents reported on their familiarity with the IILCM.
                            48
                             GAO, Results-Oriented Cultures: Implementation Steps to Assist Mergers and
                            Organizational Transformations, GAO-03-669 (Washington, D.C.: July 2, 2003).




                            Page 41                                                    GAO-12-833 Homeland Security
Figure 16: Initiatives that Slipped from DHS’s Original January 2011 Schedule to the June 2012 Update




                                         Moving forward, all seven acquisition management initiatives face
                                         significant implementation challenges that could affect DHS’s efforts to
                                         address the challenges we have identified. We have previously
                                         established that agencies must have top leadership commitment,
                                         adequate capacity, corrective action plans, and performance measures to
                                         address high-risk issues. 49 We have also established that agencies must
                                         demonstrate progress in implementing corrective actions. Table 5
                                         summarizes our assessment of DHS’s initiatives, identifying challenges to
                                         successful implementation.




                                         49
                                           GAO-11-278.




                                         Page 42                                                GAO-12-833 Homeland Security
Table 5: GAO Assessment of DHS Acquisition Management Initiatives

DHS initiative                   Implementation challenges                                     GAO’s assessment
The Integrated Investment Life   DHS is still developing guidance for implementing and         Questionable capacity
Cycle Model                      operating the IILCM, and it has not yet developed a
                                 resource estimate or dedicated a funding source
Acquisition Workforce            DHS reports it currently has adequate resources to            Questionable capacity
Development                      develop and deliver training, but future training
                                 requirements are better understood for personnel that
                                 award and administer contracts, than other disciplines,
                                 such as systems engineers and cost estimators
Program Management Corps         DHS has reported critical resource shortfalls; the            Questionable capacity and lacks
                                 department previously identified the need for 150             objective measures
                                 additional positions; additionally, performance metrics are
                                 subjective, such as: “Ensure Program Managers are
                                 engaged in advancing the department’s acquisition and
                                 program management capabilities”
Procurement Staffing Model       DHS reports it has developed the model; however, the        Questionable capacity and inadequate
                                 initiative is only intended to identify workforce needs and corrective action plan
                                 DHS has not established an initiative to meet those needs
Centers of Excellence            DHS officials anticipate the Centers of Excellence will face Questionable capacity and lacks
                                 capacity challenges; additionally, performance metrics       objective measures
                                 have not yet been established
Component Acquisition Executive DHS has established meaningful performance measures,           Questionable capacity
Structure                       such as percentage of components with appropriate
                                staffing levels, but it has reported critical resource
                                shortfalls
Business Intelligence            DHS reports that the initiative has critical resource          Questionable capacity and inadequate
                                 shortfalls, and the corrective action plan does not explicitly corrective action plan
                                 address the enduring need to improve the quality of
                                 underlying performance data, particularly cost estimates
                                 and schedules
                                          Source: GAO analysis of DHS initiatives.



                                          Capacity issues resulting from resource shortfalls have the potential to
                                          undermine DHS’s efforts to address longstanding acquisition
                                          management challenges. DHS officials said the department established
                                          Centers of Excellence in an effort to increase the effectiveness of their
                                          limited number of skilled acquisition personnel. Additionally, the
                                          department plans to reassign high-performing staff to high-priority
                                          programs in order to cope with workforce shortfalls. However, DHS
                                          officials anticipate that capacity issues will endure, and in June 2012, the
                                          department reported that it may require 250 personnel to implement the
                                          IILCM, which would further strain DHS’s limited capacity.




                                          Page 43                                                      GAO-12-833 Homeland Security
Acquisition Management    In December 2011, DHS presented a vision of acquisition management
and Program Performance   that includes a new tiered-governance structure dividing acquisition
Must Improve Before       decision authority between the headquarters-level IRB and a group of
                          component-level Executive Steering Committees (ESC). See figure 17.
Delegating Major          According to the Playbook, DHS will adopt this tiered-governance
Milestone Decision        structure because the current process does not provide consistent or
Authority                 timely decisions, ensure appropriate stakeholders are involved, or allow
                          high risk/impact issues to get sufficient senior-level attention. In fiscal
                          year 2011, DHS established ESCs for 14 IT programs.

                          Figure 17: DHS’s Proposed IRB/ESC Governance Structure




                          According to the Playbook, the ESCs would be the primary decision-
                          making authority for each program, approving ADE decisions.
                          Additionally, many ESCs would not include headquarters-level officials
                          and the program management offices overseen by the ESCs would
                          provide the necessary administrative support. Programs would be
                          elevated for IRB-level reviews if they experienced difficulty with schedule,
                          budget, or scope.

                          Since the Playbook was issued, PARM officials and DHS’s Chief
                          Information Officer clarified that ADE decision authority should not be
                          delegated to component-level officials unless the USM has approved a
                          program’s APB and the program is being executed within agreed-upon
                          cost, schedule, and performance thresholds. However, DHS has not
                          clearly documented these conditions as prerequisites for delegating ADE
                          decision authority. Additionally, as we have identified, DHS lacks the
                          knowledge needed to effectively manage its programs, nearly all of which



                          Page 44                                            GAO-12-833 Homeland Security
              are at risk of poor outcomes. Because of this, DHS generally is not
              prepared to delegate ADE decision authority to component-level officials.


              DHS has a diverse, critical, and challenging mission that requires it to
Conclusions   respond to an ever-evolving range of threats. Given this mission, it is
              important that DHS maintain an agile and flexible management approach
              in its day-to-day operations. However, DHS must adopt a more
              disciplined and systematic approach for managing its major investments,
              which are intended to help meet critical mission needs. DHS has taken
              some steps to improve investment management, but most of its major
              acquisition programs continue to cost more than expected, take longer to
              deploy than planned, or deliver less capability than promised. These
              outcomes are largely the result of DHS’s lack of adherence to key
              knowledge-based program management practices, even though many
              are reflected in the department’s own acquisition policy. DHS leadership
              has authorized and continued to invest in major acquisition programs
              even though the vast majority of those programs lack foundational
              documents demonstrating the knowledge needed to help manage risks
              and measure performance. This limits DHS’s ability to proactively identify
              and address the challenges facing individual programs. Further, although
              the department’s acquisition policy contains many key practices that help
              reduce risks and increase the chances for successful outcomes, the
              policy does not include certain program management practices that could
              further enhance acquisition management. For example, the policy does
              not require that programs demonstrate technologies in a realistic
              environment prior to initiating development activities, or that exit criteria
              be quantifiable to the extent possible.

              Cost growth and schedule slips at the individual program level complicate
              DHS’s efforts to manage its investment portfolio as a whole. When
              programs encounter setbacks, the department has often redirected
              funding to troubled programs at the expense of others, which in turn are
              more likely to struggle. Additionally, DHS acquisition policy does not fully
              reflect key portfolio-management practices that would help improve
              investment management across the department. For example, the policy
              does not empower portfolio managers to invest resources in a disciplined
              manner or establish that investments should be ranked and selected
              using a disciplined process. DHS acknowledges the importance of having
              strong portfolio-management practices. However, DHS does not have a
              process to systematically prioritize its major investments to ensure that
              the department’s acquisition portfolio is consistent with DHS’s anticipated
              resource constraints, which is particularly important because of the


              Page 45                                            GAO-12-833 Homeland Security
                      diversity of the department’s missions. Since 2008, we have emphasized
                      the need for DHS to re-establish an oversight board dedicated to
                      addressing portfolio management challenges. DHS has produced plans to
                      establish such a board, but the concept is still under development.

                      It is essential that DHS take a more disciplined acquisition management
                      approach moving forward, particularly as the department must adjust to a
                      period of governmentwide funding constraints. Without greater discipline,
                      decisionmakers will continue to lack critical information and the
                      department will likely continue to pay more than expected for less
                      capability than promised, which will ultimately hinder DHS’s day-to-day
                      operations and its ability to execute its mission. Further, Congress’s
                      ability to assess DHS funding requests and conduct oversight will remain
                      limited. To its credit, DHS has undertaken a variety of initiatives over the
                      past two years designed to address the department’s longstanding
                      acquisition management challenges, such as increasing acquisition
                      management capabilities at the component-level. However, more
                      disciplined program and portfolio management at the department-level is
                      needed before DHS can regularly delegate major milestone decision
                      authority to component-level officials. Widespread challenges—including
                      funding instability and acquisition workforce shortfalls—cost growth, and
                      schedule slips indicate how much further DHS must go to improve
                      acquisition outcomes.


                      We recommend that the Secretary of Homeland Security direct the Under
Recommendations for   Secretary for Management to take the following five actions to help
Executive Action      mitigate the risk of poor acquisition outcomes and strengthen the
                      department’s investment management activities:

                      •   Modify DHS acquisition policy to more fully reflect the following
                          program management practices:
                          •  Require that (1) programs demonstrate technologies in a realistic
                             environment prior to initiating development activities, and (2)
                             manufacturing processes be tested prior to production;
                          •  Require that (1) exit criteria be quantifiable to the extent possible,
                             and (2) consistent information be used across programs at ADE
                             2B and 2C;
                          •  State that program managers should remain with their programs
                             until the next major milestone when possible;




                      Page 46                                            GAO-12-833 Homeland Security
                     •   Modify DHS acquisition policy to more fully reflect the following
                         portfolio management practices:
                         •   Empower portfolio managers to decide how best to invest
                             resources;
                         •   Establish that investments should be ranked and selected using a
                             disciplined process;
                         •   Establish that (1) resource allocations should align with strategic
                             goals, and (2) the investment review policy should use long-range
                             planning; and
                         •   Require portfolio reviews (1) annually to consider proposed
                             changes, (2) as new opportunities are identified, and (3) whenever
                             a program breaches its objectives;
                     •   Ensure all major acquisition programs fully comply with DHS
                         acquisition policy by obtaining department-level approval for key
                         acquisition documents before approving their movement through the
                         acquisition life cycle;
                     •   Once the department’s acquisition programs comply with DHS
                         acquisition policy, prioritize major acquisition programs
                         departmentwide and ensure that the department’s acquisition portfolio
                         is consistent with DHS’s anticipated resource constraints; and
                     •   Clearly document that department-level officials should not delegate
                         ADE decision authority to component-level officials for programs
                         lacking department approved APBs or not meeting agreed-upon cost,
                         schedule, and performance thresholds.

                     DHS provided us with written comments on a draft of this report. In its
Agency Comments      comments, DHS concurred with all five of our recommendations and
and Our Evaluation   noted that two should be closed based on actions taken. The
                     department’s written comments are reprinted in appendix V. DHS also
                     provided technical comments that we incorporated into the report as
                     appropriate.

                     DHS identified specific actions the department would take to address
                     three of our recommendations. DHS stated that it was in the process of
                     revising its policy to more fully reflect key program management
                     practices. Additionally, DHS stated that it would continue to mature and
                     solidify the portfolio review process over the next few years, and that it
                     would revise its policy to reflect this process. DHS anticipates that this
                     effort will also help the department prioritize its major acquisition
                     programs departmentwide, and help ensure that the department’s
                     acquisition portfolio is consistent with anticipated resource constraints.




                     Page 47                                           GAO-12-833 Homeland Security
DHS concurred with and requested we close our recommendation that
the department ensure all acquisition programs fully comply with DHS
acquisition policy by obtaining department-level approval for key
acquisition documents before approving their movement through the
acquisition life cycle. DHS stated that, in effect, its executive review board
is approving a program’s documents when it advances the program, thus
satisfying this recommendation. As we noted in our report, DHS officials
told us in April 2012 that the department has begun to implement its
acquisition policy in a more disciplined manner and that it will no longer
advance programs through the acquisition life cycle until DHS leadership
verifies the programs have developed critical knowledge. However, it
would be premature to close this recommendation until DHS
demonstrates, over time, the consistent verification of the critical
knowledge captured in key documents, especially as we found that nearly
all of the department’s major acquisition programs lack at least some of
these acquisition documents.

DHS also concurred with and requested we close our recommendation
that the department clearly document that department-level officials
should not delegate ADE decision authority to component-level officials
for programs lacking department approved APBs or not meeting agreed-
upon cost, schedule, and performance thresholds. DHS stated that it
amended AD 102 to clarify that decision authority for any program that
breaches an approved APB’s cost, schedule or performance parameters
will not be delegated to component-level officials, thus satisfying this
recommendation. However, the amendment DHS provided does not
include this language or clearly document the department’s stated
position. For this reason, it would be premature to close this
recommendation at this time.

In addition to commenting on our recommendations, the department
made a number of observations on our draft report. For example, DHS
stated that the report references many practices that occurred prior to the
time period of the audit, and that the department has made measurable
progress on a number of fronts. While we reviewed investment
management activities going back to November 2008 to coincide with the
issuance of AD 102, we also accounted for progress made through
August 2012 by assessing ongoing DHS initiatives intended to address
investment management challenges in the future.

DHS also noted that our survey of 71 programs captured valuable
information, but suggested the survey data cannot be generalized and
expressed concern that it would be used as the basis for a


Page 48                                            GAO-12-833 Homeland Security
recommendation. To clarify, none of the recommendations in this report
are based on the survey data. In the absence of reliable program data,
we surveyed program managers to obtain their perspectives on
challenges facing the department’s acquisition programs, and we
obtained responses from 92 percent of the major acquisition programs
DHS identified in 2011. DHS noted that programs can experience cost
growth and schedule slips without a “breach.” We recognize the validity of
this point and our findings are consistent with this position.

DHS incorrectly suggested that our data sources for quantifying cost
growth – the Future Years Homeland Security Programs (FYHSP) issued
in 2008 and 2011 – did not consistently account for costs beyond the
initial five-year period. However, these two FYHSPs aggregated funding
levels for each program to produce a total project cost. To measure total
project cost growth for the 16 programs, as depicted in figure 4, we
compared the total project costs reported in the 2008 FYHSP to the total
project costs reported in the 2011 FYHSP. Thus, we measured changes
in total project costs, not just costs over two different five-year periods.


As agreed with your offices, unless you publicly announce the contents of
this report earlier, we plan no further distribution until September 19,
2012. At that time, we will send copies to the Secretary of Homeland
Security. In addition, the report will be available at no charge on the GAO
website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact
me at (202) 512-4841 or huttonj@gao.gov. Contact points for our Offices
of Congressional Relations and Public Affairs may be found on the last
page of this report. GAO staff who made key contributions to this report
are listed in appendix VI.




John P. Hutton
Director, Acquisition and Sourcing Management




Page 49                                           GAO-12-833 Homeland Security
List of Requesters

The Honorable Joseph I. Lieberman
Chairman
The Honorable Susan M. Collins
Ranking Member
Committee on Homeland Security and Governmental Affairs
United States Senate

The Honorable Thomas R. Carper
Chairman
Subcommittee on Federal Financial Management, Government
Information, Federal Services and International Security
Committee on Homeland Security and Governmental Affairs
United States Senate

The Honorable Michael T. McCaul
Chairman
Subcommittee on Oversight, Investigations, and Management
Committee on Homeland Security
House of Representatives




Page 50                                     GAO-12-833 Homeland Security
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              The objectives of this review were to assess the Department of Homeland
              Security’s (DHS) acquisition management activities. Specifically, we
              assessed the extent to which: (1) DHS’s major acquisition programs face
              challenges that increase the risk of poor outcomes; (2) DHS has policies
              and processes in place to effectively manage individual acquisition
              programs; (3) DHS has policies and processes in place to effectively
              manage its portfolio of acquisition programs as a whole; and (4) DHS has
              taken actions to resolve the high-risk acquisition management issues we
              have identified in previous reports. To answer these questions, we
              reviewed 77 of the 82 programs DHS included in its fiscal year 2011
              Major Acquisition Oversight List (MAOL), which identified each program
              the department designated a major acquisition in 2011. 1 We excluded 5
              programs that were canceled in 2011; these are identified in appendix IV.
              The 77 selected programs were sponsored by 12 different components
              and departmental offices.

              To determine the extent to which major DHS acquisition programs face
              challenges increasing the risk of poor outcomes, we surveyed the
              program managers for all 77 programs, and received usable responses
              from 71 programs (92 percent response rate). Appendix III presents the
              survey questions we asked, and summarizes the responses we received.
              The web-based survey was administered from January 12, 2012, to
              March 30, 2012. Respondents were sent an e-mail invitation to complete
              the survey on a GAO web server using a unique username and
              password. During the data collection period, nonrespondents received a
              reminder e-mail and phone call. Because this was not a sample survey, it
              has no sampling errors. The practical difficulties of conducting any survey
              may also introduce nonsampling errors, such as difficulties interpreting a
              particular question, which can introduce unwanted variability into the
              survey results. We took steps to minimize nonsampling errors by
              pretesting the questionnaire in person with program management officials
              for five different programs, each in a different component. We conducted
              pretests to make sure that the questions were clear and unbiased, the
              data and information were readily obtainable, and that the questionnaire
              did not place an undue burden on respondents. Additionally, a senior
              methodologist within GAO independently reviewed a draft of the
              questionnaire prior to its administration. We made appropriate revisions to



              1
               Undersecretary for Management, DHS, Fiscal Year 2011—Major Acquisition Oversight
              List. Memo. (Washington, D.C.: Jan. 28, 2011).




              Page 51                                                GAO-12-833 Homeland Security
Appendix I: Objectives, Scope, and
Methodology




the content and format of the questionnaire after the pretests and
independent review. All data analysis programs used to generate survey
results were independently verified for accuracy.

To determine the extent to which major DHS acquisition programs face
challenges increasing the risk of poor outcomes, we also reviewed the
2008 and 2011 versions of the Future Years Homeland Security Program
(FYHSP), all acquisition decision memoranda documenting DHS
executive review board decisions from November 2008 to April 2012, the
Office of Program Accountability and Risk Management’s (PARM) initial
Quarterly Program Assessment Report (QPAR), issued March 2012, and
other management memos identifying available program-performance
data. The survey results and documentation review allowed us to identify
program performance, and the reasons for any poor performance. We
also interviewed individuals at the component and department-level to
enhance our understanding of common challenges. At the component
level, we interviewed six of the eight Component Acquisition Executives
that had been designated by the USM, and interviewed representatives of
the remaining two. At the department level, we interviewed policy, budget,
and acquisition oversight officials, including the Deputy Assistant
Secretary for the Office of Strategic Plans, the department’s Chief
Information Officer, the Executive Director of PARM, and the Director of
Program Analysis and Evaluation (PA&E). These officials provided a
strategic perspective on program management challenges, and shared
valuable insights regarding the limitations of available program
performance data. Based on their input, we chose to use FYHSP data to
calculate cost growth for individual programs where possible because the
document is provided to Congress and constitutes DHS’s most
authoritative, out-year funding plan.

To determine the extent to which DHS policies and processes are in
place to effectively manage individual acquisition programs, as well as the
department’s acquisition portfolio as a whole, we identified key acquisition
management practices and assessed the extent to which DHS policies
and processes reflected those practices. We identified the key practices
through a review of previous GAO reports, which are listed in appendix II.
We compared DHS Acquisition Directive 102-01 (AD 102), an associated
guidebook—DHS Instruction Manual 102-01-001—and the guidebook’s
12 appendixes to those key practices, and identified the extent to which
they were reflected in the department’s acquisition policy using a basic
scoring system. If the DHS policy reflected a particular key practice, we
assigned the policy a score of 5 for that practice. If the policy did not
reflect the key practice, we assigned it a score of 1. We then took the


Page 52                                           GAO-12-833 Homeland Security
Appendix I: Objectives, Scope, and
Methodology




average score for all the key practices in a particular area—as identified
in appendix II—to establish an overall score for each key practice area.
We concluded that key practice areas that scored a 5 were reflected in
the policy, scored a 4 were substantially reflected, scored a 3 were
partially reflected, and scored a 2 were minimally reflected. We
subsequently met with PARM officials to discuss our analysis, identify
relevant sections of the policy that we had not yet accounted for, and
solicit their thoughts on those key practices that were not reflected in the
policy. In order to assess DHS’s processes for implementing its policy, we
surveyed program managers, and interviewed component and
department-level officials. We also reviewed DHS’s plans for the
Integrated Investment Life Cycle Model (IILCM), which is being designed
to better integrate the department’s investment management functions.
Further, we reviewed all acquisition decision memoranda documenting
DHS executive review board decisions from November 2008 to April
2012, the March 2012 QPAR, and other management memos identifying
available program-performance data, and any limitations of that data.

To determine the extent to which DHS has taken actions to resolve the
high-risk acquisition management issues we have identified in previous
reports and this audit, we reviewed the first three versions of the DHS
Integrated Strategy for High Risk Management—issued in January, June,
and December 2011. We also reviewed the DHS Program Management
and Execution Playbook, issued in December 2011. We identified
initiatives intended to improve acquisition management, the department’s
progress in implementing those initiatives, and enduring challenges
confronting the department. We also surveyed program managers, and
interviewed component and department-level officials to obtain their
perspectives on the initiatives.

We conducted this performance audit from August 2011 to September
2012 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.




Page 53                                           GAO-12-833 Homeland Security
Appendix II: Key Acquisition Management
                         Appendix II: Key Acquisition Management
                         Practices



Practices

                         To determine the extent to which the Department of Homeland Security
                         (DHS) has policies and processes in place to effectively manage
                         individual acquisition programs, and the department’s acquisition portfolio
                         as a whole, we identified key acquisition management practices
                         established in our previous reports examining DHS, the Department of
                         Defense, NASA, and private sector organizations. The specific program-
                         and portfolio-management practices, as well as the reports where we
                         previously identified the value of those practices, are presented below.


Key Program Management   The following list identifies several key practices that can improve
Practices                outcomes when managing an individual program.

                         Identify and validate needs

                         •   A need statement should be informed by a comprehensive
                             assessment that considers the organization’s overall mission
                         •   Current capabilities should be identified to determine if there is a gap
                             between the current and needed capabilities
                         •   Noncapital alternatives and the modernization of existing assets
                             should be evaluated before deciding how best to meet the gap
                         •   Needs should be communicated within the context of a business case
                         Assess alternatives to select most appropriate solution

                         •   Analyses of alternatives (AOA) should compare the performance,
                             costs, and risks of competing solutions, and identify the most
                             promising system solution to acquire
                         •   AOAs should be conducted early in the acquisition process, before
                             requirements are set
                         •   AOAs should be sufficiently broad to assess many alternatives across
                             multiple concepts
                         Clearly establish well-defined requirements

                         •   Programs should be grounded in well-understood concepts of how
                             systems would be used and likely requirements costs
                         •   Operators and stakeholders should be involved in the development of
                             requirements
                         •   Firm requirements should be presented in a business case at the
                             outset of a program
                         •   Requirements should be well defined to ensure clear communication
                             about what the government needs




                         Page 54                                           GAO-12-833 Homeland Security
Appendix II: Key Acquisition Management
Practices




Develop realistic cost estimates and schedules

•   A cost estimate should be well documented, and include source data,
    clearly detailed calculations, and explanations of why particular
    methods were chosen
•   A cost estimate should be comprehensive enough to ensure that cost
    elements are neither omitted nor double counted
•   A cost estimate should be accurate, unbiased, and based on an
    assessment of most likely costs
•   A cost estimate should be credible, and discuss any limitations of the
    analysis because of uncertainty or assumptions; uncertainty analyses
    and independent cost estimates should be conducted
•   A schedule should account for how long all activities will take—
    including a risk analysis—and identify resources needed to do the
    work
•   A schedule should identify relationships between sequenced
    activities—including the critical path representing the schedule’s
    longest total duration—and how much a predecessor activity can slip
    before affecting successor activities
•   A schedule should be planned so that critical project dates can be
    met, and continuously updated using logic and durations
Secure stable funding that matches resources to requirements

•   Technologies, time, funding, and other resources should be consistent
    with established requirements before development begins
•   Programs should invest in systems engineering resources early, and
    make trade-offs by reducing or deferring requirements, in order to
    address capability needs in achievable increments with shorter cycle
    times and more predictable funding needs
•   Prior to the initiation of system development activities, realistic cost
    estimates—validated against independent cost estimates—should be
    developed to establish a sound basis for acquiring new systems
•   Acquisition strategies should provide sufficient time and money for
    design activities before construction start
•   Projects should be budgeted in useful segments when dealing with a
    capped budget environment
Demonstrate technology, design, and manufacturing maturity

•   Program officials should maintain regular communication with the
    contractor to track technical performance, risks, and issues
•   Prior to the start of system development, critical technologies should
    be demonstrated to work in their intended environment




Page 55                                           GAO-12-833 Homeland Security
                           Appendix II: Key Acquisition Management
                           Practices




                           •   Prior to a critical design review, design should be stable, and a
                               prototype should demonstrate that the design can meet requirements
                           •   Prior to a production decision, (a) a fully integrated, capable prototype
                               should demonstrate that the system will work as intended in a reliable
                               manner; and (b) the program should demonstrate that manufacturing
                               processes are stable
                           •   Prior to formal operator acceptance, operators should participate in
                               the testing of system functionality
                           Utilize milestones and exit criteria

                           •   Exit criteria and decision reviews should be used to determine that
                               product managers captured required and appropriate knowledge—
                               including a more refined cost estimate—before a program moves
                               forward to the next acquisition phase
                           •   To the extent possible, exit criteria should be quantifiable, and
                               decision reviews should be consistent across programs
                           •   Independent program assessments should be conducted at each key
                               decision point
                           Establish an adequate program workforce

                           •   The right people, with the right skill sets, should be assigned to the
                               right programs
                           •   Program managers should stay on until the next major milestone to
                               assure accountability; government and contractor staff should also
                               remain consistent

Key Portfolio Management   The following list identifies several key practices that can improve
Practices                  outcomes when managing a portfolio of multiple programs.

                           Clearly define and empower leadership

                           •   Those responsible for product investment decisions and oversight
                               should be clearly identified and held accountable for outcomes
                           •   Portfolio managers should be empowered to make decisions about
                               the best way to invest resources
                           •   Portfolio managers should be supported with cross-functional teams
                               composed of representatives from key functional areas
                           Establish standard assessment criteria, and demonstrate
                           comprehensive knowledge of the portfolio

                           •   Specific criteria should be used to ensure transparency and
                               comparability across alternatives


                           Page 56                                            GAO-12-833 Homeland Security
                         Appendix II: Key Acquisition Management
                         Practices




                         •   Investments should be ranked and selected using a disciplined
                             process to assess the costs, benefits, and risks of alternative products
                         •   Knowledge should encompass the entire portfolio, including needs,
                             gaps, and how to best meet the gaps
                         Prioritize investments by integrating the requirements, acquisition,
                         and budget processes

                         •   Requirements, acquisition, and budget processes should be
                             connected to promote stability and accountability
                         •   Organizations should use an integrated approach to prioritize needs
                             and allocate resources, so they can avoid pursuing more products
                             than they can afford, and optimize return on investment
                         •   Resource allocation across the portfolio should align with strategic
                             goals/objectives, and investment review policy should use long-range
                             planning
                         Continually make go/no-go decisions to rebalance the portfolio

                         •   Program requirements should be reviewed annually to make
                             recommendations on proposed changes/descoping options
                         •   As potential new products are identified, portfolios should be
                             rebalanced based on those that add the most value
                         •   If project estimates breach established thresholds, the product should
                             be immediately reassessed within the context of the portfolio to
                             determine whether it is still relevant and affordable
                         •   Agencies should use information gathered from post-implementation
                             reviews of investments, as well as information learned from other
                             organizations, to fine-tune the investment process and the portfolios
                             to shape strategic outcomes

Previous Reports         Information Technology: Critical Factors Underlying Successful Major
Establishing Key         Acquisitions. GAO-12-7. Washington, D.C.: October 21, 2011.
Acquisition Management
                         Acquisition Planning: Opportunities to Build Strong Foundations for Better
Practices                Services Contracts. GAO-11-672. Washington, D.C.: August 9, 2011.

                         NASA: Assessments of Selected Large-Scale Projects. GAO-11-239SP.
                         Washington, D.C.: March 3, 2011.

                         Defense Acquisitions: Assessments of Selected Weapon Programs.
                         GAO-11-233SP. Washington, D.C.: March 29, 2011.




                         Page 57                                           GAO-12-833 Homeland Security
Appendix II: Key Acquisition Management
Practices




Defense Acquisitions: Strong Leadership Is Key to Planning and
Executing Stable Weapon Programs. GAO-10-522. Washington, D.C.:
May 6, 2010.

Defense Acquisitions: Assessments of Selected Weapon Programs.
GAO-10-388SP. Washington, D.C.: March 30, 2010.

Defense Acquisitions: Many Analyses of Alternatives Have Not Provided
a Robust Assessment of Weapon Systems Options. GAO-09-665.
Washington, D.C.: September 24, 2009.

Department of Homeland Security: Billions Invested in Major Programs
Lack Appropriate Oversight. GAO-09-29. Washington, D.C.: November
18, 2008.

GAO Cost Estimating and Assessment Guide: Best Practices for
Developing and Managing Capital Program Costs. GAO-09-3SP.
Washington, D.C.: March 2009.

Defense Acquisitions: Sound Business Case Needed to Implement
Missile Defense Agency’s Targets Program. GAO-08-1113. Washington,
D.C.: September 26, 2008.

Defense Acquisitions: A Knowledge-Based Funding Approach Could
Improve Major Weapon System Program Outcomes. GAO-08-619.
Washington, D.C.: July 2, 2008.

Defense Acquisitions: Realistic Business Cases Needed to Execute Navy
Shipbuilding Programs. GAO-07-943T. Washington, D.C.: July 24, 2007.

Best Practices: An Integrated Portfolio Management Approach to Weapon
System Investments Could Improve DOD’s Acquisition Outcomes.
GAO-07-388. Washington, D.C.: March 30, 2007.

Best Practices: Better Support of Weapon System Program Managers
Needed to Improve Outcomes. GAO-06-110. Washington, D.C.:
November 30, 2005.

NASA’s Space Vision: Business Case for Prometheus 1 Needed to
Ensure Requirements Match Available Resources. GAO-05-242.
Washington, D.C.: February 28, 2005.




Page 58                                       GAO-12-833 Homeland Security
Appendix II: Key Acquisition Management
Practices




Information Technology Investment Management: A Framework for
Assessing and Improving Process Maturity. GAO-04-394G. Washington,
D.C.: March 2004.

Executive Guide: Leading Practices in Capital Decision-Making.
GAO/AIMD-99-32. Washington, D.C.: December 1998.




Page 59                                        GAO-12-833 Homeland Security
Appendix III: Program Office Survey Results
               Appendix III: Program Office Survey Results




               To help determine the extent to which major Department of Homeland
               Security (DHS) acquisition programs face challenges increasing the risk
               of poor outcomes, we surveyed the program managers for all 77
               programs, and received usable responses from 71 programs (92 percent
               response rate). The web-based survey was administered from January
               12, 2012, to March 30, 2012. We present the survey questions we asked
               and summarize the responses we received below.


Introduction   The Senate Homeland Security and Governmental Affairs Committee has
               asked the Government Accountability Office (GAO) to examine the
               progress the Department of Homeland Security (DHS) has made in
               improving its acquisition management.

               In order to gain the program manager's perspective on acquisition
               management activities being implemented and challenges that may
               impede achieving program's cost, schedule, and performance objectives,
               we are conducting a survey of all major DHS acquisition programs. Since
               we do not plan on scheduling meetings with each program, this survey is
               your primary opportunity to share how you are managing your program
               and any challenges you face. Additionally, information from this survey
               will help determine whether DHS's planned acquisition management
               initiatives may mitigate challenges facing program management offices or
               if any additional efforts may be needed.

               We ask that you please complete this survey within 2 weeks of receipt.
               The survey should take about 60 to 90 minutes to complete and can be
               completed over multiple sittings. Your responses can be saved and
               accessed at a later date. If you are unsure of how to respond to a
               question, please contact us for assistance.

               We will not release individually identifiable data outside of GAO, unless
               required by law or requested by a member of Congress. We will generally
               report the results of this survey in the aggregate, but if we incorporate
               individual responses into the report, we will do so in a manner designed
               to ensure that individual respondents cannot be identified.

               Thank you for your time and assistance.




               Page 60                                         GAO-12-833 Homeland Security
                                         Appendix III: Program Office Survey Results




Section I: Program Profile
1. What is your program's name?

data intentionally not reported

2. How many years and/or months of experience do you have as a program manager at DHS and outside of DHS?

a. At DHS:

Year(s)
                                                                        Number
                                                                             of
 Mean                      Median      Minimum        Maximum       respondents
 5.6                               5         0                31               70


Month(s)
                                                                        Number
                                                                             of
 Mean                      Median      Minimum        Maximum       respondents
 5.3                               6         0                10               56


b. Outside of DHS:

Year(s)
                                                                        Number
                                                                             of
 Mean                      Median      Minimum        Maximum       respondents
 10.6                             10         0                40               52


Year(s)
                                                                        Number
                                                                             of
 Mean                      Median      Minimum        Maximum       respondents
 2.1                               0         0                11               28




                                         Page 61                                          GAO-12-833 Homeland Security
                                            Appendix III: Program Office Survey Results




3. In what phase(s) of the DHS Acquisition Directive (AD) 102 acquisition lifecycle is your program currently? (select all that
apply)

1. Need (Prior to Acquisition Decision Event (ADE) 1)
                                       Number of
 Not checked            checked      respondents
 64                             6               70


2. Analyze/Select (Between ADE 1 and ADE 2)
                                       Number of
 Not checked            checked      respondents
 60                            10               70


3. Obtain (Between ADE 2 and ADE 3)
                                       Number of
 Not checked            checked      respondents
 42                            28               70


4. Production/deploy/support (Post ADE 3)
                                       Number of
 Not checked            checked      respondents
 23                            47               70


5. Do not know
                                       Number of
 Not checked            checked      respondents
 69                             1               70




                                            Page 62                                                 GAO-12-833 Homeland Security
                                           Appendix III: Program Office Survey Results




Section II: DHS Acquisition Guidance, AD 102
4. How often, if at all, do you refer to each of the following resources to help you manage your acquisition program?

DHS AD 102 guidebook and appendices
                                                                                          I do not refer      Number of
 Daily                     Weekly        Monthly Semi-annually            Annually       to this source     respondents
 1                                12          17                20                 7                 11               68


Component's guidance
                                                                                          I do not refer      Number of
 Daily                     Weekly        Monthly Semi-annually            Annually       to this source     respondents
 4                                17          28                 7                 4                  9               69


Component Acquisition Executive (CAE) staff
                                                                                          I do not refer      Number of
 Daily                     Weekly        Monthly Semi-annually            Annually       to this source     respondents
 7                                23          19                10                 5                  4               68


DHS Acquisition Program Management Division (APMD)/ Program Analysis and Risk Management (PARM)
                                                                                          I do not refer      Number of
 Daily                     Weekly        Monthly Semi-annually            Annually       to this source     respondents
 1                                   9        17                23               10                   9               69


Other (please specify below)
                                                                                          I do not refer      Number of
 Daily                     Weekly        Monthly Semi-annually            Annually       to this source     respondents
 3                                   4         7                 2                 0                 10               26


If other resource, please specify:

data intentionally not reported




                                           Page 63                                                         GAO-12-833 Homeland Security
                                               Appendix III: Program Office Survey Results




5. Which of the following DHS-provided opportunities, if any, has your program management office used to understand DHS
acquisition guidance, AD 102; and if used, how useful was the opportunity, if at all?

1. Check box at left if your program is not required to follow AD 102, and then click here to skip to question 13
                                         Number of
 Not checked              checked      respondents
 68                               3                   71


Training session(s) on AD 102 hosted by DHS headquarters

Did your program management office use this opportunity?
                          No, the
                     opportunity                No,
 Yes                  is available      opportunity        Do not know
                   but we did not            is not                         Number of
                           use it.        available                       respondents
 30                              10                   12            11               63


If used, how useful was the opportunity, if at all?
                        Somewhat            Not at all                      Number of
 Very useful               useful             useful        No opinion    respondents
 11                              16                    1             2               30


Manuals and templates for implementing AD 102 provided by DHS headquarters

Did your program management office use this opportunity?
                          No, the
                     opportunity                No,
 Yes                  is available      opportunity        Do not know
                   but we did not            is not                         Number of
                           use it.        available                       respondents
 50                               8                    4             4               66


If used, how useful was the opportunity, if at all?
                        Somewhat            Not at all                      Number of
 Very useful               useful             useful        No opinion    respondents
 17                              29                    2             2               50




                                               Page 64                                                      GAO-12-833 Homeland Security
                                               Appendix III: Program Office Survey Results




Direct support for your program from DHS Acquisition Program Management Division (APMD)/ Program Analysis and Risk
Management (PARM)

Did your program management office use this opportunity?
                          No, the
                     opportunity                No,
 Yes                  is available      opportunity       Do not know
                   but we did not            is not                         Number of
                           use it.        available                       respondents
 49                                9                  4              3               65


If used, how useful was the opportunity, if at all?
                        Somewhat            Not at all                      Number of
 Very useful               useful             useful       No opinion     respondents
 24                               22                  2              1               49


Other (please specify below)

Did your program management office use this opportunity?
                          No, the
                     opportunity                No,
 Yes                  is available      opportunity       Do not know
                   but we did not            is not                         Number of
                           use it.        available                       respondents
 17                                1                  3              6               27


If used, how useful was the opportunity, if at all?
                        Somewhat            Not at all                      Number of
 Very useful               useful             useful       No opinion     respondents
 11                                2                  1              2               16


If other opportunity, please specify:

data intentionally not reported




                                               Page 65                                           GAO-12-833 Homeland Security
                                            Appendix III: Program Office Survey Results




6. How clear or unclear is the DHS AD 102 acquisition guidance and framework for managing the following types of
acquisitions?

Developmental
                      Somewhat      Neither clear       Somewhat                                          Number of
 Very clear               clear      nor unclear          unclear      Very unclear       No opinion    respondents
 11                           22                 4                4                 0             22                63


Non-developmental
                      Somewhat      Neither clear       Somewhat                                          Number of
 Very clear               clear      nor unclear          unclear      Very unclear       No opinion    respondents
 4                            23                 5                5                 1             24                62


IT systems
                      Somewhat      Neither clear       Somewhat                                          Number of
 Very clear               clear      nor unclear          unclear      Very unclear       No opinion    respondents
 12                           19                 5                5                 1             22                64


Services
                      Somewhat      Neither clear       Somewhat                                          Number of
 Very clear               clear      nor unclear          unclear      Very unclear       No opinion    respondents
 7                            21                 6                4                 4             20                62


7. How clear or unclear is DHS AD 102 acquisition guidance, including the guidebook and appendices, regarding each of the
following?

Understanding the Acquisition Review Board (ARB)'s roles and responsibilities
                      Somewhat      Neither clear       Somewhat                                          Number of
 Very clear               clear      nor unclear          unclear      Very unclear       No opinion    respondents
 23                           26                 3                5                 1              7                65


Understanding Component Acquisition Executive (CAE)'s roles and responsibilities
                      Somewhat      Neither clear       Somewhat                                          Number of
 Very clear               clear      nor unclear          unclear      Very unclear       No opinion    respondents
 25                           20                 7                5                 1              7                65


Implementing the Systems Engineering Life Cycle (SELC) framework to ensure capabilities are effectively delivered
                      Somewhat      Neither clear       Somewhat                                          Number of
 Very clear               clear      nor unclear          unclear      Very unclear       No opinion    respondents
 15                           24                 6                7                 3             10                65




                                            Page 66                                                    GAO-12-833 Homeland Security
                                              Appendix III: Program Office Survey Results




Conducting testing and evaluation
                        Somewhat       Neither clear      Somewhat                                          Number of
 Very clear                 clear       nor unclear         unclear      Very unclear       No opinion    respondents
 13                               28              4                 6                 3            11               65


Managing and prioritizing requirements across multiple DHS components
                        Somewhat       Neither clear      Somewhat                                          Number of
 Very clear                 clear       nor unclear         unclear      Very unclear       No opinion    respondents
 4                                18             10                12                 5            16               65


Aligning AD-102 guidance with Capital Planning and Investment Control (CPIC) and Planning, Programming, Budgeting, and Execution
(PPBE) guidance to match resources and requirements
                        Somewhat       Neither clear      Somewhat                                          Number of
 Very clear                 clear       nor unclear         unclear      Very unclear       No opinion    respondents
 10                               24              9                 9                 5             8               65


Aligning DHS and your component's guidance
                        Somewhat       Neither clear      Somewhat                                          Number of
 Very clear                 clear       nor unclear         unclear      Very unclear       No opinion    respondents
 8                                20              8                10                 8             9               63


Other (please specify below)
                        Somewhat       Neither clear      Somewhat                                          Number of
 Very clear                 clear       nor unclear         unclear      Very unclear       No opinion    respondents
 2                                 2              1                 1                 1            15               22


If other, please specify:

data intentionally not reported




                                              Page 67                                                    GAO-12-833 Homeland Security
                                             Appendix III: Program Office Survey Results




8. How clear or unclear is DHS AD 102 acquisition guidance, including the guidebook and appendices, on how to develop
each of the following key acquisition documents?

Mission Needs Statement (MNS)
                      Somewhat        Neither clear      Somewhat                                          Number of
 Very clear               clear        nor unclear         unclear      Very unclear       No opinion    respondents
 27                              25              2                 2                 0             8               64


Operational Requirements Document (ORD)
                      Somewhat        Neither clear      Somewhat                                          Number of
 Very clear               clear        nor unclear         unclear      Very unclear       No opinion    respondents
 23                              24              4                 3                 2             7               63


Life Cycle Cost Estimate (LCCE)
                      Somewhat        Neither clear      Somewhat                                          Number of
 Very clear               clear        nor unclear         unclear      Very unclear       No opinion    respondents
 13                              28              8                 4                 4             7               64


Acquisition Program Baseline (APB)
                      Somewhat        Neither clear      Somewhat                                          Number of
 Very clear               clear        nor unclear         unclear      Very unclear       No opinion    respondents
 24                              22              6                 4                 2             6               64


Analysis of Alternatives (AOA)
                      Somewhat        Neither clear      Somewhat                                          Number of
 Very clear               clear        nor unclear         unclear      Very unclear       No opinion    respondents
 19                              24              9                 2                 3             7               64


Test and Evaluation Master Plan (TEMP)
                      Somewhat        Neither clear      Somewhat                                          Number of
 Very clear               clear        nor unclear         unclear      Very unclear       No opinion    respondents
 18                              21              7                 5                 2            11               64


Integrated Logistics Support Plan (ILSP)
                      Somewhat        Neither clear      Somewhat                                          Number of
 Very clear               clear        nor unclear         unclear      Very unclear       No opinion    respondents
 15                              23              8                 6                 3             9               64




                                             Page 68                                                    GAO-12-833 Homeland Security
                                            Appendix III: Program Office Survey Results




9. How long is the average component and DHS review period for key acquisition documents required by AD 102 (e.g. MNS,
ORD, LCCE and APB)?

Component review
 Less than 1        1 month to 3    3 months to 6     6 months to 1     More than 1               Not      Number of
 month                  months           months               year             year        Applicable    respondents
 6                             22               20                6                 2              5               61


DHS review
 Less than 1        1 month to 3    3 months to 6     6 months to 1     More than 1               Not      Number of
 month                  months           months               year             year        Applicable    respondents
 1                             15               24               10                 5              8               63


10. After an Acquisition Review Board (ARB) review, how adequately does an Acquisition Decision Memo (ADM)
communicate action items?
 Very                 Somewhat           Not at all                                         Number of
 adequately           adequately       adequately       No opinion Not applicable         respondents
 26                            25                2                6                 5             64


11. How has the introduction of AD 102 helped or hindered your ability to manage your program's cost and schedule and the
overall acquisition program?

Program's cost and schedule
 Significantly        Somewhat Neither helped           Somewhat       Significantly                       Number of
 helped                  helped nor hindered             hindered         hindered         No opinion    respondents
 8                             16               22                7                 3              9               65


Overall ability to manage your acquisition program
 Significantly        Somewhat Neither helped           Somewhat       Significantly                       Number of
 helped                  helped nor hindered             hindered         hindered         No opinion    respondents
 7                             21               18                6                 5              8               65




                                            Page 69                                                     GAO-12-833 Homeland Security
                                             Appendix III: Program Office Survey Results




12. If you would like to elaborate on any of your previous responses regarding the clarity and/or implementation of DHS
acquisition guidance (AD 102) please use the following space.

data intentionally not reported


Section III: Program Performance
13. Does your program have a DHS-approved Acquisition Program Baseline (APB)?
                       No (please
                          explain        Number of
 Yes                       below)      respondents
 40                               31             71


13a. If your program does not have a DHS-approved APB, please explain why it does not have one in the box below.

data intentionally not reported

After answering 13a above,

14. How does your program's current projected cost compare against its DHS-approved APB?
                                 Cost exceeds          Cost exceeds
                      Cost meets   the APB by          the APB by 8
 Cost is below               the          less            percent or      Number of
 the APB                    APB than 8 percent                 more     respondents
 8                                23              4                5               40


15. How does your program's current projected schedule compare to its DHS-approved APB?
 Schedule is            Schedule       Schedule is
 ahead                     meets           behind        Number of
 of the APB              the APB          the APB      respondents
 3                                21             16               40


16. How do your program's current planned system capabilities compare to its DHS-approved APB?
                                            System
 System                  System        capabilities
 capabilities         capabilities     fall short of     Number of
 exceed APB            meet APB                APB     respondents
 1                                31              7               39




                                             Page 70                                            GAO-12-833 Homeland Security
                                             Appendix III: Program Office Survey Results




17. How frequently, if at all, does your program management office use each of the following performance metrics to monitor
your program's progress?

Earned Value Management data
 Used all or
 most of the               Used Used rarely or                                               Number of
 time                 sometimes      not at all        Do not know Not applicable          respondents
 24                              9                9                 1              27              70


Integrated Master Schedule
 Used all or
 most of the               Used Used rarely or                                               Number of
 time                 sometimes      not at all        Do not know Not applicable          respondents
 42                            17                 2                 1                8             70


Technology readiness levels to assess the maturity of critical technologies
 Used all or
 most of the               Used Used rarely or                                               Number of
 time                 sometimes      not at all        Do not know Not applicable          respondents
 19                            22                11                 1              17              70


Percentage of design drawings completed
 Used all or
 most of the               Used Used rarely or                                               Number of
 time                 sometimes      not at all        Do not know Not applicable          respondents
 20                              5               12                 3              29              69


Percentage of production processes under statistical control
 Used all or
 most of the               Used Used rarely or                                               Number of
 time                 sometimes      not at all        Do not know Not applicable          respondents
 16                              8               14                 1              31              70


Other (please specify below)
 Used all or
 most of the               Used Used rarely or                                               Number of
 time                 sometimes      not at all        Do not know Not applicable          respondents
 11                              0                2                 1                9             23




                                             Page 71                                                     GAO-12-833 Homeland Security
                                                Appendix III: Program Office Survey Results




If other performance metrics, please specify:

data intentionally not reported


Section IV: Cost Estimates and Program Schedule
18. If your cost estimate was developed after AD 102 went into effect, to what extent did the GAO Cost Estimating Guide, i.e.
Appendix I to DHS's AD 102 guidebook, inform your efforts to develop your program's LCCE?
                                                                                    Not
                                                                            applicable,
                                                                          cost estimate
                                                                                    was
                                                          Program does      developed
 Greatly                Somewhat                             not have a     prior to AD         Number of
 informed                informed Did not inform                  LCCE              102       respondents
 29                               17                  1               2               21               70


19. What obstacles, if any, did your program management office encounter when using the GAO Cost Estimating Guide to
develop your program's LCCE?

data intentionally not reported

20. How fully, if at all, does your integrated master schedule (IMS) account for each of the following practices?

Capturing and sequencing all activities
 Fully                     Partially        Does not
 accounts                 accounts           account                         Number of
 for                             for             for       Do not know     respondents
 38                               21                  4               5               68


Assigning resources (i.e. dollars, FTEs, etc.) to all activities
 Fully                     Partially        Does not
 accounts                 accounts           account                         Number of
 for                             for             for       Do not know     respondents
 12                               32                19                5               68


Establishing duration of all activities, noting start and end dates, and keeping durations as short as possible
 Fully                     Partially        Does not
 accounts                 accounts           account                         Number of
 for                             for             for       Do not know     respondents
 38                               20                  4               5               67




                                                Page 72                                                     GAO-12-833 Homeland Security
                                               Appendix III: Program Office Survey Results




Establishing a critical path to examine the effects of any activities slipping along this path
 Fully                    Partially         Does not
 accounts                accounts            account                          Number of
 for                            for              for      Do not know       respondents
 38                              18                  7                 5                68


Identifying float (i.e. the time that a predecessor activity can slip before the delay affects successor activities) between activities
 Fully                    Partially         Does not
 accounts                accounts            account                          Number of
 for                            for              for      Do not know       respondents
 33                              25                  4                 5                67


Updating the schedule to ensure valid status dates are captured
 Fully                    Partially         Does not
 accounts                accounts            account                          Number of
 for                            for              for      Do not know       respondents
 42                              19                  2                 5                68



Section V: Requirements
21. Have each of the following documents been approved by DHS leadership for your program?

Mission Needs Statement (MNS)
                     Submitted to
                             DHS
                      leadership,
 Yes, approved            not yet     Not approved        Do not know         Number of
                        approved                                            respondents
 44                               6                14                  4                68


Operational Requirements Document (ORD)
                     Submitted to
                             DHS
                      leadership,
 Yes, approved            not yet     Not approved        Do not know         Number of
                        approved                                            respondents
 29                              10                22                  5                66




                                               Page 73                                                         GAO-12-833 Homeland Security
                                              Appendix III: Program Office Survey Results




Test and Evaluation Master Plan (TEMP)
                    Submitted to
                            DHS
                     leadership,
 Yes, approved           not yet     Not approved       Do not know         Number of
                       approved                                           respondents
 31                              7                22                 6               66


Integrated Logistics Support Plan (ILSP)
                    Submitted to
                            DHS
                     leadership,
 Yes, approved           not yet     Not approved       Do not know         Number of
                       approved                                           respondents
 28                              9                23                 8               68


22. When setting operational requirements, which of the following processes best describes your program's efforts to
consider alternatives at the program level?
 An Analysis of
 Alternatives                            A trade-off       A trade-off
 (AOA) was           An AOA was        analysis was      analysis was                                                  Not
 conducted             conducted         conducted         conducted                                           applicable,
 prior to                    after           prior to            after      No AOA or                          operational
 operational          operational       operational       operational         trade-off                      requirements
 requirements       requirements      requirements      requirements      analysis was                        have not yet       Number of
 being set               were set         being set          were set       conducted       Do not know          been set.     respondents
 27                             18                 5                 1                 5                3                 4            63


23. At the initiation of design and development activities, how useful were each of the following stakeholders' contributions
when defining proposed system capabilities/ requirements?

1. Check box at left if design and development activities have not yet been initiated, and then click here to skip to question 28
                                        Number of
 Not checked             checked      respondents
 61                             10                71


End-user(s)
                                                                          Did not seek
                       Somewhat                                             input from        Number of
 Very useful              useful         Not useful       No opinion      stakeholder       respondents
 46                              9                 1                 2                 0               58




                                              Page 74                                                       GAO-12-833 Homeland Security
                                              Appendix III: Program Office Survey Results




Subject Matter Expert(s)
                                                                         Did not seek
                      Somewhat                                             input from         Number of
 Very useful             useful         Not useful       No opinion      stakeholder        respondents
 51                             5                 0                 2                 0             58


Systems Engineer(s)
                                                                         Did not seek
                      Somewhat                                             input from         Number of
 Very useful             useful         Not useful       No opinion      stakeholder        respondents
 37                            13                 0                 2                 4             56


Component's CAE office
                                                                         Did not seek
                      Somewhat                                             input from         Number of
 Very useful             useful         Not useful       No opinion      stakeholder        respondents
 12                            18                 9                 8               11              58


APMD/ PARM
                                                                         Did not seek
                      Somewhat                                             input from         Number of
 Very useful             useful         Not useful       No opinion      stakeholder        respondents
 7                             14                 5                12               20              58


Test & Evaluation and Standards Division
                                                                         Did not seek
                      Somewhat                                             input from         Number of
 Very useful             useful         Not useful       No opinion      stakeholder        respondents
 16                            12                 2                11               16              57


DHS Office of the Chief Information Officer
                                                                         Did not seek
                      Somewhat                                             input from         Number of
 Very useful             useful         Not useful       No opinion      stakeholder        respondents
 13                             9                 9                 6               21              58


DHS Office of Finance (including PA&E)
                                                                         Did not seek
                      Somewhat                                             input from         Number of
 Very useful             useful         Not useful       No opinion      stakeholder        respondents
 4                             10                 7                10               27              58




                                              Page 75                                                     GAO-12-833 Homeland Security
                                             Appendix III: Program Office Survey Results




DHS Office of Policy
                                                                        Did not seek
                       Somewhat                                           input from         Number of
 Very useful              useful         Not useful     No opinion      stakeholder        respondents
 6                                 8             7                10               25              56


DHS Acquisition Review Board (ARB)
                                                                        Did not seek
                       Somewhat                                           input from         Number of
 Very useful              useful         Not useful     No opinion      stakeholder        respondents
 15                               14             4                 7               17              57


Federal Partners
                                                                        Did not seek
                       Somewhat                                           input from         Number of
 Very useful              useful         Not useful     No opinion      stakeholder        respondents
 27                               14             1                 7                 9             58


Other (please specify below)
                                                                        Did not seek
                       Somewhat                                           input from         Number of
 Very useful              useful         Not useful     No opinion      stakeholder        respondents
 9                                 2             0                 1                 5             17


If other stakeholders, please specify:

data intentionally not reported

24. How many key performance parameters (KPPs) did the program have at development start, and how many KPPs does the
program currently have?

24a. At start:
                                                                            Number
                                                                                 of
 Mean                      Median         Minimum         Maximum       respondents
 7.0                              5              0                51               52


24b. Currently:
                                                                            Number
                                                                                 of
 Mean                      Median         Minimum         Maximum       respondents
 8.0                              5              0                51               53




                                             Page 76                                                     GAO-12-833 Homeland Security
                                              Appendix III: Program Office Survey Results




25. If your program's KPPs have changed or been redefined since development activities began (ADE 2A), how clear or
unclear was the process for each of the following?

1. Check box at left if there have been no revisions to the KPPs, and then click here to skip to question 27
                                        Number of
 Not checked             checked      respondents
 28                             32               60


Revising key acquisition documents to reflect the change in requirements
                       Somewhat       Neither clear       Somewhat                                               Number of
 Very clear                clear       nor unclear          unclear      Very unclear        No opinion        respondents
 10                             10                 1                1                 1                3               26


Obtaining component leadership approval
                       Somewhat       Neither clear       Somewhat                                               Number of
 Very clear                clear       nor unclear          unclear      Very unclear        No opinion        respondents
 13                             10                 1                1                 0                1               26


Obtaining DHS leadership approval
                       Somewhat       Neither clear       Somewhat                                               Number of
 Very clear                clear       nor unclear          unclear      Very unclear        No opinion        respondents
 10                              8                 2                3                 1                2               26


26. Which of the following are reasons your program's KPPs have changed or been redefined since development activities
began (ADE 2A)?

To clearly communicate performance parameters to the contractor
                                                          Number of
 A reason           Not a reason      Do not know       respondents
 9                              12                 3               24


To make requirements measurable for testing
                                                          Number of
 A reason           Not a reason      Do not know       respondents
 12                             10                 2               24


Desired performance could not be met with current technology
                                                          Number of
 A reason           Not a reason      Do not know       respondents
 6                              15                 3               24




                                              Page 77                                                      GAO-12-833 Homeland Security
                                               Appendix III: Program Office Survey Results




Associated capabilities were determined unnecessary
                                                           Number of
 A reason              Not a reason     Do not know      respondents
 6                                14               2                22


To demonstrate traceability between MNS, ORD, and TEMP
                                                           Number of
 A reason              Not a reason     Do not know      respondents
 10                               11               3                24


Other (please specify below)
                                                           Number of
 A reason              Not a reason     Do not know      respondents
 4                                  1              1                 6


If other reasons, please specify:

data intentionally not reported

27. Since your program's design and development activities began (ADE 2A), how have each of the following factors affected
your planned capabilities, if at all?

Funding availability
 Resulted in                Planned      Resulted in
 increased               capabilities     decreased
 planned               remained the         planned                                            Number of
 capabilities                  same      capabilities    Not a factor     Do not know        respondents
 11                               16              21                 9                 1             58


Changes in the program's schedule
 Resulted in                Planned      Resulted in
 increased               capabilities     decreased
 planned               remained the         planned      Not a factor     Do not know          Number of
 capabilities                  same      capabilities                                        respondents
 8                                18              15                16                 0             57


Mission(s) changed
 Resulted in                Planned      Resulted in
 increased               capabilities     decreased
 planned               remained the         planned      Not a factor     Do not know          Number of
 capabilities                  same      capabilities                                        respondents
 9                                11               2                35                 0             57




                                               Page 78                                                     GAO-12-833 Homeland Security
                                               Appendix III: Program Office Survey Results




Key Performance Parameter(s) (KPP) changed
 Resulted in              Planned        Resulted in
 increased             capabilities       decreased
 planned             remained the           planned      Not a factor     Do not know          Number of
 capabilities                same        capabilities                                        respondents
 5                                  13             2                37                 0                 57


End-user(s) input
 Resulted in              Planned        Resulted in
 increased             capabilities       decreased
 planned             remained the           planned      Not a factor     Do not know          Number of
 capabilities                same        capabilities                                        respondents
 21                                 19             2                15                 0                 57


Technology development efforts/ availability
 Resulted in              Planned        Resulted in
 increased             capabilities       decreased
 planned             remained the           planned      Not a factor     Do not know          Number of
 capabilities                same        capabilities                                        respondents
 11                                 15             6                25                 0                 57


Other (please specify below)
 Resulted in              Planned        Resulted in
 increased             capabilities       decreased
 planned             remained the           planned      Not a factor     Do not know          Number of
 capabilities                same        capabilities                                        respondents
 0                                   0             2                 8                 2                 12


If other factors, please specify:

data intentionally not reported


Section VI: Technology Maturity
28. Prior to the initiation of development activities (ADE 2A), how many of your program's critical technologies demonstrated
full functionality in realistic environments, if any?
                                                                                   Not                  Not
                                                                           applicable,          applicable,
 All critical        Some critical    No critical                              critical       the program
 technologies        technologies technologies           Do not know     technologies                   has
 demonstrated       demonstrated demonstrated                            had not been         not initiated
 full                          full           full                        identified at      development          Number of
 functionality        functionality functionality                            that time            activities    respondents
 25                                 19             1                 5                 5                 11               66




                                               Page 79                                                         GAO-12-833 Homeland Security
                                             Appendix III: Program Office Survey Results




29. Prior to the initiation of low-rate initial production (ADE 2C), how many reliability goals were met, if any, by production-
representative prototypes demonstrated in the intended environment?
                                                                               Not
                                                                       applicable,              Not
                                                                     the program        applicable,
 Prototypes           Prototypes       Prototypes                         has not     the program
 met all               met some            met no                        initiated     will not use
 reliability           reliability      reliability                low-rate initial low-rate initial        Number of
 goals                      goals            goals     Do not know    production       production         respondents
 14                            13                 0                5               14               21              67


30. Has the program used an independent testing authority?
                                                                          Number of
 Yes                           No    Do not know Not applicable         respondents
 35                            15                 0               15               65



Section VII: Resource Allocation
31. Does your program rely on any funds other than DHS appropriations?
                                                         Number of
 Yes                           No    Do not know       respondents
 21                            49                 0               70


32. Does your program use a five-year funding plan to project resource needs?
                                                         Number of
 Yes                           No    Do not know       respondents
 65                             3                 0               68


33. Did your program's funding levels in each of the following budget documents meet your program's required funding
needs as reflected in your APB?

1. Check box at left if your program does not have an APB, and then click here to skip to question 34
                                       Number of
 Not checked             checked     respondents
 56                            15               71




                                             Page 80                                                     GAO-12-833 Homeland Security
                                           Appendix III: Program Office Survey Results




At program start, component's commitment in a Resource Allocation Plan (RAP)
                   No, funds in
                    document         No, funds in
 Yes, funds               were     the document
 were                above the        were below                                           Number of
 equivalent               APB            the APB     Do not know Not applicable          respondents
 38                            2               6                 5                 4             55


At program start, DHS's commitment in a Resource Allocation Decision (RAD)
                   No, funds in
                    document         No, funds in
 Yes, funds               were     the document
 were                above the        were below                                           Number of
 equivalent               APB            the APB     Do not know Not applicable          respondents
 33                            0               8                 6                 7             54


Component's FY11 RAP
                   No, funds in
                    document         No, funds in
 Yes, funds               were     the document
 were                above the        were below                                           Number of
 equivalent               APB            the APB     Do not know Not applicable          respondents
 29                            3              16                 2                 2             52


DHS's FY11 RAD
                   No, funds in
                    document         No, funds in
 Yes, funds               were     the document
 were                above the        were below     Do not know Not applicable            Number of
 equivalent               APB            the APB                                         respondents
 30                            1              17                 2                 2             52


OMB's FY11 decision
                   No, funds in
                    document         No, funds in
 Yes, funds               were     the document
 were                above the        were below     Do not know Not applicable            Number of
 equivalent               APB            the APB                                         respondents
 30                            2              16                 2                 1             51




                                           Page 81                                                     GAO-12-833 Homeland Security
                                              Appendix III: Program Office Survey Results




FY11 budget request submitted to Congress
                     No, funds in
                      document         No, funds in
 Yes, funds                 were     the document
 were                  above the        were below      Do not know Not applicable            Number of
 equivalent                 APB            the APB                                          respondents
 31                              1               15                 1                 4               52


FY11 Congressional enacted funds
                     No, funds in
                      document         No, funds in
 Yes, funds                 were     the document
 were                  above the        were below      Do not know Not applicable            Number of
 equivalent                 APB            the APB                                          respondents
 28                              2               17                 1                 4               52


34. Which of the following events, if any, have attributed to overall funding instability (e.g. a change in planned out-year
funding from one five-year funding plan to the next five-year funding plan)?

1. Check box at left if anticipated funding levels have not changed from one year to the next, and then click here to skip to question 36
                                        Number of
 Not checked             checked      respondents
 61                             10               71


Congressional mark did not match the President's budget
 Event                     Event            Event
 occurred               occurred         occurred
 and out-year      but no change      and out-year
 funding              to out-year         funding       Event did not                         Number of
 increased               funding        decreased              occur Not applicable         respondents
 5                               4               17                17                15               58


Continuing Resolution
 Event                     Event            Event
 occurred               occurred         occurred
 and out-year      but no change      and out-year
 funding              to out-year         funding       Event did not                         Number of
 increased               funding        decreased              occur Not applicable         respondents
 3                              30               10                 2                14               59




                                              Page 82                                                      GAO-12-833 Homeland Security
                                               Appendix III: Program Office Survey Results




Program delays resulting from technical challenges
 Event                     Event             Event
 occurred               occurred          occurred
 and out-year      but no change       and out-year
 funding              to out-year          funding        Event did not                        Number of
 increased               funding         decreased               occur Not applicable        respondents
 0                               12                   3             25               18              58


Mission/requirements change
 Event                     Event             Event
 occurred               occurred          occurred
 and out-year      but no change       and out-year
 funding              to out-year          funding        Event did not                        Number of
 increased               funding         decreased               occur Not applicable        respondents
 3                                7                   7             25               16              58


Another program's acquisition funding levels affected this program's own planned funding
 Event                     Event             Event
 occurred               occurred          occurred
 and out-year      but no change       and out-year
 funding              to out-year          funding        Event did not                        Number of
 increased               funding         decreased               occur Not applicable        respondents
 1                                2                  18             17               19              57


Original cost estimates did not reflect true costs
 Event                     Event             Event
 occurred               occurred          occurred
 and out-year      but no change       and out-year
 funding              to out-year          funding        Event did not                        Number of
 increased               funding         decreased               occur Not applicable        respondents
 4                                7                   3             26               18              58


Other (please specify below)
 Event                     Event             Event
 occurred               occurred          occurred
 and out-year      but no change       and out-year
 funding              to out-year          funding        Event did not                        Number of
 increased               funding         decreased               occur Not applicable        respondents
 2                                0                   3              0               12              17




                                               Page 83                                                     GAO-12-833 Homeland Security
                                              Appendix III: Program Office Survey Results




If other events, please specify:

data intentionally not reported

35. If your program has experienced funding instability (e.g. a change in planned out-year funding from one five-year funding
plan to the next five-year funding plan), did it affect your program in each of the following ways?

Increased costs by less than eight percent
                                                                           Number of
 Yes                               No   Do not know Not applicable       respondents
 10                                24             3                21               58


Increased costs by eight percent or more
                                                                           Number of
 Yes                               No   Do not know Not applicable       respondents
 9                                 24             4                21               58


Resequencing of program's discrete segments, increments or delivery of capabilities
                                                                           Number of
 Yes                               No   Do not know Not applicable       respondents
 29                                 9             2                18               58


Pushed out delivery and caused a schedule breach
                                                                           Number of
 Yes                               No   Do not know Not applicable       respondents
 23                                13             3                18               57


Reduced performance parameters
                                                                           Number of
 Yes                               No   Do not know Not applicable       respondents
 8                                 29             2                16               55


Other (please specify below)
                                                                           Number of
 Yes                               No   Do not know Not applicable       respondents
 7                                  1             0                14               22


If other events, please specify:

data intentionally not reported




                                              Page 84                                            GAO-12-833 Homeland Security
                                             Appendix III: Program Office Survey Results




36. If a gap existed between FY11 enacted funding and FY11 required funding, how effectively were you, as a program
manager, able to directly communicate the impact on your program to DHS and component leadership?

1. Check box at left if your program did not experience a gap between FY11 enacted and required funding, and then click here to skip
to question 37
                                         Number of
 Not checked              checked      respondents
 36                               35            71


Component leadership (Component head, CAE, etc.)
 Very                  Somewhat                                        No direct              Number of
 effectively           effectively Not effectively      No opinion communication            respondents
 19                                5              1                8                    2            35


DHS leadership (Deputy Secretary, USM, PARM officials, etc.)
 Very                  Somewhat                                        No direct              Number of
 effectively           effectively Not effectively      No opinion communication            respondents
 14                               3              1                 7                 10              35


37. If you would like to elaborate on how resource allocation (i.e. funding) has affected the program's ability to achieve cost,
schedule and performance goals, please use the following space.

data intentionally not reported

Section VIII: Workforce

38. Since the program was initially staffed, how many program managers have overseen the program management office
(PMO)?
                                                                          Number of
 Mean                      Median        Minimum          Maximum       respondents
 2.5                              2              0                12               61


1. Check box at left if do not know
                                         Number of
 Not checked              checked      respondents
 63                                8            71




                                             Page 85                                                      GAO-12-833 Homeland Security
                                            Appendix III: Program Office Survey Results




39. What is the number of government FTEs in your PMO for each of the following functional areas?

Program Management

Number of government FTEs staffed at initiation of development activities
                                                                         Number of
 Mean                     Median         Minimum         Maximum       respondents
 3.5                            2                1               15               51


Number of government FTEs currently staffed
                                                                         Number of
 Mean                     Median         Minimum         Maximum       respondents
 4.7                            3                0               23               60


Number of government FTEs currently identified as a need by the program
                                                                         Number of
 Mean                     Median         Minimum         Maximum       respondents
 5.9                            4                1               18               49


Business functions (includes auditing, business, cost estimating, financial management, property management, and purchasing)

Number of government FTEs staffed at initiation of development activities
                                                                         Number of
 Mean                     Median         Minimum         Maximum       respondents
 5.6                            2                1               63               43


Number of government FTEs currently staffed
                                                                         Number of
 Mean                     Median         Minimum         Maximum       respondents
 10.7                           3                1              252               53


Number of government FTEs currently identified as a need by the program
                                                                         Number of
 Mean                     Median         Minimum         Maximum       respondents
 14.4                           4                1              346               47




                                            Page 86                                                  GAO-12-833 Homeland Security
                                             Appendix III: Program Office Survey Results




Engineering and technical (includes systems planning, research, development and engineering; life cycle logistics; test and evaluation;
production, quality and manufacturing; and facilities engineering)

Number of government FTEs staffed at initiation of development activities
                                                                          Number of
 Mean                      Median        Minimum          Maximum       respondents
 4.9                              3               0               25                40


Number of government FTEs currently staffed
                                                                          Number of
 Mean                      Median        Minimum          Maximum       respondents
 15.5                             8               1              150                48


Number of government FTEs currently identified as a need by the program
                                                                          Number of
 Mean                      Median        Minimum          Maximum       respondents
 13.7                             6               1               98                43


40. Please describe the source(s) and/ or method(s) used for identifying the numbers of FTEs in question 39.

data intentionally not reported

41. Please use the following space to comment on how personnel shortfalls, if any, have affected your program.

1. Check box at left if there have been no personnel shortages, and then click here to skip to question 42
                                         Number of
 Not checked             checked       respondents
 52                               19             71


data intentionally not reported




                                             Page 87                                                         GAO-12-833 Homeland Security
                                          Appendix III: Program Office Survey Results




Section IX: Communication to DHS
42. How effective, if at all, are each of the following tools for communicating your program's performance to DHS leadership?

Next-generation Periodic Reporting System (nPRS)
                        Somewhat       Not at all                                         Number of
 Very effective          effective     effective     No opinion        Do not use       respondents
 8                                34          13               10                 4             69


Investment Management System (IMS)
                        Somewhat       Not at all                                         Number of
 Very effective          effective     effective     No opinion        Do not use       respondents
 10                               35           7               13                 4             69


ARB reviews
                        Somewhat       Not at all                                         Number of
 Very effective          effective     effective     No opinion        Do not use       respondents
 25                               27           2                7                 8             69


Informal communication with DHS HQ
                        Somewhat       Not at all                                         Number of
 Very effective          effective     effective     No opinion        Do not use       respondents
 28                               23           1                6               11              69


Other (please specify below)
                        Somewhat       Not at all                                         Number of
 Very effective          effective     effective     No opinion        Do not use       respondents
 8                                 2           0                2                 3             15


If other tools, please specify:

data intentionally not reported




                                          Page 88                                                     GAO-12-833 Homeland Security
                                             Appendix III: Program Office Survey Results




43. If you reported that any of the tools listed above are not at all effective in communicating your program's current
performance to DHS headquarters, or if you have any concerns related to communication tools, please elaborate below.

data intentionally not reported

44. For the data entered into nPRS, who is responsible for entering and/or validating the information?

1. Check box at left if your program does not report into nPRS, and then click here to skip to question 45
                                         Number of
 Not checked             checked       respondents
 65                                6             71


Program Manager
                                   Does not have
 Enters                             a role in data
 Information             Validates        entry or                         Number of
                      Information      validation      Do not know       respondents
 4                                50              6                 1               61


Program Management Staff other than Program Manager
                                   Does not have
 Enters                             a role in data
 Information             Validates        entry or                         Number of
                      Information      validation      Do not know       respondents
 37                               17              5                 1               60


Contractor in the Program Management Office
                                   Does not have
 Enters                             a role in data
 Information             Validates        entry or                         Number of
                      Information      validation      Do not know       respondents
 39                                1             15                 1               56


Other (please specify below)
                                   Does not have
 Enters                             a role in data
 Information             Validates        entry or                         Number of
                      Information      validation      Do not know       respondents
 3                                 3              6                 3               15


If other personnel, please specify:

data intentionally not reported




                                             Page 89                                                         GAO-12-833 Homeland Security
                                              Appendix III: Program Office Survey Results




45. For the data entered into IMS, who is responsible for entering and/or validating the information?

1. Check box at left if your program does not report into IMS, and then click here to skip to question 46
                                         Number of
 Not checked             checked       respondents
 66                                5              71


Program Manager
                                   Does not have
 Enters                             a role in data
 Information             Validates        entry or                         Number of
                      Information      validation       Do not know      respondents
 1                                51               7                2                61


Program Management Staff other than Program Manager
                                   Does not have
 Enters                             a role in data
 Information             Validates        entry or                         Number of
                      Information      validation       Do not know      respondents
 30                               19               6                4                59


Contractor in the Program Management Office
                                   Does not have
 Enters                             a role in data
 Information             Validates        entry or                         Number of
                      Information      validation       Do not know      respondents
 38                                4              11                4                57


Other (please specify below)
                                   Does not have
 Enters                             a role in data
 Information             Validates        entry or                         Number of
                      Information      validation       Do not know      respondents
 1                                 3               7                5                16


If other personnel, please specify:

data intentionally not reported




                                              Page 90                                                       GAO-12-833 Homeland Security
                                             Appendix III: Program Office Survey Results




Section X: DHS Acquisition Management Initiatives
46. How familiar are you with each of the following DHS initiatives?

Establishing the Integrated Investment Life Cycle Model (IILCM)
                      Somewhat           Not at all                       Number of
 Very familiar          familiar          familiar      No opinion      respondents
 9                             24               32                 2               67


Establishing PARM
                      Somewhat           Not at all                       Number of
 Very familiar          familiar          familiar      No opinion      respondents
 19                            38                9                 2               68


Establishing the Program Manager Corps
                      Somewhat           Not at all                       Number of
 Very familiar          familiar          familiar      No opinion      respondents
 12                            22               31                 2               67


Empowering the Component Acquisition Executives (CAEs)
                      Somewhat           Not at all                       Number of
 Very familiar          familiar          familiar      No opinion      respondents
 19                            27               18                 2               66


Establishing Functional Coordination Office (e.g. Screening Coordination Office)
                      Somewhat           Not at all                       Number of
 Very familiar          familiar          familiar      No opinion      respondents
 8                             11               45                 2               66


Creating Executive Steering Councils for program governance
                      Somewhat           Not at all                       Number of
 Very familiar          familiar          familiar      No opinion      respondents
 23                            21               23                 1               68


Forming the Capabilities and Requirements Council
                      Somewhat           Not at all                       Number of
 Very familiar          familiar          familiar      No opinion      respondents
 4                             22               40                 1               67




                                             Page 91                                       GAO-12-833 Homeland Security
                                             Appendix III: Program Office Survey Results




Developing APEX, a decision support tool owned by PARM to capture and synthesize information from nPRS and IMS
                      Somewhat           Not at all                       Number of
 Very familiar          familiar          familiar      No opinion      respondents
 5                             24               37                 2               68


47. How helpful, if at all, will the following DHS initiatives be in helping you manage your acquisition program?

Establishing the Integrated Investment Life Cycle Model (IILCM)
                      Somewhat           Not at all                       Number of
 Very helpful            helpful          helpful       No opinion      respondents
 8                             18                8                32               66


Establishing PARM
                      Somewhat           Not at all                       Number of
 Very helpful            helpful          helpful       No opinion      respondents
 9                             32                5                21               67


Establishing the Program Manager Corps
                      Somewhat           Not at all                       Number of
 Very helpful            helpful          helpful       No opinion      respondents
 13                            20                6                26               65


Empowering the Component Acquisition Executives (CAEs)
                      Somewhat           Not at all                       Number of
 Very helpful            helpful          helpful       No opinion      respondents
 25                            21                7                13               66


Establishing Functional Coordination Office (e.g. Screening Coordination Office)
                      Somewhat           Not at all                       Number of
 Very helpful            helpful          helpful       No opinion      respondents
 4                             13                9                37               63




                                             Page 92                                               GAO-12-833 Homeland Security
                                           Appendix III: Program Office Survey Results




Creating Executive Steering Councils for program governance
                       Somewhat        Not at all                       Number of
 Very helpful             helpful       helpful       No opinion      respondents
 18                               17           8                24               67


Forming the Capabilities and Requirements Council
                       Somewhat        Not at all                       Number of
 Very helpful             helpful       helpful       No opinion      respondents
 7                                14           8                36               65


Developing APEX, a decision support tool owned by PARM to capture and synthesize information from nPRS and IMS
                       Somewhat        Not at all                       Number of
 Very helpful             helpful       helpful       No opinion      respondents
 4                                16           9                38               67


48. Please use the following space to describe any additional actions that DHS could implement that would help you better
manage your acquisition program (i.e. improvements for acquisition governance and document development).

data intentionally not reported


Section XI: Summary Statements
49. Please identify any significant challenges affecting your program's ability to achieve program objectives (i.e. cost,
schedule, and capabilities) that have not been adequately addressed above.

data intentionally not reported

50. If you would like, please identify any practices your program has found significantly helpful in managing your program.

data intentionally not reported




                                           Page 93                                                  GAO-12-833 Homeland Security
Appendix IV: Major DHS Acquisition
                                              Appendix IV: Major DHS Acquisition Programs
                                              and their Key Acquisition Documents



Programs and their Key Acquisition
Documents
                                              Table 6 below identifies the 71 major Department of Homeland Security
                                              (DHS) acquisition programs that responded to our survey. It consists of all
                                              the programs DHS included in its 2011 Major Acquisition Oversight List,
                                              with the exception of the 6 programs that did not respond to our survey
                                              (see table 7), and the 5 programs that were cancelled in 2011 (see table
                                              8). Table 6 also identifies whether each program’s Mission Need
                                              Statement (MNS), Operational Requirements Document (ORD),
                                              Acquisition Program Baseline (APB), Integrated Logistics Support Plan
                                              (ILSP), and Test and Evaluation Master Plan (TEMP) have been
                                              approved at the department level.

Table 6: Programs that responded to our survey

                                                                              Information     Department-approved documents
                                                                              Technology
Sponsor                     Program                                 Level   (IT) vs. Non-IT   MNS   ORD   APB    ILSP   TEMP
Analysis and Operations     Common Operational Picture (COP)
                                                                      2            IT
(A&O)
Chief Administrative        St. Elizabeth’s
                                                                      2         Non-IT
Officer (CAO)
Chief Human Capital         HR-IT
                                                                      2            IT
Officer
Chief Information Officer   Infrastructure Transformation
                                                                      1            IT
                            Program (ITP)
Customs and Border          Automated Commercial Environment
Protection (CBP)            (ACE) / International Trade Data          1            IT          X
                            System (ITDS)
CBP                         Automated Targeting System (ATS)
                                       h                              2            IT
                            Maintenance
CBP                         Border Patrol Facilities                  1         Non-IT
CBP                         Facilities Management and
                            Engineering Tactical Infrastructure       1         Non-IT
                            (FM&E TI)
CBP                         Fleet Management Program (FMP)h           1         Non-IT
CBP                         Land Ports of Entry Modernization         2         Non-IT
CBP                         Non-Intrusive Inspection (NII)
                                                                      1            IT
                            Systems Program
                                h
CBP                         SAP                                       2            IT
CBP                         Strategic Air and Marine Plan             1         Non-IT
CBP                         Tactical Communication (TAC-COM)          2            IT          X
CBP                         TECS Modernizationd                       1            IT          X                   X       X
CBP                         Transportation                            1         Non-IT                      X




                                              Page 94                                               GAO-12-833 Homeland Security
                                           Appendix IV: Major DHS Acquisition Programs
                                           and their Key Acquisition Documents




                                                                             Information     Department-approved documents
                                                                             Technology
Sponsor                    Program                                 Level   (IT) vs. Non-IT   MNS   ORD   APB    ILSP   TEMP
CBP                        Western Hemisphere Travel
                                                                    1            IT                        X      X
                           Initiative (WHTI)
Domestic Nuclear           Advanced Spectroscopic Portal
                                                                    1          Non-IT
Detection Office           (ASP)
Federal Emergency          Housing Inspection Services (HIS)
Management Agency                                                   2          Non-IT
(FEMA)
FEMA                       Integrated Public Alert and Warning
                                                                    2          Non-IT               X
                           System (IPAWS)
FEMA                       Logistics Supply Chain Management
                                                                    2            IT
                           System (LSCMS)
FEMA                       Risk Mapping, Analysis and
                                                                    1          Non-IT                      X      X
                           Planning (Risk Map)a
Immigration and Customs    Atlas
                                                                    2            IT
Enforcement (ICE)
ICE                        Detention and Removal Operations
                                                                    2            IT           X
                           (DROM)
ICE                        DRO Electronic Health Record
                                                                    2            IT
                           (EHR) System
ICE                        Enforcement Information Sharing
                                                                    2            IT
                           (EIS)
ICE                        Student and Exchange Visitor
                                                            d       2            IT
                           Information System (SEVIS I & II)
ICE                        TECS Modernization                       1            IT           X            X      X       X
National Protection and    Federal Protective Services
Programs Directorate                                                2          Non-IT
(NPPD)
NPPD                       Infrastructure Information Collection
                           Program and Visualization (IICV          2            IT           X
                           IICP)g
NPPD                       National Cybersecurity and
                                                                    1            IT           X     X      X      X       X
                           Protection System (NCPS)
NPPD                       Next Generation Network (NGN)            1            IT           X     X      X      X       X
NPPD                       United States Visitor and Immigrant
                           Status Indicator Technology (US-         1            IT           X     X      X      X
                           VISIT)
Office of Health Affairs   Bio Watch Gen-3                          1          Non-IT
Science and Technology     National Bio and Agro-Defense
                                                                    1          Non-IT
(S&T)                      Facility (NBAF)
S&T                        National Biodefense Analysis and
                           Countermeasures Center (NBACC)           1          Non-IT
                           Facility




                                           Page 95                                                 GAO-12-833 Homeland Security
                                          Appendix IV: Major DHS Acquisition Programs
                                          and their Key Acquisition Documents




                                                                           Information     Department-approved documents
                                                                           Technology
Sponsor                   Program                                Level   (IT) vs. Non-IT   MNS   ORD   APB    ILSP   TEMP
Transportation Security   Electronic Baggage Screening
                                                                  1          Non-IT                             X
Administration (TSA)      Program (EBSP)
TSA                       Field Real Estate Management
                                                                  2          Non-IT
                          (FREM)
                                    h
TSA                       HRAccess                                1          Non-IT
TSA                       National Explosives Detection
                                                                  2          Non-IT
                          Canine Team Program (K9) System
TSA                       Information Technology
                                                       h          1            IT
                          Infrastructure Program (ITIP)
TSA                       Passenger Screening Program
                                                                  1          Non-IT         X     X      X      X       X
                          (PSP)
                                                            h
TSA                       Screening Partnership Program           1          Non-IT
TSA                       Secure Flight                           1            IT           X            X      X
TSA                       Security Technology Integrated
                                                                  2            IT                 X      X
                          Program (STIP)
TSA                       Specialized Trainingh                   2          Non-IT
TSA                       Transportation Worker Identification
                                              h                   1            IT
                          Credentialing (TWIC)
TSA                       TTAC Infrastructure Modernization
                                                                  2            IT           X     X      X      X
                          (TIM)
United States Coast       C4ISRd
                                                                  1            IT           X     X             X       X
Guard (USCG)
USCG                      CG Logistics information
                                                      e           2            IT                 X
                          Management System (CG-LIMS)
USCG                      Core Accounting System (CAS)h           2            IT
                                                        d
USCG                      Fast Response Cutter (FRC)              1          Non-IT         X     X             X
USCG                      HC-130H Conversion/Sustainment
                                                                  1          Non-IT         X            X              X
                          Projects
USCG                      HC-130 J Fleet Introduction             2          Non-IT         X            X      X
USCG                      HC-144A Maritime Patrol Aircraft
                                                                  1          Non-IT         X     X             X       X
                          (MPA)d
USCG                      HH-60 Conversion Projectsd              1          Non-IT         X     X             X       X
USCG                      HH-65 Conversion/Sustainment
                                                                  1          Non-IT         X            X      X       X
                          Projects
USCG                      Interagency Operations Center
                                                                  2            IT           X     X      X      X
                          (IOC)
USCG                      Medium Endurance Cutter
                                      b                           1          Non-IT         X
                          Sustainment
USCG                      National Security Cutter (NSC)          1          Non-IT         X            X              X
USCG                      Nationwide Automatic Identification
                                        d                         1            IT           X                           X
                          System (NAIS)




                                          Page 96                                                GAO-12-833 Homeland Security
                                             Appendix IV: Major DHS Acquisition Programs
                                             and their Key Acquisition Documents




                                                                                              Information     Department-approved documents
                                                                                              Technology
Sponsor                     Program                                          Level          (IT) vs. Non-IT   MNS   ORD   APB     ILSP     TEMP
USCG                        Offshore Patrol Cutter (OPC)                        1               Non-IT         X     X      X       X        X
                                                          b
USCG                        Patrol Boats Sustainment                            2               Non-IT         X
USCG                        Rescue 21                                           1                 IT                        X       X
USCG                        Response Boat – Medium (RB-M)b                      1               Non-IT
USCG                        Unmanned Aircraft Systems (UAS)c, f                 1               Non-IT         X
United States Citizenship   Application Support Center (ASC)h
and Immigration Services                                                        2               Non-IT         X            X
(USCIS)
USCIS                       Benefit Provision – Verification
                                                                                2                 IT           X
                            Information System (VIS)
USCIS                       Integrated Document Production
                                  h                                             2                 IT
                            (IDP)
USCIS                       Transformation                                      1                 IT                 X      X
United States Secret        IT Modernization (ITM)
                                                                                2                 IT                 X      X       X
Service

                                             Legend:

                                                     Document was required at the time of our review.
                                                     Document was not required at the time of our review.

                                             Source: GAO analysis of DHS and survey data.
                                             a
                                                 Program’s document requirements were waived because it is in sustainment.
                                             b
                                                 Program’s document approval authority was delegated to a component.
                                             c
                                                 Program does not yet require any documents because it is in the Need phase.
                                             d
                                              DHS reported that the program breached its baseline after the department approved its APB. An
                                             APB breach of cost, schedule, or performance is an inability to meet the threshold value set for each.
                                             Within 90 days of the breach, DHS acquisition policy requires that either a new APB be approved,
                                             baseline-revision recommendations be made to the Acquisition Decision Authority, or the program is
                                             back within APB parameters.
                                             e
                                                 Program was designated a non-major acquisition program April 2012.
                                             f
                                             Program has two MNSs. The land-based MNS has been approved at the department-level. The
                                             cutter-based MNS has not.
                                             g
                                              Program was reorganized as the Critical Infrastructure Technology and Architecture (CITA) program
                                             in fiscal year 2012.
                                             h
                                                 PARM officials stated the program was in sustainment at the time AD 102 was signed.




                                             Page 97                                                                GAO-12-833 Homeland Security
                                            Appendix IV: Major DHS Acquisition Programs
                                            and their Key Acquisition Documents




                                            Table 7 identifies the programs that were included in DHS’s 2011 Major
                                            Acquisition Oversight List, but did not respond to our survey.

Table 7: Programs that did not respond to our survey

Sponsor               Program                                                              Level           IT vs. Non-IT
A&O                   Homeland Security Information Network (HSIN)                          2              IT
CBP                   Advanced Passenger Information System (APIS)                          2              IT
I&A                   National Security System Program (NSSP)                               1              IT
ICE                   Detention and Removal Operations (DRO)                                2              Non-IT
ICE                   Tactical Communications (TAC-COM)                                     1              IT
USCG                  Coastal Patrol Boat                                                   1              Non-IT
                                            Source: GAO analysis of DHS and survey data.



                                            Table 8 identifies the programs that were included in DHS’s 2011 Major
                                            Acquisition Oversight List, but were cancelled in 2011.

Table 8: Programs cancelled in 2011

Sponsor                              Program                                                       Level        IT vs. Non-IT
CAO                                  Electronic Records Management System (ERMS)                    2           IT
Chief Financial Officer              Transformation and Systems Consolidation (TASC)                1           IT
CBP                                  Secure Border Initiative Network (SBInet)                      1           IT
FEMA                                 Grants Management Integrated Environment (GMIE)                2           IT
Intelligence and Analysis (I&A)      Online Tracking Information System                             2           IT
                                            Source: GAO analysis of DHS data.




                                            Page 98                                                  GAO-12-833 Homeland Security
Appendix V: Comments from the Department
             Appendix V: Comments from the Department
             of Homeland Security



of Homeland Security




             Page 99                                    GAO-12-833 Homeland Security
Appendix V: Comments from the Department
of Homeland Security




Page 100                                   GAO-12-833 Homeland Security
Appendix V: Comments from the Department
of Homeland Security




Page 101                                   GAO-12-833 Homeland Security
Appendix V: Comments from the Department
of Homeland Security




Page 102                                   GAO-12-833 Homeland Security
Appendix VI: GAO Contact and Staff
                  Appendix VI: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  John Hutton, (202) 512-4841 or huttonj@gao.gov
GAO Contact
                  In addition to the contact named above, Katherine Trimble (Assistant
Staff             Director), Nathan Tranquilli (Analyst-in-Charge), John Crawford, David
Acknowledgments   Garcia, Jill Lacey, Sylvia Schatz, Rebecca Wilson, Candice Wright, and
                  Andrea Yohe made key contributions to this report.




                  Page 103                                       GAO-12-833 Homeland Security
Related GAO Products
             Related GAO Products




             Immigration Benefits: Consistent Adherence to DHS’s Acquisition Policy
             Could Help Improve Transformation Program Outcomes. GAO-12-66.
             Washington, D.C.: November 22, 2011.

             Coast Guard: Action Needed As Approved Deepwater Program Remains
             Unachievable. GAO-11-743. Washington, D.C.: July 28, 2011.

             Opportunities to Reduce Potential Duplication in Government Programs,
             Save Tax Dollars, and Enhance Revenue. GAO-11-318SP. Washington,
             D.C.: March 1, 2011.

             High-Risk Series: An Update. GAO-11-278. Washington, D.C.: February
             2011.

             Defense Acquisitions: Assessments of Selected Weapon Programs.
             GAO-11-233SP. Washington, D.C.: March 29, 2011.

             Secure Border Initiative: Controls over Contractor Payment for the
             Technology Component Need Improvement. GAO-11-68. Washington,
             D.C.: May 25, 2011.

             Department of Homeland Security: Assessments of Selected Complex
             Acquisitions. GAO-10-588SP. Washington, D.C.: June 30, 2010.

             The Office of Management and Budget’s Acquisition Workforce
             Development Strategic Plan for Civilian Agencies. GAO-10-459R.
             Washington, D.C.: April 23, 2010.

             Defense Acquisitions: Measuring the Value of DOD’s Weapon Programs
             Requires Starting with Realistic Baselines. GAO-09-543T. Washington,
             D.C.: April 1, 2009.

             Department of Homeland Security: Billions Invested in Major Programs
             Lack Appropriate Oversight. GAO-09-29. Washington, D.C.: November
             18, 2008.

             Homeland Security: Challenges in Creating an Effective Acquisition
             Organization. GAO-06-1012T. Washington, D.C.: July 27, 2006.

             Homeland Security: Successes and Challenges in DHS’s Efforts to
             Create an Effective Acquisition Organization. GAO-05-179. Washington,
             D.C.: March 29, 2005.



             Page 104                                       GAO-12-833 Homeland Security
           Related GAO Products




           Results-Oriented Cultures: Implementation Steps to Assist Mergers and
           Organizational Transformations. GAO-03-669. Washington, D.C.: July 2,
           2003

           Best Practices: Better Matching of Needs and Resources Will Lead to
           Better Weapon System Outcomes. GAO-01-288. Washington, D.C.:
           March 8, 2001.




(121010)
           Page 105                                       GAO-12-833 Homeland Security
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (http://www.gao.gov). Each weekday
GAO Reports and       afternoon, GAO posts on its website newly released reports, testimony,
                      and correspondence. To have GAO e-mail you a list of newly posted
Testimony             products, go to http://www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: http://www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.