oversight

Defense Management: Guidance and Progress Measures Are Needed to Realize Benefits from Changes in DOD's Joint Requirements Process

Published by the Government Accountability Office on 2012-02-24.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States Government Accountability Office

GAO             Report to Congressional Committees




February 2012
                DEFENSE
                MANAGEMENT
                Guidance and
                Progress Measures
                Are Needed to Realize
                Benefits from Changes
                in DOD’s Joint
                Requirements Process




GAO-12-339
                                              February 2012

                                              DEFENSE MANAGEMENT
                                              Guidance and Progress Measures Are Needed to
                                              Realize Benefits from Changes in DOD’s Joint
                                              Requirements Process
Highlights of GAO-12-339, a report to
congressional committees




Why GAO Did This Study                        What GAO Found
The Department of Defense’s Joint             After studying the Joint Capabilities Integration and Development System
Requirements Oversight Council                (JCIDS) process since September 2010, the Joint Staff began initiating actions in
(JROC) is charged with assisting in the       October 2011 to better prioritize capability needs and align those needs with
prioritization of capability needs from a     available budgetary resources. Specifically, according to Joint Staff officials, the
joint perspective and helping guide           Joint Requirements Oversight Council (JROC) has begun to consider the
investments. The JROC is supported            benefits and affordability of new capabilities within the context of joint capability
by the Joint Capabilities Integration         areas and to evaluate possible duplication before validating new capability
and Development System (JCIDS)                requirements. The Joint Staff has begun to implement a new approach to support
process. However, a congressional
                                              JROC prioritization of capability needs, but the new approach is still evolving and
committee and GAO have expressed
                                              has not been fully developed and clearly documented. New guidance does not
concerns about the extent to which
JCIDS has been effective in prioritizing
                                              clearly outline goals of the new approach, develop and communicate the analytic
capability needs. The Ike Skelton             approach envisioned to support JROC decision making, or set out criteria and
National Defense Authorization Act for        accompanying measures of progress. GAO previously reported that JCIDS’s
Fiscal Year 2011 required GAO to              ability to prioritize needs could be improved if it had an analytic approach to
provide a report on the effectiveness of      reviewing and validating proposals that would help ensure that the most
JCIDS in several areas. In addition to        important capability needs of the department are addressed. Until the Joint Staff
responding to this direction, GAO has         takes steps to fully develop, document, and institutionalize the new analytic
more broadly evaluated the extent to          approach, it is not clear whether the current momentum for improving the JCIDS
which (1) the Joint Staff has developed       process will be sustained.
and implemented an analytic approach
to prioritize capability needs and (2)
the JROC has considered aspects of            JCIDS guidance in effect through December 2011 required that sponsors of
the availability and operational support      potential major defense acquisition programs address sustainment information in
of weapon systems—called                      capability development documents according to four metrics—materiel
sustainment—when validating the               availability, operational availability, reliability, and ownership cost. Each of these
requirements of proposed capability           metrics includes a set of potentially reportable criteria or data, which are listed as
solutions. To do so, GAO analyzed             review criteria and are suggested, but not clearly required by the guidance, to be
capability documents, reviewed                included in the metric. Based on GAO’s analysis of six capability development
relevant guidance and law, and                documents, GAO found that all of the documents provided information on the
interviewed officials.                        four required sustainment metrics, but the completeness of information for all of
                                              the metrics’ key criteria varied. Further, in some cases information that should
What GAO Recommends                           have been included, according to Department of Defense officials, was not
GAO recommends that the Vice                  provided. The Joint Staff issued updated JCIDS guidance in January 2012, but
Chairman of the Joint Chiefs of Staff         the guidance still does not clearly require program sponsors to report on the
(1) revise and implement JCIDS                individual criteria for each of the four sustainment metrics. Without complete and
guidance to reflect recent changes to         detailed information on each of the individual criteria elements, the JROC may
the process and establish criteria and        not have the information it needs to make the most informed decisions when
measures for determining the relative         validating the requirements of proposed solutions intended to mitigate capability
importance of capability needs and (2)        gaps.
require program sponsors to address
each criterion in JCIDS guidance
related to sustainment in capability
documents. DOD partially concurred
with GAO’s recommendations.

View GAO-12-339. For more information,
contact John Pendleton at (202) 512-3489 or
pendletonj@gao.gov.

                                                                                        United States Government Accountability Office
Contents


Letter                                                                                     1
               Background                                                                  3
               The Joint Staff Has Initiated Efforts to Prioritize Capability Needs,
                 but Implementation Processes Are Not Fully Developed and
                 Clearly Documented                                                        8
               JROC Has Not Always Had Complete Information on Future Costs
                 to Sustain Proposed Capabilities before Approving
                 Documentation for Program Development                                    14
               Conclusions                                                                20
               Recommendations for Executive Action                                       21
               Agency Comments and Our Evaluation                                         21

Appendix I     Observations on Reporting Provisions in the National Defense
               Authorization Act for Fiscal Year 2011                                     25



Appendix II    Scope and Methodology                                                      37



Appendix III   Comments from the Department of Defense                                    40



Appendix IV    GAO Contact and Staff Acknowledgments                                      43



Tables
               Table 1: Elements of a Results-Oriented Management Framework               13
               Table 2: Examples of Review Criteria for Each of the Four
                        Sustainment Metrics                                               17
               Table 3: Section 862(b)(1)(A)—Evaluate the Effectiveness of
                        JCIDS in Achieving Timeliness in Delivering Capability to
                        the Warfighter                                                    27
               Table 4: Section 862(b)(1)(B)—Evaluate the Effectiveness of
                        JCIDS in Achieving Efficient Use of the Investment
                        Resources of the Department of Defense                            28
               Table 5: Section 862(b)(1)(C)—Evaluate the Effectiveness of
                        JCIDS in Achieving Control of Requirements Creep—and
                        Section 862(b)(1)(D)—Evaluate the Effectiveness of



               Page i                                           GAO-12-339 Defense Management
                  JCIDS in Achieving Responsiveness to Changes Occurring
                  after the Approval of a Requirements Document (Including
                  Changes to the Threat Environment, the Emergence of
                  New Capabilities, or Changes in the Resources Estimated
                  to Procure or Sustain a Capability)                           29
         Table 6: Section 862(b)(1)(E)—Evaluate the Effectiveness of
                  JCIDS in Achieving Development of the Personnel Skills,
                  Capacity, and Training Needed for an Effective and
                  Efficient Requirements Process                                30
         Table 7: Section 862(b)(2)(A)—the Time That Requirements
                  Documents Take to Receive Approval through JCIDS              31
         Table 8: Section 862(b)(2)(B)—the Quality of Cost Information
                  Considered in JCIDS and the Extent of Its Consideration       32
         Table 9: Section 862(b)(2)(C)—the Extent to Which JCIDS
                  Establishes a Meaningful Level of Priority for
                  Requirements                                                  33
         Table 10: Section 862(b)(2)(D)—the Extent to Which JCIDS Is
                  Considering Trade-offs between Cost, Schedule, and
                  Performance Objectives                                        34
         Table 11: Section 862(b)(2)(E)—the Quality of Information on
                  Sustainment Considered in JCIDS and the Extent to Which
                  Sustainment Information Is Considered                         35
         Table 12: Section 862(b)(2)(F)—an Evaluation of the Advantages
                  and Disadvantages of Designating a Commander of a
                  Unified Combatant Command for Each JROC-Interest
                  Requirements Document to Provide a Joint Evaluation
                  Task Force for Certain Purposes                               36


Figure
         Figure 1: Interaction between DOD’s Requirements Generation and
                  Acquisition Processes                                          6




         Page ii                                      GAO-12-339 Defense Management
Abbreviations

CJCSI          Chairman of the Joint Chiefs of Staff Instruction
DOD            Department of Defense
JCIDS          Joint Capabilities Integration and Development System
JROC           Joint Requirements Oversight Council



This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page iii                                                 GAO-12-339 Defense Management
United States Government Accountability Office
Washington, DC 20548




                                   February 24, 2012

                                   Congressional Committees

                                   Like the rest of the federal government, the Department of Defense
                                   (DOD) is operating in a constrained budget environment and facing
                                   difficult decisions about how to invest its resources to meet national
                                   security objectives. The Joint Requirements Oversight Council (JROC)
                                   directly and indirectly supports the Chairman of the Joint Chiefs of Staff’s
                                   role as the principal military advisor to the President and other senior
                                   national security officials. The JROC consists of the Vice Chairman of the
                                   Joint Chiefs of Staff and other senior military officials and, among other
                                   duties, is expected to assist in the prioritization of joint military
                                   requirements and ensure that resource levels needed to fulfill those
                                   requirements are consistent with the level of priority. These duties can
                                   help ensure that proposed weapon systems are prioritized to create an
                                   affordable portfolio. The JROC uses the Joint Capabilities Integration and
                                   Development System (JCIDS), a process created in 2003 to guide the
                                   development of capabilities from a joint perspective, to help it identify
                                   capability gaps and validate the requirements of proposed capability
                                   solutions to mitigate those gaps. JCIDS operates in conjunction with two
                                   other DOD processes—the acquisition process, which facilitates the
                                   development and acquisition of proposed capabilities, and the budgeting
                                   process, through which these capabilities are funded. The development
                                   and acquisition of weapon systems consume a substantial portion of
                                   DOD’s budget. From fiscal years 2011 through 2016, DOD plans to spend
                                   almost $390 billion on its current portfolio of major defense acquisition
                                   programs. 1 Moreover, DOD spends billions of dollars each year to sustain
                                   these weapon systems. As we have reported, costs to operate and
                                   support these systems make up at least 70 percent of a system’s life
                                   cycle costs. 2 The House Armed Services Committee and GAO have


                                   1
                                    Major defense acquisition programs are those programs identified by DOD that are
                                   estimated to require eventual total expenditure for research, development, test, and
                                   evaluation, including all planned increments, of more than $365 million or expenditure for
                                   procurement, including all planned increments, of more than $2.19 billion in fiscal year
                                   2000 constant dollars. Programs may also be designated by DOD as major defense
                                   acquisition programs.
                                   2
                                    GAO, Defense Management: DOD Needs Better Information and Guidance to More
                                   Effectively Manage and Reduce Operating and Support Costs of Major Weapon Systems,
                                   GAO-10-717 (Washington, D.C.: July 20, 2010).




                                   Page 1                                                    GAO-12-339 Defense Management
expressed concerns about the extent to which JCIDS has been effective
in assigning priorities to capabilities proposed to meet the needs of the
joint force.

A provision in the Ike Skelton National Defense Authorization Act for
Fiscal Year 2011 required GAO to provide a report on the effectiveness of
JCIDS in several areas, such as prioritizing joint requirements and
considering information on costs of sustaining future programs. 3 In
appendix I, we have reprinted the elements of the provision to be covered
by the report and our response to each one. 4 As part of our analysis, we
also evaluated (1) the extent to which the Joint Staff has developed and
implemented an analytic approach to prioritize capability needs and
(2) the extent to which the JROC has considered aspects of the
availability and operational support requirements of weapon systems—
called sustainment—when validating the requirements of proposed
capability solutions.

To assess the extent to which the Joint Staff has developed and
implemented an analytic approach to prioritize capability needs, we
reviewed relevant law and Joint Staff policy documents and related
guidance on the roles and responsibilities of the JCIDS process and the
JROC as it pertains to prioritization. We interviewed Joint Staff officials to
discuss changes to its approach for managing requirements within
capability portfolios and making recommendations for trade-offs among
alternatives. We also reviewed prior GAO reports that discussed
capability development and requirements prioritization and compared the
current efforts to minimize risks and identify unnecessary overlap and
duplication with what we reported in the past. To assess the extent to
which the JROC has considered aspects of the availability and
operational support requirements of weapon systems—called
sustainment—when validating the requirements of proposed capability
solutions, we reviewed DOD and Joint Staff policy documents and related
guidance on the requirement to develop sustainment metrics for capability
documents for certain programs processed through JCIDS. We
interviewed DOD and Joint Staff officials to discuss the preparation,
presentation, and consideration of sustainment data. We also selected a


3
See Pub. L. No. 111-383, § 862 (2011).
4
 We provided a draft of this report to the congressional defense committees on January 6,
2012, to satisfy this requirement.




Page 2                                                  GAO-12-339 Defense Management
             nonprobability sample of six requirements documents for programs that
             have been required to report sustainment metrics to determine whether
             the documents contained the required metrics and the supporting
             information included with the metrics.

             We conducted this performance audit from April 2011 through February
             2012 in accordance with generally accepted government auditing
             standards. Those standards require that we plan and perform the audit to
             obtain sufficient, appropriate evidence to provide a reasonable basis for
             our findings and conclusions based on our audit objectives. We believe
             that the evidence obtained provides a reasonable basis for our findings
             and conclusions based on our audit objectives. Additional details on our
             scope and methodology are in appendix II.


             JCIDS; the Planning, Programming, Budgeting and Execution process;
Background   and the Defense Acquisition System broadly make up DOD’s overall
             defense acquisition management framework. JCIDS was implemented in
             2003 to guide future defense programs from a joint capabilities
             perspective. JCIDS is one of the first steps in DOD’s acquisition
             processes; JCIDS participants work to identify and determine whether to
             validate the need for capabilities proposed by the services, the defense
             agencies, and the combatant commands. Once a requirement is
             validated, the services rely on the DOD’s Planning, Programming,
             Budgeting and Execution process, through which DOD allocates financial
             resources across the department—including the services—to identify
             funding for validated capability solutions. DOD then manages the
             development and procurement of proposed capabilities through the
             Defense Acquisition System.

             DOD implemented the JCIDS process in 2003 in an effort to assist the
             JROC by changing DOD’s requirements validation process from a
             service-specific perspective to a joint capabilities perspective. The JROC,
             which is chaired by the Vice Chairman of the Joint Chiefs of Staff,
             consists of a general or admiral from each of the military services and
             may include combatant commanders or deputy commanders when




             Page 3                                         GAO-12-339 Defense Management
directed by the JROC Chairman. 5 The JROC is charged with assisting the
Chairman of the Joint Chiefs of Staff with a number of tasks, including
(1) identifying, assessing, and approving joint military requirements to
meet the national military strategy; (2) establishing and assigning priority
levels for joint military requirements; and (3) reviewing the estimated level
of resources required to fulfill each joint military requirement and ensuring
that the resource level is consistent with the requirement’s priority, among
others. The JROC also assists acquisition officials in identifying
alternatives to any acquisition programs that meet joint military
requirements for the purposes of certain statutory provisions, addressing
matters such as cost growth. In 2009, the Weapon Systems Acquisition
Reform Act expanded the role of the JROC by directing it to assist the
Chairman of the Joint Chiefs of Staff in ensuring that trade-offs among
cost, schedule, and performance objectives are considered for joint
military requirements and establishing an objective for the overall period
of time within which an initial operational capability should be delivered to
meet each joint military requirement. 6

The JROC reviews requirements for programs designated as JROC-
interest based on their expected cost and complexity 7 and, under


5
 The following officials serve as advisors to the JROC on matters within their authority and
expertise: the Under Secretary of Defense for Acquisition, Technology and Logistics; the
Under Secretary of Defense (Comptroller); the Under Secretary of Defense for Policy; the
Director of Cost Assessment and Program Evaluation; the Director of Operational Test
and Evaluation; and such other civilian officials of DOD as designated by the Secretary of
Defense. See 10 U.S.C. § 181(d)(1). The JROC must also seek and consider input from
the commanders of the combatant commands in carrying out several of its missions. See
§ 181(d)(2).
6
 See Pub. L. No. 111-23, §§ 201(b)(1)(C), 201(b)(4) (2009) (codified at 10 U.S.C. § 181).
Both actions are to be taken in consultation with certain officials. See id.
7
 The JROC-interest designation applies to all programs that are estimated to require
eventual total expenditures for research, development, test, and evaluation, including all
planned increments, of more than $365 million or expenditures for procurement, including
all planned increments, of more than $2.19 billion in fiscal year 2000 constant dollars, or
that are designated as major defense acquisition programs or special interest. The JROC-
interest designation also applies to major automated information systems programs, which
are programs designated as such or estimated to exceed certain expenditure thresholds.
Finally, the JROC-interest designation applies to all joint doctrine, organization, training,
materiel, leadership and education, personnel, facilities, and policy change
recommendations and programs and capabilities that have a potentially significant impact
on interoperability in allied and coalition partner operations. For JROC-interest documents,
recommendations on capability needs are passed on to the Joint Capabilities Board that
reviews and, if appropriate, forwards them with recommendations to the JROC for
validation and approval.




Page 4                                                    GAO-12-339 Defense Management
guidance in effect through December 2011, also reviewed programs at
the request of certain senior DOD officials. Within JCIDS, the JROC is
supported in its duty to review and validate joint capability needs by the
Joint Capabilities Board and six Functional Capabilities Boards. 8 The
Joint Capabilities Board is chaired by the Director of the Joint Staff’s
Directorate for Force Structure, Resources, and Assessment, and each
Functional Capabilities Board is chaired by a general/flag officer or civilian
equivalent. The Joint Capabilities Board reviews capability documents
before they are passed on to the JROC for its review and also serves as
the validation authority for certain programs that do not reach JROC-
interest thresholds, 9 although the JROC may review any JCIDS document
or other issues requiring joint resolution. Functional Capabilities Boards
are responsible for reviewing proposed requirements specific to joint
capability areas, such as protection, logistics, or battlespace awareness.

In JCIDS, the JROC and its supporting organizations review requirements
documents related to capability gaps and the major defense acquisition
programs intended to fill those gaps prior to key acquisition milestones.
These requirements documents—initial capabilities documents, capability
development documents, and capability production documents for
materiel solutions and change recommendations for nonmateriel
solutions—are submitted into the JCIDS process by capability sponsors. 10
The initial capabilities document identifies a specific capability gap, or set


8
 Functional Capabilities Boards assess needs and make recommendations about
validating capability gaps for documents to be validated by the JROC or the Joint
Capabilities Board.
9
 Under the JCIDS guidance prior to January 10, 2012, the Joint Capabilities Board served
as the validation authority for Joint Capabilities Board-interest programs, which were
defined as all acquisition category II and below programs where the capabilities, systems,
or both associated with the document would have affected the joint force and an
expanded joint review was required. See Chairman of the Joint Chiefs of Staff Instruction
3170.01G at GL-6 (Mar. 1, 2009) (hereinafter cited as CJCSI 3170.01G (Mar. 1, 2009)).
Under the current version of the guidance, Joint Capabilities Board-interest documents are
those describing acquisition category II and below programs that have a potentially
significant impact on interoperability (interagency, allied, partner nation, coalition, etc.).
See Manual for the Operation of the Joint Capabilities Integration and Development
System, enc. C, para. 3(e)(2) (Jan. 19, 2012) (hereinafter cited as JCIDS Manual (Jan. 19,
2012)). Under both the previous and present versions of the guidance, programs that are
not designated as JROC- or Joint Capabilities Board-interest are validated by the sponsor.
See JCIDS Manual, enc. C, para. 3(e)(3)-(5) (Jan. 19, 2012).
10
  Sponsors are generally the military services, but can also be other DOD agencies or
combatant commands.




Page 5                                                     GAO-12-339 Defense Management
of gaps, and if a materiel solution is required, helps inform the initial
stages of the acquisition process, which include an analysis of the
alternative solutions to fulfilling the capability need and the selection of a
preferred system concept. When the technology development phase of
the acquisition process is complete, a program sponsor completes a
capability development document that includes more detail on the desired
capabilities of the proposed system and defines the system’s key
performance parameters or attributes against which the delivered
increment of capability will be measured. Finally, the sponsor prepares a
capability production document to describe the actual performance of the
system that will deliver the required capability. Figure 1 depicts how
JCIDS reviews align with the acquisition process.

Figure 1: Interaction between DOD’s Requirements Generation and Acquisition
Processes




a
 The 2012 JCIDS guidance directs the Functional Capabilities Boards to review the analysis of
alternatives and other analyses for certain capability requirements. See Chairman of the Joint Chiefs
of Staff Instruction 3170.01H, Joint Capabilities Integration and Development System, enc. A, para.
2(c)(1)(b) (Jan. 10, 2012) (hereinafter cited as CJCSI 3170.01H (Jan. 10, 2012)).


The House Armed Services Committee and a panel established by the
committee have discussed long-standing challenges with the JCIDS
process and the JROC’s fulfillment of its statutory responsibilities. The
House Armed Services Committee, in a report accompanying a bill for the
National Defense Authorization Act for Fiscal Year 2008, described a
legislative provision that would allow for joint decision making as opposed
to service-centric budget considerations by incorporating clear priorities




Page 6                                                          GAO-12-339 Defense Management
and budget guidance into the JROC process. 11 In 2009, the House Armed
Services Committee established a panel on defense acquisition reform
because of a sense that the acquisition system was not responsive
enough for today’s needs, not rigorous enough in protecting taxpayers,
and not disciplined enough in the acquisition of weapon systems for
tomorrow’s wars. The panel received testimony that the Joint Staff lacked
some of the analytical expertise necessary to ensure that the JCIDS
process rigorously vets proposed requirements. Additionally, since 2008
we have reported on these challenges. We reported in 2008 that the
JCIDS process was not effective in prioritizing capability gaps, and we
noted that capability needs continued to be proposed and defined by the
services with little involvement from the joint community. We
recommended that the Secretary of Defense direct the Chairman of the
Joint Chiefs of Staff to develop an analytic approach within JCIDS to
better prioritize and balance the capability needs of the military services,
combatant commands, and other defense components. 12 DOD partially
agreed with our recommendation but did not fully implement it, and
prioritization remains service driven. More recently, in June 2011 we
reported that the JROC did not always consider trade-offs among cost,
schedule, and performance objectives; prioritize requirements; consider
redundancies across proposed programs; or prioritize and analyze
capability gaps in a consistent manner. 13 We recommended that the
JROC require higher-quality resource estimates from requirements
sponsors to ensure that best practices are being followed, provide a
sound basis to ensure that trade-offs are considered, prioritize
requirements across proposed programs, and address potential
redundancies during requirements reviews, among other steps. DOD
partially agreed with our recommendations and commented that
improvements to the quality of resource estimates would be addressed in
upcoming changes to the JCIDS process.



11
  See H.R. Rep. No. 110-146, at 381 (2007) (accompanying H.R. 1585). The provision,
which revised parts of section 181 of Title 10, U.S. Code, was included as amended in the
National Defense Authorization Act for Fiscal Year 2008. See Pub. L. No. 110-181, § 942
(2008).
12
  GAO, Defense Acquisitions: DOD’s Requirements Determination Process Has Not Been
Effective in Prioritizing Joint Capabilities, GAO-08-1060 (Washington, D.C.: Sept. 25,
2008).
13
 GAO, DOD Weapon Systems: Missed Trade-off Opportunities During Requirements
Reviews, GAO-11-502 (Washington, D.C.: June 16, 2011).




Page 7                                                  GAO-12-339 Defense Management
                        In May 2011, we also reported that combatant command 14 officials raised
                        concerns that JCIDS focuses more on long-term service-centric capability
                        gaps than on combatant commands’ more immediate and largely joint
                        gaps. 15 JCIDS was designed as a deliberate process to meet longer-term
                        joint needs. To address urgent needs, DOD established a separate
                        process—the joint urgent operational needs process—in 2005. The joint
                        urgent operational needs process was intended to respond to needs
                        associated with combat operations in Afghanistan and Iraq and the War
                        on Terror. The revised JCIDS guidance canceled separate guidance for
                        joint urgent operational needs and incorporates and describes the joint
                        urgent operational needs process. Urgent operational needs, as defined
                        by the new JCIDS guidance, are capability requirements needed for
                        ongoing or anticipated contingency operations that if left unfulfilled could
                        potentially result in loss of life or critical mission failure. 16 For this report,
                        we focus on requirements that have not been identified as urgent and
                        instead follow the deliberative JCIDS process.


                        The Joint Staff is undertaking efforts to improve the ability to prioritize
The Joint Staff Has     capability needs from a joint perspective through JCIDS and to align
Initiated Efforts to    those needs with available budget resources. However, implementation
                        processes for JCIDS’s new approach to managing requirements and
Prioritize Capability   considering affordability are still evolving and have not been fully
Needs, but              developed and clearly documented.
Implementation
Processes Are Not
Fully Developed and
Clearly Documented

                        14
                          Combatant commands are DOD’s operational commands. Of the nine combatant
                        commands, the following six have geographic responsibilities: U.S. Africa Command, U.S.
                        Central Command, U.S. European Command, U.S. Northern Command, U.S. Pacific
                        Command, and U.S. Southern Command. The following three have functional
                        responsibilities: U.S. Special Operations Command, U.S. Strategic Command, and U.S.
                        Transportation Command.
                        15
                         GAO, Defense Management: Perspectives on the Involvement of Combatant
                        Commands in the Development of Joint Requirements, GAO-11-527R (Washington, D.C.:
                        May 20, 2011).
                        16
                         See CJCSI 3170.01H, at GL-7 (Jan. 19, 2012).




                        Page 8                                                 GAO-12-339 Defense Management
Changes in JCIDS            Determining priorities among joint requirements has been a responsibility
Processes Are Intended to   of the JROC since Congress amended section 181 of Title 10 of the U.S.
Improve Prioritization of   Code in 2008 to require the JROC to assist in establishing and assigning
                            priority levels for joint military requirements and to help ensure that
Capability Needs and        resource levels associated with those requirements are consistent with
Guide Future Investments    the level of priority. 17 DOD officials acknowledge that JCIDS has been
                            ineffective in helping the JROC carry out these responsibilities. We have
                            previously reported that JCIDS’s ability to align resources to balance
                            competing needs could be improved if it had an analytic approach that
                            provided a means to review and validate proposals to ensure that the
                            most important capability needs of the department are being addressed.
                            We further said that such an approach should establish criteria and
                            measures for identifying capability gaps and determining the relative
                            importance of capability needs. Finally, the approach should result in
                            measurable progress in allocating resources in order to eliminate
                            redundancies, gain efficiencies, and achieve a balanced mix of
                            executable programs. 18

                            DOD officials told us that downward pressure on the defense budget has
                            led the Joint Staff to change how the JCIDS process is used to strengthen
                            its ability to support JROC members in making trade-off decisions among
                            requirements and balancing risks across the force within expected
                            resources. In fall 2011, according to officials, the incoming Vice Chairman
                            of the Joint Chiefs of Staff, as the Chairman of the JROC, began to make
                            changes in the JCIDS processes to focus on what capabilities currently
                            exist and weigh the benefits of investing in new capabilities with their
                            estimated costs early in the review process. The Joint Staff issued draft
                            guidance in October 2011 and began implementation based on the draft
                            guidance. The Joint Staff issued the final guidance in January 2012.
                            Descriptions of these changes and their implementation follow:

                            •     New capabilities will be considered as part of a “capability
                                  portfolio approach.” Under the portfolio approach, officials stated
                                  that JROC members are to ensure that proposed investments in
                                  capabilities address joint needs or they will not be validated to



                            17
                                See Pub. L. No. 110-181, § 942(a) (codified as amended at 10 U.S.C. § 181(b)).
                            18
                              GAO, Best Practices: An Integrated Portfolio Management Approach to Weapon System
                            Investments Could Improve DOD’s Acquisition Outcomes, GAO-07-388 (Washington,
                            D.C.: Mar. 30, 2007).




                            Page 9                                                   GAO-12-339 Defense Management
    proceed to the acquisition process. In addition to validating capability
    proposals, according to Joint Staff officials, the JROC has begun
    examining how the services’ existing programs can support joint
    operations to reduce duplication of capabilities. As of December 2011,
    according to Joint Staff officials, the last five closed meetings of the
    JROC began conversations about how to meet requirements by
    considering available capabilities and the costs and benefits of
    proposed programs. As a result, according to Joint Staff officials, at
    least five classified programs have been reviewed and altered by
    either comparing redundant capabilities, reducing capacity, adjusting
    delivery schedules, or directing follow-on analysis before moving
    programs forward. To support JROC decision making, officials
    reported that Functional Capabilities Boards will be tasked with
    examining requirements, their associated capability gaps, and
    proposed solutions within their capability portfolios, and independently
    assessing how a proposed capability fits into its corresponding joint
    capability area. The Functional Capabilities Boards previously had
    responsibility for identifying, assessing, and prioritizing (if required)
    joint capability needs proposals within assigned joint capability areas
    but, according to officials, have not always carried out these
    responsibilities. Previously, Functional Capabilities Boards acted
    primarily as technical reviewers of requirements documents, and
    program sponsors briefed the JROC on the attributes of the program.
    However, the new guidance does not specify how the independent
    assessment is to be conducted, and it is too soon to tell how the
    Functional Capabilities Boards will respond to the new requirement.
    Officials reported that Functional Capabilities Boards are expected to
    develop methodologies on a case-by-case basis. DOD officials said
    that the analytic approach and uses of analytic information will evolve
    over time.

    DOD has previously attempted to manage capabilities
    departmentwide through a portfolio approach, but has never fully
    implemented the approach. In 2006, DOD established an effort to
    manage resources across capability areas by establishing capability
    portfolio managers to enable DOD to develop and manage
    capabilities across the department rather than by military service or
    individual program, and by doing so, to improve the interoperability of
    future capabilities, minimize capability redundancies and gaps, and
    maximize capability effectiveness. However, as we reported in 2008,
    capability portfolio managers make recommendations on capability
    development issues within their portfolios but do not have
    independent decision-making authority. In 2011, Joint Staff officials




Page 10                                         GAO-12-339 Defense Management
     told us that they were unaware of capability portfolio managers’ active
     involvement in the JCIDS process.

•    Attendance at JROC meetings will be limited to key decision
     makers and stakeholders. Beginning in October 2011, the JROC
     Chairman began to limit attendance at JROC meetings to facilitate
     candid discussion among senior leaders about priorities for joint
     requirements and alternative solutions. Joint Staff officials told us that
     previously, meetings were open to a broad range of interested parties
     and service sponsors provided briefings on their proposals for new
     capabilities. Under the new approach being implemented, a
     representative of one of the Functional Capabilities Boards will
     provide a briefing on the proposal to the JROC. The representatives
     would then present the board’s independent assessment of benefits,
     costs, and risk for the JROC to discuss and decide upon. According to
     JCIDS officials, JROC members are expected to make decisions from
     the perspective of the joint force and avoid taking a service-centric
     approach.

•    Affordability of proposals will be a primary factor in validation
     decisions. JROC members have always been expected to consider
     the resource implications of validation decisions, but officials stated
     that until recently, these considerations have not been a focus
     because capabilities were not competing with each other for funding.
     According to officials, the Functional Capabilities Boards have been
     directed to take similar steps to ensure that capability proposals not
     only meet technical requirements but also represent the most efficient
     alternative for providing a capability within the joint capability area
     without creating duplication or overlap. According to JCIDS officials,
     the JROC is also reconsidering previous validation decisions and
     asking for changes to proposals to minimize costs. 19 We reviewed
     recent Functional Capabilities Board briefings to the JROC, which
     provided information on how needs might be met with current
     capabilities and alternatives that might meet needs while minimizing
     costs. However, it is too soon to assess how the JROC will consider
     affordability of programs when making validation decisions.




19
  We are not including specific examples of issues and programs under review because of
the sensitivity of deliberations and security classification concerns.




Page 11                                                GAO-12-339 Defense Management
New JCIDS Processes Are   The Vice Chairman of the Joint Chiefs of Staff has sought some policy
Not Fully Developed and   changes and, according to officials, provided other direction to implement
Clearly Documented        changes in the JCIDS process, but the new approach to managing
                          requirements and considering affordability is still evolving and has not
                          been fully developed and clearly documented. According to Joint Staff
                          officials, the dynamic fiscal environment and the evolutionary method
                          being used to develop the new approach and implementation processes
                          make it important that decision makers maintain flexibility in decision
                          making. We believe that the new approach has promise in positioning the
                          JROC to more accurately identify capability gaps and trade-offs, but it has
                          not been fully developed to include steps to ensure that the approach is
                          fully implemented, that the intent is fully communicated to all stakeholders
                          involved, and that the results of the new approach will be measurable.

                          We have previously reported that key practices for results-oriented
                          management involve leadership from top officials as well as the
                          involvement of stakeholders at all levels throughout a period of
                          transition. 20 We have also reported that in order to demonstrate a
                          successful results-oriented framework, officials must include clearly
                          defined measures to assess intended outcomes. 21 As shown in table 1,
                          key practices for supporting change should include, among other actions,
                          obtaining and sustaining support from senior leadership to facilitate the
                          transformation, establishing clear lines of communication between all
                          affected parties, and demonstrating value and credibility of new
                          processes through the use of metrics.




                          20
                           GAO, Results-Oriented Cultures: Implementation Steps to Assist Mergers and
                          Organizational Transformations, GAO-03-669 (Washington, D.C.: July 2, 2003).
                          21
                           GAO, Defense Management: Tools for Measuring and Managing Defense Agency
                          Performance Could Be Strengthened, GAO-04-919 (Washington, D.C.: Sept. 13, 2004).




                          Page 12                                               GAO-12-339 Defense Management
Table 1: Elements of a Results-Oriented Management Framework

 Principle                    Critical elements
 Ensure top leadership drives •    Define and articulate a succinct and compelling
 the transformation.               reason for change.
                              •    Balance continued delivery of services with
                                   transformation activities.
 Establish a communication    •    Communicate early and often to build trust.
 strategy to create shared    •    Ensure consistency of message.
 expectations and report
                              •    Encourage two-way communication.
 related progress.
                              •    Provide information to meet specific needs of
                                   employees.
 Align goals and measures     •    Each of the agency’s strategic plan goals is supported
 with agency and                   by performance measures.
 departmentwide goals.        •    Measures align with departmentwide goals.
                              •    Goals and measures cascade from the corporate
                                   level to the lowest level of the agency.
 Demonstrate results.         •    Includes a combination of output- and outcome-
                                   oriented measures.
                              •    Measures are clearly defined.
                              •    Provides trend data to demonstrate progress against
                                   targeted performance.
Source: GAO.



According to Joint Staff officials, the Chairman of the JROC has been a
driving force in positioning the JROC to take on the responsibility of
aligning needs and balancing risk and resources, which fulfills one of the
key steps for results-oriented management, but the approach is still new
and officials have not completed all of the steps that facilitate institutional
acceptance and implementation of the new approach. The Joint Staff has
begun the process of change by articulating a clear rationale for
change—that the JROC can more effectively represent the warfighters’
requirements and make strategic trade-off decisions as budgets stay flat
or decrease by taking a more active role in shaping an affordable joint
force. However, best practices for managing a results-oriented change
state that goals and procedures should be communicated to stakeholders
throughout the organization so that they understand how they should
implement the new approach and how the organization will measure
progress. The Joint Staff issued guidance that outlines new procedures
intended to establish an approach for prioritizing capabilities from a joint
perspective and to increase the timeliness of JCIDS reviews by
categorizing proposals according to the level of urgency of the need and
streamlining procedures for urgent needs. However, the guidance does
not clearly outline criteria and measures for demonstrating progress



Page 13                                                 GAO-12-339 Defense Management
                        toward meeting the goal of aligning needs with available resources or
                        clearly communicate the goals and the analytic approach envisioned to
                        support JROC decision making. Further, the guidance does not describe
                        how the proposed change will affect the services, combatant commands,
                        and other stakeholders. Finally, the JCIDS guidance does not establish
                        criteria and measures for demonstrating progress toward the goal of
                        creating a balanced portfolio of programs that takes into account needs,
                        risks, and available resources, nor do other documents provided to us by
                        DOD. Measures such as the proportion of requirements that address joint
                        priority needs versus service-specific needs, the savings obtained
                        through the elimination of redundant capabilities, and the comparison of
                        estimated costs of a proposed program with the actual costs of operating
                        the program over its life cycle could be helpful in assessing whether the
                        process is balancing requirements with available resources or whether
                        further adjustments to the process are needed.

                        Considering new capabilities across the department in the context of joint
                        capability areas can help DOD begin to identify priorities for future
                        investment. However, unless the Joint Staff takes steps to define and
                        institutionalize the new approach by adhering to the key principles of
                        results-oriented management, it is not clear whether the current
                        momentum in implementing an analytic approach through JCIDS will be
                        sustained.


                        Even though sustainment costs make up a significant portion of the total
JROC Has Not Always     ownership costs of a weapon system, the JROC has not always had
Had Complete            complete information on such costs when validating documentation used
                        in the decision to initiate program development. “Sustainment” as a
Information on Future   category represents a range of activities intended to ensure the
Costs to Sustain        availability of weapon systems and to support their operations, including
                        some logistics and personnel services. During the identification of
Proposed Capabilities   capability gaps and consideration of selected alternatives, it is difficult for
before Approving        the sponsors to provide detailed information on program capabilities and
Documentation for       cost estimates. As a major defense acquisition program moves toward
                        the development stage, JCIDS requires that more complete and accurate
Program                 sustainment information be presented in capability development
Development             documents. According to DOD officials, decision makers need more
                        accurate cost information to assess whether the benefit of a proposed




                        Page 14                                           GAO-12-339 Defense Management
capability justifies the cost of sustaining the capability over its life. 22
JCIDS guidance requires that sponsors of potential major defense
acquisition programs 23 include sustainment information in capability
development documents, 24 which detail proposed solutions to fulfill
capability needs.

The JROC has generally relied on sponsor-provided assessments of
sustainment information in the capability development documents to
make its validation decisions, but these documents have not always
included all the information suggested in JCIDS guidance or sufficient
detail to enable the JROC to assess the quality of the information. A DOD
manual 25 regarding the development of sustainment information suggests
that when sustainment requirements and underlying assumptions are not
clearly documented, subsequent decisions about the project may be
based on incorrect assumptions. Prior GAO work 26 suggests that gaps in
joint warfighting capabilities and proposals to fulfill the gaps should be
clearly identified to decision-making bodies, such as the JROC, to inform
deliberations. Further, information should be complete so those making
the important decisions may do so as effectively as possible.




22
  Our prior work has found that the operation and support costs of sustaining a fielded
system account for about 70 percent of a system’s life cycle costs. See GAO-10-717.
23
  During our review, the Joint Staff revised guidance on sustainment reporting
requirements. Under the guidance in effect through December 2011, which we used for
our review, sustainment metrics were to be developed for all acquisition category I
programs involving materiel solutions and acquisition category II and below programs as
determined by the sponsor. See Manual for the Operation of the Joint Capabilities
Integration and Development System, app. B to enc. B, para. 2 (February 2009, updated
Jan. 31, 2011) (hereinafter cited as JCIDS Manual (Jan. 31, 2011)). According to the
recently revised JCIDS Manual, sustainment metrics must be developed for all acquisition
category I programs. The manual also states that acquisition category II and below
programs, with material solutions, shall include the sustainment metrics or sponsor-
defined sustainment metrics. See JCIDS Manual, app. E to enc. B, para. 2 (Jan. 19,
2012). Under both versions of the manual, these metrics are developed in the form of key
performance parameters and key system attributes.
24
  The JCIDS Manual required the inclusion of sustainment information in capability
development documents beginning in 2007.
25
  DOD, Reliability, Availability, Maintainability, and Cost Rationale Report Manual
(Washington, D.C.: June 1, 2009).
26
 GAO, Defense Acquisition: Employing Best Practices Can Shape Better Weapon
System Decisions, GAO/T-NSIAD-00-137 (Washington, D.C.: Apr. 26, 2000).




Page 15                                                   GAO-12-339 Defense Management
JCIDS guidance requires sponsors of major defense acquisition programs
to address sustainment based on four metrics—materiel availability,
operational availability, reliability, and ownership cost (renamed operation
and support cost in January 2012). 27 The guidance includes a series of
criteria, which are listed as review criteria in the guidance. The criteria
provide additional information on each metric. For example, JCIDS
guidance lists as review criteria for the materiel availability metric whether
there is a clear definition and accounting for the intended service life of
the program, an identification of planned downtime, and a comparison of
downtime value with experiences of other analogous systems, among
other criteria. Table 2 outlines examples of key review criteria within each
of the four sustainment metrics under the guidance in effect through
December 2011. 28




27
  See JCIDS Manual, app. B to enc. B (Jan. 31, 2011). See also JCIDS Manual, app. E to
enc. B (Jan. 19. 2012). The version of the JCIDS Manual in effect through December 2011
defined “materiel availability” as a measure of the percentage of the total inventory of a
system that is operationally capable (ready for tasking) of performing an assigned mission
at a given time, based on materiel condition; “operational availability” as a measure of the
percentage of time that a system or group of systems within a unit are operationally
capable of performing an assigned mission and can be expressed as (uptime/(uptime +
downtime)); “reliability” as a measure of the probability that the system will perform without
failure over a specific interval and must be sufficient to support the warfighting capability
needed; and “ownership cost” as a measure that provides balance to the sustainment
solution by ensuring that the operations and support costs associated with availability are
considered in making decisions, and should cover the planned lifecycle timeframe,
consistent with the timeframe used in the materiel availability metric. The revised guidance
largely maintained these descriptions.
28
  The revised guidance issued in January 2012 includes many of the same review criteria.
See JCIDS Manual, app. E to enc. B (Jan. 19, 2012). As previously noted, the revised
guidance renamed the ownership cost metric as operation and support cost.




Page 16                                                    GAO-12-339 Defense Management
Table 2: Examples of Review Criteria for Each of the Four Sustainment Metrics

 Sustainment           Total number of
 metric                 review criteria     Examples of review criteria
 Materiel                               8   •  Accounts for the total population of end items
 availability                                  being acquired for operational use
                                            •  Identifies and includes planned time periods
                                               during which the weapon system is not
                                               operationally available to perform an assigned
                                               mission
                                            •  Identifies sources of data and models used to
                                               establish and track materiel availability
 Operational                            5   •   Provides specific definitions for what constitutes
 availability                                   failure
                                            •   Addresses time periods during which the
                                                weapon system is nonfunctional as a result of
                                                failure
                                            •   Addresses specific issues associated with
                                                failure, including time needed to recover a
                                                weapon system after suffering a failure and
                                                time to perform diagnostics
 Reliability                            8   •   Defines how time intervals for assessing the
                                                weapon system will be measured
                                            •   Identifies sources for baseline reliability data
                                                and addresses whether the proposed reliability
                                                value is consistent with comparable systems
                                            •   Addresses whether the proposed reliability of
                                                the weapon system is consistent with intended
                                                operational use of system
 Ownership                              9   •   Defines through analysis which type of
 cost                                           ownership cost structure was used for
                                                assessing program cost
                                            •   Includes all required costs for a program,
                                                regardless of funding source
                                            •   Includes an approach to monitor, collect, and
                                                validate operation and support cost data
Source: GAO analysis of DOD guidance.



Program sponsors provide initial information on the sustainment metrics
for proposed capability solutions when they submit a capability
development document, one of three capability documents the JROC
considers in its review and potential validation of capability proposals.
Officials from both the Joint Staff and the Office of the Under Secretary of
Defense for Acquisition, Technology and Logistics review the capability
development documents and, according to officials, verify that all required
sustainment elements have been included before the documents are
validated by the JROC. Officials also provide their independent



Page 17                                                        GAO-12-339 Defense Management
assessments of the quality of the cost estimates to the JROC. 29 Officials
from the Office of the Under Secretary of Defense for Acquisition,
Technology and Logistics said they work with program sponsors to
ensure that life cycle sustainment planning and costs are as accurate as
possible. Officials from the Office of the Under Secretary of Defense for
Acquisition, Technology and Logistics and the Joint Staff told us that they
consider reported sustainment information important to a program’s
development and review all reported information.

The JCIDS Manual notes that listed criteria, information, and activities
cannot necessarily be applied to all systems. Sponsors have a degree of
latitude in determining which items are applicable for their specific
concept, technology, system, or combination of these. For example, a
program sponsor for a major defense acquisition program is required to
report a measure for operational availability, but would not necessarily
have to report on the respective criteria, such as addressing downtime
associated with failure, including recovery time or movement of
maintenance teams to the work site. Because the guidance does not
specifically require program sponsors to report on the individual criteria,
they generally include some, but not all, of the individual criteria.

Our analysis of six capability development documents found that all of the
documents provided information on all of the required sustainment
metrics. However, we found that the completeness of information
reported for all of the metrics’ key criteria varied. Specifically, none of the
documents included complete information for each of the four
sustainment metrics’ review criteria elements. In addition, each of the
documents had some common omissions; for example, none of the six
capability documents we reviewed included information on all nine of the
ownership cost metric’s criteria elements. Further, several of these
documents only included information on one criteria element for a single
metric, and none reported information on all of the elements for any of the
metrics. Finally, when information on the metrics’ key criteria was
provided, the level of detail varied among the documents. For example,
for some criteria, some documents provided a paragraph of supporting
information and analysis whereas others provided single-sentence


29
  The Joint Staff Directorate for Logistics (J4), Maintenance Division, is the Joint Staff
entity responsible for evaluating the sustainment key performance parameter. In this
report, GAO refers to the sustainment key performance parameter as sustainment
information.




Page 18                                                     GAO-12-339 Defense Management
responses that did not provide as substantial detail for the supporting
criteria. According to officials, for the capability documents we reviewed,
there were cases in which information on some of the suggested criteria
should have been included but were not.

Joint Staff officials and officials from the Office of the Under Secretary of
Defense for Acquisition, Technology and Logistics noted that while criteria
for each of the required sustainment metrics may not be applicable to
every program, it would be beneficial to the JROC if the services reported
on the criteria for each metric outlined in the guidance or indicated a
reason why a specific criterion was not applicable. Officials from the
Office of the Under Secretary of Defense for Acquisition, Technology and
Logistics told us that when they conduct their reviews of sponsor-reported
sustainment information, not all of the supporting documentation they
need for a thorough independent assessment is available in the capability
development document. These officials said that they can generally find
detailed documentation on sustainment planning and costs from sources
outside of the JCIDS process, but that the information is not always
readily available within the JCIDS database. Ultimately, review efficiency
could be improved if all the information were available in the JCIDS
database. Updated JCIDS guidance issued in January 2012 still does not
clearly require program sponsors to report on the individual criteria for
each of the sustainment metrics. The Joint Staff is developing a new
reporting tool intended to provide a standard format for reporting
sustainment information. The tool will require program sponsors to at
least minimally address each of the four sustainment metrics in order to
submit the capability development document for review through JCIDS,
according to officials. However, officials stated that this tool will not
require that sponsors address each of the individual criteria elements
within the four sustainment metrics.

Without complete and detailed information on each of the individual
criteria elements, the JROC may not be in the best position to weigh the
costs and benefits of a proposal within a capability portfolio. The quality
and completeness of the data that sponsors provide through the
capability development document in the JCIDS process will become more
important as the JROC increases its examination of the benefits of
programs balanced against their associated costs. As we have previously
reported, incomplete and inaccurate sustainment information has been a




Page 19                                         GAO-12-339 Defense Management
              long-standing problem for DOD. 30 In November 2009, a DOD team
              assessing weapon system acquisition reform reported that DOD lacked
              valid, measurable sustainment information to accurately assess how
              programmatic decisions affected life cycle costs and made
              recommendations to improve weapon system life cycle sustainment. 31
              Until the JROC requires program sponsors to report complete
              sustainment information, including both the overall metrics and the
              supporting criteria, the JROC may not always have the complete and
              detailed information it needs to make the most informed decisions.


              The prospect of declining budgets has amplified the need for DOD to
Conclusions   prioritize among capability gaps and to use its resources to maximize the
              capabilities of the joint force. The Chairman of the JROC has begun to
              take steps to better balance risks across the joint force by examining
              proposals for new capabilities within the context of existing joint capability
              areas and to consider affordability, including sustainment costs, as a
              factor in validating requirements proposals. The revised approach is new
              and evolving, but in order for it to achieve the intended results of
              prioritizing capability needs and aligning those needs with available
              resources, the Joint Staff needs to take steps to fully develop the
              approach and document it more explicitly. Specifically, DOD does not yet
              have a documented implementation plan with measures of success that
              support change. In addition, having good sustainment information is a key
              element needed to improve JCIDS’s success over the long term.
              Sustainment costs historically represent 70 percent of a system’s life
              cycle costs, but DOD has been making decisions with incomplete
              information on sustainment and does not require that sponsors address
              all of the criteria outlined in JCIDS guidance. Until the JROC has
              developed and fully documented an approach for prioritizing capability
              needs and aligning these needs with available resources and has
              complete sustainment information associated with the operation of new
              capabilities, it will not be in the best position to align resources with
              priorities or balance costs with benefits in affordable investment plans.



              30
               GAO-10-717.
              31
                DOD, Weapon System Acquisition Reform Product Support Assessment (Washington,
              D.C.: Nov. 1, 2009). This report was produced by a DOD Product Support Assessment
              Team led by the Office of the Deputy Under Secretary of Defense for Logistics and
              Materiel Readiness.




              Page 20                                             GAO-12-339 Defense Management
                      To help sustain momentum for efforts to bring a capability portfolio
Recommendations for   approach to the JCIDS process and to improve the quality of sustainment
Executive Action      information reported in capability development documents, we
                      recommend that the Vice Chairman of the Joint Chiefs of Staff, as the
                      JROC Chairman, take the following two actions:

                      •   Revise and implement guidance to reflect changes to the JCIDS
                          process as well as to establish criteria and measures for determining
                          the relative importance of capability needs across capability areas and
                          assessing progress.
                      •   Explicitly require that program sponsors address each of the criteria
                          outlined for the individual sustainment metrics when submitting
                          capability development documents.

                      The Joint Staff provided written comments on a draft of this report. In its
Agency Comments       comments, the Joint Staff said our report represented a fair and objective
and Our Evaluation    assessment of the JCIDS process. It partially agreed with both of our
                      recommendations, citing ongoing and planned changes to the joint
                      requirements development process. However, the comments did not
                      detail any specific steps that DOD plans to take to address our
                      recommendations. The comments are reprinted in their entirety in
                      appendix III. The Joint Staff also provided technical comments, which we
                      have incorporated into the report as appropriate.

                      The Joint Staff partially concurred with our recommendation that the Vice
                      Chairman of the Joint Chiefs of Staff revise and implement guidance to
                      reflect changes to the JCIDS process as well as to establish criteria and
                      measures for determining the relative importance of capability needs
                      across capability areas and assessing progress. In its written response,
                      the Joint Staff described recent initiatives to substantially change the joint
                      requirements development process to require that capability requirements
                      be evaluated within a capability portfolio by Functional Capability Boards,
                      the Joint Capabilities Board, and the JROC. The Joint Staff also
                      discussed its planned efforts to improve prioritization of capability needs
                      and stated that JROC reviews will incorporate an evolving portfolio
                      assessment tool. The Joint Staff expects that the departmentwide
                      priorities outlined in DOD’s strategic guidance as well as a revised
                      process for assessing capability gaps and combatant command priorities
                      will enable the JROC to make more informed decisions about priorities.
                      While we agree that the Joint Staff has taken important steps to enable
                      prioritization of capabilities, such as addressing prioritization in a new
                      enclosure in its revised JCIDS Manual, the enclosure does not explicitly



                      Page 21                                          GAO-12-339 Defense Management
outline implementation processes. We continue to believe that clear
guidance that establishes criteria for determining priority levels and
measures for demonstrating progress will be essential in sustaining
momentum toward the goal of creating a balanced portfolio of programs
that takes into account needs, risks, and available resources. Moreover,
providing guidance that fully documents the new procedures for assigning
priority levels to capability gaps is an essential step toward clarifying how
the procedures will be implemented.

The Joint Staff also partially concurred with our recommendation that the
Vice Chairman of the Joint Chiefs of Staff explicitly require that program
sponsors address each of the criteria outlined for the individual
sustainment metrics when submitting capability development documents.
According to its written comments, the Joint Staff criteria for the
sustainment metrics were designed to guide the development of
requirements, but were not intended to be prescriptive because individual
programs are unique and criteria applicable to one problem may not apply
to another. We agree that each of the criteria may not be applicable to
every program. However, if program sponsors addressed each criterion in
some manner, including explaining that a criterion is not applicable to the
program, the JROC would be assured that program sponsors considered
all criteria when developing program proposals. Further, the Joint Staff
commented that JCIDS reviews of capability development documents by
Joint Staff and Office of the Secretary of Defense officials ensure that a
document is thoroughly vetted for consideration by the JROC. It
suggested that the inclusion of analyses and rationale for sustainment
metrics development in capability development documents might be
duplicative because this information is contained in acquisition documents
that exist outside of JCIDS. However, as we noted in our report, the
documents that contain the analysis and rationale for the required
sustainment metrics are not necessarily reviewed by or available to the
JROC members during their consideration of a capability development
document. We continue to believe that the inclusion of a sponsor-
provided rationale for each metric criterion would enhance the
thoroughness and efficiency of the JROC’s review of sustainment
information through JCIDS.


We are sending copies of this report to interested congressional
committees, the Secretary of Defense, the Vice Chairman of the Joint
Chiefs of Staff, and other interested parties. In addition, the report is
available at no charge on the GAO website at http://www.gao.gov.



Page 22                                          GAO-12-339 Defense Management
If you or your staff have any questions about this report, please contact
me at (202) 512-3489 or pendletonj@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made key contributions to this
report are listed in appendix IV.




John Pendleton
Director, Defense Capabilities and Management




Page 23                                        GAO-12-339 Defense Management
List of Committees

The Honorable Carl Levin
Chairman
The Honorable John McCain
Ranking Member
Committee on Armed Services
United States Senate

The Honorable Daniel K. Inouye
Chairman
The Honorable Thad Cochran
Ranking Member
Subcommittee on Defense
Committee on Appropriations
United States Senate

The Honorable Howard P. “Buck” McKeon
Chairman
The Honorable Adam Smith
Ranking Member
Committee on Armed Services
House of Representatives

The Honorable C.W. “Bill” Young
Chairman
The Honorable Norman D. Dicks
Ranking Member
Subcommittee on Defense
Committee on Appropriations
House of Representatives




Page 24                                 GAO-12-339 Defense Management
Appendix I: Observations on Reporting
              Appendix I: Observations on Reporting
              Provisions in the National Defense
              Authorization Act for Fiscal Year 2011


Provisions in the National Defense
Authorization Act for Fiscal Year 2011
              Section 862 of the Ike Skelton National Defense Authorization Act for
              Fiscal Year 2011 requires the Comptroller General to carry out a
              comprehensive review of the Joint Capabilities Integration and
              Development System (JCIDS) and to submit to the congressional defense
              committees a report on the review. 1 This appendix, in conjunction with the
              letter, addresses each of the reporting provisions as described in the act.
              Specifically, section 862 requires the following contents for the review:

              Purpose. The purpose of the review is to evaluate the effectiveness of
              JCIDS in achieving the following objectives:

              •   Timeliness in delivering capability to the warfighter.
              •   Efficient use of the investment resources of the Department of
                  Defense (DOD).
              •   Control of requirements creep. 2
              •   Responsiveness to changes occurring after the approval of a
                  requirements document 3 (including changes to the threat
                  environment, the emergence of new capabilities, or changes in the
                  resources estimated to procure or sustain a capability).
              •   Development of the personnel skills, capacity, and training needed for
                  an effective and efficient requirements process.
              Matters considered. In performing the review, the Comptroller General is
              required to gather information on and consider the following matters:

              •   The time that requirements documents take to receive approval
                  through JCIDS.
              •   The quality of cost information considered in JCIDS and the extent of
                  its consideration.



              1
               See Pub. L. No. 111-383, § 862(a) (2011). The statute specifically defined JCIDS by
              reference to Chairman of the Joint Chiefs of Staff Instruction 3170.01G. See § 862(c)(1).
              2
               Requirements creep is defined in the statute as “the addition of new technical or
              operational specifications after a requirements document is approved.” See § 862(c)(3).
              3
               The term requirements document is defined in the statute as “a document produced in
              JCIDS that is provided for an acquisition program to guide the subsequent development,
              production, and testing of the program and that—(A) justifies the need for a materiel
              approach, or an approach that is a combination of materiel and non-materiel, to satisfy
              one or more specific capability gaps; (B) details the information necessary to develop an
              increment of militarily useful, logistically supportable, and technically mature capability,
              including key performance parameters; or (C) identifies production attributes required for a
              single increment of a program.” See § 862(c)(2).




              Page 25                                                   GAO-12-339 Defense Management
Appendix I: Observations on Reporting
Provisions in the National Defense
Authorization Act for Fiscal Year 2011




•   The extent to which JCIDS establishes a meaningful level of priority
    for requirements.
•   The extent to which JCIDS is considering trade-offs between cost,
    schedule, and performance objectives.
•   The quality of information on sustainment considered in JCIDS and
    the extent to which sustainment information is considered.
•   An evaluation of the advantages and disadvantages of designating a
    commander of a unified combatant command for each requirements
    document for which the Joint Requirements Oversight Council (JROC)
    is the validation authority to provide a joint evaluation task force to
    participate in a materiel solution and to
    •    provide input to the analysis of alternatives; participate in testing
         (including limited user tests and prototype testing);provide input on
         a concept of operations and doctrine; provide end user feedback
         to the resource sponsor; and participate, through the combatant
         commander concerned, in any alteration of the requirement for
         such solution.
Section 862 also provided definitions for JCIDS, requirements document,
requirements creep, and materiel solution. 4

Tables 3 through 12 contain our response to each of the requirements
mandated by the Ike Skelton National Defense Authorization Act for
Fiscal Year 2011. 5 The provision mandating our report defined the JCIDS
process by referring to the JCIDS guidance in effect from March 2009
through December 2011. 6 Accordingly, our response to the mandated
elements as presented in this appendix generally focuses on JCIDS
under that guidance. However, we also provide some information relating
to JCIDS as described in the revised guidance issued in January 2012. In
addition, our assessments generally focused on those programs that
were determined to be JROC-interest, or those that were designated as
major defense acquisition programs or major automated information
systems and capabilities that have a potentially significant impact on
interoperability in allied and coalition operations. Generally, these
programs have greater costs or are more complex than smaller programs,


4
See § 862(c).
5
See Pub. L. No. 111-383, § 862.
6
 See § 862(c)(1) (defining JCIDS as “the system for the assessment, review, validation,
and approval of joint warfighting requirements that is described in Chairman of the Joint
Chiefs of Staff Instruction 3170.01G”).




Page 26                                                   GAO-12-339 Defense Management
                                         Appendix I: Observations on Reporting
                                         Provisions in the National Defense
                                         Authorization Act for Fiscal Year 2011




                                         and therefore provide an opportunity to assess the effectiveness of more
                                         aspects of the JCIDS process.

Table 3: Section 862(b)(1)(A)—Evaluate the Effectiveness of JCIDS in Achieving Timeliness in Delivering Capability to the
Warfighter

Summary assessment                       The JCIDS guidance in effect through December 2011 did not directly facilitate the timely
                                         delivery of capabilities to the warfighter. Joint Staff officials are implementing new
                                         procedures to expedite reviews of urgent needs and establishing targets for anticipated
                                         delivery times for weapon systems to facilitate timely delivery of capabilities to meet
                                         needs.
Key GAO observations                     JROC review and validation of capability needs validated in 2010 and 2011 ranged from 3
                                         months to 17 months. Officials told us that the length of time it has taken for a capability
                                         need to be validated through JCIDS depended on several factors, including the quality
                                         and completeness of the original proposal and the maturity of the technology involved.
                                         According to officials, JCIDS reviews generally did not delay or expedite the delivery of a
                                         capability; instead, the development phases of the acquisition system took the most time
                                         between identifying a need and delivering a capability. JCIDS guidance in effect through
                                         December 2011 did not require the JROC to approve a weapon system’s anticipated
                                         schedule when making decisions about validating the related requirements documents,
                                         and according to Joint Staff officials, the JROC typically has not tried to influence the
                                         delivery dates of capabilities. Recent changes to JCIDS processes require the JROC to
                                         set targets for the anticipated delivery schedule when validating a weapon system’s
                                         capability development document and to revalidate the document if the delivery target is
                                         altered by more than 12 months. In addition, JCIDS now tailors requirements reviews
                                         according to the urgency of the need.a
GAO scope and methodology                To evaluate the extent to which JCIDS had contributed to the timely delivery of
                                         capabilities to the warfighter, we reviewed policy documents about the joint requirements
                                         and acquisition processes. Using the Joint Staff’s database of joint requirements
                                         documents reviewed through JCIDS, we selected a nonprobability sample of 12
                                         requirements documents for major weapon systems that the JROC reviewed from May 1,
                                         2010, through May 31, 2011. We focused on major weapon systems because they were
                                         reviewed through each of the JCIDS phases, whereas other weapon systems are
                                         generally reviewed at the Joint Capabilities Board level but not by the JROC. Specifically,
                                         we analyzed the time that elapsed during the JCIDS review of four initial capability
                                         documents, five capability development documents, and three capability production
                                         documents. We interviewed DOD, Joint Staff, and service officials with detailed
                                         knowledge of the programs and the review process to confirm our findings. Finally, we
                                         reviewed JCIDS guidance issued in January 2012.
                                         Sources: DOD and GAO.
                                         a
                                          See CJCSI 3170.01H, enc. A, para. 1(d)(2) (Jan. 10, 2012), and JCIDS Manual, enc. E (Jan. 19,
                                         2012). Specifically, the guidance establishes different document staffing and validation processes
                                         where a need is a joint urgent operational need or a joint emergent operational need. In 2005, DOD
                                         established an urgent-needs, expedited review process outside JCIDS because the JCIDS process
                                         was ineffective at addressing urgent wartime needs; the revised JCIDS instruction cancels the
                                         instruction driving that process. See CJCSI 3170.01H, para. 2(b) (Jan. 10, 2012) (canceling Chairman
                                         of the Joint Chiefs of Staff Instruction 3470.01, Rapid Validation and Resourcing of Joint Urgent
                                         Operational Needs (JUONs) in the Year of Execution (July 15, 2005)).




                                         Page 27                                                        GAO-12-339 Defense Management
                                         Appendix I: Observations on Reporting
                                         Provisions in the National Defense
                                         Authorization Act for Fiscal Year 2011




Table 4: Section 862(b)(1)(B)—Evaluate the Effectiveness of JCIDS in Achieving Efficient Use of the Investment Resources of
the Department of Defense

Summary assessment                      The JROC has not been effective at prioritizing capability needs or guiding the efficient
                                        use of resources to meet joint capability needs and balance resources and risks. The
                                        Joint Staff is taking steps to improve prioritization by analyzing joint requirements in the
                                        context of alternative existing and proposed programs within capability areas.
Key GAO observations                    As required by law and reflected in Joint Staff guidance, the JROC has the responsibility
                                        to assist the Chairman of the Joint Chiefs of Staff in developing and assigning priority
                                        levels for joint military requirements and to help ensure that resource levels associated
                                        with those requirements are consistent with the level of priority. However, the JROC has
                                        not explicitly considered requirements’ affordability in the context of DOD’s budget. As we
                                        have reported since 2008, the JROC has not prioritized requirements, considered
                                        redundancies across proposed programs, or prioritized and analyzed capability gaps
                                        consistently through JCIDS. A key reason it did not prioritize needs was that the JROC
                                        reviewed requirements separately and largely left prioritization and trade-off decisions to
                                        the military services.a Further, DOD’s ability to use JCIDS to ensure the efficient use of
                                        resources within and among the services has been hampered by the JROC’s inability to
                                        enforce joint priorities in service budget decisions. According to officials, the Joint Staff
                                        has recently initiated efforts to assess the importance of each capability proposal JCIDS
                                        reviews in relation to existing capabilities and other related capability needs through a
                                        “capability portfolio” approach. Criteria for validation of a proposal would include the
                                        uniqueness of the proposal and its costs and benefits. JCIDS’s Functional Capabilities
                                        Boards are already providing this assessment to the JROC for its consideration in closed
                                        meetings. However, guidance on how the Functional Capabilities Boards are to assess
                                        priorities among capabilities is not specific as to how to identify joint priorities. See the
                                        section of our letter on the Joint Staff’s efforts to prioritize capability needs for additional
                                        information.
GAO scope and methodology               To evaluate the effectiveness of the JCIDS process in efficiently investing DOD’s
                                        resources, we reviewed relevant legislation and policy documents and guidance and
                                        discussed issues with senior DOD officials. We also reviewed previous GAO assessments
                                        of JCIDS and acquisition issues detailed in multiple reports, including GAO-11-502;
                                        Defense Acquisitions: DOD’s Requirements Determination Process Has Not Been
                                        Effective in Prioritizing Joint Capabilities, GAO-08-1060 (Washington, D.C.: Sept. 25,
                                        2008); and High-Risk Series: An Update, GAO-11-278 (Washington, D.C.: February
                                        2011).
                                         Sources: DOD and GAO.
                                         a
                                         GAO, DOD Weapon Systems: Missed Trade-off Opportunities During Requirements Reviews,
                                         GAO-11-502 (Washington, D.C.: June 16, 2011).




                                         Page 28                                                     GAO-12-339 Defense Management
                                        Appendix I: Observations on Reporting
                                        Provisions in the National Defense
                                        Authorization Act for Fiscal Year 2011




Table 5: Section 862(b)(1)(C)—Evaluate the Effectiveness of JCIDS in Achieving Control of Requirements Creep—and Section
862(b)(1)(D)—Evaluate the Effectiveness of JCIDS in Achieving Responsiveness to Changes Occurring after the Approval of a
Requirements Document (Including Changes to the Threat Environment, the Emergence of New Capabilities, or Changes in
the Resources Estimated to Procure or Sustain a Capability)

Summary assessment                      The JCIDS process does not actively prevent or encourage requirements creep,a as these
                                        changes are driven by the program sponsor and may not appear during review of
                                        documents.
Key GAO observations                    We have previously reported that despite the importance of well-developed and stable
                                        requirements for obtaining good weapon program outcomes, program requirements often
                                        change during development. For example, the threat the program originally addressed
                                        may change, or the user and acquisition communities may change opinions on what is
                                        needed from a program.b Programs that modified critical system characteristics after
                                        development start experienced higher levels of cost growth and longer delays in
                                        delivering capabilities. The JROC typically oversees changes only for those system
                                        characteristics considered critical to delivering a capability.c In March 2011, we reported
                                        that 21 of 39 major defense acquisition programs we assessed had at least one change to
                                        a critical characteristic, either increasing or decreasing a requirement.d Specifically, 10 of
                                        the 21 programs reported adding or enhancing a critical characteristic; 3 reported
                                        reducing, deferring, or deleting a critical characteristic; and 8 reported making both types
                                        of changes.
                                        The acquisition community has taken some steps to address requirements creep and
                                        changes made to a program’s attributes by establishing configuration steering boards for
                                        major acquisition programs to review and limit requirements changes and avoid cost
                                        increases. However, as we reported in July 2011, individual programs varied in how they
                                        used these boards to control requirements and mitigate cost and schedule risks.e
GAO scope and methodology               To evaluate the effectiveness of JCIDS in controlling requirements creep and responding
                                        to changes, we reviewed policy documents on the joint requirements process and the
                                        acquisition system and our prior reviews of JCIDS and acquisition issues, including
                                        GAO-11-233SP.
                                        Sources: DOD and GAO.
                                        a
                                         We asked officials about their perspectives on the responsiveness of JCIDS to changes occurring
                                        after the approval of a requirements document and they had no specific insights to offer. Other than
                                        the examples presented in this table, neither we nor officials were able to identify specific examples
                                        regarding the effectiveness of JCIDS in achieving responsiveness to changes.
                                        b
                                         GAO, Defense Acquisitions: Assessments of Selected Weapon Programs, GAO-08-467SP
                                        (Washington, D.C.: Mar. 31, 2008).
                                        c
                                         Specifically, changes may only appear before the JROC and the Joint Capabilities Board for
                                        validation if they affect the key performance parameters, key system attributes, or other performance
                                        attributes. Changes to characteristics of a lower priority are typically addressed by the military
                                        services. JROC-interest programs with approved capability development and production documents
                                        must return to the JROC if they experience certain cost growth.
                                        d
                                         GAO, Defense Acquisitions: Assessments of Selected Weapon Programs, GAO-11-233SP
                                        (Washington, D.C.: Mar. 29, 2011).
                                        e
                                        GAO, Defense Acquisitions: DOD Can Improve Its Management of Configuration Steering Boards,
                                        GAO-11-640 (Washington, D.C.: July 7, 2011).




                                        Page 29                                                           GAO-12-339 Defense Management
                                         Appendix I: Observations on Reporting
                                         Provisions in the National Defense
                                         Authorization Act for Fiscal Year 2011




Table 6: Section 862(b)(1)(E)—Evaluate the Effectiveness of JCIDS in Achieving Development of the Personnel Skills,
Capacity, and Training Needed for an Effective and Efficient Requirements Process

Summary assessment                      In response to a requirement in the John Warner National Defense Authorization Act for
                                                          a
                                        Fiscal Year 2007, DOD developed and implemented a training program for personnel
                                        who develop requirements for major defense acquisition programs. DOD is currently
                                        planning to expand the training program to personnel involved in developing joint
                                        requirements for programs that do not meet the threshold to be major defense acquisition
                                        programs. This program helps to develop the personnel skills, capacity, and training for
                                        the requirements process.b
Key GAO observations                    The Office of the Under Secretary of Defense for Acquisition, Technology and Logistics,
                                        working with the Defense Acquisition University, developed the requirements
                                        management certification training program to enhance the ability of individuals to write,
                                        review, and validate requirements. Defense Acquisition University officials reported that
                                        about 1,000 personnel with responsibility for generating requirements for major defense
                                        acquisition programs took the requirements development training in 2008. Since then,
                                        training has been available to individuals involved in smaller programs, but DOD officials
                                        believe that the acquisition community’s ability to identify and document joint requirements
                                        would be improved if all staff completed the training and applied the concepts in
                                        developing proposals.
                                        The curriculum addresses multiple topics related to generating requirements, including the
                                        JCIDS process for reviewing and approving documents, the collaboration between JCIDS
                                        and the Defense Acquisition System, and DOD’s resource allocation process. The
                                        program presents a series of courses that can be tailored to each individual’s level of
                                        involvement in the requirements generation process. The Defense Acquisition University
                                        provides the training courses and tracks individual completion of each course, according
                                        to officials, but beyond the required completion of DOD training, each component
                                        determines the specific steps necessary to certify requirements managers, such as
                                        additional training and on-the-job experience.
GAO scope and methodology               To evaluate the effectiveness of JCIDS in developing the personnel skills, capacity, and
                                        training needed for an effective and efficient requirements process, we reviewed relevant
                                        legislation and Joint Staff policy documents to identify the requirement to implement
                                        training for personnel involved with developing joint requirements. We interviewed officials
                                        from the Office of the Under Secretary of Defense for Acquisition, Technology and
                                        Logistics; the Joint Staff; and the Defense Acquisition University to discuss DOD’s
                                        approach for implementing the training, and we interviewed service and combatant
                                        command officials to discuss the extent to which the required training had been
                                        implemented.
                                         Sources: DOD and GAO.
                                         a
                                         See Pub. L. No. 109-364, § 801 (2006).
                                         b
                                          Neither we nor officials identified any specific programs under JCIDS itself to develop the personnel
                                         skills, capacity, and training for the requirements process. Officials identified no specific DOD-wide
                                         development programs other than the one described in the table.




                                         Page 30                                                          GAO-12-339 Defense Management
                                       Appendix I: Observations on Reporting
                                       Provisions in the National Defense
                                       Authorization Act for Fiscal Year 2011




Table 7: Section 862(b)(2)(A)—the Time That Requirements Documents Take to Receive Approval through JCIDS

Summary assessment                    JCIDS review of documents validated from May 1, 2010 through May 31, 2011 took an
                                      average of 9 months to receive approval. Joint Staff and service officials believed that the
                                      length of the review period was generally appropriate given the cost and complexity of the
                                      capabilities proposed. New procedures designed to improve responsiveness set goals for
                                      the duration of JCIDS reviews, and are based on the urgency of the need.
Key GAO observations                  The average of 9 months for review of documents within JCIDS is consistent with our
                                      previous findings. We reported in 2008 that it took 9.5 months on average for the JROC to
                                      validate capability proposals submitted from 2003 through 2005 and 8 months for
                                                                             a
                                      submissions from 2006 through 2008.
                                      JCIDS guidance in effect through December 2011 specified timeliness goals for the initial
                                                                                                                               b
                                      commenting and adjudication phases of the process, but not for the later review phases.
                                      However, goals for the commenting and adjudication phases were not always met. For
                                      most programs we analyzed, reviews exceeded timeliness goals for the commenting or
                                      adjudication periods, or both, by about 25 percent or more. The duration of the later
                                      review phases varied and accounted for the most time between submission of a proposal
                                      and validation. According to Joint Staff and service officials, the reasons for extended
                                      review times included need for additional information from the documents’ service
                                      sponsors and the lack of sponsor personnel available to address comments and approve
                                      changes. We previously reported that given the size, complexity, and level of funding
                                      associated with a major weapon system, the investment of time for a thorough, joint
                                      review may be warranted.c
GAO scope and methodology             To evaluate the time it took to review and approve requirements documents through
                                      JCIDS, we reviewed policy documents on the JCIDS process. We selected a
                                      nonprobability sample of 12 requirements documents—4 initial capabilities documents, 5
                                      capability development documents, and 3 capability production documents—for major
                                      weapon systemsd validated from May 1, 2010, through May 31, 2011, to determine the
                                      time it took the JROC to approve them. We focused on major weapon systems because
                                      they were reviewed through each of the JCIDS phases. We interviewed Joint Staff and
                                      service officials to discuss the time frames, including factors that contributed to timelines.
                                       Sources: DOD and GAO.
                                       a
                                        GAO, Defense Acquisitions: DOD’s Requirements Determination Process Has Not Been Effective in
                                       Prioritizing Joint Capabilities, GAO-08-1060 (Washington, D.C.: Sept. 25, 2008).
                                       b
                                        See JCIDS Manual, enc. D (Jan. 31, 2011). These reviews are performed by the Functional and
                                       Joint Capabilities Boards and the JROC.
                                       c
                                           GAO-08-1060.
                                       d
                                        We selected requirements for major weapon systems because these requirements are validated by
                                       the JROC whereas requirements for small programs are generally delegated to lower validation
                                       authorities.




                                       Page 31                                                      GAO-12-339 Defense Management
                                         Appendix I: Observations on Reporting
                                         Provisions in the National Defense
                                         Authorization Act for Fiscal Year 2011




Table 8: Section 862(b)(2)(B)—the Quality of Cost Information Considered in JCIDS and the Extent of Its Consideration

Summary assessment                       The cost information considered during JCIDS reviews has not always been reliable, and
                                         the extent to which JCIDS has considered cost information for proposed programs was
                                         limited to funding considerations.
Key GAO observations                     As we reported in June 2011, the services did not consistently provide high-quality
                                         resource estimates to the JROC to aid its validation decisions and cost estimates were
                                         often unreliable when assessed against best practices criteria.a In some cases, the quality
                                         of service estimates varied from those validated by the services’ cost analysis agencies to
                                         those that represented less rigorous rough-orders-of-magnitude.b In most cases, the
                                         services had not effectively conducted uncertainty and sensitivity analyses, which
                                         establish confidence levels for resource estimates, based on the knowledge available, or
                                         examined the effects of changing assumptions and ground rules, all of which could further
                                         the JROC’s efforts to provide a sound basis for making cost, schedule, and performance
                                         trade-off decisions. Moreover, we reported that the JROC had not explicitly considered a
                                         requirement’s affordability in a broader context during JCIDS reviews.
                                         Owing to the poor quality of the estimates of programs we reviewed, we recommended in
                                         the June 2011 report that the Vice Chairman of the Joint Chiefs of Staff require that
                                         capability sponsors present resource estimates that have been reviewed by a military
                                         service’s cost analysis organization to ensure that best practices are being followed. The
                                         Joint Staff partially concurred with our recommendation, stating that program office cost
                                         estimates were compared to independent cost estimates during reviews of capability
                                         development documents. However, it is not clear whether the JCIDS guidance issued in
                                         January 2012 fully responds to our recommendation from the June 2011 report. Officials
                                         told us that the guidance does not explicitly address the reliability of cost information, but
                                         according to Joint Staff officials, the JROC intends to rely more heavily on estimates
                                         developed by the Director of Cost Assessment and Program Evaluation to inform its
                                         deliberations.
GAO scope and methodology                To evaluate the quality of cost information considered in JCIDS and the extent of its
                                         consideration, we relied on prior GAO work—GAO-11-502—that addressed the extent to
                                         which the JROC considered cost information. We also interviewed Joint Staff and Office
                                         of the Under Secretary of Defense for Acquisition, Technology and Logistics officials, and
                                         reviewed JCIDS guidance to characterize the JROC’s approach for addressing our prior
                                         recommendation.
                                         Sources: DOD and GAO.
                                         a
                                         GAO, DOD Weapon Systems: Missed Trade-off Opportunities During Requirements Reviews,
                                         GAO-11-502 (Washington, D.C.: June 16, 2011).
                                         b
                                          A rough-order-of-magnitude estimate is a quick, high-level estimate that generally involves less time
                                         and effort than a budget-quality estimate.




                                         Page 32                                                          GAO-12-339 Defense Management
                                         Appendix I: Observations on Reporting
                                         Provisions in the National Defense
                                         Authorization Act for Fiscal Year 2011




Table 9: Section 862(b)(2)(C)—the Extent to Which JCIDS Establishes a Meaningful Level of Priority for Requirements

Summary assessment                      JCIDS has not established a meaningful level of priority for joint requirements, but it is
                                        adopting a new approach to prioritize joint requirements within the context of current
                                        capabilities.
Key GAO observations                    In 2008, we reported that JCIDS was ineffective in setting priorities across the services
                                        and that virtually all capability proposals submitted to JCIDS had been approved or
                                        validated.a In June 2011, we reported that the JROC still did not prioritize requirements
                                        during the JCIDS process, consider redundancies across proposed programs, or prioritize
                                        and analyze capability gaps in a consistent manner b DOD officials told us that
                                        prioritization across programs occurred primarily through DOD’s budgeting process, which
                                        is the responsibility of the military services and the Office of the Under Secretary of
                                        Defense (Comptroller). JCIDS guidance in effect through December 2011 required that
                                        capability sponsors prioritize capability gaps and identify redundancies when developing
                                        initial capabilities documents, but it did not specify the criteria sponsors should use to
                                        prioritize needs. Our review found that sponsors used different methods, with some
                                        ranking priorities in numerical order and others assigning priority categories. Some did not
                                        report on redundancies and overlap.
                                        The JROC and Joint Staff have acknowledged the need to improve the JROC’s ability to
                                        prioritize capabilities through the JCIDS process. Officials noted that the Joint Staff is now
                                        implementing a new approach to impose prioritization of joint needs within the context of
                                        existing capabilities in joint capability areas. However, the new approach has not been
                                        fully developed and clearly documented, and the extent to which current momentum
                                        toward assigning priorities will be sustained is uncertain. See the section of our letter on
                                        the Joint Staff’s efforts to prioritize capability needs for more information.
GAO scope and methodology               To evaluate the extent to which JCIDS establishes a meaningful level of priority for
                                        requirements, we reviewed law and Joint Staff policy documents to understand the roles
                                        and requirements of the JCIDS process and the JROC as it pertains to prioritization. We
                                        interviewed Joint Staff officials to discuss changes to the approach for prioritizing
                                        capability requirements. We also reviewed prior GAO reports, including GAO-11-502 and
                                        GAO-08-1060, that discussed prioritization and compared the current efforts to prioritize
                                        with what was reported in the past.
                                         Sources: DOD and GAO.
                                         a
                                          GAO, Defense Acquisitions: DOD’s Requirements Determination Process Has Not Been Effective in
                                         Prioritizing Joint Capabilities, GAO-08-1060 (Washington, D.C.: Sept. 25, 2008). GAO’s 2008 report
                                         assessed JCIDS under guidance prior to CJCSI 3170.01G.
                                         b
                                         GAO, DOD Weapon Systems: Missed Trade-off Opportunities During Requirements Reviews,
                                         GAO-11-502 (Washington, D.C.: June 16, 2011).




                                         Page 33                                                       GAO-12-339 Defense Management
                                        Appendix I: Observations on Reporting
                                        Provisions in the National Defense
                                        Authorization Act for Fiscal Year 2011




Table 10: Section 862(b)(2)(D)—the Extent to Which JCIDS Is Considering Trade-offs between Cost, Schedule, and
Performance Objectives

Summary assessment                      The JROC has not always considered trade-offs among cost, schedule, and performance
                                        objectives during JCIDS. It is premature to assess current initiatives for considering trade-
                                        offs within JCIDS.
Key GAO observations                    The Weapon Systems Acquisition Reform Act of 2009 amended section 181 to require the
                                        JROC to assist the Chairman of the Joint Chiefs of Staff in ensuring the consideration of
                                        trade-offs among cost, schedule, and performance objectives for joint military
                                        requirements.a However, as we reported in June 2011, most trade-offs are made by the
                                        military services outside the JCIDS process as they analyze alternative solutions for
                                        meeting capability needs. The JROC’s reviews of requirements do not align with these
                                        activities, and as a result, the JROC does not have an opportunity to provide the services
                                        input on trade-offs or on the proposed solution to a capability gap before it is selected,
                                        and significant time and resources may be expended before the JROC gets to formally
                                        weigh in during the JCIDS process. However, we discussed several instances where the
                                        JROC held requirements firm during reviews of programs reporting substantial cost
                                        growth. According to officials, by accepting increased cost and schedule delays, the
                                        JROC essentially traded cost and possibly schedule for performance. We made
                                        recommendations to improve the JROC’s ability to make trade-offs, including establishing
                                        a mechanism to review the results of the services’ alternatives earlier in the process,
                                        among other actions.b
                                        The Joint Staff issued new JCIDS guidance intended to better position the JROC for
                                        considering cost, schedule, and performance trade-offs and provide improved accuracy
                                        early in the process. As explained in DOD documentation, changes include the following:
                                        •    Conducting a review of the program sponsor’s analysis of alternatives study—
                                             including study results, such as cost-, schedule-, and performance-level
                                             recommendations—prior to an acquisition program proceeding to the next phase of
                                             development.
                                        •    Considering cost, schedule, performance, and quantity targets in making validation
                                             decisions and annotating decisions in JROC memorandums, as appropriate.
                                        •    Reviewing requirements that have unacceptably deviated from schedule, quantity,
                                             cost, and performance parameters.
GAO scope and methodology               To evaluate the extent to which JCIDS is considering trade-offs between cost, schedule,
                                        and performance objectives, we relied primarily on GAO’s June 2011 report—
                                        GAO-11-502—that discussed the extent to which the JROC has considered trade-offs
                                        within programs. We also reviewed Joint Staff briefings and guidance documents to
                                        identify changes in guidance with regard to cost, schedule, and performance trade-offs
                                        since June 2011.
                                        Sources: DOD and GAO.
                                        a
                                         See Pub. L. No. 111-23, § 201(b)(1)(C) (2009) (codified at 10 U.S.C. § 181(b)(1)(C)). Because of this
                                        statutory responsibility, we focused on the JROC.
                                        b
                                        GAO, DOD Weapon Systems: Missed Trade-off Opportunities During Requirements Reviews,
                                        GAO-11-502 (Washington, D.C.: June 16, 2011).




                                        Page 34                                                         GAO-12-339 Defense Management
                                        Appendix I: Observations on Reporting
                                        Provisions in the National Defense
                                        Authorization Act for Fiscal Year 2011




Table 11: Section 862(b)(2)(E)—the Quality of Information on Sustainment Considered in JCIDS and the Extent to Which
Sustainment Information Is Considered

Summary assessment                      The quality of information on sustainment planning that program sponsors provided
                                        JCIDS varied and the JROC has not systematically considered sustainment before
                                        validating programs.a
Key GAO observations                    The JCIDS Manualb in effect through December 2011 required that program sponsors of
                                        major defense acquisition programs develop information on sustainment based on four
                                        factors—materiel availability, operational availability, reliability, and ownership costc—and
                                        listed descriptive criteria, against which documents would be reviewed, for program
                                        sponsors to consider.d We have previously reported that the rigor of the analysis
                                        supporting sustainment planning and cost estimates varied. Our current review of six
                                        capability development documents showed that all the sponsors addressed the
                                        information required, but the level of detail varied. For example, the Navy provided an
                                        ownership cost estimate for its proposed Littoral Combat Ship capability but not
                                        information to support the estimate or a description of plans to monitor, collect, or validate
                                        the ownership cost data. Further, the level of detail provided for the supporting criteria
                                        varied, with some documents providing a paragraph of supporting information for some
                                        criteria and others providing single-sentence responses. Additionally, none of the
                                        documents we reviewed included a clear definition of the type of cost structure used to
                                        determine the ownership cost estimates for the proposed programs.
                                        JCIDS has not generally sought detailed information to present to the JROC for
                                        consideration before validating proposals. Although the review criteria associated with
                                        each of the four required sustainment metrics may not be applicable to every program,
                                        according to DOD officials the information could be useful for JROC validation discussions
                                        and decisions. Without complete information, the JROC may not always have the
                                        sustainment information it needs to effectively manage and validate capability documents.
                                        See the section of our letter on sustainment information available to the JROC for more
                                        information.
GAO scope and methodology               To assess the quality of reported sustainment information and the extent to which it is
                                        being considered in JCIDS, we worked with DOD officials to identify six capability
                                        development documents that included sustainment information. We reviewed the
                                        documents to determine whether and the extent to which sustainment information was
                                        included and interviewed DOD officials to discuss reported sustainment data. We also
                                        reviewed relevant law and Joint Staff policy documents to understand reporting
                                        requirements.
                                        Sources: DOD and GAO.
                                        a
                                         Under a version of the JCIDS Manual in effect through December 2011, the four sustainment factors
                                        were required to be developed for major defense acquisition programs. Applicability of some of the
                                        information for acquisition category II and lower programs was left to the determination of the
                                        sponsor. Under the current version of the manual, issued January 19, 2012, acquisition category II
                                        and lower programs, with materiel solutions, are to include either the sustainment metric or sponsor-
                                        defined sustainment metrics. As a consequence, under both versions, sustainment information may
                                        not necessarily be considered or have been considered outside the context of major defense
                                        acquisition programs.
                                        b
                                         The JCIDS Manual required the inclusion of sustainment information in capability development
                                        documents beginning in 2007.
                                        c
                                            The current version of the JCIDS Manual renamed the fourth factor as “operation and support cost.”
                                        d
                                            See JCIDS Manual, enc. B (Jan. 31, 2011).




                                        Page 35                                                           GAO-12-339 Defense Management
                                        Appendix I: Observations on Reporting
                                        Provisions in the National Defense
                                        Authorization Act for Fiscal Year 2011




Table 12: Section 862(b)(2)(F)—an Evaluation of the Advantages and Disadvantages of Designating a Commander of a Unified
Combatant Command for Each JROC-Interest Requirements Document to Provide a Joint Evaluation Task Force for Certain
Purposes

Summary assessment                     Developing a joint evaluation task force could have some advantages for combatant
                                       commands, such as advancing a dialogue among the commands, but the disadvantages
                                       of burdening commands with additional duties of participation may outweigh the benefits.
Key GAO observations                   Combatant command officials reported that participation in a task force might provide
                                       greater opportunities for the commands to provide input, but they noted that the
                                       commands already participate in identification and discussion of capability requirements
                                                                                                a
                                       through the current JCIDS process and other forums. Specifically, commands can
                                       participate in
                                       •    experimentation and testing efforts and analysis of alternative teams;
                                       •    discussions of capability proposals that deviate from cost, schedule, or quantity
                                            targets in existing forums;
                                       •    advocating for desired material solutions as end users through integrated priority lists
                                            and service component collaboration; and
                                       •    commenting on concept of operations and doctrine through participation in regular
                                            Functional Capabilities Board meetings.
                                       DOD officials consistently cited the following three key disadvantages to formation of joint
                                       evaluation task forces:
                                       •    Implementation of the task force would represent a shift in responsibilities from the
                                            military services—which have historically completed these duties—to the combatant
                                            commands.
                                       •    Combatant commands have limited manpower and few acquisition-trained specialists
                                            to appropriately fill the role of combat developer.
                                       •    It is unclear how the leadership responsibilities would be divided among the
                                            combatant commands, particularly where a solution is of interest to multiple
                                            commands.
                                       In response to congressional questions about establishing such a task force, the current
                                       Vice Chairman of the Joint Chiefs of Staff stated that such a task force could result in an
                                       additional layer of coordination and staffing between the existing requirements and
                                       acquisition processes, further delaying the identification of capability gaps and
                                       subsequent fielding of systems addressing those gaps.
GAO scope and methodology              To evaluate the advantages and disadvantages of implementing a joint evaluation task
                                       force, we reviewed a March 2010 report prepared by the House Armed Services
                                       Committee Panel on Defense Acquisition Reform and interviewed Joint Staff and
                                       combatant command officials to discuss advantages and disadvantages of the concept.
                                        Sources: DOD and GAO.

                                        Note: See the introduction to this appendix for the full text of this requirement, including the specific
                                        purposes identified.
                                        a
                                         Among the findings of a May 2011 GAO report was that combatant commands noted the importance
                                        of participating in the development of joint requirements, but questioned the value of what they
                                        described as a resource-intensive and time-consuming process that is not always responsive to their
                                        more immediate capability needs. They also questioned the value of a process that results in
                                        decisions that while influential, are advisory to the acquisition and budget process driven by service
                                        investment priorities. See GAO, Defense Management: Perspectives on the Involvement of
                                        Combatant Commands in the Development of Joint Requirements, GAO-11-527R (Washington, D.C.:
                                        May 20, 2011). Although the report did not address involvement in a joint evaluation task force, it
                                        provides combatant command perspectives on the overall JCIDS process, including views on greater
                                        combatant command involvement.




                                        Page 36                                                             GAO-12-339 Defense Management
Appendix II: Scope and Methodology
              Appendix II: Scope and Methodology




              To assess the extent to which the Joint Staff has developed and
              implemented an analytic approach to prioritize capability needs, we
              reviewed relevant legislation and Joint Staff guidance on the roles and
              requirements of the JCIDS process and the JROC as it pertains to
              assigning levels of priority to capability proposals. Specifically, we
              reviewed section 181 of the U.S. Code, Title 10; Chairman of the Joint
              Chiefs of Staff Instruction 3170.01G; Chairman of the Joint Chiefs of Staff
              Instruction 3170.01H (which we reviewed in draft form); and the JCIDS
              Manual. We then compared the current and prior versions of the
              instruction and manual to identify changes in the guidance with respect to
              prioritization of capability proposals. We met with officials from the Joint
              Staff; Department of the Air Force; Department of the Army; Department
              of the Navy; and Office of the Under Secretary of Defense for Acquisition,
              Technology and Logistics to discuss their perspectives on the
              implementation of changes to JCIDS with respect to prioritizing capability
              requirements. In order to understand how the JROC is implementing its
              new approach for prioritizing capabilities, we reviewed briefing materials
              presented at a JROC forum in November 2011. To corroborate our
              understanding of the documents we reviewed, we conducted interviews
              with Joint Staff and Office of the Under Secretary of Defense for
              Acquisition, Technology and Logistics officials. To understand the Joint
              Staff’s recent internal review of JCIDS, we reviewed the charter and
              recommendations, and met with Joint Staff officials to discuss how those
              recommendations from the review might affect the JROC’s prioritization of
              capability proposals. We also reviewed prior reports by GAO, the House
              Armed Services Committee, and the Defense Business Board that
              discussed prioritization of capability proposals through JCIDS, and
              compared the JROC’s current efforts to prioritize with what has been
              reported in the past. We assessed whether the guidance in the JCIDS
              Manual and JCIDS instruction (in draft form during our review and issued
              in January 2012) on prioritization meets the intent of recommendations
              contained in our prior reports.

              To assess the extent to which the JROC has considered aspects of the
              availability and operational support requirements of weapon systems—
              called sustainment—when validating the requirements of proposed
              capability solutions, we reviewed relevant DOD and Joint Staff policy
              documents and related guidance outlining the requirement to develop and
              report sustainment metrics for capability documents. Specifically, we
              reviewed the reporting requirements for major defense acquisition




              Page 37                                         GAO-12-339 Defense Management
Appendix II: Scope and Methodology




programs processed through JCIDS. 1 To determine JCIDS reporting
requirements for capability development documents and to understand
the JCIDS process, we reviewed the JCIDS Manual enclosure pertaining
to sustainment and instructions from the Chairman of the Joint Chiefs of
Staff. We also reviewed prior GAO work on related topics. Further, we
interviewed DOD and Joint Staff officials to discuss preparation,
presentation, and consideration of sustainment data.

We also conducted a case study analysis of select capability
development documents that included sustainment information. We
sought a universe of all capability development documents subject to
reporting the sustainment key performance parameter validated since
2007, when JCIDS began requiring program sponsors to include this
information. We initially obtained a universe of 22 JCIDS capability
development documents from the Office of the Under Secretary of
Defense for Acquisition, Technology and Logistics/Logistics and Materiel
Readiness. We relied on this provided list of documents because the
Joint Staff’s Knowledge Management/Decision Support database did not
produce reliable results of all requirements documents containing
sustainment information. We narrowed the list by eliminating programs
that have been downgraded, truncated, or canceled; programs in which
the sustainment data were not at the key performance parameter level or
were more narrowly defined by only one type of platform; programs
whose information was contained in capability production documents
rather than capability development documents; and programs whose
capability development documents were entered in JCIDS before the full
implementation of the sustainment key performance parameter
requirement. Additionally, several programs did not have supporting
documentation that would allow a review; Joint Staff officials we met with
stated that these documents were not available either because of a
misidentification of the type of capability document that was being
reviewed (a capability production document as opposed to a capability
development document), or because the document was not included in
the Joint Staff’s database for JROC review. These factors led to a refined


1
 Under the version of the JCIDS Manual in effect during the time of our review,
sustainment metrics were to be developed for all acquisition category I programs involving
materiel solutions and acquisition category II and below programs as determined by the
sponsor. See JCIDS Manual, app. B to enc. B, para. 2 (Jan. 31, 2011). The current
version of the manual states that acquisition category II and below programs, with materiel
solutions, shall include the sustainment metrics or sponsor-defined sustainment metrics.
See JCIDS Manual, app. E to enc. B, para. 2 (Jan. 19. 2012).




Page 38                                                  GAO-12-339 Defense Management
Appendix II: Scope and Methodology




universe of 12 requirements documents. We then randomly selected
capability development documents for two programs per service—Army,
Air Force, and Navy—resulting in a total of six programs to serve as case
studies. Because this was a nonprobability sample of programs, the
results are not generalizable to all programs; however, they are illustrative
of the kinds of issues that are possible in such programs. In order to
assess reported sustainment information in the six selected cases, we
performed a content analysis of the documentation available for the six
cases. Two GAO analysts independently reviewed each of the six
capability development documents, assessing whether each of the
individual elements of the JCIDS Manual sustainment metrics was
included, coding the inclusion of each metric as “yes,” “no,” “partial,” and
“don’t know.” The two analysts then discussed and reconciled all initial
disagreements regarding the assigned codes. We then discussed the
results of this content analysis with officials from the Joint Staff and the
Office of the Under Secretary of Defense for Acquisition, Technology and
Logistics/Logistics and Materiel Readiness to verify that the results of our
analysis were valid.

We conducted this performance audit from April 2011 through February
2012 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.




Page 39                                         GAO-12-339 Defense Management
Appendix III: Comments from the
              Appendix III: Comments from the Department
              of Defense



Department of Defense




              Page 40                                      GAO-12-339 Defense Management
Appendix III: Comments from the Department
of Defense




Page 41                                      GAO-12-339 Defense Management
Appendix III: Comments from the Department
of Defense




Page 42                                      GAO-12-339 Defense Management
Appendix IV: GAO Contact and Staff
                  Appendix IV: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  John Pendleton, (202) 512-3489 or pendletonj@gao.gov
GAO Contact
                  In addition to the contact named above, Margaret Morgan, Assistant
Staff             Director; Melissa Blanco; Mae Jones; Kate Lenane; Jennifer Madison;
Acknowledgments   Ron Schwenn; Michael Shaughnessy; Michael Silver; Jennifer Spence;
                  Amie Steele; and Kristy Williams made key contributions to this report.




(351609)
                  Page 43                                       GAO-12-339 Defense Management
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.