oversight

Department of Energy: Problems in the Management and Use of Supercomputers

Published by the Government Accountability Office on 1999-07-14.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                    United States General Accounting Office

GAO                 Testimony
                    Before the Subcommittee on Basic Research, Committee
                    on Science, House of Representatives




To Be Released
on Delivery
Expected at
                    DEPARTMENT OF
2 p.m. EDT
Wednesday           ENERGY
July 14, 1999



                    Problems in the
                    Management and Use of
                    Supercomputers
                    Statement For the Record by
                    Susan D. Kladiva, Associate Director,
                    Energy, Resources, and Science Issues,
                    Resources, Community, and Economic
                    Development Division




GAO/T-RCED-99-257
             Mr. Chairman and Members of the Subcommittee:

             We appreciate the opportunity to provide a statement for the record on the
             Department of Energy’s (DOE) use, acquisition, and management of its
             supercomputing resources. The Department spent about $826 million
             between 1994 and 1997 acquiring and operating supercomputers. These
             computers support a variety of uses, and DOE’s largest supercomputer
             effort—the Accelerated Strategic Computing Initiative—is a critical part of
             DOE’s attempt to build the capability for “virtual testing” as a replacement
             for the actual testing of nuclear weapons. DOE estimates that it will spend
             about $5.2 billion on the strategic computing initiative between fiscal years
             1996 and 2004. Our testimony covers how effectively DOE has managed the
             use and acquisition of its supercomputers, and DOE’s management of its
             strategic computing initiative. Our testimony is based on two reports we
             issued in 1998 and 1999.1

             In summary, we found that DOE had not effectively overseen the use and
             acquisition of supercomputers and that oversight of the $5.2 billion
             strategic computing initiative is hampered by weaknesses in management
             and information processes. In July 1998, we reported that DOE’s national
             laboratories used only about 59 percent of their available supercomputing
             capacity and were missing opportunities to share these resources. At that
             time, DOE had about 17 percent of the world’s supercomputing capacity
             and was planning to almost triple its capacity over the next 3 years. In
             terms of managing acquisitions of supercomputers, we concluded that DOE
             has not effectively overseen the process. Furthermore, the strategic
             computing initiative’s strategic plan is out of date, annual plans have been
             prepared only sporadically, and milestones are not well defined. Currently,
             little information exists to track the program’s progress or to compare the
             program’s accomplishments with its milestones. Consequently, it is
             difficult to determine which of the hundreds of milestones have been met,
             which are behind schedule, or even which are still relevant, given changes
             in the program.


             In July 1998, we reported that nine DOE laboratories have supercomputers.
Background   DOE’s program offices fund supercomputer purchases (or leases), and the
             laboratories’ management and operating contractors acquire and operate
             the systems. According to DOE, it acquires new supercomputers relatively
             frequently because of rapidly changing technology. Since 1993, when

             1
              Information Technology: Department of Energy Does Not Effectively Manage Its Supercomputers
             (GAO/RCED-98-208, July 17, 1998) and Nuclear Weapons: DOE Needs to Improve Oversight of the $5
             Billion Strategic Computing Initiative (GAO/RCED-99-195, June 28, 1999).



             Page 1                                                                    GAO/T-RCED-99-257
                         statistics were first collected, DOE has consistently had several
                         supercomputers that have ranked among the world’s most powerful as
                         measured by a list of the top 500 supercomputers in the world.

                         DOE  uses supercomputers to support two major research missions. First,
                         DOE  uses them in the strategic computing initiative to ensure the safety and
                         reliability of nuclear weapons. The initiative is part of DOE’s Office of
                         Defense Programs’ nuclear stockpile stewardship program. DOE created
                         the initiative as a substitute for the physical testing of nuclear weapons.
                         The initiative is intended to provide the unprecedented simulation
                         capabilities needed to help verify the safety and reliability of U.S. nuclear
                         weapons without nuclear testing. The estimated cost of the initiative is
                         about $5.2 billion for fiscal years 1996 through 2004. Second, DOE’s Office
                         of Science funds non-defense computational research projects, including
                         specific science and engineering problems that require large-scale
                         supercomputing capability.

                         In 1996, the Congress passed the Clinger-Cohen Act, which requires
                         federal agencies to adopt a comprehensive approach to acquiring and
                         managing information technology (including supercomputers), and
                         charged the Office of Management and Budget with oversight
                         responsibility.


                         We found that DOE had not effectively overseen the use and acquisition of
DOE Does Not             supercomputers. During 1997, DOE’s national laboratories used only about
Effectively Manage Its   59 percent of their available supercomputing capacity and were missing
Supercomputers           opportunities to share these resources. Utilization rates varied among the
                         laboratories from about 31 percent to about 75 percent. At that time, DOE
                         had about 17 percent of the world’s supercomputing capacity and was
                         planning to almost triple its capacity during fiscal years 1998 through 2000.
                         Sharing of supercomputers among DOE laboratories and with DOE-funded
                         off-site users was not generally considered by the Department as a way to
                         better use existing resources and/or to forgo the need to acquire more
                         supercomputers. In addition, the largest supercomputers were not being
                         used to run the very large-scale programs that were used to justify their
                         acquisition. In 1997, for example, less than 5 percent of the jobs run on the
                         largest supercomputers used more than one-half of the machines’
                         capabilities.

                         Because DOE does not manage supercomputers centrally, no single person
                         or office within the Department knows at a given time how many



                         Page 2                                                     GAO/T-RCED-99-257
                       supercomputers the Department has agency-wide, what they cost, or how
                       they are being utilized. The Department lacks an investment strategy and a
                       defined process to ensure that supercomputer acquisitions are fully
                       justified and represent the best use of funds among competing priorities.
                       Instead, individual program offices (e.g., Office of Defense Programs,
                       Office of Science) independently procure and operate supercomputers and
                       the Department’s chief information officer does not oversee the
                       acquisition of these supercomputers. As a result, new supercomputers are
                       planned and acquired with little departmental oversight, while
                       underutilized capacity already exists within the Department.

                       We also reported that DOE’s proposed implementation of the
                       Clinger-Cohen Act would not improve departmental oversight. In
                       April 1998, the Department outlined its plan to implement a new
                       investment planning and oversight process for information technology in
                       response to the Clinger-Cohen Act. DOE’s process separately manages
                       administrative and scientific computers, leaving the responsibility for
                       scientific computers—including supercomputers—to individual program
                       offices. This proposed approach reflects the view of the Department’s
                       program offices that supercomputers are research “tools” rather than
                       information technology investments. This approach may also allow DOE’s
                       program offices to continue acquiring supercomputers outside the
                       Department’s normal process for complying with the Clinger-Cohen Act.
                       Contrary to what is envisioned in the Clinger-Cohen Act, this approach
                       effectively places the vast majority of DOE’s information technology
                       resources outside the purview of the chief information officer.


                       In June 1999, we reported that significant weaknesses in DOE’s
DOE Needs to           management and information processes hamper the oversight of the
Improve Oversight of   strategic computing initiative. Although program managers report that
the Strategic          many milestones have been met, the lack of a comprehensive planning and
                       progress tracking system makes assessment of the program’s progress
Computing Initiative   difficult and subjective. Currently, the program’s strategic plan is out of
                       date, annual plans have been prepared only sporadically, and milestones
                       are not well defined. Furthermore, little information exists to track the
                       program’s progress or to compare the program’s accomplishments with its
                       milestones. Consequently, it is difficult to determine which of the
                       hundreds of milestones have been met, which are behind schedule, or
                       even which are still relevant, given changes in the program.




                       Page 3                                                    GAO/T-RCED-99-257
                 Program cost estimates have also increased substantially. In 1995, DOE
                 estimated that costs for the first 5 years of the program (fiscal years 1996
                 through 2001) would be $1.7 billion. By 1999, estimated costs for that same
                 5-year period increased to $2.9 billion. DOE currently estimates that the
                 program will cost about $5.2 billion for fiscal years 1996 through 2004.
                 Some of the cost increases result from the shift to computer-based
                 simulations, while some reflect weaknesses in DOE’s cost estimation.


                 For future contacts regarding this testimony, please contact Susan D.
Contact and      Kladiva at (202) 512-3841. Individuals making contributions to this
Acknowledgment   testimony included Linda Chu, Daniel Feehan, Anne McCaffrey, and
                 Edward Zadjura.




(141364)         Page 4                                                     GAO/T-RCED-99-257
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 37050
Washington, DC 20013

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested