oversight

Weather Service Modernization: Risks Remain That Full Systems Potential Will Not be Achieved

Published by the Government Accountability Office on 1997-04-24.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                          United States General Accounting Office

GAO                       Testimony
                          Before the Subcommittee on Oversight of Government
                          Management, Restructuring and the District of Columbia,
                          Committee on Governmental Affairs, U.S. Senate


For Release on Delivery
Expected at
12:30 p.m.
                          WEATHER SERVICE
Thursday,
April 24, 1997            MODERNIZATION

                          Risks Remain That Full
                          Systems Potential Will Not
                          Be Achieved
                          Statement of Joel C. Willemssen
                          Director, Information Resources Management
                          Accounting and Information Management Division




GAO/T-AIMD-97-85
             Mr. Chairman and Members of the Subcommittee:

             We are pleased to be here today to discuss the National Weather Service’s
             (NWS) systems modernization program. At an estimated cost of about
             $4.5 billion, it is one of the largest modernization programs in the federal
             government. The modernization is vital to the Weather Service’s plans for
             improving operations; at the same time, it is intended to help NWS
             streamline and downsize its organization, and is an effort that we continue
             to endorse. As with most large systems-development projects, however,
             this program faces persistent challenges that must be overcome if the
             considerable anticipated benefits of full modernization are to be realized.
             Our concerns led us to place the Weather Service effort on our 1995 list of
             high-risk government programs, where it remains today.1

             The work of the National Weather Service is critically important to all
             Americans, as the United States experiences considerable severe weather.
             In a typical year, the United States is pummeled by about 10,000 violent
             thunderstorms; 5,000 floods; 1,000 tornadoes; and several hurricanes. As
             we have seen in recent months, unpredictable weather can wreak havoc in
             people’s lives; sometimes the difference between tragedy and recoverable
             loss lies in the ability of early forecasts and warnings of potentially
             dangerous weather to help protect life and property.


             NWS  uses a variety of systems and manual processes to collect, process,
Background   and disseminate weather data to and among its network of field offices
             and regional and national centers. Prior to the modernization, these
             systems and processes were largely outdated. Radar equipment dated back
             to the 1950s, and much of the current information processing, display, and
             data communications system has been in use since the 1970s.

             To enhance its ability to deliver weather services, NWS determined some 15
             years ago to use the power of technology to “do more with less.” To reach
             the goal of better forecasting and earlier warnings with a smaller,
             downsized operation, the Weather Service has been acquiring new
             observing systems—including radars, satellites, and ground-based
             sensors—as well as powerful forecaster workstations. The goals of the
             modernization were to (1) achieve more uniform weather services
             nationwide, (2) improve forecasting, (3) provide more reliable detection
             and prediction of severe weather and flooding, (4) permit more

             1
             High-Risk Series: An Overview (GAO/HR-95-1, February 1995) and High-Risk Series: Information
             Management and Technology (GAO/HR-97-9, February 1997).



             Page 1                                                                       GAO/T-AIMD-97-85
                            cost-effective operations, and (5) achieve higher productivity. The
                            modernization includes four major systems-development programs, which
                            I will briefly describe.


The Advanced Weather        This program integrates, for the first time, satellite, radar, and other data
Interactive Processing      to support weather forecaster decision-making and communications; it is
System (AWIPS)              the linchpin of the NWS modernization. Operating under a $550-million
                            funding cap, the system is expected to be fully deployed in 1999. AWIPS
                            development systems have been delivered to 16 locations nationwide; this
                            represents the first two of six modules, or “builds.” AWIPS is planned for a
                            total of 152 locations once fully deployed.


The Next Generation         This is a program to acquire, launch, and control five satellites for
Geostationary Operational   identifying and tracking severe weather events, such as hurricanes. The
Environmental Satellite     first satellite was launched in 1994, and the second in 1995. Three more
                            satellites are planned for launch between now and 2002. The total cost for
(GOES-Next)                 these five satellites is estimated to be just under $2 billion.


The Next Generation         This is a program to acquire 163 Doppler radars.2 Largely deployed, these
Weather Radar (NEXRAD)      radars have helped NWS increase the accuracy and timeliness of warnings
                            for severe thunderstorms, tornadoes, and other hazardous weather events.
                            Scheduled for completion this year, 121 of a planned 123 NWS NEXRAD
                            radars have been delivered to operational locations. The cost of this
                            program is just under $1.5 billion.


The Automated Surface       This is a program to automate and enhance methods for collecting,
Observing System (ASOS)     processing, and displaying surface weather conditions, such as
                            temperature and precipitation, and to replace human weather observers.
                            Scheduled for completion in fiscal year 1998, the system has been installed
                            at 265 of 314 planned NWS operational locations. Estimated costs for ASOS
                            are about $351 million; this includes the NWS units and 554 units for the
                            Federal Aviation Administration and the Department of Defense.

                            The modernization also includes upgrades to existing systems, improved
                            weather models, and the acquisition of several smaller systems. In
                            addition, NWS is restructuring its field offices to be more efficient; table 1
                            indicates the before-and-after plan.

                            2
                             This includes radars for NWS, the Air Force, and the Federal Aviation Administration.



                            Page 2                                                                           GAO/T-AIMD-97-85
Table 1: NWS Office Restructuring
Plan                                Pre-modernization                         Future
                                    52 Weather Service Forecast Offices       119 Weather Forecast Officesa
                                    204 Weather Service Offices
                                    3 National Centers                        9 National Centers
                                    13 River Forecast Centers                 13 River Forecast Centersa
                                    a
                                        These offices are to be co-located.

                                    Source: NWS.




                                    The Weather Service has generated better data—particularly with the new
Important Successes                 radars and satellites—and greatly improved forecasts and warnings. These
Achieved, Yet                       can be related directly to saving lives and reducing the effects of natural
Problems Have                       disasters. As shown in figure 1, lead times of warnings for severe storms
                                    and tornadoes improved by about 5 minutes between 1986 and 1996,
Hindered the                        which is not insignificant. With tornadoes, for example, it can mean the
Modernization                       difference in whether people have time to reach shelter. In some
                                    instances, lead times are much earlier. Last year, for instance, NWS issued
                                    flood potential “statements” 2-3 days in advance of Hurricane Fran. Flash
                                    flood warnings were issued with 6 hours’ lead time. Similarly, in the East
                                    Coast blizzard of 1996, NWS issued forecasts 3 to 5 days in advance.




                                    Page 3                                                           GAO/T-AIMD-97-85
Figure 1: NWS Warning Lead Time for Severe Local Storms, 1986-1996 (in Minutes)


 Minutes
 20




 15


                                      All severe local storms
 10




   5
                                             Tornadoes

   0
       1986      87        88       89        90          91          92          93          94          95          96



                                         Source: NWS.




                                         Notwithstanding such successes, however, each of the four programs has
                                         experienced cost increases and schedule delays.3 Some of these increases


                                         3
                                          A list of related GAO reports and testimony on the NWS modernization, including its four primary
                                         components, appears at the end of this statement.



                                         Page 4                                                                          GAO/T-AIMD-97-85
and delays can be attributed to changes in requirements; others were
caused by program management and development problems.

We reported in 1995 that six of eight sensors in the ASOS system did not
meet contract specifications for accuracy or performance. For example,
the precipitation accumulation sensor underreported rainfall amounts
during heavy downpours, and the temperature and dew point sensor
readings frequently fell short of dew point reliability requirements. Some
of these shortfalls occurred because of the contractor’s failure to deliver
products that met specifications, and others resulted from the failure of
government-furnished equipment to meet specifications. In addition, we
found that ASOS users from the aviation, meteorology, and climate
communities had needs that the ASOS system, as specified, did not satisfy.

We recommended that NWS define and prioritize—in conjunction with ASOS
users—all system corrections, enhancements, and supplements necessary
to meet valid user needs. We further recommended that NWS
formulate—again in conjunction with ASOS users—explicit system
performance and cost/benefit criteria governing the release of human
observers. Because of these problems, NWS delayed plans for releasing
human weather observers and implemented actions to correct shortfalls in
meeting specifications and to address other user concerns.

In reference to NEXRAD, we testified in 1995 that many NWS and Air Force
radars were not available nearly as often as required. For example,
between 10 and 62 percent of Air Force NEXRAD radars were falling short of
availability requirements. (NWS did not know if its radars were meeting the
availability requirement because it was not monitoring availability on a
site-by-site basis.) Further, we found that a radar upgrade to address one
cause of unavailability—the lack of an uninterruptible power supply—was
not to be completed until fiscal years 1999 and 2002 for the Air Force and
NWS, respectively.


We recommended that NWS analyze and monitor system availability data
on a site-specific basis for operational NEXRADs and correct any shortfalls
in system availability revealed by the analysis. We also recommended that
the Air Force improve the reliability of Air Force NEXRAD availability data
and correct any shortfalls found. NWS and the Air Force did initiate steps in
1995 to implement our recommendations to improve NEXRAD availability.

In terms of staffing, the sizable reductions promised as a result of the
modernization will not be realized. While NWS originally planned to reduce



Page 5                                                       GAO/T-AIMD-97-85
                  staff by 21 percent, we reported in 1995 that the goal had been scaled back
                  to 8 percent. NWS attributes the reduced goal primarily to the need for
                  more staff than originally envisioned to operate new systems, and to other
                  unanticipated requirements.


                  Mr. Chairman, the National Oceanic and Atmospheric Administration’s
Remaining Risks   (NOAA) ultimate success in completing the modernization depends, in part,
                  on how well and how quickly it can complete a systems architecture4 and
                  address specific risks associated with the crucial AWIPS system. The
                  modernization needs an overall architecture to guide systems
                  development; NWS agrees that such a technical blueprint is necessary, and
                  is currently working on one. Yet until such an architecture is developed
                  and enforced, the modernization will likely continue to be subject to
                  higher costs and reduced performance. This is an important point as
                  component systems continue to evolve to meet additional demands and
                  take advantage of improved technology. We cannot emphasize too
                  strongly the need for an overall architecture to guide system evolution. An
                  architecture would help ensure that changes to NEXRAD, for example, are
                  compatible with the many systems with which NEXRAD must exchange
                  data.

                  As we have reported several times over the past few years, full utilization
                  of the data from the new observing systems has been prevented by delays
                  and continuing problems with AWIPS. We have made several
                  recommendations that we feel will strengthen the Weather Service’s ability
                  to acquire AWIPS. First, we recommended that NWS ensure that each “build”
                  is fully tested and all material defects corrected before beginning software
                  development associated with the next build. Second, we recommended
                  that NWS establish a software quality assurance program to increase the
                  probability of delivering promised AWIPS capability on time and within
                  budget. Third, we recommended that NWS obtain an independent
                  assessment of the cost to develop and deploy AWIPS.




                  4
                   A systems architecture is a blueprint to guide and constrain the development and evolution (i.e.,
                  maintenance) of a collection of related systems; it can be viewed as having both logical and technical
                  components. At the logical level, the architecture provides a high-level description of the
                  organizational mission being accomplished, the business functions being performed and the
                  relationships among functions, the information needed to perform the functions, and the flow of
                  information among functions. At the technical level, the architecture provides the rules and standards
                  needed to ensure that the interrelated systems are built to be interoperable, portable, and
                  maintainable. These include specifications of critical aspects of the component systems’ hardware,
                  software, communications, data, security, and performance characteristics.



                  Page 6                                                                           GAO/T-AIMD-97-85
Progress to date in these areas has, however, been uneven, and we remain
concerned about AWIPS development risks—risks that threaten the
system’s ability to be completed on time, within budget, and with the
functional capability that AWIPS must be able to provide. Until AWIPS is
deployed and functioning properly, NWS will not be able to take full
advantage of the nearly $4 billion investment it has made in the other
components of the modernization.

After early successes in demonstrating the technical feasibility of system
functions, design problems and disagreements between NOAA and the
development contractor in 1993-1994 stymied progress. Some development
responsibility was brought in-house—to NWS/NOAA labs—in 1995. The AWIPS
program strategy was changed again in 1996, when even more
development responsibility—for AWIPS data acceptance, processing, and
display capabilities—was brought in-house, primarily to NOAA’s Forecast
Systems Laboratory (FSL). At that time, NWS decided to use FSL’s prototype
system, called Weather Forecast Office (WFO)-Advanced, which was being
developed in parallel with AWIPS as a risk-reduction tactic.

NWS officials chose WFO-Advanced because of its demonstrated superior
data-acceptance, processing, and display capability over the contractor’s
version, hoping that it would enable the agency to deploy these AWIPS
capabilities to field operations more quickly. The contractor did, however,
retain responsibility for communications, system monitoring and control,
and other capabilities. With these changes, NWS expects AWIPS to make its
1999 target date for full deployment, within the $550-million cap.

As we reported in December 1994, NOAA/NWS labs are research and
development operations that primarily develop prototype systems; as
such, they did not employ software development processes characteristic
of a software-production environment. Specifically, the labs did not have
the software quality assurance and configuration management processes,
among others, sufficient to ensure production of stable, reliable software
code.5 Developing software code for use in one or two prototype
installations requires a far less rigorous approach than what is needed
when nationwide deployment is planned. However, some of the software
the NOAA/NWS labs were developing was intended for operational use in


5
 Software quality assurance refers to a program that independently (1) monitors whether the
software and the processes used to develop it fully satisfy established standards and procedures and
(2) ensures that any deficiencies in the software product, process, or their associated standards are
swiftly brought to management’s attention. Software configuration management refers to a process
by which changes to software products are controlled. It includes identification of products to be
controlled, accounting for changes to these products, and reporting on the products’ status.



Page 7                                                                            GAO/T-AIMD-97-85
AWIPSand was essentially being handed off directly from the labs to the
contractor. We therefore recommended that NWS and NOAA strengthen their
processes for developing production-quality software code.

With the 1995 and 1996 AWIPS development changes, significantly more
design and development responsibility has been transferred to the
government, in particular to NOAA’s FSL. In visiting FSL in Boulder,
Colorado, we found that—with the exception of one subsystem that we
specifically discussed in 1994—the question of capability remained: lab
quality assurance and configuration management processes for
production-level software were still lacking. However, NWS and NOAA
officials said that they have heeded our 1994 recommendations and are
improving their processes in other ways. They said that in order to
preserve the labs’ research and development missions, they do not wish to
impose any unnecessary, rigorous software development procedures on
the labs. Instead, NOAA management plans to play a more active role in
preparing the government-furnished software for the contractor.

According to NWS officials, they plan to improve the software development
processes for WFO-Advanced and other government-developed software
using staff from NWS headquarters, NOAA’s systems acquisitions division,
and the contractor. Specifically, NWS plans to (1) more fully document the
lab’s design and software code, (2) design the integration of
government-furnished software and contractor-developed software,
(3) fully test all government software before it is turned over to the
contractor, and (4) strengthen quality assurance and configuration
management. To help accomplish this, NWS has established several specific
contract task orders.

Weather Service officials acknowledge that preparing WFO-Advanced for
the contractor is a large task because it comprises such a significant
portion of the AWIPS software. In addition, officials understand that there is
no room for schedule delays due to unforeseen problems. They feel
confident, however, that they can meet this challenge because of the steps
I have just described, and because they have experience in turning
government software over to a contractor. For example, NWS’ Office of
Hydrology provided hydro-meteorological software to the contractor for
the first AWIPS module (“build 1”), which was successfully tested last
summer. In addition, NWS officials said that they are applying to AWIPS
lessons learned from their configuration management experiences in the
NEXRAD and ASOS development projects.




Page 8                                                        GAO/T-AIMD-97-85
                   NOAA  has put into place appropriate plans and procedures to mitigate these
                   risks; how it implements these plans and procedures will be critical if NOAA
                   is to avoid turning the risks into actual problems. Unfortunately, systems
                   development risks in large projects such as AWIPS frequently do turn into
                   problems. And, as discussed, AWIPS has suffered development setbacks in
                   the past. Given these circumstances, we believe it will be extremely
                   difficult for NOAA and NWS to develop and deploy the AWIPS system within
                   the $550-million cap.

                   What can be done to minimize such risks? First, NOAA and NWS
                   management need to be vigilant to identify new problems with AWIPS
                   software development. New software and WFO-Advanced must be fully
                   tested to ensure that they are up to production quality and will not cause
                   complications when integrated with other AWIPS software. Second, we
                   believe that NOAA needs to renegotiate as quickly as possible the contract
                   for AWIPS builds 4 through 6. While NOAA officials expect no major cost or
                   schedule changes, this is not a guarantee; NOAA must exercise close
                   oversight of this process.


                   Another important element of the Weather Service modernization is the
Geostationary      acquisition of geostationary operational environmental satellites (GOES).
Operational        These satellites are uniquely positioned to be able to observe the
Environmental      development of severe weather, such as hurricanes and thunderstorms,
                   and provide information allowing forecasters to issue timely warnings.
Satellite (GOES)   Satellites in the current series will, however, begin to reach the end of
                   their useful lives within 5 years; NOAA is now planning to procure
                   replacements, which will be very similar to the current satellites. At issue,
                   Mr. Chairman, is the type of satellite system to build for the longer term,
                   especially in light of NOAA’s budget, which is likely to remain constrained
                   in the immediate years ahead. Our report on both short- and long-term
                   satellite replacements was released last month.6

                   In brief, we found NOAA’s approach for the near term reasonable, although
                   we recommended that the agency clarify its policy for replacing partially
                   failed satellites and backing up planned launches. For the longer term, we
                   concluded that changing the GOES system design offers many potential
                   benefits: improved performance, lower costs, and more closely meeting
                   the needs of forecasters.



                   6
                    Weather Satellites: Planning for the Geostationary Satellite Program Needs More Attention
                   (GAO/AIMD-97-37, March 13, 1997).



                   Page 9                                                                          GAO/T-AIMD-97-85
Several new approaches have been suggested in recent years, by
government, academic, and industry experts; many include technologies
unavailable when the present series of satellites was designed. These
approaches have pros and cons; all options would require careful
engineering analysis before an informed decision about the future of the
GOES program can be made.


Our concern centers on NOAA’s delay in conducting such analyses and
developing specific proposals. At present, NOAA anticipates beginning its
follow-up program in 2003 at the earliest. Given that developing a new
satellite takes up to 10 years, deferring a start until 2003 likely means that
NOAA will have to rely on its current, early-1980s-design satellites until
about 2013.

Mr. Chairman, given the range of options that exist for a significantly
improved GOES system, the Congress may wish to evaluate the costs and
benefits of different approaches to the timing, funding, and scope of the
follow-up program. This could include a potential role for the National
Aeronautics and Space Administration’s advanced spacecraft technology
programs.

In summary, we see clear benefits in the National Weather Service
modernization—improved forecasts and warnings. We also see
risks—risks that can only be reduced through development and
enforcement of a systems modernization architecture, careful
implementation of planned mitigation techniques in the case of AWIPS, and
commitment to earlier planning in the case of the GOES satellites.


This concludes my statement, Mr. Chairman. I would be happy to respond
to any questions you or other Members of the Subcommittee may have at
this time.




Page 10                                                        GAO/T-AIMD-97-85
Page 11   GAO/T-AIMD-97-85
Related GAO Products


              National Oceanic and Atmospheric Administration: Weather Service
              Modernization and NOAA Corps Issues (GAO/T-AIMD/GGD-97-63, March 13,
              1997).

              Weather Satellites: Planning for the Geostationary Operational
              Environmental Satellite Program Needs More Attention (GAO/AIMD-97-37,
              March 13, 1997).

              High-Risk Series: Information Management and Technology (GAO/HR-97-9,
              February 1997).

              NOAA   Satellites (GAO/AIMD-96-141R, September 13, 1996).

              Weather Forecasting: Recommendations to Address New Weather
              Processing System Development Risks (GAO/AIMD-96-74, May 13, 1996).

              Weather Forecasting: NWS Has Not Demonstrated that New Processing
              System Will Improve Mission Effectiveness (GAO/AIMD-96-29, February 29,
              1996).

              Weather Forecasting: New Processing System Faces Uncertainties and
              Risks (GAO/T-AIMD-96-47, February 29, 1996).

              Weather Forecasting: Radars Far Superior to Predecessors, but Location
              and Availability Questions Remain (GAO/T-AIMD-96-2, October 17, 1995).

              Weather Service Modernization Staffing (GAO/AIMD-95-239R, September 26,
              1995).

              Weather Forecasting: Radar Availability Requirements Not Being Met
              (GAO/AIMD-95-132, May 31, 1995).

              Weather Forecasting: Unmet Needs and Unknown Costs Warrant
              Reassessment of Observing System Plans (GAO/AIMD-95-81, April 21, 1995).

              Weather Service Modernization Questions (GAO/AIMD-95-106R, March 10,
              1995).

              Weather Service Modernization: Despite Progress, Significant Problems
              and Risks Remain (GAO/T-AIMD-95-87, February 21, 1995).

              Meteorological Satellites (GAO/NSIAD-95-87R, February 6, 1995).



              Page 12                                                       GAO/T-AIMD-97-85
           Related GAO Products




           High-Risk Series: An Overview (GAO/HR-95-1, February 1995).

           Weather Forecasting: Improvements Needed in Laboratory Software
           Development Processes (GAO/AIMD-95-24, December 14, 1994).

           Weather Forecasting: Systems Architecture Needed for National Weather
           Service Modernization (GAO/AIMD-94-28, March 11, 1994).

           Weather Forecasting: Important Issues on Automated Weather Processing
           System Need Resolution (GAO/IMTEC-93-12BR, January 6, 1993).

           Weather Satellites: Action Needed To Resolve Status of the U.S.
           Geostationary Satellite Program (GAO/NSIAD-91-252, July 24, 1991).

           Weather Satellites: Cost Growth and Development Delays Jeopardize U.S.
           Forecasting Ability (GAO/NSIAD-89-169, June 30, 1989).




(511425)   Page 13                                                       GAO/T-AIMD-97-85
Ordering Information

The first copy of each GAO report and testimony is free.
Additional copies are $2 each. Orders should be sent to the
following address, accompanied by a check or money order
made out to the Superintendent of Documents, when
necessary. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office
P.O. Box 6015
Gaithersburg, MD 20884-6015

or visit:

Room 1100
700 4th St. NW (corner of 4th and G Sts. NW)
U.S. General Accounting Office
Washington, DC

Orders may also be placed by calling (202) 512-6000
or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone phone. A recorded menu will provide information on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov




PRINTED ON    RECYCLED PAPER
United States                       Bulk Rate
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. G100
Official Business
Penalty for Private Use $300

Address Correction Requested