oversight

Information Security: Effective Patch Management is Critical to Mitigating Software Vulnerabilities

Published by the Government Accountability Office on 2003-09-10.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                United States General Accounting Office

GAO                             Testimony
                                Before the Subcommittee on Technology
                                Information Policy, Intergovernmental
                                Relations, and the Census, House
                                Committee on Government Reform
For Release on Delivery
Expected at 10:00 a.m. EDT
Wednesday, September 10, 2003   INFORMATION SECURITY
                                Effective Patch
                                Management is Critical to
                                Mitigating Software
                                Vulnerabilities
                                Statement of Robert F. Dacey
                                Director, Information Security Issues




GAO-03-1138T
                                                September 10, 2003


                                                INFORMATION SECURITY

                                                Effective Patch Management is Critical to
Highlights of GAO-03-1138T, testimony           Mitigating Software Vulnerabilities
before the Subcommittee on Technology,
Information Policy, Intergovernmental
Relations, and the Census, House
Committee on Government Reform




Attacks on computer systems—in                  The increase in reported information systems vulnerabilities has been
government and the private                      staggering, especially in the past 3 years (see chart). Automated attacks are
sector—are increasing at an                     successfully exploiting such software vulnerabilities, as increasingly
alarming rate, placing both federal             sophisticated hacking tools become more readily available and easier to use.
and private-sector operations and               The response to two recent critical vulnerabilities in Microsoft Corporation
assets at considerable risk. By
                                                and Cisco Systems, Inc., products illustrates the collaborative efforts
exploiting software vulnerabilities,
hackers can cause significant                   between federal entities and the information security community to combat
damage. While patches, or software              potential attacks.
fixes, for these vulnerabilities are
often well publicized and available,            Patch management is one means of dealing with these increasing
they are frequently not quickly or              vulnerabilities to cybersecurity. Critical elements to the patch management
correctly applied.                              process include management support, standardized policies, dedicated
                                                resources, risk assessment, and testing. In addition to working with software
The federal government recently                 vendors and security research groups to develop patches or temporary
awarded a contract for a                        solutions, the federal government has taken a number of other steps to
governmentwide patch notification               address software vulnerabilities. For example, offered without charge to
service designed to provide                     federal agencies, the federal patch notification service provides subscribers
agencies with information to
support effective patching. Forty-
                                                with information on trusted, authenticated patches available for their
one agencies now subscribe to this              technologies. At present, the government is considering broadening the
service.                                        scope of these services and capabilities, along with the number of users.
                                                Other specific tools also exist that can assist in performing patch
At the request of the Chairman of               management.
the Subcommittee on Technology,
Information Policy,                             In addition to implementing effective patch management practices,
Intergovernmental Relations, and                several additional steps can be taken when addressing software
the Census, GAO reviewed (1) two                vulnerabilities. Such steps include stronger software engineering practices
recent software vulnerabilities and
                                                and continuing research and development into new approaches toward
related responses; (2) effective
patch management practices,                     computer security.
related federal efforts, and other
available tools; and (3) additional             Security Vulnerabilities, 1995—First Half of 2003 (11,155 in the aggregate)
steps that can be taken to better
protect sensitive information
systems from software
vulnerabilities.




www.gao.gov/cgi-bin/getrpt–GAO-03-1138T.

To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Robert F.
Dacey at (202) 512-3317 or daceyr@gao.gov.
                   Mr. Chairman and Members of the Subcommittee:

                   I am pleased to be here today to participate in the Subcommittee’s hearing
                   on recent cyber incidents and the role of software patch management1 in
                   mitigating the risks that these types of events will recur. Current incidents
                   inundating the Internet, coupled with the increasing number and
                   sophistication of attacks, place both federal and private-sector operations
                   and assets at considerable risk. Several of these incidents exploited
                   software vulnerabilities for which patches were already publicly available.

                   In my testimony today I will discuss (1) two recent software vulnerabilities
                   and related responses; (2) effective patch management practices, related
                   federal efforts, and other available tools; and (3) additional steps that can
                   be taken to better protect sensitive information systems from software
                   vulnerabilities.

                   In preparing for this testimony, we analyzed professional information
                   technology security literature, including research studies and reports
                   about cybersecurity-related vulnerabilities. We also interviewed private-
                   sector and federal officials about their patch management experiences.
                   And we analyzed relevant documents and interviewed officials of the
                   Patch Authentication and Dissemination Capability (PADC) service and
                   supporting contractors to determine the service’s current capabilities and
                   usage. Finally, we reviewed actions taken by PADC and agency officials in
                   response to recent cybersecurity vulnerabilities. Our work was performed
                   in accordance with generally accepted government auditing standards,
                   from June to September 2003.


                   Since 1995, over 11,000 security vulnerabilities in software products have
Results in Brief   been reported. Along with these increasing vulnerabilities, the
                   sophistication of attack technology has steadily advanced. Attacks such as
                   viruses and worms2 that once took weeks or months to propagate over the
                   Internet now take only hours, or even minutes. In just the past 3 months,


                   1
                    A patch is a piece of software code that is inserted into a program to temporarily fix a
                   defect. Patches are developed and released by software vendors when vulnerabilities are
                   discovered. Patch management is the process of effectively applying available patches.
                   2
                    A virus is a program that “infects” computer files, usually executable programs, by
                   inserting a copy of itself into the file. In contrast, a worm is an independent computer
                   program that reproduces by copying itself from one system to another across a network.
                   Unlike computer viruses, worms do not require human involvement to propagate.



                   Page 1                                                                      GAO-03-1138T
                      two critical and widespread vulnerabilities were identified in products
                      from Microsoft Corporation and Cisco Systems, Inc. Federal agencies
                      were affected by the Blaster and Welchia worms, which exploited the
                      Microsoft vulnerability. The response to these recent events illustrates
                      how federal entities are communicating and coordinating with software
                      vendors and security research groups to combat such attacks.

                      Effective patch management, one means of dealing with these increasing
                      security threats, includes several critical elements, such as top
                      management support, standardized policies, dedicated resources, risk
                      assessment, and testing. In the federal arena, the Department of Homeland
                      Security now provides agencies with information on trusted, authenticated
                      patches for their specific technologies without charge. This service,
                      known as PADC, currently has 41 agency subscribers. Other tools and
                      resources also exist that can assist in performing patch management
                      functions.

                      Patch management is but one—albeit important and essential—
                      component in the protection of systems from security vulnerabilities.
                      However, in the longer term, the nation’s ability to withstand attacks may
                      ultimately come from more rigorous software engineering practices and
                      better tools and technologies. My statement today will highlight steps we
                      can take now and in the future to help reduce our vulnerability to cyber
                      intrusion.


                      Flaws in software code that could cause a program to malfunction
Background:           generally result from programming errors that occur during software
Vulnerabilities and   development. The increasing complexity and size of software programs
                      contribute to the growth in software flaws. For example, Microsoft
Exploits              Windows 2000 reportedly contains about 35 million lines of code,
                      compared with about 15 million lines for Windows 95. As reported by the
                      National Institute of Standards and Technology (NIST), based on various
                      studies of code inspections, most estimates suggest that there are as many
                      as 20 flaws per thousand lines of software code. While most flaws do not
                      create security vulnerabilities,3 the potential for these errors reflects the




                      3
                       A vulnerability is the existence of a flaw or weakness in hardware or software that can be
                      exploited resulting in a violation of an implicit or explicit security policy.



                      Page 2                                                                      GAO-03-1138T
difficulty and complexity involved in delivering trustworthy code.4 By
exploiting software vulnerabilities, hackers and others who spread
malicious code can cause significant damage, ranging from Web site
defacement to taking control of entire systems, and thereby being able to
read, modify, or delete sensitive information, destroy systems, disrupt
operations, or launch attacks against other organizations’ systems.

Between 1995 and the first half of 2003, the CERT Coordination Center5
(CERT/CC) reported 11,155 security vulnerabilities that resulted from
software flaws. Figure 1 illustrates the dramatic growth in security
vulnerabilities over these years.

Figure 1: Security Vulnerabilities, 1995—first half of 2003




The growing number of known vulnerabilities increases the number of
potential attacks created by the hacker community. As vulnerabilities are
discovered, attackers may attempt to exploit them. Attacks can be
launched against specific targets or widely distributed through viruses and
worms.


4
 National Institute for Standards and Technology, Procedures for Handling Security
Patches: Recommendations of the National Institute of Standards and Technology, NIST
Special Publication 800-40 (Gaithersburg, MD: August 2002).
5
 The CERT/CC is a center of Internet security expertise at the Software Engineering
Institute, a federally funded research and development center operated by Carnegie-Mellon
University.



Page 3                                                                    GAO-03-1138T
Worms and viruses are commonly used to launch denial-of-service attacks,
which generally flood targeted networks and systems with so much
transmission of data that regular traffic is either slowed or completely
interrupted. Such attacks have been utilized ever since the groundbreaking
Morris worm, which brought 10 percent of the systems connected to
Internet systems to a halt in November 1988. In 2001, the Code Red worm
used a denial-of-service attack to affect millions of computer users by
shutting down Web sites, slowing Internet service, and disrupting business
and government operations.6 This type of attack continues to be used by
recent worms, including Blaster, which I will discuss further later in my
testimony.

The sophistication and effectiveness of cyber attack have steadily
advanced. Because automated tools now exist, CERT/CC has noted,
attacks that once took weeks or months to propagate over the Internet
now take just hours, or even minutes. Code Red achieved an infection rate
of over 20,000 systems within 10 minutes, foreshadowing more damaging
and devastating attacks. Indeed, earlier this year, the Slammer worm,
which successfully attacked at least 75,000 systems, became the fastest
computer worm in history, infecting more than 90 percent of vulnerable
systems within 10 minutes.

Frequently, skilled hackers develop exploitation tools and post them on
Internet hacking sites. These tools are then readily available for others to
download, allowing even inexperienced programmers to create a
computer virus or to literally point and click to launch an attack.
According to a NIST publication, 30 to 40 new attack tools are posted to
the Internet every month.7

The threat to systems connected to the Internet is illustrated by the
increasing number of computer security incidents reported to CERT/CC.
This number rose from just under 10,000 in 1999 to over 52,000 in 2001, to
about 82,000 in 2002, and to over 76,000 for the first and second quarters of
2003. And these are only the incidents that are reported. According to the
Director of CERT/CC, as much as 80 percent of actual incidents go



6
 U.S. General Accounting Office, Information Security: Code Red, Code Red II, and
SirCam Attacks Highlight Need for Proactive Measures, GAO-01-1073T (Washington D.C.:
August 29, 2001).
7
 U.S. General Accounting Office, Information Security: Weaknesses Place Commerce Data
and Operations at Serious Risk, GAO-01-751 (Washington D.C.: August 13, 2001).



Page 4                                                                GAO-03-1138T
    unreported, in most cases because the organization was either unable to
    recognize that its systems had been penetrated (because there were no
    indications of penetration or attack) or because it was reluctant to report
    an incident. Figure 2 illustrates the number of incidents reported to
    CERT/CC from 1995 through the second quarter of 2003.

    Figure 2: Information Security Incidents, 1995—first half of 2003




    According to CERT/CC, about 95 percent of all network intrusions could
    be avoided by keeping systems up to date with appropriate patches;
    however, such patches are often not quickly or correctly applied.
    Maintaining current patches is becoming more difficult, as the length of
    time between the awareness of a vulnerability and the introduction of an
    exploit is shrinking. For example, the Blaster worm was released almost
    simultaneously with the announcement of the vulnerability it exploited.

    Successful attacks on unpatched software vulnerabilities have caused
    billions of dollars in damage. Following are examples of significant
    damage caused by worms that could have been prevented had available
    patches been effectively installed:

•   In September 2001 the Nimda worm appeared, reportedly infecting
    hundreds of thousands of computers around the world, using some of the
    most significant attack methods of Code Red II and 1999’s Melissa virus
    that allowed it to spread widely in a short amount of time. A patch had
    been made publicly available the previous month.



    Page 5                                                              GAO-03-1138T
•   On January 25, 2003, Slammer triggered a global Internet slowdown and
    caused considerable harm through network outages and other unforeseen
    consequences. As we discussed in our April testimony, the worm
    reportedly shut down a 911 emergency call center, canceled airline flights,
    and caused automated teller machine (ATM) failures.8 According to media
    reports, First USA Inc., an Internet service provider, experienced network
    performance problems after an attack by the Slammer worm, due to a
    failure to patch three of its systems. Additionally, the Nuclear Regulatory
    Commission reported that Slammer also infected a nuclear power plant’s
    network, resulting in the inability of the computers to communicate with
    each other, disrupting two important systems at the facility. In July 2002,
    Microsoft had released a patch for its software vulnerability that was
    exploited by Slammer. Nevertheless, according to media reports, some of
    Microsoft’s own systems were infected by Slammer.

    In addition to understanding the threat posed by security vulnerabilities, it
    is useful to understand the process of vulnerability identification and
    response. In general, when security vulnerabilities are discovered, a
    process is initiated to effectively address the situation through appropriate
    reporting and response. Typically, this process begins when security
    vulnerabilities are discovered by software vendors, security research
    groups, users, or other interested parties, including the hacker community.
    When a software vendor is made aware of a vulnerability in its product,
    the vendor typically first validates that the vulnerability indeed exists. If
    the vulnerability is deemed critical, the vendor may convene a group of
    experts, including major clients and key incident-response groups such the
    Federal Computer Incident Response Center (FedCIRC) and CERT/CC, to
    discuss and plan remediation and response efforts.

    After a vulnerability is validated, the software vendor develops and tests a
    patch and/or workaround. A workaround may entail blocking access to or
    disabling vulnerable programs.

    The incident response groups and the vendor typically prepare a detailed
    public advisory to be released at a set time. The advisory often contains a
    description of the vulnerability, including its level of criticality; systems
    that are affected; potential impact if exploited; recommendations for
    workarounds, and Web site links where a patch (if publicly available) can


    8
    U.S. General Accounting Office, Information Security: Progress Made, But Challenges
    Remain to Protect Federal Systems and the Nation’s Critical Infrastructures,
    GAO-03-564T (Washington, D.C.: April 8, 2003).



    Page 6                                                                 GAO-03-1138T
                                 be downloaded. Incident-response groups as well as software vendors may
                                 continue to issue updates as new information about the vulnerability is
                                 discovered. When a worm or virus is reported that exploits a vulnerability,
                                 virus detection software vendors also participate in the process. Such
                                 vendors develop and make available to their subscribers downloadable
                                 “signature files” that are used, in conjunction with their software, to
                                 identify and stop the virus or worm from infecting systems protected by
                                 their software. The Organization for Internet Safety (OIS), which consists
                                 of leading security researchers and vendors, recently issued a voluntary
                                 framework for vulnerability reporting and response.9


                                 Recently, two critical vulnerabilities were discovered in widely used
Collaborative                    commercial software products. The federal government and the private-
Response to Two                  sector security community took steps, described below in chronological
                                 order, to collaboratively respond to the threat of potential attacks against
Recent Software                  these vulnerabilities.
Vulnerabilities
Microsoft Remote                 Last Stage of Delirium Research Group discovered a security vulnerability
Procedure Call                   in Microsoft’s Windows Distributed Component Object Model (DCOM)10
Vulnerability Exploited by       Remote Procedure Call (RPC)11 interface. This vulnerability would allow
                                 an attacker to gain complete control over a remote computer.
Hacker
                             •   On June 28, 2003, the group notified Microsoft about the RPC
                                 vulnerability. Within hours of being notified, Microsoft verified the
                                 vulnerability.

                             •   On July 16, Microsoft issued a security bulletin publicly announcing the
                                 critical vulnerability and providing workaround instructions and a patch.

                             •   The following day, CERT/CC issued its first advisory.




                                 9
                                  Organization for Internet Safety, Guidelines for Security Vulnerability Reporting and
                                 Response, Version 1.0 (July 2003).
                                 10
                                  Distributed Component Object Model (DCOM) allows direct communication over the
                                 network between software components.
                                 11
                                  Remote Procedure Call (RPC) is a protocol of the Windows operating system that allows
                                 a program from one computer to request a service from a program on another computer in
                                 a network, thereby facilitating interoperability.



                                 Page 7                                                                    GAO-03-1138T
•   Nine days after Microsoft’s announcement, on July 25, Xfocus, an
    organization that researches and demonstrates security vulnerabilities,
    released code that could be used to exploit the vulnerability.

•   On July 31, CERT/CC issued a second advisory reporting that multiple
    exploits had been publicly released, and encouraged all users to apply the
    patches.

•   On August 11, 2003, the Blaster worm (also known as Lovsan) was
    launched to exploit this vulnerability. When the worm is successfully
    executed, it can cause the operating system to crash. Experts consider
    Blaster, which affected a range of systems, to be one of the worst exploits
    of 2003. Although the security community had received advisories from
    CERT/CC and other organizations to patch this critical vulnerability,
    Blaster reportedly infected more than 120,000 unpatched computers in the
    first 36 hours. By the following day, reports began to state that many users
    were experiencing slowness and disruptions to their Internet service, such
    as the need to frequently reboot. The Maryland Motor Vehicle
    Administration was forced to shut down, and systems in both national and
    international arenas have also been affected. The worm was programmed
    to launch a denial-of-service attack on Microsoft’s Windows Update Web
    site www.windowsupdate.com (where users can download security
    patches) on August 16. Microsoft preempted the worm’s attack by
    disabling the Windows Update Web site.

•   On August 14, two variants to the original Blaster worm were released.
    Federal agencies reported problems associated with these worms to
    FedCIRC.

•   On August 18, Welchia, a worm that also exploits this vulnerability, was
    reported. Among other things, it attempts to apply the patch for the RPC
    vulnerability to vulnerable systems, but reportedly creates such high
    volumes of network traffic that it effectively denies services in infected
    networks. Media reports indicate that Welchia affected several federal
    agencies, including components of the Departments of Defense and
    Veterans Affairs.

    The federal government’s response to this vulnerability included
    coordination with the private sector to mitigate the effects of the worm.

•   On July 17, FedCIRC issued an advisory to encourage federal agencies to
    patch the vulnerability, followed by several advisories from the
    Department of Homeland Security (DHS).



    Page 8                                                         GAO-03-1138T
•   The following week, on July 24, DHS issued its first advisory to heighten
    public awareness of the potential impact of an exploit of this
    vulnerability.12

•   On July 28, on behalf of the Office of Management and Budget (OMB),
    FedCIRC requested that federal agencies report on the status of their
    actions to patch the vulnerability.

•   From August 12 to August 18, DHS’s National Cyber Security Division
    hosted several teleconferences with federal agencies, CERT/CC, and
    Microsoft.

    Figure 3 is a timeline of selected responses to the Blaster Internet worm.




    12
     Department of Homeland Security, Potential For Significant Impact On Internet
    Operations Due To Vulnerability In Microsoft Operating Systems (Washington, D.C.: July
    24, 2003).



    Page 9                                                                 GAO-03-1138T
Figure 3: Event Timeline for the Blaster Internet Worm




                                          Based on an analysis of the agencies reported actions, as requested on July
                                          28, FedCIRC indicated that many respondents had completed patch
                                          installation on all systems at the time of their report and that only a
                                          minimal number of infections by the Blaster worm were reported.


Cisco IOS Vulnerability                   Cisco Systems, Inc., which controls approximately 82 percent of the
Exploits Attempted                        worldwide share of the Internet router13 market, discovered a critical
                                          vulnerability in its Internet operating system (IOS) software. This
                                          vulnerability could allow an intruder to effectively shut down unpatched
                                          routers, blocking network traffic. Cisco had informed the federal
                                          government of the vulnerability prior to public disclosure, and worked
                                          with different security organizations and government organizations to
                                          encourage prompt patching.

                                      •   On July 16, 2003, Cisco issued a security bulletin to publicly announce the
                                          critical vulnerability in its IOS software, and provide workaround
                                          instructions and a patch. Cisco had planned to officially notify the public
                                          of the vulnerability on July 17, but early media disclosure led them to
                                          announce the vulnerability a day earlier. In addition, FedCIRC issued


                                          13
                                            Routers are hardware devices or software programs that forward Internet and network
                                          traffic between networks and are critical to their operation.



                                          Page 10                                                                  GAO-03-1138T
                           advisories to federal agencies and DHS advised private-sector entities of
                           the vulnerability. In the week that the vulnerability was disclosed,
                           FedCIRC, OMB, and DHS’s National Cyber Security Division held a
                           number of teleconferences with representatives from the executive
                           branch.

                       •   On July 17, OMB requested that federal agencies report to CERT/CC on the
                           status of their actions to patch the vulnerability by July 24.

                       •   On July 18, DHS issued an advisory update in response to an exploit that
                           was posted online, and OMB moved up the agencies’ reporting deadline to
                           July 22.

                           CERT/CC has received reports of attempts to exploit this vulnerability, but
                           as of September 5, no incidents have yet been reported.


                           Patch management is a process used to help alleviate many of the
Patch Management: A        challenges involved with securing computing systems from attack. It is a
Critical Process for       component of configuration management14 that includes acquiring, testing,
                           and applying patches to a computer system. I will now discuss common
Mitigating Cyber           patch management practices, federal efforts to address software
Vulnerabilities            vulnerabilities in agencies, and services and tools to assist in carrying out
                           the patch management process.


Common Practices for       Effective patch management practices have been identified in security-
Effective Patch            related literature from several groups, including NIST, Microsoft,15 patch
Management                 management software vendors, and other computer-security experts.
                           Common elements identified include the following:

                       •   Senior executive support. Management recognition of information
                           security risk and interest in taking steps to manage and understand risks,
                           including ensuring that appropriate patches are deployed, is important to
                           successfully implementing any information security–related process and
                           ensuring that appropriate resources are applied.


                           14
                             Configuration management is the control and documentation of changes made to a
                           system’s hardware, software, and documentation throughout the development and
                           operational life of a system.
                           15
                            Microsoft Corporation, Solutions for Security, Solutions for Management: The Microsoft
                           Guide to Security Patch Management (Redmond, WA: 2003).



                           Page 11                                                                 GAO-03-1138T
•   Standardized patch management policies, procedures, and tools.
    Without standardized policies and procedures in place, patch management
    can remain an ad-hoc process—potentially allowing each subgroup within
    an entity to implement patch management differently or not at all. Policies
    provide the foundation for ensuring that requirements are communicated
    across an entity. In addition, selecting and implementing appropriate patch
    management tools is an important consideration for facilitating effective
    and efficient patch management.

•   Dedicated resources and clearly assigned responsibilities. It is
    important that the organization assign clear responsibility for ensuring
    that the patch management process is effective. NIST recommends
    creating a designated group whose duties would include supporting
    administrators in finding and fixing vulnerabilities in the organization’s
    software. It is also important that the individuals involved in patch
    management have the skills and knowledge needed to perform their
    responsibilities, and that systems administrators be trained regarding how
    to identify new patches and vulnerabilities.

•   Current technology inventory. Creating and maintaining a current
    inventory of all hardware equipment, software packages, services, and
    other technologies installed and used by the organization is an essential
    element of successful patch management. This systems inventory assists
    in determining the number of systems that are vulnerable and require
    remediation, as well as in locating the systems and identifying their
    owners.

•   Identification of relevant vulnerabilities and patches. It is important
    to proactively monitor for vulnerabilities and patches for all software
    identified in the systems inventory. Various tools and services are
    available to assist in identifying vulnerabilities and their respective
    patches. Using multiple sources can help to provide a more
    comprehensive view of vulnerabilities.

•   Risk assessment. When a vulnerability is discovered and a related patch
    and/or alternative workaround is released, the entity should consider the
    importance of the system to operations, the criticality of the vulnerability,
    and the risk of applying the patch. Since some patches can cause
    unexpected disruption to entities’ systems, organizations may choose not
    to apply every patch, at least not immediately, even though it may be
    deemed critical by the software vendor that created it. The likelihood that
    the patch will disrupt the system is a key factor to consider, as is the
    criticality of the system or process that the patch affects.


    Page 12                                                         GAO-03-1138T
                             •   Testing. Another critical step is to test each individual patch against
                                 various systems configurations in a test environment before installing it
                                 enterprisewide to determine any impact on the network. Such testing will
                                 help determine whether the patch functions as intended and its potential
                                 for adversely affecting the entity’s systems. In addition, while patches are
                                 being tested, organizations should also be aware of workarounds, which
                                 can provide temporary relief until a patch is applied. Testing has been
                                 identified as a challenge by government and private-sector officials, since
                                 the urgency in remediating a security vulnerability can limit or delay
                                 comprehensive testing. Time pressures can also result in software
                                 vendors’ issuing poorly written patches that can degrade system
                                 performance and require yet another patch to remediate the problem. For
                                 instance, Microsoft has admittedly issued security patches that have been
                                 recalled because they have caused systems to crash or are too large for a
                                 computer’s capacity. Further, a complex, heterogeneous systems
                                 environment can lengthen this already time-consuming and time-sensitive
                                 process because it takes longer to test the patch in various systems
                                 configurations.

                             •   Distributing patches. Organizations can deploy patches to systems
                                 manually or by using an automated tool. One challenge to deploying
                                 patches appropriately is that remote users may not be connected at the
                                 time of deployment, leaving the entity’s networks vulnerable from the
                                 remote user’s system because they have not yet been patched. One private-
                                 sector entity stated that its network first became affected by the Microsoft
                                 RPC vulnerability when remote users plugged their laptops into the
                                 network after being exposed to the vulnerability from other sources.

                             •   Monitoring through network and host vulnerability scanning.
                                 Networks can be scanned on a regular basis to assess the network
                                 environment, and whether patches have been effectively applied. Systems
                                 administrators can take proactive steps to preempt computer security
                                 incidents within their entities by regularly monitoring the status of patches
                                 once they are deployed. This will help to ensure patch compliance with the
                                 network’s configuration.


Federal Efforts to Address       The federal government has taken several steps to address security
Software Vulnerabilities         vulnerabilities that affect federal agency systems, including efforts to
                                 improve patch management. NIST has taken a number of steps, including,
                                 as I previously mentioned, providing a handbook for patch management.
                                 In addition, NIST offers a source of vulnerability data, which I will discuss
                                 later in this testimony. Further, in accordance with OMB’s reporting


                                 Page 13                                                         GAO-03-1138T
instructions for the first year implementation of the Federal Information
Security Management Act (FISMA), maintaining up-to-date patches is a
part of FISMA’s system configuration requirements. As such, OMB
requires that agencies report how they confirm that patches have been
tested and installed in a timely manner.16 In addition, certain
governmentwide services are offered to federal agencies to assist them in
ensuring that software vulnerabilities are patched. For example, FedCIRC
was established to provide a central focal point for incident reporting,
handling, prevention, and recognition for the federal government. Its
purpose is to ensure that the government has critical services available in
order to withstand or quickly recover from attacks against its information
resources.

In addition, for the two recent vulnerabilities just discussed in my
testimony, OMB and FedCIRC held teleconferences with agency Chief
information officers to discuss vulnerabilities and request that agencies
report on the status of their actions to patch them. An OMB official
indicated that they planned to hold meetings with agencies to discuss
ways to improve communication of and followup on critical
vulnerabilities, including addressing some of the challenges identified in
the two recent exercises, such as delays in reaching key security personnel
in certain instances.

FedCIRC also initiated PADC to provide users with a method of obtaining
information on security patches relevant to their enterprise and access to
patches that have been tested in a laboratory environment. The federal
government offers PADC to federal civilian agencies at no cost. According
to FedCIRC, as of last month, 41 agencies were using PADC. Table 2 lists
its features and benefits, as reported by FedCIRC. OMB reported that
while many agencies have established PADC accounts, actual usage of
those accounts is extremely low.




16
  Title III—Federal Information Security Management Act of 2002, E-Government Act of
2002, P.L. 107-347, December 17, 2002. This act superseded an earlier version of FISMA
that was enacted as Title X of the Homeland Security Act of 2002.



Page 14                                                                 GAO-03-1138T
Table 2: Reported Features and Benefits of the Patch Authentication and Dissemination Capability

 Features                                                             Benefits
 •   Authorized government users subscribe from a secure Web          •   Notifications to subscribers will occur when a patch is available
     interface.                                                           for subscriber-selected systems or applications.
 •   Subscribers create customized notification profiles, including   •   FedCIRC will ensure that the patch originates from a reliable
     operating systems, firewalls, routers, antivirus software,           source.
     intrusion-detection systems, and servers.
 •   Subscribers are notified when new threats or vulnerabilities     •   FedCIRC will validate that the patch eliminates the intended
     are discovered; notifications are updated as vendor patches          vulnerability.
     are released and authenticated.
 •   Subscribers may visit a secure site to download validated        •   All aspects of the system are secure from subscriber information
     patches.                                                             to the secure download of patches.
 •   Subscribers may contact the PADC Help Desk to verify             •   Single consolidated source for all patch updates.
     information or to seek assistance.                               •   No cost to federal civilian government agencies.
Source: FedCIRC.

                                               To participate in PADC, subscribers (who could be one or more
                                               individuals within an agency) receive an account license that allows them
                                               to receive notifications and log into the secure Web site to download the
                                               patch. To establish an account, each subscriber must set up a profile
                                               defining the technologies that they use. The profiles act much like a
                                               filtering service and allow PADC to notify agencies of only the patches that
                                               pertain to their systems. The profiles do not include system–specific
                                               information because of the sensitivity of that information. Subscribers
                                               using PADC receive notification of threats, vulnerabilities, and the
                                               availability of patches on the basis of the submitted profiles. They are
                                               notified by E-mail or pager message that a vulnerability or patch has been
                                               posted to a secure Web site that affects one or all of their systems.

                                               When a patch is identified, FedCIRC, through contractor support, ensures
                                               that it originates from a reliable source. The patch is then tested on a
                                               system to which it applies. The installation of the patch and the operation
                                               of the system are monitored to ensure that the patch causes no problem.
                                               Next, if an exploit had been developed, exploit testing is performed to
                                               ensure that the patch fixes the vulnerability. Any issues identified with a
                                               patch are summarized and provided to the users. The validated patch is
                                               then uploaded to PADC servers and made available to users. A patch is
                                               considered validated when it has been downloaded from a trusted source,
                                               authenticated, loaded onto an appropriate system, tested, exploit–tested,
                                               verified, and posted to the PADC server. This type of testing and validation
                                               is performed for over 60 technologies that, according to FedCIRC officials,
                                               account for approximately 80 percent of the technologies used by federal
                                               agencies. Also available is notification of patches that are not validated for
                                               over 25,000 additional technologies.


                                               Page 15                                                                        GAO-03-1138T
                              According to FedCIRC officials, high-priority patches are to be tested and
                              posted on the PADC server within the same business day of availability.
                              Medium- and low-priority patches are to be completed by the following
                              business day, but are generally available sooner. However, because PADC
                              has several early warning mechanisms in place and arrangements with
                              software vendors, some patches may be available as soon as a
                              vulnerability is made public. FedCIRC officials emphasize that although
                              the contractor tests the security patches, these tests do not ensure that the
                              patch can be successfully deployed in another environment; therefore,
                              agencies still need to test the patch for compatibility with their own
                              business processes and technology.

                              PADC offers a reporting capability that is hierarchical. Senior managers
                              can look at their complete system and see which subsystems have been
                              patched. These enterprisewide reports and statistics can be generated for
                              a “reporting user” subscriber who has read-only capability within the
                              system.

                              According to agency officials, there are limitations to the PADC service.
                              Although it is free to agencies, only about 2,000 licenses or accounts are
                              available because of monetary constraints. According to FedCIRC
                              officials, this requires them to work closely with participating agencies to
                              balance the number of licenses that a single agency requires with the need
                              to allow multiple agencies to participate. For example, the National
                              Aeronautics and Space Administration initially requested over 3,000
                              licenses—one for each system administrator. Another agency, NIST,
                              thought that each of its users should have his or her own PADC account.
                              Another limitation is the level of services currently provided by PADC. At
                              present, the government is considering broadening the scope of these
                              services and capabilities, along with the number of users.


Services and Tools Also       Several services and automated tools are available to assist entities in
Provide Means for             performing the patch management function, including tools designed to be
Improving Patch               stand-alone patch management systems. In addition, systems management
                              tools can be used to deploy patches across an entity’s network. Some of
Management                    the features in services and tools typically include methods to

                          •   inventory computers and the software applications and patches installed;

                          •   identify relevant patches and workarounds and gather them in one
                              location;



                              Page 16                                                         GAO-03-1138T
•   group systems by departments, machine types, or other logical divisions to
    easily manage patch deployment;

•   scan a network to determine the status of the patches and other
    corrections made to network machines (hosts and/or clients);

•   assess the machines against set criteria;

•   access a database of patches;

•   test patches;

•   deploy effective patches; and

•   report information to various levels of management about the status of the
    network.

    Patch management vendors also offer central databases of the latest
    patches, incidents, and methods for mitigating risks before a patch can be
    deployed or a patch has been released. Some vendors provide support for
    multiple software platforms, such as Microsoft, Solaris, Linux, and others,
    while others focus on certain platforms exclusively, such as Microsoft.

    Patch management tools can be either scanner–based (non agent) or
    agent–based. While scanner–based tools can scan a network, check for
    missing patches, and allow an administrator to patch multiple computers,
    these tools are best suited for smaller organizations due to their inability
    to serve a large number of users without breaking down or requiring major
    changes in procedure. Another difficulty with scanner-based tools is that
    part-time users and turned-off systems will not be scanned.

    Agent-based products place small programs, or agents, on each computer,
    to periodically poll a patch database—a server on the network—for new
    updates, giving the administrator the option of applying the patch. Agent-
    based products require up-front work to integrate agents into the
    workstations and in the server deployment process, but are better suited
    to large organizations due to their ability to generate less network traffic
    and provide a real-time network view. The agents maintain information
    that can be reported when needed. Finally, some patch management tools
    are hybrids—allowing the user to utilize agents or not.

    Instead of an automated stand-alone system, entities can also use other
    methods and tools to perform patch management. For example, they can


    Page 17                                                        GAO-03-1138T
                            maintain a database of the versions and latest patches for each server and
                            each client in their network and track the security alerts and patches
                            manually. While labor-intensive, this can be done. In addition, entities can
                            employ systems management tools with patch-updating capabilities to
                            deploy the patches. This method requires monitoring for the latest security
                            alerts and patches. Entities may also need to develop better relationships
                            with their vendors to be alerted to vulnerabilities and patches prior to
                            public release. In addition, software vendors may provide automated tools
                            with customized features to alert system administrators and users of the
                            need to patch, and if desired, automatically apply patches.

                            A variety of resources are also available to provide information related to
                            vulnerabilities and their exploits. As I mentioned earlier, one resource is
                            CERT/CC, a major center for analyzing and reporting vulnerabilities as
                            well as providing information on possible solutions. Another useful
                            resource is NIST’s ICAT, which offers a searchable index leading users to
                            vulnerability resources and patch information. ICAT links users to publicly
                            available vulnerability databases and patch sites, thus enabling them to
                            find and fix vulnerabilities existing on their systems. It is based on
                            common vulnerability and exposures (commonly referred to as CVE)
                            naming standards. These are standardized names for vulnerabilities and
                            other information security exposures, compiled in an effort to make it
                            easier to share data across separate vulnerability databases and tools.

                            Many other organizations exist, including the Last Stage of Delirium
                            Research Group, that research security vulnerabilities and maintain
                            databases of such vulnerabilities. In addition, mailing lists, such as
                            BugTraq, provide forums for announcing and discussing vulnerabilities,
                            including information on how to fix them. In addition, Security Focus
                            monitors thousands of products to maintain a vulnerability database and
                            provide security alerts. Finally, vendors such as Microsoft and Cisco
                            provide software updates on their products, including notices of known
                            vulnerabilities and their corresponding patches.


                            In addition to implementing effective patch management practices, several
Additional Steps That       additional steps can be considered when addressing software
Can Be Taken                vulnerabilities, including:

                        •   deploying other technologies, such as antivirus software, firewalls, and
                            other network security tools to provide additional defenses against
                            attacks;



                            Page 18                                                        GAO-03-1138T
•   employing more rigorous engineering practices in designing,
    implementing, and testing software products to reduce the number of
    potential vulnerabilities;

•   improving tools to more effectively and efficiently manage patching;

•   researching and developing technologies to prevent, detect, and recover
    from attacks as well as identify their perpetrators, such as more
    sophisticated firewalls to keep serious attackers out, better intrusion-
    detection systems that can distinguish serious attacks from nuisance
    probes and scans, systems that can isolate compromised areas and
    reconfigure while continuing to operate, and techniques to identify
    individuals responsible for specific incidents; and

•   ensuring effective, tested contingency planning processes and procedures.

    Actions are already underway in many, if not all, of these areas. For
    example, CERT/CC has a research program, one goal of which is to try to
    find ways to improve technical approaches for identifying and preventing
    security flaws, for limiting the damage from attacks, and for ensuring that
    systems continue to provide essential services in spite of compromises
    and failures. Also, Microsoft recently initiated its Trustworthy Computing
    strategy to incorporate security-focused software engineering practices
    throughout the design and deployment of its software, and is reportedly
    considering the use of automated patching in future products.


    In summary, it is clear from the increasing number of reported attacks on
    information systems that both federal and private-sector operations and
    assets are at considerable—and growing—risk. Patch management can be
    an important element in mitigating the risks associated with software
    vulnerabilities, part of overall network configuration management and
    information security programs. The challenge will be ensuring that a patch
    management process has adequate resources and appropriate policies,
    procedures, and tools to effectively identify vulnerabilities and patches
    that place an entity’s systems at risk. Also critical is the capability to
    adequately test and deploy the patches, and then monitor progress to
    ensure that they work. Although this can currently be performed, the
    eventual solution will likely come from research and development to
    better build security into software and tools from the beginning.

    Mr. Chairman, this concludes my statement. I would be pleased to answer
    any questions that you or other members of the Subcommittee may have at


    Page 19                                                        GAO-03-1138T
           this time. Should you have any further questions about this testimony,
           please contact me at (202) 512-3317 or at daceyr@gao.gov.

           Individuals making key contributions to this testimony included Shannin
           G. Addison, Michael P. Fruitman, Michael W. Gilmore, Sophia Harrison,
           Elizabeth L. Johnston, Min S. Lee, and Tracy C. Pierson.




(310507)
           Page 20                                                       GAO-03-1138T
This is a work of the U.S. government and is not subject to copyright protection in the
United States. It may be reproduced and distributed in its entirety without further
permission from GAO. However, because this work may contain copyrighted images or
other material, permission from the copyright holder may be necessary if you wish to
reproduce this material separately.
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to e-mail
                         alerts” under the “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548