oversight

Law Enforcement: Better Performance Measures Needed to Assess Results of Justice's Office of Science and Technology

Published by the Government Accountability Office on 2003-11-14.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States General Accounting Office

GAO             Report to the Honorable Jane Harman,
                House of Representatives



November 2003
                LAW ENFORCEMENT

                Better Performance
                Measures Needed to
                Assess Results of
                Justice’s Office of
                Science and
                Technology




GAO-04-198
                                                November 2003


                                                LAW ENFORCEMENT

                                                Better Performance Measures Needed to
Highlights of GAO-04-198, a report to the       Assess Results of Justice's Office of
Honorable Jane Harman, House of
Representatives                                 Science and Technology



The mission of the Office of                    OST's budgetary resources grew significantly in recent years, along with the
Science & Technology (OST),                     range of its program responsibilities. From fiscal year 1995 through fiscal
within the Department of Justice’s              year 2003, OST received over $1 billion through Department of Justice
National Institute of Justice (NIJ),            appropriations and the reimbursement of funds from other federal agencies
is to improve the safety and                    in exchange for OST’s agreement to administer these agencies' projects. Of
effectiveness of technology used by
federal, state, and local law
                                                the over $1 billion that OST received, approximately $749 million, or 72
enforcement and other public                    percent, was either directed to specific recipients or projects by public law,
safety agencies. Through NIJ, OST               subject to guidance in congressional committee reports, or directed though
funds programs in forensic                      reimbursable agreements. At the same time that spending expanded, OST’s
sciences, crime prevention, and                 program responsibilities have changed—from primarily law enforcement
standards and testing. To support               and corrections to broader public safety technology.
these programs, Congress
increased funding for OST from                  OST delivers three groups of products through various methods. The three
$13.2 million in 1995 to $204.2                 groups include (1) information dissemination and technical assistance; (2)
million in 2003 (in constant 2002               the application, evaluation, and demonstration of existing and new
dollars). GAO reviewed (1) the                  technologies for field users; and (3) technology research and development.
growth in OST’s budgetary
resources and the changes in OST’s
                                                According to OST, as of April 2003, it has delivered 945 products since its
program responsibilities, (2) the               inception. Furthermore, OST identified an additional 500 products
types of products OST delivers and              associated with ongoing awards. OST makes its products available through a
the methods used for delivering                 variety of methods, such as posting information on its Web site and
them; and (3) how well OST’s                    providing research prototypes to field users for testing and evaluation.
efforts to measure the success of
its programs in achieving intended              OST has been unable to fully assess its performance in achieving its goals as
results meet applicable                         required by applicable criteria because it does not use outcome measures to
requirements.                                   assess the extent to which it achieves the intended results of its programs.
                                                OST’s current measures primarily track outputs, the goods and services
                                                produced, or in some cases OST uses intermediate measures, which is a step
GAO recommends that the Director                toward developing outcome measures. The Government Performance and
of NIJ reassess the measures used               Results Act of 1993 provides that federal agencies measure or assess the
to evaluate OST’s progress toward               results of each program activity. While developing outcome measures for the
achieving its goals and to better               types of activities undertaken by OST is difficult, we have previously
focus on outcome measures to                    reported on various strategies that can be used to develop outcome
assess results where possible. In               measures, or, at least intermediate measures, for similar types of activities.
those cases where measuring
outcomes is, after careful
                                                OST’s annual budgetary resources in constant 2002 dollars, fiscal years 1995–2003
consideration, deemed infeasible,
                                                Dollars in millions
we recommend developing
                                                300
appropriate intermediate measures
that will help to discern program               250
effectiveness.                                  200

                                                150

                                                100
www.gao.gov/cgi-bin/getrpt?GAO-04-198.           50

To view the full product, including the scope     0
and methodology, click on the link above.         1995          1996   1997      1998       1999      2000       2001      2002     2003
For more information, contact Laurie Ekstrand         Fiscal year
at (202) 512-8777 or Ekstrandl@gao.gov.
                                                Source: OST.
Contents


Letter                                                                                       1
               Results in Brief                                                              2
               Background                                                                    3
               OST’s Budgetary Resources Have Grown and Program
                 Responsibilities Have Changed                                              6
               OST Delivers Three Groups of Products Through Various Methods               13
               OST’s Performance Measurement Efforts Do Not Fully Meet
                 Requirements                                                              17
               Conclusions                                                                 24
               Recommendation                                                              25
               Agency Comments and Our Evaluation                                          25

Appendix I     Scope and Methodology                                                       28



Appendix II    Bugetary Resources for OST’s Programs in Current
               Year Dollars                                                                32



Appendix III   OST’s 10 Categories of Products
                                                                                           33



Appendix IV    OST’s Portfolio Areas                                                       35



Appendix V     OST’s Operations                                                            37



Appendix VI    OST’s Goals in its Fiscal Year 2004 Performance Plan
               and GAO’s Assessment                                 47



Appendix VII   Comments from the Department of Justice                                     51




               Page i                    GAO-04-198 Justice's Office of Science and Technology
Appendix VIII   GAO Contacts and Staff Acknowledgments                                        54
                GAO Contacts                                                                  54
                Staff Acknowledgments                                                         54


Tables
                Table 1: Flow of Budgetary Resources to OST’s Programs                          7
                Table 2: Budgetary Resources in Constant 2002 Dollars for OST’s
                         Programs by NIJ Allocation, Fiscal Years 1995-2003                   10
                Table 3: Budgetary Resources in Constant 2002 Dollars for OST’s
                         Investigative and Forensic Sciences by NIJ Allocation,
                         Fiscal Years 1995-2003                                               13
                Table 4: GAO’s Assessment of the 42 Measures OST Developed for
                         11 of Its Initiatives                                                19
                Table 5: OST’s Outside Studies of Its Initiatives                             23
                Table 6: Budgetary Resources in Current Dollars for OST’s
                         Programs by NIJ Allocation, Fiscal Years 1995-2003                   32
                Table 7: GAO’s Groupings of OST’s Categories of Products and
                         Examples of Each Category                                            33
                Table 8: Total Funds Awarded for the Operations, Maintenance,
                         and Technical Support of OST’s 10 Technology Centers,
                         Fiscal Years 1995-2003                                               40
                Table 9: OST’s Technology Centers, Their Affiliated Partners, and
                         the Amounts Awarded to Support the Centers                           42
                Table 10: OST’s Performance Goals, Initiatives, and Measures for
                         Fiscal Year 2004, and GAO’s Assessment                               47


Figures
                Figure 1: OST’s Budgetary Resources in Constant 2002 Dollars,
                         Fiscal Years 1995–2003                                                 8
                Figure 2: GAO’s Grouping of OST’s 945 Delivered Products, as of
                         April 2003                                                           14
                Figure 3: OST’s Organizational Structure                                      38
                Figure 4: OST’s 10 Technology Centers and the Regions They Serve              39
                Figure 5: Stakeholders and Customers that Contribute to the
                         Setting of OST’s priorities                                          44




                Page ii                     GAO-04-198 Justice's Office of Science and Technology
Abbreviations

AAG               Assistant Attorney General
CITA              Crime Identification Technology Act
CLIP              Crime Lab Improvement Program
CODIS             Combined DNA Index System
COPS              Community-Oriented Policing Services
DNA               deoxyribonucleic acid
DOD               Department of Defense
FBI               Federal Bureau of Investigation
GAO               General Accounting Office
GPRA              Government Performance and Results Act
LLEBG             Local Law Enforcement Block Grant
NFSIA             Paul Coverdell National Forensic Sciences Improvement
                  Act
NIJ               National Institute of Justice
NLECTC            National Law Enforcement and Corrections Technology
                  Centers
OJP               Office of Justice Programs
OMB               Office of Management and Budget
OST               Office of Science and Technology
R&D               research and development
SSLEA             State and Local Law Enforcement Assistance




This is a work of the U.S. government and is not subject to copyright protection in the
United States. It may be reproduced and distributed in its entirety without further
permission from GAO. However, because this work may contain copyrighted images or
other material, permission from the copyright holder may be necessary if you wish to
reproduce this material separately.




Page iii                           GAO-04-198 Justice's Office of Science and Technology
United States General Accounting Office
Washington, DC 20548




                                   November 14, 2003

                                   The Honorable Jane Harman
                                   House of Representatives

                                   Dear Ms. Harman:

                                   To enhance public safety and bring criminals to justice, it is important for
                                   law enforcement officials to benefit from the latest advances in science
                                   and technology. The mission of the Office of Science and Technology
                                   (OST), within the Department of Justice’s National Institute of Justice
                                   (NIJ), is to improve the safety and effectiveness of technology used by
                                   federal, state, and local law enforcement, corrections, and other public
                                   safety agencies. OST awards funds to research and develop more effective
                                   technology and improve access to technology in a wide range of areas. For
                                   example, OST funds programs in the areas of crime prevention
                                   technologies, investigative and forensic sciences, and electronic crime.
                                   Examples of products resulting from OST’s programs include a guide on
                                   school safety, an evaluation of police protective gear, a prototype for
                                   ground-penetrating radar, and a report on gunshot residue detection and
                                   interpretation. To support OST’s programs, Congress has significantly
                                   increased its funding, from $13.2 million in fiscal year 1995 to $204.2
                                   million in fiscal year 2003 (in constant 2002 dollars).

                                   In response to your interest about whether OST’s programs are achieving
                                   their intended results, we reviewed certain aspects of OST’s operations.
                                   Specifically, this report assesses (1) the growth in OST’s budgetary
                                   resources, from fiscal year 1995 to fiscal year 2003, and changes in OST’s
                                   program responsibilities; (2) what types of products OST delivers and the
                                   methods used to deliver these products to public safety agencies; and
                                   (3) how well OST’s efforts to measure the success of its programs in
                                   achieving intended results meet applicable requirements.

                                   To address our objectives, we collected and analyzed relevant data and
                                   reports and interviewed OST officials and NIJ officials, including NIJ
                                   executive staff and the Assistant NIJ Director for OST, division chiefs, and
                                   managers. We also collected data and interviewed officials at OST
                                   technology centers in Rockville, Maryland; and El Segundo and San Diego,
                                   California. Appendix I contains detailed information on the scope and
                                   methodology we used for this assessment. We conducted this engagement
                                   in accordance with generally accepted government auditing standards.


                                   Page 1                       GAO-04-198 Justice's Office of Science and Technology
                   OST has grown in terms of both budgetary resources and the range of
Results in Brief   programs it operates.1 From fiscal year 1995 through fiscal year 2003, OST
                   received over $1 billion through several Department of Justice (Justice)
                   appropriations accounts as well as the reimbursement of funds from other
                   federal agencies in exchange for OST’s agreement to administer these
                   agencies’ projects. Of the over $1 billion that OST has received,
                   approximately $749.7 million, or about 72 percent, was either directed for
                   specific recipients or projects by public law, subject to guidance in
                   congressional committee reports designating specific recipients or
                   projects, or directed from reimbursable agreements with other federal
                   agencies for OST to manage their projects. At the same time that spending
                   has expanded, OST’s program responsibilities have changed—from
                   primarily law enforcement and corrections technologies to broader public
                   safety technologies, including safe school initiatives.

                   OST delivers three groups of products through various methods. The three
                   groups include (1) information dissemination and technical assistance;
                   (2) the application, evaluation, and demonstration of existing and new
                   technologies for field users; and (3) technology research and development
                   (R&D). According to OST, as of April 2003, it had delivered 945 products
                   since its inception. Furthermore, OST identified an additional 500 products
                   associated with ongoing awards. Depending on its research agenda, OST
                   makes its products available through a variety of methods, such as posting
                   information on its Web site and providing research prototypes to field
                   users for testing and evaluation. While OST does not directly
                   commercialize the results of its technology R&D, it does help link
                   prototypes with potential developers.

                   OST has been unable to fully assess its performance in achieving its goals
                   because it does not measure the extent to which it achieves the intended
                   outcomes of its programs. OST’s current measures primarily track outputs
                   (goods and services produced). In some cases OST uses intermediate
                   measures—a step closer to developing outcome measures—but has not
                   taken this step toward better measurement in many cases where it may be


                   1
                    We are using “programs” to indicate the broad categories of OST’s individual projects. NIJ
                   and OST have referred to these categories as both portfolio areas and programs. Our use of
                   the term “programs” encompasses “portfolio areas” (see app. IV for OST’s portfolio areas)
                   and the safe school technology, counterterrorism technology, and correction technology
                   programs. NIJ and OST delineations between the various programs and various portfolio
                   areas are flexible. For example, some of the projects to develop metal detectors and
                   personnel locator devices would apply to both school safety technologies and corrections
                   technologies programs and therefore could be placed in different portfolio areas.




                   Page 2                             GAO-04-198 Justice's Office of Science and Technology
             possible to do so. The Government Performance and Results Act of 1993
             (GPRA) provides, among other things, that federal agencies establish
             performance measures, including, the assessment of relevant outputs and
             outcomes of each program activity. Office of Management and Budget
             (OMB) guidance suggests that, to the extent possible, federal agencies
             measure or assess the extent to which they are achieving the intended
             outcomes of their programs. As part of Justice’s efforts to comply with
             GPRA, OST established goals and developed output, and some
             intermediate, measures to track its progress. While developing outcome
             measures for the types of activities undertaken by OST is difficult, we have
             previously reported on various strategies that can be used to develop
             outcome measures or at least intermediate measures for activities that are
             similar to those in OST’s portfolio of programs.

             So that OST does all that is possible to assess whether its programs are
             achieving their intended results, we are recommending that the Attorney
             General instruct the Director of NIJ to reassess OST’s performance
             measures to better focus on outcome measures. In commenting on a draft
             of this report, the Assistant Attorney General (AAG) for Justice’s Office of
             Justice Programs (OJP) agreed with our recommendation. The AAG made
             additional comments concerning the challenge of developing outcome
             measures for R&D activities, OST’s overall performance record, and the
             amount of OST’s funds that are directed for specific recipients and
             projects. We respond to these comments in the Agency Comments and
             Evaluation section of the report. OJP also provided technical comments,
             which have been incorporated in this report where appropriate.


             The Office of Science and Technology (OST) was created in fiscal year
Background   1995 following a long history of science and technology efforts within the
             National Institute of Justice (NIJ).2 NIJ is a component of the Office of
             Justice Programs (OJP), a Justice agency that, among other things,
             provides assistance to state, tribal, and local governments. In establishing
             OST’s objectives and allocating funds for OST’s programs, the NIJ Director
             considers the priorities of many stakeholders, including the President,
             Congress, Justice, and state and local law enforcement and public safety
             agencies.



             2
              NIJ was established in statute by the Justice System Improvement Act of 1979 (P.L. 96-157,
             93 Stat. 1167 (1979)), which, among other things, amended the Omnibus Crime Control and
             Safe Streets Act of 1968 (P.L. 90-351, 82 Stat. 197 (1968)).




             Page 3                             GAO-04-198 Justice's Office of Science and Technology
OST Established in Statute   In November 2002, Congress established OST and its mission and duties in
by the Homeland Security     statute as part of the Homeland Security Act of 2002 (the Act).3 The Act
Act of 2002                  specified OST’s mission “to serve as the national focal point for work on
                             law enforcement technology; and to carry out programs that, through the
                             provision of equipment, training, and technical assistance, improve the
                             safety and effectiveness of law enforcement technology and improve
                             access to such technology by federal, state, and local law enforcement
                             agencies.” The Act defined the term “law enforcement technology” to
                             include “investigative and forensic technologies, corrections technologies,
                             and technologies that support the judicial process.”4 The Act also specified
                             OST’s duties to include the following, among others:

                         •   establishing and maintaining advisory groups to assess federal, state, and
                             local technology needs;

                         •   establishing and maintaining performance standards, and testing,
                             evaluating, certifying, validating, and marketing products that conform to
                             those standards;

                         •   carrying out research, development, testing, evaluation, and cost-benefit
                             analysis of certain technologies; and

                         •   developing and disseminating technical assistance and training materials.



OST’s Operations             OST’s operations have multiple levels of internal organization and multiple
                             kinds of external partners. (For a more detailed description of OST’s
                             operations, see app. V.) OST’s multiple levels of organization include a
                             Washington, D.C., office and a network of 10 technology centers that
                             provide technical assistance to OST’s customers around the country.5 To
                             fulfill its mission, OST also collaborates with entities such as the



                             3
                              P.L. 107-296, 116 Stat. 2135, 2159 (2002). These mission and duties are not unlike what OST
                             had been carrying out previously. The Act codified the mission and duties in statute.
                             4
                              According to NIJ, forensic science is the application of established scientific techniques to
                             the identification, collection, and examination of evidence from crime scenes; the
                             interpretation of laboratory findings; and the presentation of reported findings in judicial
                             proceedings.
                             5
                              These 10 technology centers are OST’s National Law Enforcement and Corrections
                             Technology Center (NLECTC) system.




                             Page 4                              GAO-04-198 Justice's Office of Science and Technology
    Departments of Defense and Energy and public and private laboratories to
    take advantage of established technical expertise and resources.

    NIJ has three main types of awards for funding OST’s programs: grants,
    interagency agreements, and cooperative agreements.6

•   Grants are generally awarded annually by NIJ to state and local agencies
    or private organizations for a specific product and amount.

•   Interagency agreements are used by NIJ for creating partnerships with
    federal agencies.

•   Cooperative agreements are a type of NIJ grant to nonfederal entities that
    prescribes a higher level of monitoring and federal involvement.

    NIJ also uses memorandums of understanding (MOU) to coordinate
    programs and projects between agencies. The MOUs specify the roles,
    responsibilities, and funding amounts to be provided by participating
    agencies. Through NIJ, OST can provide supplemental funding to
    interagency and cooperative agreements that may be used to contract for
    special projects.

    OST awards are administered by managers at its Washington, D.C., office
    who have final oversight and management responsibility. These managers
    may delegate some responsibility to another federal R&D agency receiving
    the award. In March 2003, 21 managers were responsible for overseeing
    336 active awards totaling $636 million.

    Guidance has been established for measuring the performance of
    government operations. To assist Justice to follow the Government
    Performance and Results Act of 1993 (GPRA),7 OST establishes goals and
    develops performance measures to track its progress. In addition, in May
    2002, the White House Office of Management and Budget (OMB) and
    Office of Science and Technology Policy issued a memorandum setting
    forth R&D investment criteria that departments and agencies should
    implement. The investment criteria require an explanation of why the
    investment is important, how funds will be allocated to ensure quality, and


    6
     We did not include contracts because NIJ uses them for the purchase of goods and
    services rather than for awarding funds for carrying out OST programs and projects.
    7
    P.L.103-62, 107 Stat. 285 (1993).




    Page 5                              GAO-04-198 Justice's Office of Science and Technology
                          how well the investment is performing. According to the memorandum,
                          program managers must define appropriate outcome measures, and
                          milestones that can be used to track progress toward goals and assess
                          whether funding should be enhanced or redirected. The memorandum
                          encourages federal R&D agencies to make the processes they use to
                          satisfy GPRA consistent with these criteria.


                          OST’s budgetary resources have grown and the range of program
OST’s Budgetary           responsibilities has changed. Budgetary resources for OST increased
Resources Have            significantly, from $13.2 million in fiscal year 1995 to $204.2 million in
                          fiscal year 2003 (in constant 2002 dollars), totaling over $1 billion.8 This
Grown and Program         increase can be attributed to the introduction of new allocations and large
Responsibilities Have     increases for existing ones. The NIJ director decides how to allocate
                          certain appropriated funds to the various NIJ components, including OST.
Changed                   About $749.7 million, or 72 percent, of OST’s total budgetary resources
                          was either directed to specific recipients or projects by public law, subject
                          to congressional committee report guidance designating specific
                          recipients or projects, or directed from the reimbursements from other
                          Justice and federal agencies in exchange for OST managing their projects.
                          Corresponding with the designation of spending for specific recipients and
                          projects, the range of OST’s programs changed, from primarily law
                          enforcement and corrections to include broader public safety technology
                          R&D, such as for improving school safety and combating terrorism.


Budgetary Resources for   OST’s budgetary resources9 include both funding received via Justice
OST’s Programs            appropriations accounts as well as reimbursements from other Justice and
                          federal agencies. First, OST receives funding via three appropriations
                          accounts enacted in the appropriations law for the Justice Department.
                          From these appropriations accounts, OJP allocates amounts to NIJ. The
                          NIJ director suballocates part of the NIJ funds for OST programs. In
                          addition, OST receives reimbursements from other Justice and federal
                          agencies in exchange for OST’s management of specific projects of those
                          agencies, such as ballistic imaging evaluation for the FBI. Table 1 lists NIJ



                          8
                           Figures do not include funding for management and administration expenses, salaries, and
                          unobligated balances carried from one year to the next.
                          9
                           For the purposes of this report, we will refer to both the funds OST receives via several
                          Justice appropriations accounts as NIJ allocations as well as the reimbursements it
                          receives as OST’s budgetary resources.




                          Page 6                              GAO-04-198 Justice's Office of Science and Technology
                                         allocations from the Justice appropriations accounts that go toward
                                         funding OST programs.

Table 1: Flow of Budgetary Resources to OST’s Programs

 Justice appropriation accounts                        NIJ’s allocations to OST programs
 Justice Assistance                                    NIJ Base: NIJ uses base funds for research, development, demonstration,
                                                       and dissemination activities.
                                                       Counterterrorism R&D:a NIJ sponsors research, development, and
                                                       evaluations and tools to help criminal justice and public safety agencies deal
                                                       with critical incidents, including terrorist acts.
 State and Local Law Enforcement Assistance            Local Law Enforcement Block Grant (LLEBG): NIJ allots its R&D portion
 (SLLEA)                                               of LLEBG funds to OST to assist local units of government to identify, select,
                                                       develop, modernize, and purchase new technologies for law enforcement
                                                       use.
 Community Oriented Policing Services                  Crime Identification Technology Act (CITA): CITA activities include
 (COPS)                                                upgrading and integrating national, state, and local criminal justice record,
                                                       identification systems, and funding multi-jurisdictional, multi-agency
                                                       communications systems, and improving forensic science capabilities,
                                                       including DNA analysis.
                                                       Safe Schools Technology R&D: OST’s Safe Schools Technology R&D
                                                       program uses three methods for improving school safety: needs
                                                       assessments and development of technical partners, technology R&D, and
                                                       technical assistance.
                                                       Crime Lab Improvement Program (CLIP): CLIP activities include providing
                                                       equipment, supplies, training, and technical assistance to state and local
                                                       crime laboratories to establish or expand their capabilities and capacities to
                                                       perform various types of forensic analyses.
                                                       DNA Backlog Reduction: This seeks to eliminate public crime laboratories’
                                                       backlogs of DNA evidence as soon as possible.
                                                       Paul Coverdell National Forensic Sciences Improvement Act (NFSIA):
                                                       This provides funding to state and local laboratories to improve the quality,
                                                       timeliness, and credibility of forensic science services for criminal justice
                                                       purposes.
                                                       Reimbursements of funds from other Justice Department and federal
                                                       agencies’ accounts: Reimbursable activities have included ballistic imaging
                                                       evaluation from the FBI, a study of communications interoperability (the
                                                       ability to communicate across different public safety agencies and
                                                       jurisdictions) requirements from the Defense Advanced Research Projects
                                                       Agency, and death investigator guidelines from the Centers for Disease
                                                       Control and Prevention.
Source: GAO analysis of OST data.
                                         a
                                         In fiscal year 1999, OST’s counterterrorism R&D programs received funding through the Justice
                                         Department’s Counterterrorism Fund appropriation account.


                                         OST’s budgetary resources almost quadrupled from fiscal year 1995 to
                                         1996, increased 70 percent from fiscal year 1999 to 2000, and increased
                                         63 percent from fiscal year 2001 to 2002. While resources decreased



                                         Page 7                                GAO-04-198 Justice's Office of Science and Technology
                        24 percent from fiscal year 2002 to 2003, OST’s fiscal year 2003 level still
                        represents a 157 percent increase over the fiscal year 1999 level.

                        Figure 1: OST’s Budgetary Resources in Constant 2002 Dollars, Fiscal Years
                        1995–2003
                        Dollars in millions
                        275

                        250

                        225

                        200

                        175

                        150

                        125

                        100

                         75

                         50

                         25

                          0
                          1995          1996      1997        1998       1999        2000        2001        2002        2003
                              Fiscal year
                        Source: OST.

                        Notes: Figures do not include funding for management and administration expenses, such as
                        salaries.

                        The $103.4 million increase from fiscal year 2001 to 2002 is largely attributable to increases of
                        $55.6 million in reimbursable agreements, $24.3 million in DNA Backlog Reduction allocation, and
                        $15.4 million in the Crime Lab Improvement Program allocation.

                        The sharp decrease in OST’s budgetary resources from fiscal years 2002 to 2003 is largely attributed
                        to the elimination of counterterrorism R&D allocation (from $45.3 million in fiscal year 2002), which
                        moved to the Department of Homeland Security, and a decrease of $26.2 million from reimbursable
                        agreements.


Certain Allocations     Our analysis of OST’s yearly budgetary resources from fiscal year 1995 to
Contributed to the      fiscal year 2003 showed that the overall increase can be attributed to the
Increase in Budgetary   introduction of new NIJ allocations and large increases for existing ones.
                        The NIJ allocations that contributed to the overall increase in OST’s
Resources since 1995    budgetary resources are most notably the Crime Lab Improvement
                        Program, DNA Backlog Reduction, Safe Schools Technology R&D, and
                        Counterterrorism R&D allocations. Table 2 shows figures for all years in
                        constant 2002 dollars.




                        Page 8                                 GAO-04-198 Justice's Office of Science and Technology
    All dollar figures used in this narrative are in constant 2002 dollars, except
    as noted otherwise.

    Fiscal years 1995-1996: The $39.4 million (298 percent) increase from
    $13.2 million to $52.6 million primarily came from two NIJ allocations
    totaling $35.4 million.

•   Local Law Enforcement Block Grant (LLEBG) initiated with $22.2 million.
•   Reimbursement of funds increased by $13.2 million (471 percent) from
    $2.8 million to $16.0 million.

    Fiscal years 1999-2000: The $55.6 million (70 percent) increase from
    $79.5 million to $135.1 million primarily came from three NIJ allocations
    totaling $51.7 million.

•   DNA Backlog Reduction initiated with $15.6 million.
•   Safe Schools Technology R&D allocation initiated with $15.6 million.10
•   Counterterrorism R&D increased by $20.5 million (193 percent) from $10.6
    million to $31.1 million.

    Fiscal years 2001-2002: The $103.4 million (63 percent) increase from
    $164.6 million to $268.0 million primarily came from three NIJ allocations
    totaling $95.3 million.

•   Reimbursement of funds increased by $55.6 million (209 percent) from
    $26.6 million to $82.2 million.
•   DNA Backlog Reduction increased by $24.3 million (227 percent) from
    $10.7 million to $35 million.
•   Crime Lab Improvement Program increased by $15.4 million (79 percent)
    from $19.6 million to $35 million.

    To be consistent with the report narrative and to show trends, figures in
    table 2 are in constant 2002 dollars. A table with the figures in current
    dollars can be found in appendix II.




    10
     In fiscal year 1999, NIJ used the LLEBG allocation to meet congressional guidance to
    spend $10 million on a new Safe School Initiative. The following year NIJ’s Safe Schools
    Technology R&D funding was introduced with $15 million. The OST funding was not
    reduced as a result of the $15 million increase for the Safe Schools Technology R&D.




    Page 9                             GAO-04-198 Justice's Office of Science and Technology
Table 2: Budgetary Resources in Constant 2002 Dollars for OST’s Programs by NIJ Allocation, Fiscal Years 1995-2003

 Dollars in millions
 NIJ allocations for OST programs       1995      1996      1997       1998        1999     2000      2001      2002     2003     Totala
 NIJ Base                               10.4        13.3    12.7       14.8        20.3      19.1      29.0     27.1      32.3     179.1
 Local Law Enforcement Block Grant
 (LLEBG)                                      0     22.2    21.7       21.4        21.2      20.8      20.2     20.0      19.6     167.1
 Crime Identification Technology Act
 (CITA)                                       0       0         0          0           0      4.4       4.3      1.4         0      10.1
 Safe Schools Technology Research
 and Development                              0       0         0          0           0     15.6      17.7     17.0      16.6      66.9
 Crime Lab Improvement Program
 (CLIP)                                       0      1.1      3.3      13.4        15.9      15.6      19.6     35.0      39.6     143.4
                                    b
 DNA Backlog Reduction                        0       0         0          0           0     15.6      10.7     35.0      35.2      96.5
 Paul Coverdell National Forensic
                                  b
 Sciences Improvement Act (NFSIA)             0       0         0          0           0        0         0      5.0       4.9        9.9
 Counterterrorism R&D                         0       0     10.9       12.9        10.6      31.1      36.5     45.3         0     147.2
 Reimbursements from other Justice
 and federal agencies                    2.8        16.0        0        8.9       11.5      13.0      26.6     82.2      56.0     217.1
         a
 Total                                  13.2        52.6    48.6       71.4        79.5     135.1    164.6     268.0     204.2    1037.1
Source: GAO analysis of OST data.
                                          a
                                          Totals might not add due to rounding.
                                          b
                                          In fiscal years 2000 and 2001, DNA Backlog Reduction was funded as DNA Combined DNA Index
                                          System (CODIS) Backlog Reduction. In fiscal years 2002 and 2003, both the DNA Backlog Reduction
                                          and Coverdell NFSIA allocations were funded within DNA CODIS Backlog Reduction.


                                          OST had a $63.8 million (24 percent) decrease in total budgetary resources
                                          from fiscal years 2002 to 2003, largely attributed to its not receiving fiscal
                                          year 2003 Counterterrorism R&D resources, which totaled $45.3 million in
                                          fiscal year 2002. According to OST, its counterterrorism resources were
                                          transferred to the Department of Homeland Security’s Office of Domestic
                                          Preparedness. There was also a $26.2 million decrease in the
                                          reimbursement of funds from other agencies. However, OST’s fiscal year
                                          2003 level still represents a 157 percent increase from fiscal year 1999.


Range of OST’s Program                    The range of OST’s program responsibilities has changed over the years
Responsibilities Has                      from primarily law enforcement and corrections to include broader public
Changed                                   safety technology R&D. This has happened as more and more of OST’s
                                          budgetary resources were directed to be spent on specific recipients and
                                          projects. Appropriated funds, for example, are sometimes designated for
                                          specific recipients or projects in public law. In addition, guidance on the
                                          spending of appropriated funds may be provided through congressional



                                          Page 10                                 GAO-04-198 Justice's Office of Science and Technology
committee reports. Of the more than $1 billion (in constant 2002 dollars)
that OST programs received from fiscal years 1995 to 2003, $532.6 million,
or 51 percent, was designated for specific recipients and projects in public
law or subject to guidance in committee reports designating specific
recipients or projects.11 Of the $532.6 million, $249.8 million was
designated in public law for specific recipients or projects while $282.8
million was specified in committee report guidance for specific recipients
or projects.12

In addition to the $532.6 million designated in public law for specific
recipients or projects or subject to guidance in committee reports for
specific recipients or projects, another $217.1 million was reimbursements
from other Justice and federal agencies in exchange for OST’s
management of specific projects of those agencies. Thus, the total
spending either directed for specific recipients and projects through public
law, subject to committee report guidance designating specific recipients
or projects, or received as reimbursements, amounts to $749.7 million, or
72 percent, of OST’s total budgetary resources.

The range of OST’s program responsibilities has changed to include such
areas as school safety and counterterrorism. In fiscal year 1999, a Safe
Schools Initiative program was established pursuant to conference
committee report guidance13 with $10 million14 directing NIJ to develop
school safety technologies. In another example, OST’s counterterrorism
R&D program, initially funded by public law in fiscal year 1997,15 received
$147.3 million through fiscal year 2002, $96.6 million of which was
specified in conference report guidance for three recipients from fiscal




11
 We separated reimbursements from this total because they included projects that were
not originally allocated to OST, although those projects also may have been specified in
public law and committee reports.
12
  Included in the $249.8 million was $143.5 million for the CLIP project. Committee report
guidance further designated $107.0 million of that $143.5 million for specific recipients.
Given that we have included the $107.0 million in the amounts designated in public law for
specific recipients or projects, we excluded it from the committee report guidance category
to avoid double counting.
13
 H.R. Conf. Rep. No. 105-825, at 1020-21 (1998).
14
 For this effort, NIJ initially allocated Local Law Enforcement Block Grant funds to OST.
15
 P.L. 104-208, 110 Stat. 3009, 3009-13 (1996).




Page 11                             GAO-04-198 Justice's Office of Science and Technology
years 2000 to 200216—Oklahoma City National Memorial Institute for the
Prevention of Terrorism ($37.8 million), Dartmouth College’s Institute for
Security Technology Studies ($51.8 million), and the New York
University’s Center for Catastrophe Preparedness and Response
($7 million).

OST’s program responsibilities have also changed to expand the focus on
investigative and forensic sciences. Our review of OST’s budgetary
resources for fiscal years 1995 through 2003 shows that budgetary
resources for investigative and forensic sciences equals at least
$342.1 million in constant fiscal year 2002 dollars,17 or about one-third, of
its $1 billion in budgetary resources, as shown in table 3. The proportion of
investigative and forensic sciences annual funding to total OST funding
rose from 6 percent ($800,000) in fiscal year 1995 to 52 percent
($106.0 million) in fiscal year 2003.




16
 H.R. Conf. Rep. No. 106-479, at 161 (1999); H.R. Conf. Rep. No. 106-1005, at 226 (2000);
and H.R. Conf. Rep. No. 107-278, at 86-87 (2001).
17
  The total amount of budgetary resources for investigative and forensic sciences is likely
to be larger. However, because of the limitations in detail in the budget documents we
received from OST, we could not determine the amount of funding for investigative and
forensic sciences within certain NIJ Base and LLEBG projects, such as within OST’s
technology center network and unspecified NIJ-directed projects.




Page 12                             GAO-04-198 Justice's Office of Science and Technology
Table 3: Budgetary Resources in Constant 2002 Dollars for OST’s Investigative and Forensic Sciences by NIJ Allocation,
Fiscal Years 1995-2003

 Dollars in millions
 NIJ allocation containing funds for
 investigative and forensic sciences             1995      1996     1997      1998       1999     2000    2001     2002    2003    Totala
 NIJ Base                                          0.6       0.6      0.4          1.5     6.2     5.6      5.5      5.0     4.3    29.6
 LLEBG                                               0         0        0           0        0     1.1        0       0        0      1.1
 CITA                                                0         0        0           0        0     0.8      1.3       0        0      2.0
 Safe Schools Technology R&D                         0         0        0           0        0       0        0       0        0       0
 CLIP                                                0       1.1      3.3         13.4    15.9    15.6     19.6    35.0     39.6   143.4
 DNA Backlog Reduction                               0         0        0           0        0    15.6     10.7    35.0     35.2    96.5
 Coverdell NFSIA                                     0         0        0           0        0       0        0      5.0     4.9      9.9
 Counterterrorism R&D                                0         0        0           0        0       0        0       0        0       0
 Reimbursement of funds from other agencies        0.2       8.9        0           0        0     1.6      1.1    25.4     22.0    59.1
 Totala                                            0.8     10.5       3.6         14.9    22.1    40.2     38.5   105.4    106.0   342.1
Source: GAO analysis of OST data.
                                         a
                                          Totals might not add due to rounding.


                                         OST delivers many products, which we categorized into three groups, and
OST Delivers Three                       uses various methods to deliver them. These three groups are
Groups of Products                       (1) information dissemination and technical assistance; (2) the
                                         application, evaluation, and demonstration of existing and new
Through Various                          technologies for field users; and (3) technology R&D. According to OST, as
Methods                                  of April 2003, it had delivered 945 products since its inception.18
                                         Furthermore, OST identified an additional 500 products expected from
                                         ongoing awards. Figure 2 shows our distribution of OST’s delivered
                                         products by group. We recognize, as OST officials told us, that the groups
                                         overlap and there is not a clean division between them. For example,
                                         while reports are associated with information dissemination, they may
                                         also result from the technology R&D group. OST has reviewed our
                                         classification of products and agrees that it is generally accurate. Because
                                         classification of some products is based on a judgment call, the
                                         proportions of products in each group should be considered
                                         approximations.




                                         18
                                           Because NIJ’s science and technology efforts predate OST’s establishment in fiscal year
                                         1995, some of the products listed as delivered have award years prior to 1995. The earliest
                                         listed is 1983.




                                         Page 13                                   GAO-04-198 Justice's Office of Science and Technology
OST’s Range of Products       The following examples, while not exhaustive, indicate the wide range of
                              OST’s products.

                          •   Reports on topics such as analysis of DNA typing data, linguistic methods
                              for determining document authorship, a pepper spray projectile and
                              disperser, and gunshot residue detection and interpretation.
                          •   Prototypes of products including ground-penetrating radar, ballistics
                              matching using 3-dimensional images of bullets and cartridge cases, and
                              an optical recognition system to identify and track stolen vehicles.
                          •   Evaluations of technology including prison telemedicine networks, police
                              vehicles, and protective gear.
                          •   Guides on topics such as electronic crime scene investigation, use of
                              security technologies in schools, and antennas for radio communications.
                              For a more detailed description of OST’s products and further examples,
                              see appendix III.

                              Figure 2: GAO’s Grouping of OST’s 945 Delivered Products, as of April 2003




                                                                  20%             Application, evaluation, and demonstration of
                                                                                  existing and new technology for field users
                                                                                  (191)



                                           63%                      17%           Technology R&D
                                                                                  (161)




                                                                                  Information dissemination and technical
                                                                                  assistance
                                                                                  (593)
                              Source: GAO analysis of OST data.

                              Notes: See appendix III, table 7 for examples of the products within each group. Proportions should
                              be considered approximations because some products overlap categories.


Information Dissemination     Information dissemination and technical assistance represents about
and Technical Assistance      63 percent of OST’s delivered products. OST provides information to its
                              customers in a variety of ways. For example, OST provides guidance to
                              R&D laboratories on the needs of public safety practitioners. To public
                              safety practitioners, OST recommends certain public safety practices,
                              tools, and technologies. Through its Office of Law Enforcement Standards,


                              Page 14                                   GAO-04-198 Justice's Office of Science and Technology
                            OST develops performance standards to ensure that commercially
                            available public safety equipment, such as handheld and walk-through
                            metal detectors, meets minimum performance requirements. OST also
                            helps its customers enhance their technical capacities by providing them
                            with training and technical assistance through its Crime Lab Improvement
                            Program (which also provides supplies and equipment), DNA Backlog
                            Reduction Program, and network of technology centers. OST also uses the
                            R&D expertise and experience of already established laboratories and
                            other R&D organizations to provide additional guidance for managing
                            specialized technology projects. Further, OST helps its customers receive
                            surplus federal equipment by acting as their liaison to the equipment
                            transfer program of the Department of Defense. For example, equipment
                            transferred ranges from armored vehicles to boots and uniforms.

                            In addition, OST sponsors conferences, workshops, and forums that bring
                            together its customers, technologists, and policymakers. For example, it
                            sponsors the Mock Prison Riot, an annual event demonstrating emerging
                            technologies in riot training scenarios held at the former West Virginia
                            Penitentiary in Moundsville, West Virginia. This event brings together
                            corrections officers and vendors for technology showcases and training
                            exercises. Also, OST sponsors the Innovative Technologies for Community
                            Corrections Annual Conference, among others.


Application, Evaluation,    Another OST product group is the application, evaluation, and
and Demonstration of New    demonstration of new and existing technologies, which represents about
and Existing Technologies   20 percent of OST’s delivered products. Some of OST’s programs apply
                            existing technology solutions in new ways to assist public safety
                            operations. Examples of the application of new and existing technologies
                            include developing methods for the collection and analysis of chemical
                            trace evidence left from explosives and a handheld computer device
                            provided to bomb technicians in order to access bomb data at the scene of
                            incidents. In addition, OST tests commercially available products through
                            NIJ-certified laboratories to determine whether they are in accordance
                            with national performance standards. Examples of products evaluated
                            against standards include body armor, handcuffs, and semiautomatic
                            pistols. OST’s evaluations also include conducting field tests to compare
                            different commercially available products of the same type to allow users
                            to select the product that best suits their needs. OST also demonstrates
                            technology resulting from R&D directly to its customers through OST-
                            sponsored events. For example, the Critical Incident Response
                            Technology Seminar, formerly known as the Operation America,
                            demonstrates live-fire simulation for bomb technicians. The annual Mock


                            Page 15                     GAO-04-198 Justice's Office of Science and Technology
                 Prison Riot demonstrates emerging technologies for use by corrections
                 officers and tactical team members.


Technology R&D   About 17 percent of OST’s delivered products were related to technology
                 R&D, which involves the development of prototype devices, among other
                 efforts.19 According to OST, R&D in its early stages includes development
                 of prototypes and demonstration that a principle can be proven. Applied
                 R&D, which also involves the development of prototypes, includes
                 technologies that are made available to public safety agencies, generally
                 through OST-assisted commercialization. Examples of products resulting
                 from OST’s applied R&D range from a bomb threat training simulator,
                 facial recognition technology for internet-based gang tracking, to a
                 personal alarm and location monitoring system for corrections officers.

                 According to OST, R&D in its early stages begins with testing technology
                 concepts, exploring solutions, and deciding whether continued
                 development is warranted. If OST decides to support product development
                 and if it has available funds, it awards funding to develop, demonstrate,
                 and evaluate an experimental prototype, which is then further developed
                 into an initial engineering prototype, and then demonstrated and
                 evaluated. If the prototype proves successful, OST demonstrates a “near
                 commercial” model to its customers for their evaluation.

                 While OST does not directly commercialize the results of its technology
                 R&D, it does provide prototypes to local users for field-testing and assists
                 in linking prototypes with potential commercial developers. OST officials
                 believe it would be a conflict of interest and therefore inappropriate for
                 them to promote one vendor or technology over another or try to dictate
                 what equipment their customers should purchase. OST’s role in
                 commercialization is to bring technologies and potential manufacturers
                 together so that the manufacturers can determine the feasibility of
                 commercializing the technologies.




                 19
                  While some of the products resulting from technology R&D are similar to those of the
                 application, demonstration, and evaluation of new and existing technologies group, the
                 primary distinction is that the former includes the development of prototypes and the latter
                 generally does not.




                 Page 16                            GAO-04-198 Justice's Office of Science and Technology
OST’s Methods for         OST delivers its products to its customers through a variety of methods.
Delivering its Products   (We recognize that products are sometimes delivery methods. For
                          example, a publication can be both a product resulting from research and
                          a method of information dissemination.) Besides publications, OST’s
                          methods for delivering information and technical assistance include mass
                          mailings; downloadable material from its Web site; panels, boards, and
                          working groups; training, support, and presentations; and programs to
                          enhance the capacity of public safety agencies.

                          OST also delivers its products related to application, evaluation, and
                          demonstration through various means. For example, private industry
                          provides new and existing technologies to OST; in turn, OST informs its
                          customers of the results of using these technologies in new ways. OST
                          publishes user guides and the test results of its evaluations of
                          commercially available equipment (both standards-based and comparison-
                          based). Seeking to further educate its customers, OST demonstrates new
                          technology at technology fairs, providing “hands on” opportunities to use
                          it.

                          For its R&D products, OST may test “near commercial” prototypes in
                          particular settings. For example, OST may install in a police agency a
                          prototype technology that facilitates communications among public safety
                          agencies and across jurisdictions. If the technology is effective, the police
                          agency may incorporate the technology directly into its operations, before
                          the technology has become a commercial product.


                          OST’s efforts to measure its performance results, including the usefulness
OST’s Performance         and effectiveness of its products, do not fully meet applicable
Measurement Efforts       requirements. To help Justice comply with the Government Performance
                          and Results Act of 1993 (GPRA), OST establishes goals and develops
Do Not Fully Meet         performance measures to track its progress. GPRA, which mandates
Requirements              performance measurements by federal agencies, requires, among other
                          things, that each agency measure or assess relevant outputs and outcomes
                          of each program activity.20 According to GPRA, the Office of Management
                          and Budget (OMB), and GAO, outcomes assess actual results as compared
                          with the intended results or consequences that occur from carrying out a


                          20
                           Performance measures are to be included in the agency performance plan covering each
                          program activity set forth in the budget of such agency. Program activity, in this case,
                          refers to projects and activities that are listed in program and financing schedules of the
                          annual Budget of the United States Government.




                          Page 17                             GAO-04-198 Justice's Office of Science and Technology
                          program or activity. Outputs count the goods and services produced by a
                          program or organization. Intermediate measures can be used to show
                          progress to achieving intended results. Subsequent OMB and committee
                          report guidance on GPRA, and previous GAO reports21 recognize that
                          output measures can provide important information in managing
                          programs. However, committee report guidance emphasizes using
                          outcome measures to aid policy makers because such measures are key to
                          assessing intended results.


OST Performance           The performance measures that OST has developed do not measure
Measures Do Not Measure   results. According to the NIJ director, the Assistant Attorney General
Results                   (AAG) in April 2002 issued a memorandum requiring NIJ, including OST,
                          to develop outcome measures for fiscal year 2004. In August 2002, the NIJ
                          Director responded by stating that OST had indeed developed outcome
                          measures for its programs. In its fiscal year 2004 performance plan,22 OST
                          established goals for 11 of its initiatives23 and developed 42 measures for
                          assessing the achievement of those goals. However, based on our review
                          of OST’s performance plan, OMB guidance on GPRA, and GAO definitions
                          of outcome, output, and intermediate measures, we determined that of the
                          42 measures, none were outcome-oriented, 28 were output-oriented, and
                          14 were intermediate. See table 4 for GAO’s determination of the measures
                          and appendix VI for further details of our results.




                          21
                           U.S. General Accounting Office, Managing for Results: An Agenda to Improve the
                          Usefulness of Agencies’ Annual Performance Plans, GAO/GGD/AIMD-98-228 (Washington,
                          D.C.: Sept. 8, 1998).
                          22
                            Annual performance plans describe a department component’s goals and performance
                          targets in support of the department’s long-term strategic goals and targets. In its fiscal
                          year 2004 performance plan, OST reported actual performance data for fiscal year 2002,
                          enacted plans for fiscal year 2003, and performance plans for fiscal year 2004.
                          23
                           Initiatives in this sense encompass portfolio areas, programs, and projects.




                          Page 18                             GAO-04-198 Justice's Office of Science and Technology
Table 4: GAO’s Assessment of the 42 Measures OST Developed for 11 of Its
Initiatives

                                                               Type of measure
 OST’s initiatives                                      Output Intermediate Outcome
 1. Convicted Offender DNA Backlog Reduction
    Program                                                   0              3          0
 2. No Suspect DNA Backlog Reduction Program                  0              1          0
 3. Paul Coverdell National Forensic Sciences
    Improvement Grants Program                                0              1          0
 4. Critical Incident Response Technology Initiative          4              1          0
 5. DNA Research & Development                                4              0          0
 6. Law Enforcement Technology Research and
    Development                                               4              1          0
 7. School Safety Technology                                  3              0          0
 8. Crime Lab Improvement Program                             4              6          0
 9. Office for Law Enforcement Standards                      3              0          0
 10. Smart Gun                                                4              0          0
 11. OST’s network of regional centers (known as
     the National Law Enforcement and Corrections
     Technology Center system)                                2              1          0
Source: GAO analysis of OST data.


According to Justice officials, R&D activities present measurement
challenges because outcomes are difficult or costly to measure. As the NIJ
Director pointed out, a May 2002, White House OMB and Office of Science
and Technology Policy memorandum concluded that agencies should not
have the same expectations for measuring the results of basic R&D as they
do for applied R&D.24 According to NIJ, relatively little of OST’s work is
basic R&D. As shown earlier, most of OST’s products are related to
information dissemination and technical assistance and the application,
evaluation, and demonstration of existing and new technologies for field
users.




24
  According to the OMB document, Budget of the United States Government (Analytical
Perspectives) for fiscal year 2004, basic R&D is defined as systematic study directed
toward greater knowledge or understanding of fundamental aspects of phenomena and of
observable facts without specific applications toward processes or products in mind.
Applied R&D is defined as systematic study to gain knowledge or understanding necessary
to determine the means by which a recognized and specific need may be met.




Page 19                             GAO-04-198 Justice's Office of Science and Technology
                            We recognize that OST’s task in relation to measuring the results of even
                            non-basic research is complex in part because of the wide array of
                            activities it sponsors, and because of inherently difficult measurement
                            challenges involved in assessing the types of programs it undertakes. For
                            example, programs that are intended to deter crime face measurement
                            issues in assessing the extent to which something (crime) does not
                            happen. Nevertheless, improvement in measurement of program results is
                            important to help OST ensure it is doing all that is possible to achieve its
                            goals. It is worth noting that an outcome measure in relation to one OST
                            program was discussed by the NIJ Director in a May 2002 statement to
                            Congress. In this statement, the Director provided an example of an
                            outcome from the Convicted Offender DNA Backlog Reduction Program.
                            The Director stated that as a direct result of the program, approximately
                            400,000 convicted offender samples and almost 11,000 cases with no
                            suspect were analyzed. According to the NIJ Director, as of May 14, 2002,
                            more than 900 “hits” had been made on the FBI’s Combined DNA Index
                            System (CODIS) database as a direct result of the program, that is, 900
                            cases previously unsolved had been reopened. This information indicates
                            how the program is achieving its intended results in addressing unsolved
                            cases. Although this example seems to be a credible outcome measure, it
                            is not included in OST’s fiscal year 2004 performance plans.


Limitations in OST’s        OST efforts to measure information dissemination effectiveness have been
Efforts to Measure          limited. One of the purposes of GPRA is to improve federal program
Effectiveness of            effectiveness and public accountability by promoting a new focus on
                            results, service quality, and customer satisfaction. Surveys to gauge
Information Dissemination   customer satisfaction represent one step toward finding out whether
                            customers have received information and whether they deem it of value.
                            However, these surveys have limitations in determining the extent to
                            which the information has been acted upon and resulted in intended
                            improvements. Thus, surveys such as these are more likely to be
                            intermediate measures (Did information get transferred?) than outcome
                            measures (Did information get transferred, acted upon, and achieve a
                            result?).

                            In 1998, NIJ initiated an effort to report the results of surveys to measure
                            the satisfaction of participants at all conferences, workshops, and seminar
                            series.25 OST reported on the “grantee level of satisfaction with NIJ


                            25
                             The surveys were done to determine if participants were satisfied with the conference as
                            a vehicle of information dissemination.



                            Page 20                            GAO-04-198 Justice's Office of Science and Technology
conferences” for fiscal years 1998-2000. However, in the fiscal years 2001-
2004 GPRA performance plans, OST discontinued tracking the surveys
because OJP and NIJ had ceased tracking these data as a performance
measure.

In fiscal year 2001, OST attempted to evaluate the effectiveness and value
of its TECHbeat newsletter. The survey sample of 5,500 was taken from a
distribution of major readership groups on TECHbeat’s mailing list of
20,674. According to OST, the response rate for the survey was too low to
produce statistically valid results: only 124 completed or substantially
completed responses were collected. The surveyors also experienced a
very low return on follow-up phone queries. According to the study, the
primary reason for the exceedingly low response rate was that so many
individuals on the mailing list had either changed jobs or were completely
unfamiliar with TECHbeat. Given these results, OST is trying to improve
the management and distribution of TECHbeat.26

In fiscal year 2001, OST attempted to launch another effort to measure
program results, service quality, and customer satisfaction, but funding for
the effort was not provided. OST requested funding for an evaluation to
measure the success of its outreach efforts, including those by its
technology centers. The evaluation was to determine customer
satisfaction with its strategies for outreach and communication and with
its products. Specifically, OST planned to measure user satisfaction of the
content, format, and delivery mechanisms of its efforts, such as
technology information and assistance.




26
 To address issues with the mailing lists, the technology centers have shipped a larger
portion of copies to agencies, in bulk, and to individuals who have actively requested
copies and supplied their addresses; continued to purchase the most current version of the
National Directory of Law Enforcement Administrators, Correctional Institutions and
Related Agencies to update their mailing list; and modified mailing labels to include the
addressee and “...or Training Officer” in case the addressee is no longer with that agency.




Page 21                            GAO-04-198 Justice's Office of Science and Technology
Most Studies of Other OST   In fiscal years 1998 and 1999, OST funded eight outside studies of some of
Initiatives Have Focused    its science and technology initiatives (see table 5).27 Our review of these
Primarily on Process        studies showed that seven of the eight studies focused on management
                            and organizational processes, and one was outcome-oriented.28
                            Management and process evaluations can be useful tools for examining
                            how a program is operating and can offer insights into best practices. They
                            do not assess whether a program is achieving its intended results.




                            27
                             Initiatives in this sense encompass portfolio areas, programs, and projects.
                            28
                             GPRA establishes two approaches for assessing an agency’s performance: annual
                            measurement of program performance against goals outlined in a performance plan and
                            program evaluations to be conducted by the agency as needed. Evaluations can play a
                            critical role in helping to address measurement and analysis challenges. Performance
                            measurement is the ongoing monitoring and reporting of program accomplishments,
                            particularly progress toward established goals. Program evaluations are individual
                            systematic studies conducted periodically or on an ad hoc basis to assess how well a
                            program is working. See U.S. General Accounting Office, Performance Measurement and
                            Evaluation: Definitions and Relationships, GAO/GGD-98-26 (Washington, D.C.: April
                            1998).




                            Page 22                            GAO-04-198 Justice's Office of Science and Technology
Table 5: OST’s Outside Studies of Its Initiatives

 Outside study topics                                 Focus of study                                   Type of study      Date completed
 1. National Law Enforcement and Corrections          Management, oversight, structure,                Process            October 1998
                                           a
     Technology Centers (NLECTC) Program              organization, and operations
 2. Counterterrorism Technology Portfolio             Organization, funding, program process           Process            June 1999
 3. Investigative and Forensic Sciences               Program and structure, management,               Process            August 1999
    Technology Portfolio                              policies, procedures, lines of control, and
                                                      funding
 4. Less-Than-Lethal Technology Portfolio             Management, processes, and organization          Process            September 1999
 5. Southwest Border States Antidrug Information      Program efficacy, including awareness of         Outcome            October 1999
    System                                            the program, and its value and usefulness
                                                      or benefits to customers
 6. Law Enforcement and Corrections Technology        Management and coordination, processes,          Process            February 2000
    Advisory Council Priorities and Technology        organizational challenges
    Portfolio Interaction
 7. Critical Incident Response and Management         Options for planning, organization, mission, Process                September 2000
    Crime fighting Technology Program for State       management, budget, and
    and Local First Responder Teams                   recommendations
 8. Standards Initiative                              Recommendations for the planning,                Process            September 2000
                                                      organization, and management of the
                                                      proposed initiative expected to be a part of
                                                      #7 above
Source: OST.
                                            a
                                            In this report we refer to the National Law Enforcement and Corrections Technology Centers as
                                            OST’s network of technology centers.


Efforts Are Under Way to                    The Homeland Security Act of 2002 requires NIJ29 to transmit to Congress
Address Performance                         by late November 2003 a report assessing the effectiveness of OST’s
Measurement of                              existing system of technology centers and to identify the number of
                                            centers necessary to meet the technology needs of federal, state, and local
Technology Centers                          law enforcement in the United States. According to NIJ, in response to the
                                            Homeland Security Act requirement, it has initiated a study to assess the
                                            impact and effectiveness of the technology center system and how it can
                                            be enhanced to meet the evolving science and technology research and
                                            technology needs of the state and local public safety community. NIJ also
                                            stated that the report would address the functions that the technology
                                            center system must provide to transfer NIJ’s research and development
                                            results to practice in the criminal justice system. NIJ and OST have failed



                                            29
                                             The Homeland Security Act actually directs the “Director” of OST to transmit the report.
                                            After reorganizing in early 2003, NIJ now calls this position the assistant NIJ director for
                                            science and technology.




                                            Page 23                               GAO-04-198 Justice's Office of Science and Technology
                         to provide us with information detailing the methodology of the study, so
                         we cannot comment on the likelihood that this study will produce the
                         information sought by Congress. Additionally, according to OJP, the
                         technology centers are in the process of developing outcome measures to
                         demonstrate the impact of their activities.

                         According to NIJ, OJP has implemented additional performance measures
                         developed in May 2003 that will apply to NIJ, including OST. However, OJP
                         said it would defer implementing the measures related to the technology
                         centers until the results of the technology center study are known and NIJ
                         has a chance to take action, if warranted.


Measuring Results Is     We acknowledge that measuring results using outcome measures is
Difficult but Feasible   difficult, and may be especially so in relation to some of the types of
                         activities undertaken by OST. Indeed, given the types and wide range of
                         program goals for OST efforts—solving old crimes, saving lives, and
                         reducing property loss—it may be the case that for some programs
                         intermediate measures represent the best feasible measure of results. We
                         note that approximately 63 percent of OST’s products fall into the category
                         of information dissemination and technical assistance, aimed at informing
                         customers and ultimately encouraging adoption of research results that
                         lead to increased efficiency and effectiveness. There are strategies
                         available that have been used by other federal agencies to take steps
                         toward assessing the effectiveness of information dissemination and
                         technical assistance efforts. For example, a recent GAO report30 outlines
                         various strategies to assess media campaigns and informational seminars,
                         including immediate post workshop surveys and follow-up surveys and the
                         use of logic models to define measures of a program’s progress toward
                         intended results and long-term goals.


                         Given the wide range of its products, OST has the potential to significantly
Conclusions              improve the technological capabilities of federal, state, and local public
                         safety agencies. However, the lack of information about the results of
                         program efforts, or the assessment of progress toward goals, means that
                         little is known about their effectiveness. While developing outcome



                         30
                          U.S. General Accounting Office, Program Evaluation: Strategies for Assessing How
                         Information Dissemination Contributes to Agency Goals, GAO-02-923 (Washington, D.C.:
                         Sept. 30, 2002)




                         Page 24                          GAO-04-198 Justice's Office of Science and Technology
                     measurements in many research-related programs is extremely difficult,
                     there are various performance measurement strategies that other federal
                     programs have used for assessing information dissemination, technical
                     assistance and other R&D activities that might be applied to OST’s
                     programs. It is important to develop outcome measurements where
                     feasible, or intermediate measurements where appropriate, to assist
                     Congress, OST and NIJ management, and OST’s customers to better assess
                     whether investment in OST’s programs is paying off with improved law
                     enforcement and public safety technology.


                     To help ensure that OST does all that is possible to measure its progress in
Recommendation       achieving goals through outcome-oriented measures, we recommend that
                     the Attorney General instruct the Director of NIJ to reassess the measures
                     OST uses to evaluate its progress toward achieving its goals and to better
                     focus on outcome measures to assess results where possible. In those
                     cases where measuring outcome is, after careful consideration, deemed
                     infeasible, we recommend developing appropriate intermediate measures
                     that will help to discern program effectiveness.


                     We provided a copy of a draft of this report to the Attorney General of the
Agency Comments      United States for review and comment. In an October 30, 2003, letter, the
and Our Evaluation   Assistant Attorney General (AAG) for OJP commented on the draft. Her
                     comments are summarized below and presented in their entirety in
                     appendix VII. OJP also provided technical comments, which have been
                     incorporated into this report where appropriate.

                     In the AAG’s written response, the Justice Department concurred with our
                     recommendation that NIJ reassess the measures OST uses to assess
                     program outcomes. In response to our recommendation, the AAG reported
                     that she has directed the NIJ Director, to reassess NIJ’s performance
                     measures for OST and to refine them, where possible, in order to focus
                     them more toward measuring outcomes.

                     While the AAG agreed with our recommendation, she also made several
                     other comments. First, she commented that developing numerical
                     outcome measures like those urged by GAO is a particular challenge for
                     R&D activities. As stated in our report, we recognize that measuring
                     results using outcome measures is difficult and may be especially so in
                     relation to some of the types of activities undertaken by OST. Our
                     reference to a numerical measure is meant only as an example of how one
                     of OST’s program activities can be linked to intended results. We believe


                     Page 25                      GAO-04-198 Justice's Office of Science and Technology
that further consideration of measures, both quantitative and qualitative,
could improve the assessment of results for R&D as well as other OST
programs. Our report also notes that relatively little of OST’s work is R&D.
The majority of OST’s products are in the category of information
dissemination and technical assistance.

Second, the AAG noted that GAO did not reach any conclusions in its
discussion of OST’s growth in budgetary resources, changes in program
responsibilities, management of programs, and delivery of its products.
The AAG noted that Justice believed that OST’s record is outstanding.
Neither OST nor we can determine whether OST’s efforts in these areas
are successful or otherwise, given that OST has not developed measures to
assess their outcomes. Therefore, it is not possible to draw conclusions.

Third, the AAG indicated that GAO did not discuss in detail that over one-
half of OST’s funds were designated by Congress for specific recipients
and projects. She noted that GAO missed an opportunity to inform the
requester of the impact of Congress’ recent decisions regarding OST. We
reported that 51 percent of OST’s budgetary resources were designated for
specific recipients and projects in public law or subject to guidance in
committee reports.


As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 10 days from its
issue date. At that time, we will send copies of the report to the Attorney
General, appropriate congressional committees and other interested
parties. We will also make copies available to others upon request. In
addition, the report will be available at no charge on GAO’s Web site at
http://www.gao.gov. Major contributors to this report are listed in
appendix VIII.




Page 26                       GAO-04-198 Justice's Office of Science and Technology
If you or your staff have any questions concerning this report, contact me
on (202) 512-8777.

Sincerely yours,




Laurie E. Ekstrand
Director, Homeland Security and Justice Issues




Page 27                      GAO-04-198 Justice's Office of Science and Technology
             Appendix I: Scope and Methodology
Appendix I: Scope and Methodology


             To answer our objectives, we interviewed National Institute of Justice
             (NIJ) and Office of Science and Technology (OST) officials and collected
             documents at OST’s office in Washington, D.C., and at three of OST’s
             technology centers—the National center in Rockville, Maryland; West
             center in El Segundo, California; and Border Research and Technology
             Center in San Diego, California. We selected the Rockville center because
             of its proximity to Washington, D.C., and the other two centers because of
             their locations and particular areas of technology and technical
             concentrations. We also interviewed a small group of OST’s customers—
             federal, state, and local law enforcement, and corrections and public
             safety officials—who were selected by officials at the El Segundo and San
             Diego centers. In addition, we analyzed information that is available on the
             National Institute of Justice’s public Web site.

             To determine OST’s budgetary resources and amounts from fiscal year
             1995 to fiscal year 2003 and the changes in OST program responsibilities,
             we reviewed NIJ and OST budget documents, interviewed officials in
             OST’s Technology Management and the OJP’s Office of Budget and
             Management Services, and reviewed pertinent appropriations laws and
             committee reports covering that period. To determine the amount of OST
             budgetary resources that were directed to specific recipients and projects,
             we compared OST’s budget documents that listed individual recipients and
             projects with the public laws and reports. We defined directed spending as
             spending for specific recipients and projects designated in appropriations
             laws or subject to congressional committee report guidance designating
             specific recipients or projects. We did not determine the amount of
             reimbursable projects designated in public laws or specified in committee
             reports because those projects were not originally allocated to OST.
             Instead, we considered all the reimbursable projects to be specific
             projects for which OST was directed pursuant to its agreements with other
             agencies on spending its reimbursable funds.

             To determine the changes in OST’s program responsibilities, we analyzed
             the year-to-year changes in its budget and program scope. To determine
             the amount of OST’s budgetary resources used for investigative and
             forensic sciences for fiscal years 1995-2003, we compared OST’s portfolio
             description and NIJ’s definition of forensic sciences with the individual
             budget program and project items listed in OST’s budget documents for
             each fiscal year. However, while we recognize that OST’s technology
             centers and their technical partners include investigative and forensic
             sciences in their provision of technical assistance, we did not attempt to
             determine the amount of center funds associated with investigative and
             forensic sciences because the budget documents we received from OST


             Page 28                             GAO-04-198 Justice's Office of Science and Technology
Appendix I: Scope and Methodology




did not break out such amounts within the funding awarded to the centers.
Therefore, our determination that $342.1 million of OST’s total funding
supported investigative and forensic sciences did not include such
amounts.

To determine the amounts of funding awarded to the technology centers,
we analyzed databases on all of the products OST has produced through
April 2003 and the associated grants, interagency agreements, and
cooperative agreements and their amounts.

To determine the composition of OST’s products and how OST delivers
the products to its customers, we analyzed OST documents and a database
of all the products associated with its past and ongoing awards, from
inception through April 2003, that were delivered or anticipated to be
delivered. While the database included the award amounts associated with
the products, it was not possible to reliably associate the award amounts
for each product or type of product because multiple types of products
could result from individual awards. We also conducted interviews with
the parties mentioned above.

For the budget and product data that OST provided us, we assessed the
reliability of these data by examining relevant fields for missing data,
conducting range checks to identify any potentially erroneous outliers and
inspecting a subset of selected data elements that were common to two or
more data sets. In addition, we independently verified selected budget
data back to appropriations legislation and Committee reports. In
conducting our analyses, we identified some potential data errors or
reliability problems. When this occurred, we contacted agency officials to
address and resolve these matters. However, we did not verify the budget
or product data back to source materials. Overall, we determined that
budget or product data provided to us is adequate for the descriptive
purposes it is used in this report.

We examined OST’s efforts to measure performance by interviewing
officials on this matter at OJP, NIJ, and OST in the Washington, D.C.,
office along with officials and staff at the technology centers, and current
and previous Advisory Council officials. We also reviewed related agency
documents, such as the OJP mission statement and performance plans;
NIJ strategic planning documents and website pages, annual performance
plans and performance reports, and GPRA documents; policies and
procedures, Department of Justice memoranda, OST internal planning and
reporting documents, program descriptions and documentation, and other
related documents.


Page 29                             GAO-04-198 Justice's Office of Science and Technology
Appendix I: Scope and Methodology




As part of our examination, we reviewed OST’s fiscal year 1997 to 2004
goals and measures as presented in OST’s GPRA performance plans.1 We
focused our review on OST’s fiscal year 2004 performance plan and
measures. As part of our review of these goals and measures, we made a
determination as to whether the performance measure was output,
outcome, or intermediate-oriented. To make this determination about the
types of performance measures contained in OST’s performance plans, we
compared the measures used in the plans with the requirements of GPRA,
accompanying committee report, OMB’s guidance on performance
measurement challenges (Circular A-11), Justice’s guidance to its
components for preparing performance measures, and previous GAO work
on GPRA.2

Also included in our examination of OST performance measurement
efforts were studies prepared by external parties under grants from OST
that reviewed selected OST initiatives such as portfolio areas, projects,
and programs. In response to our request for all of OST’s efforts to assess
its programs, OST provided eight outside studies funded from fiscal years
1998 to 1999. For example, the Pymatuning Group, Inc., conducted an
“Assessment of the National Law Enforcement and Corrections
Technology Center (NLECTC) Program,” which described the operations
of the OST’s regional technology centers network. We reviewed all eight of
the outside studies for performance information on the OST initiatives
being examined in the report. We examined the studies to determine
whether they provided information that would be considered consistent
with an outcome-oriented evaluation as defined by our criteria.3

The scope of this review was limited to OST, and therefore we cannot
compare OST’s efforts to measure the performance of its programs or the
amount of funding directed to specific recipients and projects with the
efforts and funding of any other federal R&D agencies. We performed our


1
 To determine the goal for each OST program included in the plan, we used the stated
public benefit statement provided in the plan, except for the Law Enforcement Technology
R&D program.
2
 See U.S. General Accounting Office, Agency Performance Plans: Examples of Practices
That Can Improve Usefulness to Decisionmakers, GAO/GGD/AIMD-99-69 (Washington,
D.C.: Feb. 26, 1999) for our guidance concerning intermediate-oriented measures and
Managing for Results: Critical Issues for Improving Federal Agencies’ Strategic Plans,
GAO/GGD-97-180 (Washington D.C.: Sept. 16, 1997).
3
See U.S. General Accounting Office, Performance Measurement and Evaluation:
Definitions and Relationships, GAO/GGD-98-26 (Washington, D.C.: April 1998).




Page 30                             GAO-04-198 Justice's Office of Science and Technology
Appendix I: Scope and Methodology




audit work from September 2002 to September 2003 in Washington, D.C.,
and other cited locations in accordance with generally accepted
government auditing standards.




Page 31                             GAO-04-198 Justice's Office of Science and Technology
                                              Appendix II: Bugetary Resources for OST’s
Appendix II: Bugetary Resources for OST’s     Programs in Current Year Dollars



Programs in Current Year Dollars

Table 6: Budgetary Resources in Current Dollars for OST’s Programs by NIJ Allocation, Fiscal Years 1995-2003

 Dollars in millions
 NIJ allocations for OST programs                  1995      1996     1997      1998       1999     2000     2001     2002     2003     Totala
 NIJ Base                                            9.2     12.0      11.7      13.8       19.2    18.4     28.6      27.1     32.8     172.9
 Local Law Enforcement Block Grant
 (LLEBG)                                                0    20.0      20.0      20.0       20.0    20.0     20.0      20.0     19.9     159.8
 Crime Identification Technology Act (CITA)             0        0        0           0        0      4.2      4.2      1.4        0       9.9
 Safe Schools Technology Research and
 Development                                            0        0        0           0        0    15.0     17.5      17.0     16.9      66.4
 Crime Lab Improvement Program (CLIP)                   0      1.0      3.0      12.5       15.0    15.0     19.4      35.0     40.3     141.1
                                    b
 DNA Backlog Reduction                                  0        0        0           0        0    15.0     10.6      35.0     35.8      96.3
 Paul Coverdell National Forensic Sciences
                          b
 Improvement Act (NFSIA)                                0        0        0           0        0        0        0      5.0      5.0      10.0
 Counterterrorism R&D                                   0        0     10.0      12.0       10.0    30.0     36.0      45.3        0     143.3
 Reimbursements from other Justice and
 federal agencies                                    2.5     14.5         0          8.3    10.9    12.5     26.3      82.2     56.9     214.1
         a
 Total                                              11.7     47.5      44.7      66.6       75.1   130.2    162.6    268.0    207.6 1,013.8
Source: GAO analysis of OST data.
                                              a
                                              Totals might not add due to rounding
                                              b
                                              In fiscal years 2000 and 2001, DNA Backlog Reduction allocations was funded as DNA CODIS
                                              Backlog Reduction. In fiscal years 2002 and 2003, both the DNA Backlog Reduction and Coverdell
                                              NFSIA allocations were funded within DNA CODIS Backlog Reduction..




                                              Page 32                                 GAO-04-198 Justice's Office of Science and Technology
                                            Appendix III: OST’s 10 Categories of Products
Appendix III: OST’s 10 Categories of Products


                                            While we divided OST’s products into three groups for our reporting
                                            purposes, OST divides them into 10 categories. (See table 7 for GAO’s
                                            3 groupings of OST’s 10 categories.) In regrouping OST’s 10 categories, we
                                            recognized, as OST officials told us, that the 10 categories overlap and
                                            there is not a clean division between them. We also recognized that many
                                            of OST’s products could also be considered a delivery method. For
                                            example, publications, such as the TECHbeat newsletter, are OST
                                            products that can also represent a method of delivery for OST technology
                                            information. OST has reviewed our classification of products and agrees
                                            that it is generally accurate.

Table 7: GAO’s Groupings of OST’s Categories of Products and Examples of Each Category

 GAO’s 3 groupings of
 OST’s 10 categories         OST’s 10 categories                   Examples of products
 1. Technology R&D           1. Results of the early stages of     Results of investigating forensic techniques, studying potential
                                 technology R&D include the        less-than-lethal incapacitation technologies, and researching
                                 development of prototypes and advanced weapons detection.
                                 demonstration that a principle or
                                 concept can be proven.
                             2. New applied technologies made      Improved bomb robots and electromagnetic concealed weapons
                                available to public safety         detection.
                                agencies, generally through
                                commercialization.
 2. Application, evaluation, 3. Existing technologies applied to   Communications interoperability (the ability to communicate across
     and demonstration of       new situations.                    different public safety agencies and jurisdictions), handheld
     new and existing                                              computer devices for bomb investigators, and software tools to
     technologies for field                                        measure levels of school safety.
     users
                             4. Product evaluations based on       Ballistic and stab-resistant body armor, handcuffs, semi-automatic
                                voluntary national performance     pistols, walk-through metal detectors; patrol vehicles, patrol vehicle
                                standards and comparisons with     tires, and replacement brake pads; cut-, puncture-, and pathogen-
                                like products.                     resistant protective gloves.
                             5. Technology demonstrations.         Annual Mock Prison Riot meeting demonstrates emerging
                                                                   technologies for use in hands-on riot training scenarios, and the
                                                                   annual Critical Incident Response Technology seminar (formerly
                                                                   called Operation America), in which bomb technicians practice live-
                                                                   fire simulations.
 3. Information              6. Information and guidance for       Needs assessments of what public safety practitioners require,
     dissemination and           public safety practitioners and   such as for combating electronic crime and terrorism; funding
     technical assistance        those in R&D.                     requirements for forensic science; investigative, selection, and
                                                                   application guides; and technology and training for small agencies.
                             7. Standards to ensure that           Ballistic resistance of personal body armor and handheld and
                                commercially available public      walk-through metal detectors.
                                safety equipment meets
                                minimum performance.




                                            Page 33                            GAO-04-198 Justice's Office of Science and Technology
                                                   Appendix III: OST’s 10 Categories of Products




 GAO’s 3 groupings of
 OST’s 10 categories                OST’s 10 categories                    Examples of products
                                    8. Enhanced capacity that gives        Technology assistance provided to OST’s customers by its
                                       agencies access to technologies     regional centers; Crime Lab Improvement Program for establishing
                                       and tools they otherwise might      or expanding laboratories’ capacities for forensic analysis; the DNA
                                       not have had funding for or         Backlog Reduction Program for helping to eliminate DNA backlog,
                                       access to.                          leading to the resolution of unsolved crimes.
                                    9. Conferences, forums, and            Technical working groups of experienced practitioners and
                                       workshops that bring together       researchers working to improve investigation techniques and issue
                                       practioners, technologists, and     procedural guides. Panels and councils of public safety leaders,
                                       policymakers to form                experts, and policymakers assisting OST and its regional centers
                                       partnerships, communicate           in setting development priorities, launching technologies,
                                       needs, and educate participants.    identifying equipment problems, and enhancing understanding of
                                                                           technological issues and advances. Commercialization planning
                                                                           workshops involving developers and entrepreneurs interested in
                                                                           commercializing public safety technologies.
                                    10. Technical expertise and            Space and Naval Warfare Systems Command providing oversight,
                                        oversight of technology projects   contracting, and administrative support for the NIJ User Centric
                                        provide additional oversight and   Information Technology Program and Critical Incident
                                        guidance.                          Management System Testbed; the U.S Air Force Research
                                                                           Laboratory providing oversight and administrative support to the
                                                                           NIJ Concealed Weapons Detection and Personnel Location
                                                                           Technology Programs and hosting the NIJ-sponsored National
                                                                           Cyberscience Laboratory.
Source: GAO analysis of OST data.




                                                   Page 34                             GAO-04-198 Justice's Office of Science and Technology
                 Appendix IV: OST’s Portfolio Areas
Appendix IV: OST’s Portfolio Areas


                 OST has organized its individual projects to develop, improve, and
                 implement technology for public safety agencies into nine portfolio areas.
                 As of April 2003, these portfolio areas included

             •   critical incident technology, for first responders and investigators
                 protecting the public in the event of critical incidents such as natural
                 disasters, industrial accidents, or terrorist acts;

             •   communications interoperability1 and information sharing,
                 enhancing communication among public safety agencies through wired
                 links, wireless radios, and information networks, even when disparate
                 systems are involved;

             •   electronic crime, supporting computer forensic laboratories, publishing
                 guides for handling electronic evidence, and developing computer forensic
                 tools;

             •   investigative and forensic sciences, funding at the state and local levels
                 for DNA-typing of convicted offenders and use of DNA-typing in the
                 investigation of unsolved cases, and developing tools for forensic
                 casework;

             •   crime prevention technologies, including contraband detectors, sensors
                 and surveillance systems, and biometric technologies;

             •   protective systems technologies, including body armor; “smart”
                 handguns, which fire only upon recognition of, for example, a certain
                 handprint or password; puncture resistant gloves; better handcuffs; better
                 concealed weapon detection; and personnel tracking and location
                 technologies;

             •   less-than-lethal technologies, developing alternatives to lethal force,
                 including technologies involving electrical or chemical effects, light
                 barriers, vehicle stopping, and blunt trauma, and evaluating and modeling
                 the effects of these technologies;

             •   learning technologies, developing technology tools for agencies to use in
                 training their personnel, including use of the internet, CD-ROMs, and
                 video-based and interactive simulations; and


                 1
                  Interoperability of communications is the ability to communicate across different public
                 safety agencies and jurisdictions.




                 Page 35                              GAO-04-198 Justice's Office of Science and Technology
    Appendix IV: OST’s Portfolio Areas




•   standards and testing, ensuring that the equipment public safety
    agencies buy is safe, dependable, and effective.




    Page 36                              GAO-04-198 Justice's Office of Science and Technology
                             Appendix V: OST’s Operations
Appendix V: OST’s Operations


                             As with other federal agencies, OST’s operations involve multiple levels of
                             internal organization and multiple kinds of external partners. OST’s
                             multiple levels of organization include a Washington, D.C., office that
                             manages its technology programs and a network of technology centers
                             around the country that provide technical assistance to OST’s regional
                             customers. OST also collaborates with other R&D entities, such as those in
                             the Departments of Defense and Energy and public and private
                             laboratories, by forming technical partnerships in order to leverage
                             already established technical expertise and resources to support their
                             program efforts. Another aspect of OST’s complex operations is the need
                             to determine OST’s own priorities and the priorities of its customers,
                             which involves Washington and regional center staff collaborating
                             formally and informally with a myriad of federal, state, and local officials,
                             as well as with one another.


OST Has Multiple Levels of   OST’s multiple levels of organization include a Washington, D.C., office
Organization                 and technology centers, as well as technical partnerships with
                             government, public and private R&D and public safety organizations. As of
                             September 2003, OST’s Washington office consisted of 25 full-time-
                             equivalent Justice staff divided into three divisions and under the Assistant
                             NIJ Director for OST.1 Responsibility for managing these programs is
                             divided among the three divisions. (See figure 3 for OST’s organizational
                             structure.)




                             1
                              In addition, there were 2 federal detailees, 2 visiting scientists, and 32 on-site contractors
                             supporting OST.




                             Page 37                              GAO-04-198 Justice's Office of Science and Technology
    Appendix V: OST’s Operations




    Figure 3: OST’s Organizational Structure




•   Research and Technology Development Division manages electronic
    crime (including cybercrime), critical incidents and counterterrorism,
    communications interoperability and information sharing, crime
    prevention, learning technology tools, less-than-lethal technologies,
    standards development, school safety, and corrections technologies.

•   Investigative and Forensic Sciences Division manages DNA-related
    R&D and other investigative and forensic sciences, such as fingerprint
    analysis, and includes the Crime Laboratory Improvement Program
    projects, DNA Backlog Reduction projects, and DNA research and
    development projects.

•   Technology Assistance Division, through the technology center
    network, provides training and technical advice to, and identifies
    technologies for, OST’s customers, and oversees OST’s network of 10
    technology centers (see figure 4). The technology centers are another
    source of technical advice for OST’s customers.



    Page 38                         GAO-04-198 Justice's Office of Science and Technology
                                                                    Appendix V: OST’s Operations




Figure 4: OST’s 10 Technology Centers and the Regions They Serve




                    Northwest
             Anchorage, Alaska

                                                                                                                           Northeast
                                                                                                                           Rome, N.Y.

                                                                                                                           Office of Law
                                                                                                                           Enforcement Standards
                                                                                                                           Gaithersburg, Md.

                                                                                                                           National
                                                                                                                           Rockville, Md.
                 Rocky Mountain
                   Denver, Colo.                                                                                           Office of Law Enforcement
                                                                                                                           Technology Commercialization
                           West                                                                                            Wheeling, W.Va.
               El Segundo, Calif.
                                                                                                                           Rural Law Enforcement
          Border Research and                                                                                              Technology Center
            Technology Center                                                                                              Hazard, Ky.
              San Diego, Calif.
                                                                                                                           Southeast
                                                                                                                           Charleston, S.C.

Source: National Institute of Justice, Department of Justice.




OST’s Technology Centers                                            OST’s network of 10 technology centers provides technical assistance,
                                                                    among other things, to OST’s customers. From fiscal year 1995 to fiscal
                                                                    year 2003 (as of July 2003), funding support for the centers totaled
                                                                    $171.7 million. (See table 8 for funding by center.) The technology centers
                                                                    comprise six regional centers and four specialty centers. While the
                                                                    regional centers assist OST’s customers by region—Northwest, West,
                                                                    Rocky Mountain, Northeast, Southeast, and National—they are expected
                                                                    to coordinate and collaborate among one another regardless of where the
                                                                    resources and capabilities are located. Each of these 6 centers works with
                                                                    a regional advisory council comprising state and local law enforcement,
                                                                    corrections, and public safety representatives.

                                                                    As described below, the four specialty centers provide specialized
                                                                    expertise and services.

                                                                •   The Office of Law Enforcement Standards tests commercially available
                                                                    equipment and develops minimum performance standards for such
                                                                    equipment.


                                                                    Page 39                        GAO-04-198 Justice's Office of Science and Technology
                             Appendix V: OST’s Operations




                         •   The Office of Law Enforcement Technology Commercialization, Inc.,
                             assists inventors and developers, among others, in commercializing
                             technologies.

                         •   The Border Research and Technology Center aids in the development of
                             technologies for agencies concerned with law enforcement at the northern
                             and southern borders.

                         •   The Rural Law Enforcement Technology Center aids rural and small-
                             community law enforcement and corrections agencies.

                             Table 8: Total Funds Awarded for the Operations, Maintenance, and Technical
                             Support of OST’s 10 Technology Centers, Fiscal Years 1995-2003

                              Dollars in millions
                              Regional centers                                                                    Total funding
                                 National, Rockville, Md.                                                                    20.4
                                 Northeast, Rome, N.Y.                                                                       11.7
                                 Southeast, North Charleston, S.C.                                                           23.5
                                 Northwest, Anchorage, Alaska                                                                 2.8
                                 Rocky Mountain, Denver, Colo.                                                               16.2
                                 West, El Segundo, Calif.                                                                    12.7
                              Specialty centers
                                 Border Research Technology Center, San Diego, Calif.                                         8.2
                                 Office of Law Enforcement Standards, Gaithersburg, Md.                                      53.6
                                 Office of Law Enforcement Technology Commercialization, Wheeling,
                                 W.Va.                                                                                       19.6
                                 Rural Law Enforcement Technology Center, Hazard, Ky.                                         3.0
                              Total funding                                                                               $171.7
                             Source: OST.

                             Notes: Figures are based on the current year values of each award. According to OST documents,
                             the first award year for the Office of Law Enforcement Standards in support of OST efforts was 1994.
                             All of the centers had award years of 1995 or later.



OST’s Technical              In addition to forming divisions and technology centers, OST has also
Partnerships for Long-       formed partnerships with governmental, public and private R&D
Term Support                 organizations, agencies, and working groups. According to OST officials,
                             an advantage of these partnerships is that OST can leverage the expertise
                             and resources of already established R&D facilities without having to
                             create their own. These partners have included




                             Page 40                                GAO-04-198 Justice's Office of Science and Technology
    Appendix V: OST’s Operations




•   corporations, such as Georgia Tech Research Corporation and L-3
    Communications, Analytics Corporation;

•   state and local agencies, such as the Houston Police Department and the
    Washington Metropolitan Area Transit Authority;

•   academic institutions, such as the University of Virginia and Syracuse
    University;

•   other federal government agencies, such as the Department of Defense’s
    Army Training and Doctrine Command, and the Department of
    Transportation’s Federal Aviation Administration; and

•   foreign government organizations, such as the Royal Canadian Mounted
    Police, the United Kingdom Police Scientific Development Branch, and the
    government of Israel.

    Each of OST’s technology centers is affiliated with one or more of OST’s
    technical partners. These technical partners are awarded funding in
    exchange for providing staff and facilities to the technology centers. Table
    9 lists OST’s partners and their affiliations, and funding they received to
    support the centers through June of fiscal year 2003.




    Page 41                        GAO-04-198 Justice's Office of Science and Technology
                                            Appendix V: OST’s Operations




Table 9: OST’s Technology Centers, Their Affiliated Partners, and the Amounts Awarded to Support the Centers

 Dollars in millions
                                                                                                                          Amount awarded
 Technology centers                              Affiliated OST partner                                                   to support center
 Regional centers
     National, Rockville, Md.                    Aspen Systems Corporation, Rockville, Md.                                                  20.4
     Northeast, Rome, N.Y.                       Air Force Research Laboratory, U.S. Air Force, Rome, N.Y.                                  11.7
     Southeast, North Charleston, S.C.           South Carolina Research Authority, North Charleston, S.C.                                  21.3
                                                 Space and Naval Warfare Systems Center, U.S. Navy,
                                                 Columbia, S.C.                                                                              0.6
                                                 Oak Ridge National Laboratory, U.S. Department of Energy, Oak
                                                 Ridge, Tenn.                                                                                0.3
                                                 Savannah River Site, Department of Energy, Aiken, S.C.                                      1.3


     Northwest, Anchorage, Alaska                Chenega Technology Services Corporation, and National
                                                 Business Center, U.S. Department of Interior, Anchorage, Alaska                             2.8
     Rocky Mountain, Denver, Colo.               University of Denver - Colorado Seminary, Denver, Colo.                                    16.2
     West, El Segundo, Calif.                    Aerospace Corporation, El Segundo, Calif.                                                  12.7
 Specialty centers
     Border Research Technology Center, San      Aerospace Corporation, El Segundo, Calif.
     Diego, Calif                                                                                                                            1.4
                                                 Space and Naval Warfare Systems Center, U.S. Navy, San
                                                 Diego, Calif.                                                                               1.7
                                                 Sandia National Laboratories, U.S. Department of Energy,
                                                 Albuquerque, N. Mex.                                                                        5.1
                                                 U.S. Attorney’s Office, Southern District of California,
                                                                                                                                               a
                                                 Department of Justice, San Diego, Calif.                                                   0.0
     Office of Law Enforcement Standards,        National Institute of Standards and Technology, U.S. Department
     Gaithersburg, Md.                           of Commerce, Gaithersburg, Md.                                                             53.6
     Office of Law Enforcement Technology        OLETC, Inc., Wheeling, W.Va.
     Commercialization, Wheeling, W.Va.                                                                                                      2.8
                                                 Wheeling Jesuit University, Wheeling, W.Va.                                                14.0
                                                 National Aeronautics and Space Administration                                               2.8
     Rural Law Enforcement Technology Center, Eastern Kentucky University, Hazard, Ky.
     Hazard, Ky.                                                                                                                             3.0
                 b
 Total funding                                                                                                                            $171.7
Source: OST.

                                            Note: Figures are based on the current year values of each award. Award amounts are for the
                                            operations, maintenance and technical support of the centers.
                                            a
                                            Actual amount is $25,000.
                                            b
                                            Total might not add due to rounding.




                                            Page 42                                GAO-04-198 Justice's Office of Science and Technology
                        Appendix V: OST’s Operations




OST Collaborates with   To determine its program priorities, OST collaborates with its many
Many Customers and      customers and partners. Staff at both OST’s Washington, D.C., office and
Partners to Determine   its technology centers are involved in helping OST to set program
                        priorities. The staff report the results of their collaboration through formal
Program Priorities      meetings, periodic reports, and informal communication. Input is
                        exchanged continually between OST’s customers and staff and within its
                        multiple levels of organization. Using their input, the NIJ Director
                        determines OST’s program priorities. (See figure 5 for the stakeholders,
                        partners, and customers that contribute to the setting of OST’s priorities.)




                        Page 43                        GAO-04-198 Justice's Office of Science and Technology
Appendix V: OST’s Operations




Figure 5: Stakeholders and Customers that Contribute to the Setting of OST’s
Priorities




Page 44                         GAO-04-198 Justice's Office of Science and Technology
                                Appendix V: OST’s Operations




OST Collaborates with           OST’s three divisions collaborate with other U.S. government agencies, the
Government Agencies,            research and professional communities, and its technology centers to
Research and Professional       solicit input for setting priorities. Also, the divisions work with public
                                safety practitioners at the state and local levels by, for example, meeting
Communities, and Centers        with grantees and assessing their needs.

                            •   The Investigative and Forensic Sciences Division collaborates with,
                                and receives input from, researchers, academia, and the forensic
                                laboratory community to help set program priorities. It also collaborates
                                with, for example, the FBI and the interagency Technical Support Working
                                Group.

                            •   The Research and Technology Development Division receives input
                                through its collaboration with other federal agencies, such as the FBI,
                                Drug Enforcement Administration, U.S. Secret Service, and White House
                                Office of National Drug Control Policy. The division also participates in
                                interagency working groups, such as for school safety and the Technical
                                Support Working Group. Through these collaborations, OST can develop
                                and share technologies used by both its customers and other agencies. For
                                example, OST works with the Department of Defense to conduct less-than-
                                lethal weapons R&D for law enforcement.

                            •   The Technology Assistance Division is primarily responsible for
                                receiving input from OST’s technology centers. The centers solicit input
                                from customers through their outreach efforts, such as technical
                                assistance, e-mail exchanges, and telephone calls. The centers are also
                                required to use OST’s web-based reporting system to record information
                                on their customers’ requests for technical assistance. The centers are also
                                required to submit monthly reports on their activities and finances.


Advisory Councils and           OST’s technology centers solicit input from the national and regional
Federal, State, and Local       advisory councils that OST created to determine and advocate for the
Public Safety Agencies          particular needs of its customers. Members of the national advisory
                                council are selected by the technology centers and represent federal, state,
Collaborate with OST’s          and local public safety agencies, as well as international criminal justice
Technology Centers              organizations. Among its duties, this national advisory council identifies
                                the present and future equipment and technology needs of OST’s
                                customers and reviews the programs of the technology centers. In
                                addition, the national advisory council recommends (1) ways to improve
                                the technology centers’ programs’ relevance to the needs of the centers’
                                customers and (2) broad priorities for the technology center network and
                                OST that are consistent with the needs of their customers.



                                Page 45                        GAO-04-198 Justice's Office of Science and Technology
Appendix V: OST’s Operations




Each technology center has a regional advisory council. The regional
advisory councils consist of a cross-section of law enforcement and other
public safety officials who represent the interests of state and local
officials. The regional advisory councils solicit input from the state and
local agencies serviced in their regions, advise and support their
respective center directors on their customers’ problems and needs, and
advocate for resource support and improvements required by their
customers. Through this method of sharing information, OST can better
understand the needs of its customers. For example, OST’s regional
councils can represent the unique needs of their customers that the
national advisory council or the technology centers might not be aware of.




Page 46                        GAO-04-198 Justice's Office of Science and Technology
                                               Appendix VI: OST’s Goals in its Fiscal Year
Appendix VI: OST’s Goals in its Fiscal Year    2004 Performance Plan and GAO’s
                                               Assessment


2004 Performance Plan and GAO’s
Assessment
Table 10: OST’s Performance Goals, Initiatives, and Measures for Fiscal Year 2004, and GAO’s Assessment

                                                                                                               Type of measure
                                                   Measures for assessing achievement of
OST’s initiatives      Goals for initiatives       goals                                              Output    Intermediate Outcome
1. Convicted           Reduce DNA backlog       1. Number of labs demonstrating improved                              X
    Offender DNA       and support a                access to external capabilities and
    Backlog            functioning, active          increased lab capabilities.
    Reduction          system, which can solve 2. Number of samples (1) analyzed using the                            X
    Program            old crimes and prevent
                                                   selected DNA markers that are required by
                       new ones from occurring.
                                                   the FBI’s national Combined DNA Index
                                                   System (CODIS) database, and (2) made
                                                   available for CODIS.
                                                   3. Number of states that have experienced an                       X
                                                      increase in the number of samples they
                                                      have contributed to the national database.
2. No Suspect          Reduce DNA backlog       4. Number of DNA samples from cases where                             X
    DNA Backlog        and support a                there is no known suspect.
    Reduction          functioning, active
    Program            system, which can solve
                       old crimes and prevent
                       new ones from occurring.
3. Paul Coverdell      Improve quality,            5. Number of forensic labs with improved                           X
    National           timeliness, and credibility     analytic and technological resources.
    Forensic           of forensic science
    Sciences           services.
    Improvement
    Grants
    Program
4. Critical Incident   Improve the ability of     6. Number of technology demonstrations and             X
    Response           public safety responders,      test indicators that describe the goods and
    Technology         including law                  services produced.
    Initiative         enforcement and            7. Number of prototype technologies                    X
                       corrections officers, to
                                                     developed.
                       deal with critical
                       incidents, save lives, and 8. Number    of guides, standards, and                 X
                       reduce property loss.         assessments in progress.
                                                   9. Number of guides, standards, and                   X
                                                      assessments completed.
                                                   10. Number of technologies introduced in law                       X
                                                       enforcement and corrections agencies.




                                               Page 47                             GAO-04-198 Justice's Office of Science and Technology
                                             Appendix VI: OST’s Goals in its Fiscal Year
                                             2004 Performance Plan and GAO’s
                                             Assessment




                                                                                                              Type of measure
                                                   Measures for assessing achievement of
OST’s initiatives   Goals for initiatives          goals                                             Output    Intermediate Outcome
5. DNA Research     Develop faster and more        11. Number of projects researching new              X
    &               powerful tools and                  forensic DNA markers.
    Development     techniques for the             12. Number of development/validation studies        X
                    analysis of DNA
                                                       for forensic DNA techniques.
                    evidence. These new
                    tools and techniques will      13. Number of computer programs developed           X
                    result in more crimes              for forensic DNA analysis.
                    prevented and solved           14. Number of prototypes and tools for forensic     X
                    and more perpetrators              DNA analysis.
                    brought to justice.
6. Law              Assist in applying             15. Number of technology demonstrations and         X
    Enforcement     technology to reduce the            tests.
    Technology      vulnerability of critical
                                                   16. Number of prototype technologies                X
    Research and    infrastructure; detect
                                                       developed.
    Development     weapons and other
                    contraband; improve            17. Number of guides, standards, and                X
                    technologies to locate             assessments in progress.
                    and differentiate between      18. Number of guides, standards, and                X
                    individuals in structures;         assessments completed.
                    leverage information
                    technology to enhance          19. Number of technologies introduced in law                     X
                    the responder                      enforcement and corrections agencies.
                    community’s ability to
                    anticipate and deal with
                    critical incidents; identify
                    and respond to terrorist
                    attacks involving
                    chemical, biological, and
                    other unconventional
                    weapons; and develop
                    needed standards.a
7. School Safety    Assist school              20. Number of technology demonstrations.                X
    Technology      administrators and law
                                               21. Number of conferences and forums.                   X
                    enforcement in creating a
                    safer and more             22. Number of school safety technology                  X
                    productive learning            products.
                    environment. Safe,
                    effective, appropriate,
                    and affordable
                    technologies can affect
                    the perception and reality
                    of safe schools.




                                             Page 48                             GAO-04-198 Justice's Office of Science and Technology
                                            Appendix VI: OST’s Goals in its Fiscal Year
                                            2004 Performance Plan and GAO’s
                                            Assessment




                                                                                                              Type of measure
                                                  Measures for assessing achievement of
OST’s initiatives   Goals for initiatives         goals                                              Output    Intermediate Outcome
8. Crime Lab        Provide immediate             23. Number of crime labs receiving specialized       X
    Improvement     results in solving more            forensic services.
    Program         crimes, bringing to justice
                                                  24. Number of capacity-building forensic R&D         X
                    more criminals, and
                                                      and validation projects funded.
                    improving administration
                    of justice through the        25. Number of forensic technology training tools     X
                    presentation of strong,           developed and distributed.
                    reliable forensic evidence    26. Number of labs providing continuing              X
                    at trial.                         education or advanced training to crime
                                                      analysts.
                                                  27. Number of crime labs with increased                           X
                                                      capacity for implementation of new forensic
                                                      capabilities (including DNA analysis).
                                                  28. Number of capacity-building forensic R&D                      X
                                                      and validation projects completed and
                                                      impacting crime labs.
                                                  29. Number of labs establishing new forensic                      X
                                                      capabilities.
                                                  30. Number of labs expanding current forensic                     X
                                                      capabilities.
                                                  31. Number of labs experiencing a reduction in                    X
                                                      time needed for evidence analysis.
                                                  32. Number of labs experiencing a reduction in                    X
                                                      backlogged evidentiary sample analysis.
9. Office for Law   Help the public safety   33. Number of methods for examining                       X
    Enforcement     community make                evidentiary materials developed.
    Standards       informed decisions about 34. Number of standards for equipment and
                                                                                                       X
                    products being marketed      operating procedures developed.
                    for public safety
                    personnel.               35. Law  enforcement technology deliverables              X
                                                 (standards, product performance
                                                 evaluations, product guides).
10. Smart Gun       Develop a firearm that     36. Successful demonstration of prototype               X
                    could save the lives of         recognition system for smart gun.
                    law enforcement officers 37. Failure mode analysis for prototype                   X
                    and members of the             recognition system for smart gun.
                    public that they encounter
                    while performing their     38. Incorporation and demonstration of                  X
                    duties.                        recognition system into firearm (where
                                                   applicable).
                                                  39. Identification of appropriate biometric          X
                                                      solutions for recognition system (where
                                                      applicable).




                                            Page 49                             GAO-04-198 Justice's Office of Science and Technology
                                              Appendix VI: OST’s Goals in its Fiscal Year
                                              2004 Performance Plan and GAO’s
                                              Assessment




                                                                                                                      Type of measure
                                                   Measures for assessing achievement of
 OST’s initiatives    Goals for initiatives        goals                                                     Output     Intermediate Outcome
 11. OST’s network    Help state and local law  40. Number of technology information                            X
      of technology   enforcement, corrections,      documents distributed.
      centers         and public safety
                                                41. Number of practitioners trained through the                 X
      (known as the   personnel do their jobs
                                                    Crime Mapping Program.
      National Law    more safely and
      Enforcement     efficiently, thereby      42. Savings to criminal justice agencies through                               X
      and             leading to greater            the DOD’s Section 1033 Military Surplus
      Corrections     administrative                Program. Section 1033 of the National
      Technology      efficiencies, more crimes     Defense Authorization Act for Fiscal Year
      Center          solved, and more lives        1997b authorizes DOD to transfer excess
      system)         saved.                        military property to federal and state
                                                    agencies to support law enforcement
                                                    activities including counterdrug and
                                                    counterterrorism activities.
Source: OST.
                                              a
                                               Because the goal for this initiative was not outcome-oriented according to our criteria, we used the
                                              initiative’s mission statement as the goal.
                                              b
                                              P.L. 104-201, 110 Stat. 2422 (1996).




                                              Page 50                                 GAO-04-198 Justice's Office of Science and Technology
             Appendix VII: Comments from the
Appendix VII: Comments from the
             Department of Justice



Department of Justice




             Page 51                           GAO-04-198 Justice's Office of Science and Technology
Appendix VII: Comments from the
Department of Justice




Page 52                           GAO-04-198 Justice's Office of Science and Technology
Appendix VII: Comments from the
Department of Justice




Page 53                           GAO-04-198 Justice's Office of Science and Technology
                  Appendix VIII: GAO Contacts and Staff
Appendix VIII: GAO Contacts and Staff
                  Acknowledgments



Acknowledgments

                  Laurie Ekstrand (202) 512-8777
GAO Contacts      Weldon McPhail (202) 512-8644


                  In addition to those named above, the following individuals contributed to
Staff             this report: Samuel L. Hinojosa, Debra L. Picozzi, Katherine M. Davis,
Acknowledgments   Richard Hung, Geoffrey R. Hamilton, Denise M. Fantone, Kristeen McLain,
                  Elizabeth H. Curda, Rebecka Derr, Thomas M. Beall, and Leo M. Barbour.




(440165)
                  Page 54                           GAO-04-198 Justice's Office of Science and Technology
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to e-mail
                         alerts” under the “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548