oversight

Information Management: National Technical Information Service's Dissemination of Technical Reports Needs Congressional Attention

Published by the Government Accountability Office on 2012-11-19.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States Government Accountability Office

GAO             Report to Congressional Committees




                INFORMATION
November 2012



                MANAGEMENT

                National Technical
                Information Service’s
                Dissemination of
                Technical Reports
                Needs Congressional
                Attention




GAO-13-99
                                                 November 2012

                                                 INFORMATION MANAGEMENT
                                                 National Technical Information Service’s
                                                 Dissemination of Technical Reports Needs
                                                 Congressional Attention
Highlights of GAO-13-99, a report to
congressional committees




Why GAO Did This Study                           What GAO Found
NTIS was established by statute in               As a component of the Department of Commerce, the National Technical
1950 to collect scientific and technical         Information Service (NTIS) is organized into five primary offices that offer the
research reports, maintain a                     public and federal agencies a variety of products and services. As of late October
bibliographic record and repository of           2012, NTIS was supported by 181 staff, all except 6 of which held full-time
these reports, and disseminate them to           positions. NTIS reports its progress toward agency goals to the Deputy Secretary
the public. NTIS charges fees for its            of Commerce, and the Director of NTIS reports to the Director of Commerce’s
products and services and is required            National Institute of Standards and Technology. In addition, NTIS receives
by law to be financially self-sustaining         oversight of its functions and strategic direction from an advisory board with
to the fullest extent possible.                  members appointed by the Secretary of Commerce. NTIS’s product and service
GAO was mandated by Congress to                  offerings include, among other things, subscription access to reports contained in
update its 2001 report on aspects of             its repository in both print and electronic formats, distribution of print-based
NTIS’s operations and the reports in its         informational materials to federal agencies’ constituents, and digitization and
collection. Specifically, GAO’s                  scanning services.
objectives were to determine (1) how
NTIS is currently organized and                  NTIS revenues are generated exclusively from direct sales or subscriptions for its
operates, including its functions,               products and services. NTIS reported that net revenues from all its functions
current staffing level, reported cost of         (products and services) totaled about $1.5 million in fiscal year 2011. However,
operations, and revenue sources; (2)             over most of the last 11 years, its costs have exceeded revenues by an average
the age of and demand trends for                 of about $1.3 million for its products. While NTIS has not recovered all of its costs
reports added to NTIS’s repository;              for products through subscriptions and other fees, it has been able to remain
and (3) the extent to which these                financially self-sustaining because of revenues generated from its services such
reports are readily available from other         as distribution and order fulfillment, web hosting, and e-training. The NTIS
public sources. To do this, GAO                  strategic plan states that the electronic dissemination of government technical
reviewed agency documentation,                   information by other federal agencies has contributed to reduced demand for
analyzed a sample of reports added to            NTIS’s products. As a result, the agency is taking steps to reduce its net costs,
NTIS’s collection from fiscal years              such as improving business processes and increasing the breadth and depth of
1990 through 2011 (reports from the              its collection.
period since GAO’s last study and
other older reports), and interviewed            NTIS’s repository has been growing with mostly older reports, but the demand for
relevant agency officials.                       more recent reports is greater. Specifically, NTIS added approximately 841,500
                                                 reports to its repository during fiscal years 1990 through 2011, and approximately
What GAO Recommends                              62 percent of these had publication dates of 2000 or earlier. However, the
GAO is suggesting that Congress
                                                 agency was more likely to distribute (by direct sale or through a subscription)
                                                 reports published more recently. For example, GAO estimated that 100 percent
reassess the appropriateness and
                                                 of the reports published from 2009 through 2011 had been distributed at least
viability of the fee-based model under
which NTIS currently operates for
                                                 once, while only about 21 percent of reports published more than 20 years ago
                                                 had been.
disseminating technical information to
determine whether the use of this                Of the reports added to NTIS’s repository during fiscal years 1990 through 2011,
model should be continued. In                    GAO estimates that approximately 74 percent were readily available from other
comments on a draft of this report, the          public sources. These reports were often available either from the issuing
Department of Commerce stated that               organization’s website, the federal Internet portal (http://www.USA.gov), or from
NTIS believes GAO’s conclusions do               another source located through a web search. Reports published from 1990 to
not fully reflect the value that the             2011 were more likely to be readily available elsewhere than those published in
agency provides. However, GAO                    1989 or earlier. Further, GAO estimated that 95 percent of the reports available
maintains that its conclusions and               from sources other than NTIS were available free of charge. NTIS’s declining
suggestion to Congress are warranted.            revenue associated with its basic statutory function and the charging for
                                                 information that is often freely available elsewhere suggests that the fee-based
View GAO-13-99. For more information,            model under which NTIS currently operates for disseminating technical
contact Valerie C. Melvin at (202) 512-6304 or   information may no longer be viable and appropriate.
melvinv@gao.gov

                                                                                          United States Government Accountability Office
Contents


Letter                                                                                         1
               Background                                                                       4
               NTIS Carries out a Variety of Functions Related to Its Mission, but
                 Currently Is Financially Sustained by Its Service Offerings                    7
               Additions to NTIS’s Repository Have Mostly Included Older
                 Reports, but Demand for More Recent Reports Is Greater                       21
               Most Reports Added Since 1990 Are Freely Available from Other
                 Websites                                                                     24
               Conclusions                                                                    29
               Matter for Congressional Consideration                                         29
               Agency Comments and Our Evaluation                                             29

Appendix I     Objectives, Scope, and Methodology                                             32



Appendix II    Comments from the Department of Commerce                                       36



Appendix III   GAO Contact and Staff Acknowledgments                                          41



Tables
               Table 1: NTIS Staff as of October 31, 2012                                     10
               Table 2: NTIS Office Responsibilities and Functions                            10
               Table 3: Reported Net Earned Revenue (or Net Cost) for NTIS
                        Products and Services, Fiscal Years 2001-2011                         18


Figures
               Figure 1: NTIS Organization Chart                                                8
               Figure 2: Fiscal Years 2001-2011 Net Earned Revenues and Net
                        Costs for NTIS Products and Services                                  19
               Figure 3: Year of Publication for Documents Added to NTIS
                        Repository, 1990 through 2011                                         22
               Figure 4: Estimated Percentage of Reports Distributed and Not
                        Distributed by Publication Date during Fiscal Years 2000
                        through 2011                                                          23
               Figure 5: Estimated Availability of Reports by Year of Publication             25



               Page i                  GAO-13-99 Commerce's National Technical Information Service
Figure 6: Estimated Availability of Reports Distributed and Not
         Distributed During Fiscal Years 2000 through 2011                                26
Figure 7: Of Reports Found Elsewhere, Estimated Percent
         Available for Free and Not Free, by Year of Publication                          28




Abbreviations

Commerce          Department of Commerce
NASA              National Aeronautics and Space Administration
NIST              National Institute of Standards and Technology
NTIS              National Technical Information Service
SSA               Social Security Administration




This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page ii                      GAO-13-99 Commerce's National Technical Information Service
United States Government Accountability Office
Washington, DC 20548




                                   November 19, 2012

                                   The Honorable Barbara Mikulski
                                   Chairwoman
                                   The Honorable Kay Bailey Hutchison
                                   Ranking Member
                                   Subcommittee on Commerce, Justice,
                                     Science, and Related Agencies
                                   Committee on Appropriations
                                   United States Senate

                                   The Honorable Frank Wolf
                                   Chairman
                                   The Honorable Chaka Fattah
                                   Ranking Member
                                   Subcommittee on Commerce, Justice,
                                     Science, and Related Agencies
                                   Committee on Appropriations
                                   House of Representatives

                                   The Department of Commerce’s National Technical Information Service
                                   (NTIS) serves as a permanent repository and disseminator of scientific,
                                   technical, engineering, and business-related information. 1 NTIS acquires
                                   such information—largely in the form of research reports—primarily from
                                   federal agencies and their contractors and grantees, as well as from other
                                   domestic and foreign sources. NTIS estimates that it maintains in its
                                   central repository more than 2.5 million records 2 covering 378 technical
                                   and business-related subject areas. In addition, NTIS performs various
                                   fee-based information services for other federal agencies. The legislation




                                   1
                                    In the remainder of this report, we use the term “technical information” to refer collectively
                                   to these types of information.
                                   2
                                    These 2.5 million records contain, among other things, the metadata that identify all of
                                   the reports held by NTIS, such as the report’s author, publication date, and the agency
                                   where the report originated. These records link to the actual reports, which NTIS
                                   maintains elsewhere in hardcopy or as electronic documents or microfiche. According to
                                   the agency, these records represent all reports that have been catalogued in the
                                   electronic repository since 1964. As part of our further discussion in this report, we use the
                                   term “reports” in referring to the repository holdings.




                                   Page 1                        GAO-13-99 Commerce's National Technical Information Service
that established NTIS 3 requires all of the services and functions it
provides to be financially self-sustaining, to the extent feasible.

In May 2001, we reported on NTIS’s operations, 4 noting among other
things, the availability of many of the reports maintained in its repository
from other sources, such as the originating agencies’ websites. House
Report 112-169, accompanying the Consolidated and Further Continuing
Appropriations Act, 2012, 5 directed GAO to update aspects of its prior
study. Our specific objectives were to determine (1) how NTIS is currently
organized and operates, including its various functions, current staffing
level, reported costs of operations, and revenue sources; (2) the age of
and demand trends for reports added to NTIS’s repository; and (3) the
extent to which these reports are readily available from other public
sources.

To determine how NTIS is organized and operates, we reviewed the
agency’s strategic plan and documentation on its organizational and
reporting structure, office staffing level and assigned responsibilities, and
types of products and services offered. We also reviewed cost data
contained in the agency’s financial reports. 6 In addition, we analyzed
relevant laws and regulations describing NTIS’s authority and
responsibilities, 7 and reviewed our previous reports that discussed its
mission and operations. 8 We supplemented our analyses with interviews
of the Director of NTIS and other relevant agency officials; we also
interviewed officials of the Department of Commerce and its National
Institute of Standards and Technology (NIST), which have specific
reporting relationships with NTIS.




3
15 U.S.C. 1153.
4
 GAO, Information Management: Dissemination of Technical Reports, GAO-01-490
(Washington, D.C.: May 18, 2001).
5
Pub. L. No. 112-55 (Nov. 18, 2011).
6
 The cost data reflected in our report were taken from documentation provided by NTIS.
However, we did not test the reliability of NTIS’s cost accounting system, from which the
agency derives its cost data.
7
For example, 15 U.S.C.1151-57, 15 U.S.C. 3704b-1 & 2, and 15 C.F.R. Part 1180.
8
  GAO, Information Policy: NTIS’ Financial Position Provides an Opportunity to Reassess
Its Mission, GAO/GGD-00-147 (Washington, D.C.: June 30, 2000) and GAO-01-490.




Page 2                       GAO-13-99 Commerce's National Technical Information Service
To determine the age of and demand trends for reports added to NTIS’s
repository, we first analyzed the distribution of the publication dates for all
reports added to the repository since 1990. We chose to include reports
added since 1990 to determine the current availability of older documents
in light of NTIS’s role as a permanent repository, and to focus on the
reports added since our previous review in 2001. 9 We subsequently
selected a stratified random sample from this body of reports 10 and then
used cumulative data provided by NTIS on the sales of any of these
reports to analyze the demand for reports in the repository.

To determine the extent to which reports NTIS collects are readily
available from other public sources, we searched the Internet to
determine if each of the reports included in our sample could be found
elsewhere and at no cost. Using a tiered approach, we searched the
following four sources in the order shown: (1) the issuing organization’s
website; (2) the U.S. Government Printing Office’s Federal Digital System
website—http://www.gpo.gov/fdsys; (3) the federal government Internet
portal USA.gov—http://www.USA.gov; and (4) a web search conducted
using the commercial search engine http://www.Google.com. Specifically,
we determined whether each report was first available at no cost on the
issuing organization’s website and, if so, concluded the Internet search at
this point. However, if the report was not available, then the search
continued to the second source, and so on, until either the report was
found to be available at one of the remaining sources, or all sources were
exhausted. We then used our results to estimate the percentage of the
total population of NTIS reports added to the repository during fiscal years
1990 through 2011 that was available from other public sources.

All of the results derived from the sample analyses constituted estimates
that are subject to sampling errors. These sampling errors measure the
extent to which the sample size and structure are likely to differ from the
population they represent. Because we followed a probability procedure
based on random selections, our sample is only one of a large number of
samples that we might have drawn. Since each sample could have
provided different estimates, we express our confidence in the precision
of our particular sample’s results as a 95 percent confidence interval. This


9
 We did not include reports added for fiscal year 2012 because our study only focused on
those fiscal years that had been completed at the time that our study was initiated.
10
 Our selected sample consisted of 384 reports.




Page 3                      GAO-13-99 Commerce's National Technical Information Service
             is the interval that would contain the actual population value for 95
             percent of the samples we could have drawn.

             A more detailed discussion of our objectives, scope, and methodology is
             provided in appendix I.

             We conducted this performance audit from February 2012 to November
             2012 in accordance with generally accepted government auditing
             standards. Those standards require that we plan and perform the audit to
             obtain sufficient, appropriate evidence to provide a reasonable basis for
             our findings and conclusions based on our audit objectives. We believe
             that the evidence obtained provides a reasonable basis for our findings
             and conclusions based on our audit objectives.


             NTIS’s basic statutory function is to collect research reports, maintain a
Background   bibliographic record and permanent repository of these reports, and
             disseminate them to the public. In addition, NTIS has developed a variety
             of information-related services. NTIS charges user fees for the sale of its
             products to the public and services to federal agencies.

             Under statutory authority enacted in 1950, 11 NTIS collects reports
             containing scientific, technical, and engineering information from both
             domestic and foreign sources in a repository and makes the information
             available to (1) business and industry, (2) state and local governments,
             (3) other federal agencies, and (4) the general public to increase U.S.
             competitiveness in the global economy. The statute does not define
             scientific, technical, and engineering information. However, the Secretary
             of Commerce has interpreted this to include “all types of information
             which have a more or less direct bearing on business and industry
             generally.” 12 The Secretary of Commerce described such information as
             including “economic information, market information, and related
             information so long as it is reasonably specific and bears some direct
             relationship to the organization and operation of industrial or business
             enterprise.” NTIS’s enabling legislation authorized it to charge fees for its




             11
              15 U.S.C. 1152.
             12
               In 1954, the Comptroller General found this to be a reasonable interpretation of the law’s
             use of the word “technical.” 34 Comp. Gen. 58 (1954).




             Page 4                       GAO-13-99 Commerce's National Technical Information Service
products and established a policy to recover all costs, as feasible, through
the fees. 13

NTIS’s authority was revised in the National Technical Information Act of
1988. 14 This act gave the agency authority to also enter into contracts,
cooperative agreements, joint ventures, and other transactions as
necessary in the conduct of the business of NTIS, and declared the NTIS
repository to be a permanent federal function that could not be transferred
to the private sector without congressional approval. The act was
subsequently amended by the American Technology Preeminence Act of
1991, 15 which

•     required all costs associated with acquisition, processing, storage,
      bibliographic control, and archiving to be recovered primarily by fees;

•     required agencies to transfer unclassified scientific, technical, and
      engineering information which results from federally funded research
      and development to NTIS; and

•     provided that NTIS’s use of new methods or media for information
      dissemination should include producing and disseminating information
      in electronic format.

Further, the Commerce, Justice, and State, the Judiciary, and Related
Agencies Appropriations Act of 1993 16 established a revolving fund for the
payment of all expenses incurred by NTIS and gave it the authority to use
that fund without further appropriations action by Congress.

Our prior work found that, from fiscal years 1988 through 1994, NTIS
showed excess revenue from the sale of its products and services in all
years except one and generated cumulative net earned revenues of $5.8
million. However, over the next 5-year period (from fiscal year 1995
through fiscal year 1999), NTIS had a cumulative net cost 17 of about $4.2


13
    15 U.S.C. 1153.
14
    Pub. L. No. 100-519 (Oct. 24, 1988); 15 U.S.C. 3704b.
15
    Pub. L. No. 102-245 (Feb. 14, 1992); 15 U.S.C. 3704b-1.
16
    Pub. L. No. 102-395 (Oct. 6, 1992).
17
  In the remainder of this report, we use the term “net costs” to refer to financial losses or
gross costs that exceeded revenues.




Page 5                         GAO-13-99 Commerce's National Technical Information Service
million. Accordingly, in August 1999, the Secretary of Commerce
proposed closing NTIS by September 30, 2000, because he believed that
declining sales revenues would not continue to be sufficient to recover all
of the agency’s operating costs. The Secretary attributed this decline
partly to other agencies’ practice of making their research results
available to the public for free through the Web. He also proposed
transferring NTIS’s research report archives to the Library of Congress
and requiring federal agencies to give the public free online access to
new research reports.

In June 2000, 18 we reported that NTIS could possibly remain solvent for a
few years because Commerce and NTIS officials had acted in fiscal year
1999 to reduce costs when it appeared there would not be sufficient funds
to operate. However, we noted that the agency’s solvency beyond fiscal
year 2003 was in doubt based on our forecast of its revenues and costs,
which suggested that costs would exceed revenues beginning in fiscal
year 2001 and continue. We pointed out that, if these net costs were
realized, cumulative net earned revenues from previous years’ operations
could be absorbed and the agency could run out of operating funds after
fiscal year 2003. Based on these findings, we suggested that Congress
consider whether NTIS (or another agency) was needed to serve as a
permanent repository and disseminator for research reports; we also
suggested that Congress could be aided in this assessment by reviewing
NTIS annual financial and operational reports that Commerce was
required to submit, but which it had not done since fiscal year 1995.

We again reported on NTIS’s operations in May 2001. 19 We noted that,
according to its Deputy Director, the agency had significantly reduced its
costs after our June 2000 report and had shown net earned revenues of
$2.2 million for fiscal year 2000. Moreover, the Deputy Director had
projected a net earned revenue of $1.5 million for fiscal year 2001. We
also noted that the agency was providing a variety of other fee-based
services for agencies and that, while demand for electronic products was
on the rise, research reports and other scientific, technical, and
engineering information maintained by NTIS were also becoming
increasingly available on agency websites and through other public
sources—often at no cost. We included as a matter for congressional


18
 GAO/GGD-00-147.
19
 GAO-01-490.




Page 6                  GAO-13-99 Commerce's National Technical Information Service
                          consideration that Congress look at how this information was defined;
                          whether there was a need for a central repository of this information; and
                          if a central repository was maintained, whether all information should be
                          retained permanently, and what business model should be used to
                          manage it.

                          The Secretary of Commerce agreed with our assessment and raised as a
                          primary question whether there was a need for a central repository in
                          view of the increasing availability of newer publications from sources
                          other than NTIS. The Secretary also noted that the need for a central
                          repository depended on whether the information would be permanently
                          maintained by agencies and whether the information would be easy to
                          locate without the kind of bibliographic control that NTIS provides.

                          Subsequent to the issuance of our reports, in December 2003, Congress
                          passed the 21st Century Nanotechnology Research and Development
                          Act, which provided a coordinated federal approach to stimulating
                          nanotechnology research and development. The act directed the
                          Secretary of Commerce to establish a clearinghouse for information
                          related to the commercialization of nanotechnology research using the
                          resources of NTIS to the extent possible. As of September 2012, NTIS
                          noted that it held over 700 publications in its nanotechnology collection.


                          NTIS currently operates as one of 12 independent bureaus within
NTIS Carries out a        Commerce, with the mission to help promote the nation’s economic
Variety of Functions      growth by providing access to information that stimulates innovation and
                          discovery. Headquartered in Alexandria, Virginia (with warehouses in
Related to Its Mission,   Alexandria, Virginia and in Brandywine, Maryland), the agency is
but Currently Is          organized into five primary offices. To carry out its statutory functions of
                          collecting and maintaining a permanent repository and bibliographic
Financially Sustained     record of research reports, and to disseminate them, the agency offers a
by Its Service            variety of products, such as fee-based access to the reports in its
                          repository. In addition, NTIS offers information-related services to federal
Offerings                 agencies, such as distribution and order fulfillment, web hosting, and e-
                          training, that are less directly related to its basic statutory function. While
                          NTIS’s service offerings have resulted in increased revenues, allowing
                          the agency to remain financially self-sustaining, it has experienced a net
                          cost relative to its products, calling into question whether the agency’s
                          function of acting as a self-financing repository of technical information is
                          still viable.




                          Page 7                    GAO-13-99 Commerce's National Technical Information Service
NTIS Receives Guidance         NTIS is led by a director, who is aided by two executives—a chief
and Oversight from             information officer and a chief financial officer. In addition, three
Commerce Department            operational offices have a variety of responsibilities for providing products
                               and services that include collecting and disseminating technical reports,
Entities and Carries out Its   offering access to other information sources, and providing information-
Functions through Five         related services to federal agencies. Figure 1 displays the NTIS
Primary Offices                organization.

                               Figure 1: NTIS Organization Chart




                               NTIS operates as a unit within Commerce and receives oversight from
                               the Deputy Secretary of Commerce, the Director of NIST, 20 and an
                               advisory board. In this regard, the NTIS Director communicates progress
                               toward agency goals to the Deputy Secretary of Commerce. For example,
                               the Director participates in biweekly executive management meetings that
                               are held with the Deputy Secretary. At these meetings, executive leads
                               from each of Commerce’s components report on the status of
                               performance and strategic goals within their offices, among other things.




                               20
                                 NIST is a Commerce operating unit that, among other things, promotes U.S. innovation
                               and industrial competitiveness by advancing measurements for science, standards, and
                               technology.




                               Page 8                      GAO-13-99 Commerce's National Technical Information Service
In addition, under a Commerce departmental order, 21 the NTIS Director
also reports to the Director of NIST. As outlined in the Commerce
organization chart and stated in the Commerce departmental organization
order, the NTIS Director is to report and be responsible to the Director of
NIST. In turn, the Director of NIST prepares the NTIS Director’s annual
performance appraisal. Beyond this, the Director of NIST stated that the
agency does not provide any other operational or financial oversight
functions for NTIS. For example, NIST does not approve NTIS’s budget
(although it does coordinate NTIS’s budget for final inclusion in the
department’s overall budget, which is approved by Commerce).

Further, NTIS receives guidance on its operations from the NTIS Advisory
Board, which was established by law 22 in 1988 to review the general
policies and operations of NTIS, including policies related to fees and
charges for its products and services. The board, 23 comprised of a
chairperson and four members appointed by the Secretary of Commerce,
is required to meet at least every 6 months to discuss NTIS’s activities.
The board’s last meeting took place in late October 2012, and, according
to the notice of open meetings from the Federal Register, the intended
focus was on the agency’s strategic business plan. The board submits an
annual report to the Secretary of Commerce, which includes strategic and
tactical recommendations regarding NTIS’s future operations.

As of late October 2012, NTIS was supported by 181 staff, all except 6 of
which held full-time positions. These included 103 NTIS employees and
78 contractors. Table 1 shows the number of staff dedicated to the Office
of the Director and each of the agency’s other primary offices as of
October 2012.




21
 Department of Commerce Departmental Organization Order 30-7A (Washington, D.C.:
November 2008).
22
 15 U.S.C. 3704b(c).
23
  NTIS Advisory Board members can include individuals who are both internal and
external to the Department of Commerce.




Page 9                      GAO-13-99 Commerce's National Technical Information Service
                                            Table 1: NTIS Staff as of October 31, 2012

                                                                          Full-time        Part-time  Full-time   Part-time
                                             NTIS office                 employees        employees contractors contractors            Total
                                             Director                                2               0              0              0        2
                                             Chief Financial Officer               13                0              0              0        13
                                             Chief Information
                                             Officer                               26                0             26              5        57
                                             Federal Services                      15                0              3              0        18
                                             Product Management
                                             and Acquisition                       29                1              5              0        35
                                             Production Services                   17                0             39              0        56
                                             Total staff                          102                1             73              5       181
                                            Source: NTIS data.


                                            Among these offices, executive functions, such as directing NTIS
                                            activities and developing NTIS budgets and policies for the use of
                                            information technology, are carried out by the Office of the Director, along
                                            with the offices of the Chief Financial Officer and the Chief Information
                                            Officer. The three other operational offices have a variety of
                                            responsibilities for providing NTIS’s products and services. Table 2
                                            summarizes the specific responsibilities and functions of these offices.

Table 2: NTIS Office Responsibilities and Functions

Office                      Key responsibilities
Office of the Director      Manages and is responsible for all activities of NTIS. In addition to the Director, officials in this office
                            include a contracting officer, legal personnel, and program analysts.
Office of the Chief Financial Develops NTIS budgets, financial plans, financial reports, and analyses.
Officer                       Develops, maintains, and operates NTIS financial systems.
                              Provides financial advice to the Director of NTIS.
                              Provides management analysis, administrative management, procurement, facilities management,
                              property management, and office support services to NTIS components and other agencies.
                              Provides financial management services to NTIS components, customers, and source clients.
Office of the Chief         Develops and administers plans, policies, and standards for NTIS’s acquisition and use of information
Information Officer         technology.
                            Represents NTIS on information technology matters, including computer security issues.
                            Plans, develops, operates, and services centralized and decentralized network computer-based
                            capabilities and associated telecommunications networks.
                            Provides training, consultation, technical documentation, and operation of data processing and
                            telecommunications.




                                            Page 10                         GAO-13-99 Commerce's National Technical Information Service
Office                       Key responsibilities
Office of Federal Services   Develops strategies for information dissemination services and Internet-based service business
                             opportunities for federal agencies.
                             Manages service projects using in-house capabilities or through joint venture partners.
                             Assists agencies in meeting their multimedia product requirements.
                             Coordinates marketing and exhibit activities for information dissemination and Internet-based services.
Office of Product            Builds the NTIS bibliographic database by cataloging, indexing, and abstracting information products
Management and               from either hard copy or electronic media.
Acquisition                  Identifies and acquires scientific, technical, and business-related information from domestic and
                             international sources.
                             Coordinates the production and sale of clearinghouse products.
                             Manages customer relationships involving order receipt/entry, inquiry and complaint resolution, and
                             product identification functions.
                             Gathers data about customers’ or potential customers’ needs and expectations.
                             Promotes available federal government information, services, and programs to customers through
                             demand and subscription sales.
Office of Production         Provides services relative to order fulfillment, information product dissemination and service delivery,
Services                     order processing, product manufacturing, warehousing, specialized storage, inventory management, and
                             distribution.
                             Operates, maintains, and manages specialized equipment and computer-based systems for the imaging,
                             publication, and reproduction of information products.
                             Provides project- and product-management support for individual products and services.
                                            Source: GAO analysis of Commerce data.




NTIS Offers Technical                       As part of its basic statutory function to collect and disseminate technical
Reports and Other                           reports, NTIS offers a variety of fee-based products. According to the
Products to a Variety of                    agency, its customer base for these products include scientists,
                                            engineers, the business community, librarians, information specialists in
Customers                                   government, academia, and the general public. The agency has
                                            organized its products into three lines of business—”Technical Reports,”
                                            “Clearinghouse,” and “Publishing”—as described below.

Technical Reports                           To carry out its statutory function of collecting and disseminating such
                                            information, NTIS maintains a searchable repository that contains
                                            bibliographic records for the over 2.5 million scientific, technical,
                                            engineering, and business research reports it has acquired from federal
                                            government agencies, state and local governments, international
                                            organizations, and private-sector organizations. NTIS’s bibliographic
                                            database provides for the classification and cataloging of the records in
                                            this database. For example, according to the NTIS Database Search
                                            Guide, NTIS classifies its records into 39 categories. These categories
                                            can be used for searching the contents of NTIS’s database. The database
                                            covers a host of scientific and technical subjects, such as biology,



                                            Page 11                                  GAO-13-99 Commerce's National Technical Information Service
chemistry, physics, transportation, health care, and the environment. Of
the 2.5 million reports, NTIS noted that approximately 700,000 reports
have been digitized, with the remainder in physical form, such as paper or
microfilm.

Access to the reports is provided both through the direct sale of individual
reports and by subscription. Individual reports can be purchased via
postal mail; phone; e-mail; and NTIS’s online ordering system, accessed
through its website, http://www.ntis.gov. Subscription-based access to the
reports is obtained through, among other things, NTIS’s National
Technical Reports Library, which provides subscribers with the ability to
search the bibliographic records repository and to access the
approximately 700,000 digitized reports in portable document format
(PDF). 24 In addition, NTIS’s Selected Research Service allows a
subscriber to select from more than 378 subcategories and automatically
receive reports tailored to that area of interest. According to NTIS, reports
that have not been digitized can be provided in a digital format when a
customer purchases a copy.

Prices for individual reports and subscriptions vary. For example, an
electronic copy of a report from the Economics and Statistics
Administration, Benefits of Manufacturing Jobs: Executive Summary, can
be purchased for $15; a “customized CD” for this report can be purchased
for $30. Further, an electronic copy of a report from the National
Aeronautics and Space Administration (NASA) Marshall Space Flight
Center, NASA Robotics for Space Exploration, can be purchased for $15;
a “customized CD” for this report can be purchased for $30. With regard
to subscriptions, access to the National Technical Reports Library is sold
as an annual subscription to institutions based on the number of
individuals accessing the library. For example, as of October 2010, an
annual subscription providing access for up to 3,000 individuals costs
$2,100, while an annual subscription providing access for 18,001 to
28,000 individuals costs $11,200. In addition, subscriptions can be
purchased for a specific number of issues of a particular document type.
For example, six issues of the Livestock, Dairy, and Poultry Outlook can
be purchased for $91.


24
  The Merriam-Webster Dictionary defines PDF as “a computer file format for the
transmission of a multimedia document that is not intended to be edited further and
appears unaltered in most computer environments.” Merriam-Webster Online, accessed
October 5, 2012, http://www.merriam-webster.com/dictionary/pdf.




Page 12                    GAO-13-99 Commerce's National Technical Information Service
Clearinghouse   In addition to the technical reports that it collects and disseminates, NTIS
                disseminates publications covering a wide array of topics on behalf of
                other federal agencies. According to NTIS, these agencies request that
                NTIS distribute the publications in print or electronically. The following are
                examples of the federal products distributed:

                •   Standard Occupational Classification Manual. A manual containing
                    information on all occupations in the national economy classified
                    according to the system used by federal statistical agencies for the
                    purpose of collecting, calculating, analyzing, or disseminating
                    occupational data.

                •   Food and Drug Administration Food Code Manual. A code and
                    reference document that provides technical and legal information
                    about the regulations of the retail and food service industry.

                •   North American Industry Classification System. A publication that
                    details a system for the collection, analysis, and dissemination of
                    industrial statistics used by government policy analysts, academics,
                    researchers, the business community, and the public.

                •   Export Administration Regulations. A compilation of regulations issued
                    by Commerce’s Bureau of Industry and Security relating to the control
                    of certain exports, re-exports, and activities.

                •   National Correct Coding Policy Manual. A manual developed by the
                    Centers for Medicare and Medicaid Services to control improper
                    coding leading to inappropriate payment in Medicare Part B claims.
                    The manual provides guidance for providers on the correct coding of
                    claims being submitted for reimbursement.

Publishing      Through a memorandum of understanding or interagency agreement,
                NTIS also provides access to information collected from federal agencies,
                which it refers to as its “Publishing” line of business. In some instances,
                NTIS repackages the information with additional features. According to
                NTIS, agencies initiate the request for these services. These offerings
                include the following:

                •   Drug Enforcement Administration database. The Drug Enforcement
                    Administration database identifies persons and organizations
                    authorized to handle certain controlled drug substances and
                    chemicals under the Controlled Substance Act. NTIS is the authorized
                    official distributor of the database. NTIS provides online subscription



                Page 13                  GAO-13-99 Commerce's National Technical Information Service
                              access to the database on a weekly, monthly, or quarterly basis. NTIS
                              also provides this information via a searchable CD-ROM.

                          •   Death Master File. This file, maintained by the Social Security
                              Administration (SSA), contains approximately 85 million records of
                              deaths that have been reported since 1936. The file is used by
                              government; credit reporting organizations; and financial,
                              investigative, medical research, and other industries to verify deaths.
                              Through an agreement with SSA, NTIS is the only authorized official
                              distributor of the Death Master File. In this regard, NTIS provides
                              access to this information, including on DVD, and provides the means
                              to search and download the Death Master File online.

                          •   World News Connection. NTIS offers access to this online news
                              service, which provides translated and English-language news and
                              information from non-U.S. media sources. The information is obtained
                              from full-text, and summaries of newspaper articles, conference
                              proceedings, television and radio broadcasts, periodicals, and
                              nonclassified technical reports. The material in World News
                              Connection is provided to NTIS by the Open Source Center, a U.S.
                              government agency that provides analysis of foreign open source
                              intelligence.



NTIS Provides Fee-Based   In addition to its product offerings, NTIS offers a variety of fee-based
Services to Federal       services to federal agencies that are less directly related to its basic
Agencies                  statutory function of collecting and disseminating scientific and technical
                          information. These include services that leverage capabilities NTIS has
                          developed in the course of carrying out its mission. Its five service
                          offerings are distribution and order fulfillment, web-based services and
                          federal cloud computing, brokerage services, e-training and knowledge
                          management services, and digitization and scanning services. To provide
                          its services, NTIS enters into memorandums of understanding or
                          interagency agreements with agencies. Further, NTIS offers some of
                          these services through public-private partnerships with private industry,
                          which it refers to as “joint venture partners.” The five types of services are
                          described below.

Distribution and Order    Through a memorandum of understanding or interagency agreement,
Fulfillment Services      NTIS distributes large amounts of information products for federal
                          agencies. According to NTIS officials, these services differ from that
                          provided by its Clearinghouse in that they are used for distributing large
                          quantities of agencies’ products rather than selling individual copies of


                          Page 14                  GAO-13-99 Commerce's National Technical Information Service
                          publications. NTIS identified five primary clients for these services: the
                          Internal Revenue Service, the Department of Agriculture, the Department
                          of Education, the Pension Benefit Guaranty Corporation, and the
                          Administrative Office of the U.S. Courts. For example, NTIS currently has
                          an agreement with the Department of Agriculture to distribute on its behalf
                          health and nutrition educational materials in the form of brochures,
                          posters, and similar nutritional products. It also has an agreement with the
                          Department of Education to perform similar services.

Web-Based Services and    In 1988, Congress required NTIS to implement new methods or media for
Federal Cloud Computing   disseminating technical information; a 1992 amendment 25 specified that
                          this should include producing and disseminating products in electronic
                          formats. According to NTIS, this has been a primary basis for NTIS’s
                          transformation from a static paper-based distribution operation to a
                          modern, computer-based model and also a basis for NTIS to provide
                          information-dissemination services to other agencies. For example,
                          NTIS’s information systems infrastructure enables it to host federal
                          agencies’ applications and websites. The agency currently has an
                          agreement with the Federal Insurance and Mitigation Administration to
                          provide hosting for two of its websites and associated systems and to
                          host its web-based tool for reviewing underwriting and claims operations.
                          NTIS has also expanded its infrastructure to provide cloud computing 26
                          services and, according to the agency, is currently offering infrastructure-
                          as-a-service and software-as-a-service. 27 For example, along with joint
                          venture partner Carney, Inc., NTIS had an agreement with the National
                          Archives and Records Administration and currently has an agreement
                          with the Social Security Administration to configure and host the “Jive”
                          platform. 28 According to NTIS, the agency has eight primary clients for its


                          25
                               15 U.S.C. 3704b(e)(5).
                          26
                           Cloud computing is an emerging form of computing that relies on Internet-based
                          services and resources to provide computing services to customers.
                          27
                            According to NIST, infrastructure-as-a-service is a model where the service provider
                          delivers and manages the basic computing infrastructure of servers, software, storage,
                          and network equipment on which a platform to develop and execute applications can be
                          developed by the consumer. Software-as-a-service is a model where the service provider
                          delivers one or more applications and the computational resources and underlying
                          infrastructure to run them for use on demand as a turnkey service.
                          28
                            “Jive” is a social networking product for enterprise implementation. See its website:
                          http://www.jivesoftware.com. More specifically, it intends to help employees, customers,
                          and partners to connect, communicate, and collaborate.




                          Page 15                       GAO-13-99 Commerce's National Technical Information Service
                           web-based services and federal cloud computing offerings, including the
                           Department of Homeland Security, the Internal Revenue Service, and
                           other federal agency initiatives. 29

Brokerage Services         NTIS provides billing and collection services on a reimbursable basis to
                           other agencies that, like itself, charge for products and services but which
                           lack the necessary financial infrastructure to do their own billing and
                           collecting. The agency refers to these as its “Brokerage Services.” For
                           example, NTIS had an agreement with the National Agricultural Library to
                           develop, implement, and operate account maintenance, invoicing, and
                           collection procedures for the fees charged by the National Agricultural
                           Library to users of its photocopy and loan services. In addition, the
                           agency had an agreement with the National Library of Medicine to
                           perform invoicing, accounting, and collection services for its Interlibrary
                           Loan services. NTIS officials stated, however, that the agency plans to
                           stop marketing its brokerage services due to the decrease in demand for
                           this service.

E-Training and Knowledge   NTIS’s service offerings have also been expanded to e-training and
Management Services        knowledge management. Specifically, in conjunction with joint venture
                           partners, the agency provides collaborative software solutions, learning
                           management systems and support services, training evaluation software,
                           and talent management applications. For example, NTIS entered into an
                           agreement with Booz Allen Hamilton to provide, among other things,
                           program management; secure Internet hosting; and operations,
                           maintenance, and support services for the Defense Manpower Data
                           Center’s enterprise training program. According to NTIS, as of May 2012,
                           it had 28 primary government clients for this service offering, including the
                           Departments of Justice, the Interior, and the Treasury, and the U.S.
                           Patent and Trademark Office.




                           29
                            The other six primary clients are the Defense Manpower Data Center, Department of
                           Commerce, Wage Determination Online, U.S. Commission on Civil Rights, Joint Task
                           Force National Capital Region, and National Telecommunications and Information
                           Administration.




                           Page 16                    GAO-13-99 Commerce's National Technical Information Service
Digitization and Scanning   As another service, NTIS digitizes various document types, such as
Services                    microfilm or microfiche and paper forms, to assist agencies in complying
                            with section 508 standards. 30 Further, it offers storage and distribution for
                            the documents that it digitizes. For example, NTIS has an agreement with
                            SSA to provide alternative modes of receiving SSA notices and other
                            communications. NTIS provides this service with the assistance of its joint
                            venture partners, Vastec, Inc. and Braille Works, Inc. NTIS offers this
                            information in data compact disc, large print, and audio compact disc. As
                            of May 2012, the agency had six primary clients for its digitization and
                            scanning services, including SSA and the Department of Justice.


NTIS’s Reported Revenues    As a fee-for-service entity, NTIS’s revenues are generated exclusively
and Costs over the Past     from its products and services, and all its revenues, expenses, and capital
Decade                      expenditures are deposited into and paid out of its revolving fund. Overall,
                            NTIS had net earned revenues for 8 of the last 11 fiscal years. For
                            example, for fiscal year 2011, the agency reported that net earned
                            revenues from all its functions (products and services) totaled about $1.5
                            million. According to NTIS’s Financial Report for fiscal year 2011, the
                            revolving fund ending unobligated balance 31 was approximately $7.4
                            million.

                            However, over most of the last decade, the agency has incurred net costs
                            for its products. Specifically, NTIS product expenditures exceeded
                            revenues for 10 out of the past 11 fiscal years. The agency lost, on
                            average, about $1.3 million over the last 11 years on its products.

                            In contrast, NTIS’s overall financial performance has been supported by
                            revenues from its service offerings. The agency’s service revenues
                            increased, on average, about $1.8 million over the last 11 years. In
                            particular, for fiscal year 2011, revenues were about $53.5 million, costs
                            incurred were about $52 million, and the overall net earned revenue was
                            approximately $1.5 million from its service offerings. NTIS officials
                            attribute most of the net earned revenue to the agency’s agreements with


                            30
                              Section 508 of the Rehabilitation Act of l973 requires federal agencies to ensure that
                            their development, procurement, maintenance, or use of electronic and IT equipment
                            takes into account the need for people with disabilities to have access to and use of
                            information that is comparable to that of individuals without disabilities.
                            31
                              The unobligated balance represents the funds in NTIS’s revolving fund that may be
                            carried over into the next fiscal year.




                            Page 17                      GAO-13-99 Commerce's National Technical Information Service
the Departments of Agriculture and Education and the Social Security
Administration for various service offerings. Table 3 identifies the net
earned revenue or net cost for NTIS products and services over the last
11 fiscal years, as reported by NTIS. In those years in which NTIS had
net costs, the agency was sustained by cumulative net earned revenues
from previous years’ operations.

Table 3: Reported Net Earned Revenue (or Net Cost) for NTIS Products and
Services, Fiscal Years 2001-2011

                                                                                     Net earned
 Fiscal year                                   Products          Services     revenues (net cost)
 2001                                           $398,233       $1,772,011              $2,170,244
 2002                                         (1,197,257)       2,276,535               1,079,278
 2003                                         (1,409,026)       1,685,945                 276,919
 2004                                         (1,315,773)       1,824,157                 508,384
 2005                                         (1,973,646)       2,465,149                 491,503
 2006                                         (1,889,437)       2,179,183                 289,746
 2007                                         (1,683,794)         834,741               (849,053)
 2008                                         (2,072,519)         768,982              (1,303,537)
 2009                                         (2,217,263)       1,099,377              (1,117,886)
 2010                                          (730,347)        3,041,803               2,311,456
 2011                                          (690,736)        2,182,535               1,491,799
 Total                                   $(14,781,565)        $20,130,418              $5,348,853
Source: GAO analysis of NTIS-reported data.


Figure 2 illustrates the net earned revenues and net costs associated with
NTIS’s products and services from fiscal year 2001 through fiscal year
2011.




Page 18                                  GAO-13-99 Commerce's National Technical Information Service
Figure 2: Fiscal Years 2001-2011 Net Earned Revenues and Net Costs for NTIS
Products and Services




The decline in revenue for its products continues to call into question
whether NTIS’s basic statutory function of acting as a self-financing
repository and disseminator of scientific and technical information is still
viable. This is further highlighted by the fact that the services which are
financially sustaining the agency are less directly related to this function.

Recognizing its financial stance, NTIS has conducted analyses, and
identified in its 2011-2016 Strategic Plan actions to help address net
costs from its products, including its technical reports. The plan
emphasizes that the agency’s collection, culture, and information
technology infrastructure are its main strengths; that continued use of less
robust business systems and an aging work force are its primary
weaknesses; that growth opportunities still exist in the various sectors
served by NTIS, whether through products or services; and that NTIS is
threatened by and will have to overcome a shrinking customer base for its
products. The plan identifies three strategic initiatives to guide NTIS
during this period:




Page 19                   GAO-13-99 Commerce's National Technical Information Service
1. Increase revenue by enhancing the number of acquisitions, creating
   new products, reaching more customers, and adding value to what
   NTIS collects, and to reduce costs by reviewing and improving key
   business processes.

2. Improve NTIS’s utilization by other agencies by increasing the breadth
   and depth of its own collection and enhancing the suite of information
   management services that it can provide.

3. Achieve workforce excellence by focusing on identifying and acquiring
   the critical workforce skills required to accomplish the agency’s
   mission in a rapidly changing world.

Beyond these initiatives identified in the strategic plan, the Director of
NTIS also provided information on several other initiatives under way to
address the budget shortfalls from products. These initiatives include the
following product and organizational improvements:

•   Enhancing the accessibility of federal science content by shifting from
    a pricing model for stand-alone products (e.g., paper/print, microfiche,
    and compact disk media) to one that is subscription-based.

•   Repositioning NTIS to support open government initiatives in
    science—meeting with agencies such as NIST, the National Oceanic
    and Atmospheric Administration, and the Government Printing Office
    to address how NTIS can reposition its programs to support current
    science information needs.

•   Building collections of reports based on themes and categories that
    will be supported through subscriptions.

•   Adjusting the NTIS business model to support the increased demand
    for subscriptions to customers.

•   Reducing staff—for example, the agency has received authorization
    to provide early retirement for eligible employees and has stopped
    hiring additional staff. The agency anticipates employee attrition to
    further reduce current staffing levels.

Notwithstanding these efforts, NTIS could likely continue to face
challenges in recouping the costs of its products given the increasing
availability of technical information from other sources.




Page 20                 GAO-13-99 Commerce's National Technical Information Service
                        From fiscal year 1990 through 2011, 32 most of the additions to NTIS’s
Additions to NTIS’s     repository were older reports published in the year 2000 or before;
Repository Have         however, the greater demand was for more recently published reports. In
                        this regard, the agency added 841,502 reports to its repository from 1990
Mostly Included Older   through 2011. Of the reports added, approximately 62 percent, or
Reports, but Demand     524,256, had publication dates of 2000 or earlier, while approximately 38
                        percent, or 317,246, of the reports were published from 2001 to 2011.
for More Recent         Specifically, reports that were added to the repository during this period
Reports Is Greater      were as follows:

                        •    79,943 reports published in 1989 and prior years,

                        •    444,313 reports published from 1990 through 2000,

                        •    129,591 reports published from 2001 through 2004,

                        •    126,225 reports published from 2005 through 2008, and

                        •    61,430 reports published from 2009 through 2011.

                        Figure 3 shows the distribution of reports that were added by each
                        publication date age group from 1990 through 2011.




                        32
                          We did not include reports added for fiscal year 2012 because our study only focused on
                        those fiscal years that had been completed at the time that our study was initiated.




                        Page 21                     GAO-13-99 Commerce's National Technical Information Service
Figure 3: Year of Publication for Documents Added to NTIS Repository, 1990
through 2011




With regard to demand for the reports, we estimate that, during fiscal
years 2000 through 2011, NTIS distributed (sold) one or more copies of
about 419,657, or almost 50 percent, of the 841,502 reports added to its
repository during fiscal years 1990 through 2011. 33 Of these 419,657
reports, approximately 78 percent were distributed through a subscription
only. 34 NTIS officials attributed this to the fact that subscriptions are a
cost-effective way for libraries to meet their collection development
requirements within a specific or broad area of interest. The officials
noted that direct sales are generally to customers interested in specific
topics.



33
  The 95 percent confidence interval for the estimated percent of reports distributed one
or more times is (44.9, 54.8) percentage points. For purposes of this report, demand is
defined as the distribution (sale) of a report by direct sale or subscription.
34
  The 95 percent confidence interval for the estimated percent of reports distributed
through a subscription only is (69.7, 84.4) percentage points.




Page 22                      GAO-13-99 Commerce's National Technical Information Service
However, as shown in figure 4, the agency was more likely to distribute a
higher percentage of more recently published reports than older ones. For
example, we estimate that between 96 and 100 percent of the reports
published from 2001 through 2011 had been distributed, while only about
21 percent of reports published in 1989 or earlier were distributed during
this time period. That is, the demand for older holdings in the NTIS
repository is lower than for newer publications.

Figure 4: Estimated Percentage of Reports Distributed and Not Distributed by
Publication Date during Fiscal Years 2000 through 2011




Note: This figure presents estimates based upon our random sample of 384 reports. The heights of
the bars show the percentage point estimate for each category on the horizontal axis relative to the
others. The line on each bar represents the 95 percent confidence interval associated with each
estimate. Estimates are rounded to the nearest tenth of a percent, and may sum to greater than 100
percent due to rounding. See app. I for an explanation of confidence intervals.




Page 23                          GAO-13-99 Commerce's National Technical Information Service
                        Based on our sample, we estimate that most (about 74 percent) of the
Most Reports Added      reports added to NTIS’s repository during fiscal years 1990 through 2011
Since 1990 Are Freely   were readily available from other public websites, and nearly all of these
                        (95 percent) could be obtained for free. 35 Specifically, we estimate that
Available from Other    approximately 621,917, or about 74 percent, of the 841,502 reports
Websites                added to NTIS’s repository from fiscal years 1990 through 2011 are
                        readily available from one of the other four publicly available sources we
                        searched (i.e., the issuing organization’s website; the Government
                        Printing Office’s Federal Digital System website; the U.S. government’s
                        official web portal, USA.gov; or from another website located through a
                        Google search). 36 The source that most often had the report we were
                        searching for was another website located through
                        http://www.Google.com.

                        In addition, of the reports added since fiscal year 1990, those with more
                        recent publication dates were more likely to be available from other public
                        sources than older ones. For example, approximately 87 percent of the
                        reports with publication dates from 2009 to 2011 were available
                        elsewhere, while 55 percent of those published in 1989 or earlier were. 37
                        Figure 5 shows the estimated availability of reports added to NTIS’s
                        repository since fiscal year 1990 by date of publication.




                        35
                          The 95 percent confidence interval for the estimated percentage of reports available
                        elsewhere which could be obtained for free is (90.7, 97.5) percentage points.
                        36
                          The 95 percent confidence interval for the estimated percentage of reports available
                        through one or more of the four publically available sources we searched is (67.9, 80.0)
                        percentage points.
                        37
                          For reports published between 2009 and 2011, we estimate that 86.5 percent are
                        available elsewhere. The 95 percent confidence interval for this estimate is (74.2, 93.5)
                        percentage points. For reports published in 1989 or earlier, we estimate that 55 percent
                        are available elsewhere. The 95 percent confidence interval for this estimate is (34.1,
                        74.3) percentage points.




                        Page 24                      GAO-13-99 Commerce's National Technical Information Service
Figure 5: Estimated Availability of Reports by Year of Publication




As shown in figure 6, of the reports that were found to be readily available
from one of the other four sources that we searched, about 61 percent of
those reports had been distributed (sold) by NTIS. Conversely, of the
reports that were not found to be readily available from one of the other
four sources, most, or about 82 percent, had not been distributed by
NTIS. 38




38
  The 95 percent confidence interval for the estimated percent available reports that were
distributed is (54.0, 67.4) percentage points. The 95 percent confidence interval for the
estimated percent of reports not available and not distributed is (73.2, 87.8) percentage
points.




Page 25                      GAO-13-99 Commerce's National Technical Information Service
Figure 6: Estimated Availability of Reports Distributed and Not Distributed During
Fiscal Years 2000 through 2011




Not only were most of the reports in our sample available from sources
other than NTIS, but about 95 percent of the reports available elsewhere
could be obtained free of charge from one of the four other sources we
searched. 39 The remaining 5 percent were available from the public
sources for a fee. 40 Moreover, the year of publication did not appear to
have an effect on whether a report was available free of charge. For
example, the following reports available for a fee from NTIS were
available free of charge from the issuing organization’s website:

Hazardous Waste Characteristics Scoping Study, November 1996,
Environmental Protection Agency, 278 pages. (At NTIS, print on demand
costs $73, electronic $25.)



39
 The 95 percent confidence interval for this estimate is (90.7, 97.5) percentage points.
40
 The 95 percent confidence interval for this estimate is (2.5, 9.3) percentage points.




Page 26                      GAO-13-99 Commerce's National Technical Information Service
“Homeland Security: Intelligence Indications and Warning,” December
2002, Naval Postgraduate School, 5 pages. (At NTIS, print on demand
costs $17, electronic $15.)

Export Controls: System for Controlling Exports of High Performance
Computing Is Ineffective, 2000, GAO, 60 pages. (At NTIS, print on
demand costs $48, electronic $15.)

FDA Enforcement Report: July 20, 2011, July 2011, Food and Drug
Administration, 28 pages. (At NTIS, print on demand costs $33, electronic
$15.)

Principal Rare Earth Elements Deposits of the United States: A Summary
of Domestic Deposits and a Global Perspective, 2010, Geological Survey,
104 pages. (At NTIS, print on demand costs $60, electronic $25.)

Of those reports we found available elsewhere, figure 7 shows the
estimated percent that were available elsewhere for free by year that the
document was published.




Page 27                 GAO-13-99 Commerce's National Technical Information Service
Figure 7: Of Reports Found Elsewhere, Estimated Percent Available for Free and
Not Free, by Year of Publication




Note: These estimates are for reports readily available from one of the other four publicly available
sources that we searched.

The Director of NTIS acknowledged two factors that contributed to the
free and public availability of reports from other sources:

•    Federal agencies are providing information, including their federal
     scientific, technical, and engineering information products, on their
     websites in electronic format and on central federal information
     websites such as http://www.data.gov and http://www.science.gov.
     The information is available for immediate download at no cost.

•    Federal agencies are participating in programs with Internet search
     engines that permit the public to locate their products for free or for
     less than they are when purchased from NTIS. In addition,
     commercial vendors are also able to obtain these information
     products from agency websites or through Internet search and are
     able to make these products available for free or at a price lower than
     that offered by NTIS.




Page 28                          GAO-13-99 Commerce's National Technical Information Service
                     Further, NTIS acknowledged in its strategic plan that because the Internet
                     continues to change the way people acquire and use information and
                     permits federal agencies to make their information products available for
                     free, NTIS is challenged to meet its statutory mandate as a self-financing
                     repository and disseminator of technical information. Notwithstanding
                     these acknowledgments, NTIS continues to charge for reports that are
                     freely available from other public sources.


                     NTIS serves as a permanent repository and disseminator of technical
Conclusions          information, and by statute, is required to be financially self-sustaining, to
                     the fullest extent feasible, by charging fees for its products and services.
                     While the agency had cumulative net earned revenues as of September
                     30, 2011, its costs exceeded revenue by an average of about $1.3 million
                     over the last 11 years from the sale of technical information. The agency’s
                     net revenue now comes primarily from services that are less directly
                     related to its basic statutory function. The decline in sales of technical
                     information is due in part to the increasing availability of this information
                     from other sources, including websites and Internet search tools, and
                     often at no charge. Charging for information that is freely available
                     elsewhere is a disservice to the public and may also be wasteful insofar
                     as some of NTIS’s customers are other federal agencies. Taken together,
                     these considerations suggest that the fee-based model under which NTIS
                     currently operates for disseminating technical information may no longer
                     be viable or appropriate.


                     In light of the agency’s declining revenue associated with its basic
Matter for           statutory function and the charging for information that is often freely
Congressional        available elsewhere, Congress should consider examining the
                     appropriateness and viability of the fee-based model under which NTIS
Consideration        currently operates for disseminating technical information to determine
                     whether the use of this model should be continued.


                     The Acting Secretary of Commerce provided written comments on a draft
Agency Comments      of this report, which are reprinted in appendix II. In its comments,
and Our Evaluation   Commerce expressed appreciation for our study and for our focus on the
                     initiatives that NTIS has undertaken. However, the department said NTIS
                     did not believe our conclusions (that the fee-based model under which it
                     operates for disseminating technical information may no longer be viable
                     or appropriate) fully reflect the additional value that NTIS provides with
                     the work that it performs. Commerce stated that, through its federal


                     Page 29                  GAO-13-99 Commerce's National Technical Information Service
clearinghouse and repository, the agency provides federally funded
reports that are not otherwise readily available, such as most of those
issued prior to l989. Additionally, Commerce stated that NTIS recognizes
that it cannot remain financially solvent solely through sales and
subscriptions of technical reports with expectations that these products
will be widely available for free. It added that NTIS is moving in the
direction of “open and improved access” to information, but recognizes
that it needs to maintain a sustainable financial model and continue
providing enhanced value to the information generated by other federal
agencies. In this regard, Commerce described features that it believes
have added value to the technical reports that NTIS maintains, how these
features improve access to the documents and related data, and specific
information services that the agency provides to science information
professionals.

Our report highlighted various initiatives that NTIS has undertaken to
provide older reports that might not otherwise be readily available, and to
increase the value of its technical reports, information management
services, and technology transfer capabilities. However, as discussed in
the report, we found that the demand for older holdings in the agency’s
repository is lower than for new publications. For example, we estimate
that between 96 and 100 percent of the reports published from 2001
through 2011 had been distributed, while only about 21 percent of reports
published in l989 or earlier were distributed during this time period.
Further, as the agency acknowledged, its financial health is based on
both its information product and service missions. Also, as we state in
our report, the agency’s net revenue now comes primarily from services
that are less directly related to its basic statutory function, while sales of
its technical information products have resulted in net losses. This decline
in sales of NTIS’s technical reports is due in part to the increasing
availability of this information from other sources, including websites and
Internet search tools, and often at no charge. With these factors in mind,
we stand by our conclusion that the fee-based model under which NTIS
currently operates for disseminating technical information may no longer
be viable or appropriate.

NTIS also provided technical comments on the report via e-mail, which
we have incorporated as appropriate.


We are sending copies of this report to interested congressional
committees. We are also sending copies to the Secretary of Commerce;
the Director, NTIS; the Director, NIST; and other interested parties.


Page 30                  GAO-13-99 Commerce's National Technical Information Service
Copies of this report will also be available at no charge on GAO’s website
at http://www.gao.gov.

Should you or your staffs have any questions on information discussed in
this report, please contact me at (202) 512-6304 or melvinv@gao.gov.
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. GAO staff who made
major contributions to this report are listed in appendix III.




Valerie C. Melvin
Director
Information Management
  and Technology Resources Issues




Page 31                 GAO-13-99 Commerce's National Technical Information Service
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              Our objectives were to determine (1) how the National Technical
              Information Service (NTIS) is currently organized and operates, including
              its various functions, current staffing level, reported cost of operations,
              and revenue sources; (2) the age of and demand trends for reports added
              to NTIS’s repository; and (3) the extent to which these reports are readily
              available from other public sources.

              To determine how NTIS is organized and operates, we reviewed the
              agency’s strategic plan, and documentation on its organizational and
              reporting structure, office staffing level and assigned responsibilities, and
              types of products and services offered. We also reviewed cost data
              contained in the agency’s annual financial reports. 1 In addition, we
              reviewed relevant laws and regulations on NTIS’s authority and
              responsibilities 2 and our previous reports that discussed its mission and
              operations. 3 We supplemented our analyses with interviews of the
              Director of NTIS and other relevant agency officials; we also interviewed
              officials of the Department of Commerce and its National Institute of
              Standards and Technology (NIST), which have specific reporting
              relationships with NTIS.

              For our second and third objectives, we obtained from NTIS the full list of
              document accession numbers for reports added to its repository since
              fiscal year 1990. We chose to include reports added since fiscal year
              1990 to determine the current availability of older documents in light of
              NTIS’s role as a permanent repository, and to focus on the reports added
              since our previous review in 2001. 4 We then stratified the list of document
              accession numbers into four strata by the year in which a document was
              added to the repository. We subsequently selected a stratified random
              sample with 96 document accession numbers per strata, for a total



              1
               The cost data reflected in our report were taken from documentation provided by NTIS.
              However, we did not test the reliability of NTIS’s cost accounting system, from which the
              agency derives its cost data.
              2
               For example 15 U.S.C. 1151-57, 15 U.S.C. 3704b-1 & 2, and 15 C.F.R. Part 1180.
              3
                GAO, Information Policy: NTIS’ Financial Position Provides an Opportunity to Reassess
              Its Mission, GAO/GGD-00-147 (Washington, D.C.: June 30, 2000) and Information
              Management: Dissemination of Technical Reports, GAO-01-490 (Washington, D.C.: May
              18, 2001).
              4
               We did not include reports added for fiscal year 2012 because our study only focused on
              those fiscal years that had been completed at the time that our study was initiated.




              Page 32                      GAO-13-99 Commerce's National Technical Information Service
Appendix I: Objectives, Scope, and
Methodology




sample size of 384 reports. All of the estimates made with this sample are
weighted to reflect the stratified design. NTIS provided us with the full
bibliographic data for each document in our sample.

To determine the age of reports added to NTIS’s repository since fiscal
year 1990, we used the year of publication for the reports in our sample to
estimate the age range (prior to 1990; 1990-2000; 2001-2004; 2005-
2008; and 2009-2011) for all documents added from fiscal year 1990
through fiscal year 2011. To determine the demand trends for reports
added to NTIS’s repository during fiscal years 1990 through 2011, we
requested the sales data from fiscal year 2000 through fiscal year 2011
for the 384 reports in our stratified sample. NTIS provided the distribution
data for both direct sales and subscriptions for all of these documents.
We then used our sample and the sales data to estimate the extent and
prevalence of the sales among all reports added to the NTIS repository.
Specifically, we used these data to estimate the (1) total number of
reports distributed by direct sale and subscription, (2) total number of
reports distributed one or more times, (3) percentage of reports
distributed relative to the year the reports were published, and (4)
percentage of reports distributed relative to their availability elsewhere.
The observations were statistically weighted in the estimation process to
reflect the stratified sample design that we used.

To determine the extent to which reports in the repository are readily
available from other public sources, we first developed a methodology for
conducting systematic Internet searches to determine availability
elsewhere. More specifically, as part of this methodology, we searched
the Internet to determine if each of the reports included in our sample of
384 reports could be found elsewhere and at no cost. Using a tiered
approach, we searched the following four sources in the order shown: (1)
the issuing organization’s website; (2) the U.S. Government Printing
Office’s Federal Digital System website—http://www.gpo.gov/fdsys; (3) a
web search conducted using the federal government Internet portal
USA.gov—http://www.USA.gov; and (4) a web search conducted using
the commercial search engine http://www.Google.com. 5 Specifically, with
this methodology, we determined whether each report was first available
at no cost on the issuing organization’s website and, if so, concluded the



5
 These sources and the order in which they were searched were based on an effort to
duplicate the methodology used in our previous 2001 study of NTIS.




Page 33                      GAO-13-99 Commerce's National Technical Information Service
Appendix I: Objectives, Scope, and
Methodology




Internet search at this point. However, if the report was not available, then
the search continued to the second source, and so on, until either the
report was found to be available at one of the remaining sources, or all
sources were exhausted. 6 We then used our results to estimate the
percentage of the total population of NTIS reports added to the repository
during fiscal years 1990 through 2011 that was available from other public
sources.

All of the results derived from the sample analyses constituted estimates
that are subject to sampling errors. These sampling errors measure the
extent to which samples of this size and structure are likely to differ from
the population they represent. Because we followed a probability
procedure based on random selections, our sample is only one of a large
number of samples that we might have drawn. Since each sample could
have provided different estimates, we express our confidence in the
precision of our particular sample’s results as a 95 percent confidence
interval. This is the interval that would contain the actual population value
for 95 percent of the samples we could have drawn.

To determine the reliability of the data provided from NTIS’s repository of
reports, we performed basic steps to ensure the data provided were valid,
and reviewed relevant information describing the database supporting the
repository. We tested for duplicate records, missing values, and out-of-
range values in the data received from NTIS. We did not assess the
reliability of the system used to maintain these data or the processes
used in extracting the data for our engagement purposes. To determine
the reliability of the sales data provided by NTIS, we conducted interviews
with agency officials to gain an understanding of the process by which
accounts receivable records are added and managed within NTIS’s
system of accounts receivable–”CIS.PUB.” Further, we asked cognizant
agency officials specific questions to understand the controls in place for
ensuring the integrity and reliability of the data contained in CIS.PUB. In
addition, we met with NTIS officials to discuss data collected from NTIS
and obtained their assertions regarding the data it provided. Based on the
results of these efforts, we found the data sources to be sufficiently
reliable, given the way they are reported herein.



6
 In identifying the reports’ availability elsewhere, we did not assess if the source at which
the report was found was the original issuer of the report or whether the report’s content
was unaltered from its original issuance.




Page 34                       GAO-13-99 Commerce's National Technical Information Service
Appendix I: Objectives, Scope, and
Methodology




We conducted this performance audit from February 2012 to November
2012 in accordance with generally accepted government auditing
standards. Those standards require that we plan and perform the audit to
obtain sufficient, appropriate evidence to provide a reasonable basis for
our findings and conclusions based on our audit objectives. We believe
that the evidence obtained provides a reasonable basis for our findings
and conclusions based on our audit objectives.




Page 35                      GAO-13-99 Commerce's National Technical Information Service
Appendix II: Comments from the Department
             Appendix II: Comments from the Department
             of Commerce



of Commerce




             Page 36                     GAO-13-99 Commerce's National Technical Information Service
Appendix II: Comments from the Department
of Commerce




Page 37                     GAO-13-99 Commerce's National Technical Information Service
Appendix II: Comments from the Department
of Commerce




Page 38                     GAO-13-99 Commerce's National Technical Information Service
Appendix II: Comments from the Department
of Commerce




Page 39                     GAO-13-99 Commerce's National Technical Information Service
Appendix II: Comments from the Department
of Commerce




Page 40                     GAO-13-99 Commerce's National Technical Information Service
Appendix III: GAO Contact and Staff
                  Appendix III: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Valerie C. Melvin, (202) 512-6304 or melvinv@gao.gov
GAO Contact
                  In addition to the contact named above, Cynthia Scott (Assistant
Staff             Director), Carl Barden, Virginia Chanley, Elena Epps, Nancy Glover, Alina
Acknowledgments   J. Johnson, Lee McCracken, Constantine Papanastasiou, David Plocher,
                  Bradley Roach, and Tina Torabi made key contributions to this report.




(310986)
                  Page 41                      GAO-13-99 Commerce's National Technical Information Service
                      The Government Accountability Office, the audit, evaluation, and
GAO’s Mission         investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (http://www.gao.gov). Each weekday
GAO Reports and       afternoon, GAO posts on its website newly released reports, testimony,
                      and correspondence. To have GAO e-mail you a list of newly posted
Testimony             products, go to http://www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
                      Website: http://www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in   E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.