oversight

Financial Management: DOD's Metrics Program Provides Focus for Improving Performance

Published by the Government Accountability Office on 2003-03-28.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States General Accounting Office

GAO          Report to the Chairman and Ranking
             Minority Member, Subcommittee on
             Readiness and Management Support,
             Committee on Armed Services, U.S.
             Senate
March 2003
             FINANCIAL
             MANAGEMENT
             DOD’s Metrics
             Program Provides
             Focus for Improving
             Performance




GAO-03-457
             a
                                                March 2003


                                                FINANCIAL MANAGEMENT

                                                DOD's Metrics Program Provides Focus
Highlights of GAO-03-457, a report to the
Chairman and Ranking Minority Member,           for Improving Performance
Subcommittee on Readiness and
Management Support, Committee on
Armed Services, U.S. Senate




The Department of Defense (DOD)                 The DOD Comptroller’s metrics showing significant reductions in payment
has historically been unable to                 recording errors and in commercial and travel card payment delinquencies
accurately account for and record               were, in general, based on definitions and methodologies that were either
its disbursements. In March 2002,               consistent with or better than those used for prior reporting on these issues.
the DOD Comptroller cited metrics               Although the methodology used to calculate two of the cited measures
that showed dramatic reductions in
payment recording errors
                                                resulted in overstating the rates of improvement, our recalculation after
(57 percent between October 2000                correcting for the methodology errors still showed positive—although less
and October 2001), backlogs of                  dramatic—improvement trends.
commercial payments (41 percent
between April and October 2001),                While we were able to verify the reductions in travel card delinquencies
and travel card payment                         because the underlying data were available from an independent source, we
delinquencies (34 percent for those             could not verify the accuracy of the specific improvement percentages
individually billed and 86 percent              reported for payment recording errors and commercial payment
for those centrally billed between              delinquencies. DOD’s archaic and nonintegrated systems either do not
January and December 2001). As a                contain the transaction-level detail to support the completeness and
result, the Congress asked us to                accuracy of the metrics or they make it extremely onerous and time
determine whether the cited
reductions were (1) calculated
                                                consuming for the staff to gather and reconcile the needed detail. However,
using consistent definitions and                we were able to verify that DOD has made numerous policy, procedure, and
methodologies, (2) properly                     systems changes that support an overall trend toward improved
supported, and (3) effective                    performance in these areas.
indicators of short-term financial
management progress.                            If they could be verified, some of the cited metrics could be effective
                                                indicators of short-term financial management progress. However, if
                                                considered alone, delinquency rates are not necessarily good indicators for
                                                centrally billed travel cards or commercial payments. Placing too much
GAO recommends the following:
• Use definitions and criteria                  emphasis on paying bills promptly may tempt DOD staff to bypass important
  consistent with the Defense                   internal controls meant to ensure that the goods and services being paid for
  Finance and Accounting Service                were properly authorized and actually received.
  when calculating and reporting
  metrics related to payment                    Despite shortcomings, the cited metrics have focused DOD’s attention on
  recording errors.                             highly visible financial management problems. As shown below, recent
• Measure improvements in                       metrics issued by the DOD Comptroller indicate continuing improvements.
  individually billed travel card
                                                  Reported Progress from Beginning Measurement Dates
  delinquencies by using same
  month to same month                             100      Percent
  comparisons.                                     80
• Work with the military services                  60
  and other defense agencies to                    40
  develop performance measures                     20
  that complement the metrics                       0
  program for crosscutting issues.                                  Oct. 01                 Sept. 02
                                                              Commercial payment backlogs
DOD concurred with our
                                                              Payment recording errors
recommendations.                                              Centrally billed travel card delinquencies
 www.gao.gov/cgi-bin/getrpt?GAO-03-457.
                                                Source: GAO calculations using DOD data.
 To view the full report, including the scope
 and methodology, click on the link above.      Note: Individually billed travel card delinquencies are not shown because of limitations in the data
 For more information, contact Gregory Kutz,    on the measurement dates.
 (202) 512-9095 or kutzg@gao.gov.
Contents



Letter                                                                                                                   1
                             Results In Brief                                                                            4
                             Background                                                                                  7
                             Cited Metrics Were Generally Based on Improved Definitions and
                               Methodologies                                                                            11
                             Most Cited Metrics Are Not Verifiable                                                      16
                             Cited Metrics Serve Important Purpose But Further Steps Are
                               Needed to Sustain Improvements                                                           25
                             Conclusion                                                                                 28
                             Recommendations                                                                            28
                             Agency Comments and Our Evaluation                                                         29


Appendixes
              Appendix I:    Objectives, Scope, and Methodology                                                         30
             Appendix II:    Comments from the Under Secretary of Defense                                               32
             Appendix III:   GAO Contact and Acknowledgments                                                            34


Related Audit Reports                                                                                                   35

and Testimonies

Figures                      Figure 1: Reductions in Payment Recording Errors Between
                                       October 2000 and September 2002                                                  14
                             Figure 2: Reductions in the Number of Payment Recording
                                       Errors                                                                           17
                             Figure 3: Decrease in Vendor Pay Backlogs for the Period April
                                       2001 through September 2002                                                      21
                             Figure 4: Decrease in Centrally Billed Travel Card Delinquencies
                                       for the Period January 2001 through December 2002                                22
                             Figure 5: Decrease in Individually Billed Travel Card Delinquencies
                                       for the Period January 2001 through December 2002                                25



                              This is a work of the U.S. Government and is not subject to copyright protection in the
                              United States. It may be reproduced and distributed in its entirety without further
                              permission from GAO. It may contain copyrighted graphics, images or other materials.
                              Permission from the copyright holder may be necessary should you wish to reproduce
                              copyrighted materials separately from GAO’s product.




                             Page i                                                 GAO-03-457 DOD's Metrics Program
A
United States General Accounting Office
Washington, D.C. 20548



                                    March 28, 2003                                                                 Leter




                                    The Honorable John Ensign
                                    Chairman
                                    The Honorable Daniel K. Akaka
                                    Ranking Minority Member
                                    Subcommittee on Readiness
                                      and Management Support
                                    Committee on Armed Services
                                    United States Senate

                                    The Department of Defense’s (DOD) inability to accurately account for and
                                    record its disbursements has been a serious, long-standing, and much
                                    reported financial management problem. The department’s ability to
                                    improve its accounting has historically been hindered by its reliance on
                                    fundamentally flawed financial management systems and processes and a
                                    weak overall internal control environment. In fact, DOD’s complex and
                                    inefficient payment processes have generally inhibited the proper
                                    recording of transactions when they occur, including the prompt and
                                    proper matching of disbursements with obligations—a critical funds-
                                    control measure. Such payment recording errors mean that DOD does not
                                    know the true amount of funds that it has available to obligate and spend in
                                    each appropriation account. As a result, the department risks
                                    overspending or not effectively using all available funding for needed
                                    items.




                                    Page 1                                        GAO-03-457 DOD's Metrics Program
Auditors have also reported on DOD’s history of delinquent payments to its
commercial suppliers and for its government-issued individually billed
travel cards. These weaknesses increased the risk of fraud and/or
disbursement errors, including duplicate payments, payments in the wrong
amount, or charges to the wrong accounts. In addition, travel card
delinquencies and charge-offs have resulted in millions of dollars in lost
rebates and increased fees. These payment problems contributed greatly
to our decision to put DOD financial management on our high-risk list of
areas that are vulnerable to waste, fraud, abuse, and mismanagement in
1995, a designation that continues today.1

To increase attention to these problems, the Defense Finance and
Accounting Service (DFAS) accelerated and expanded the use of its
existing financial management performance metrics several years ago, to
include specific measures for payment recording errors and payment
delinquencies. In his March 6, 2002, testimony before your Subcommittee,
the DOD Comptroller reported significant improvements in these measures
based on calculations using DFAS data. In particular, the Comptroller
stated that

• payment recording errors decreased by 57 percent between October
  2000 and October 2001,

• DOD’s backlog of commercial payments (i.e., delinquent unpaid
  invoices) had been reduced by 41 percent between April 2001 and
  October 2001, and




1
 U.S. General Accounting Office, High Risk Series: An Overview, GAO/HR-95-1
(Washington, D.C.: February 1995); U.S. General Accounting Office, Major Management
Challenges and Program Risks: A Governmentwide Perspective, GAO/OCG-99-1
(Washington, D.C.: January 1999); U.S. General Accounting Office, High-Risk Series: An
Update, GAO-03-119 (Washington, D.C.: January 2003); U.S. General Accounting Office,
Major Management Challenges and Program Risks: Department of Defense, GAO-03-98
(Washington, D.C.: January 2003).




Page 2                                                GAO-03-457 DOD's Metrics Program
• payment delinquencies decreased by 34 percent for personal
  (individually billed) travel cards and 86 percent for organizational
  (centrally billed) travel cards between January 2001 and December
  2001.2

This report responds to your request that we review DOD’s reported
improvements. As agreed with your office, our objectives were to
determine whether (1) the cited performance measures were defined and
calculated in a manner consistent with previous reporting on payment
recording errors and delinquencies, (2) the cited improvement data were
properly supported and represent real improvements in performance, and
(3) the metrics are effective indicators of short-term financial management
progress that can be sustained.

In conducting this work, we visited various DFAS centers and gathered,
analyzed, and compared information on how payment recording errors,
commercial payment backlogs, and travel card delinquencies were defined,
calculated, and reported both in the past and for the cited metrics. We
spoke with center personnel about process and systems improvements and
we gathered and analyzed relevant output that demonstrated the results of
those changes. We also reviewed the various financial management
metrics programs in place or being developed throughout DOD. However,
as discussed later in this report, we were unable to independently verify the
completeness and accuracy of the data that supported the cited metrics,
with the exception of travel card information. We performed our work in
accordance with U.S. generally accepted government auditing standards
from June 2002 through February 2003. Details of our scope and
methodology are in appendix I. We requested comments from the
Secretary of Defense and the Under Secretary of Defense (Comptroller).
DOD’s comments are reprinted in appendix II. We considered and
incorporated DOD’s suggested technical comments as appropriate.




2
  Individually billed accounts are held and paid by individual cardholders based on
reimbursement of expenses incurred while on official government travel. Centrally billed
accounts are used to purchase transportation or for the travel expenses of a unit and are
paid directly by the government.




Page 3                                                 GAO-03-457 DOD's Metrics Program
Results In Brief   In general, the definitions and methodologies used to gather data for the
                   DOD Comptroller reported metrics were either consistent with or better
                   than those used for prior reporting related to payment recording errors,
                   commercial payment backlogs, and travel card payment delinquencies. For
                   example, in contrast to previous reporting, intransit transactions3 were
                   appropriately included in the definition of payment recording errors for the
                   cited metrics calculations. While the underlying data were more complete,
                   however, DOD personnel agree that there may still be payment recording
                   errors that have not been identified and properly categorized. In addition,
                   the methodology used to calculate two of the cited measures resulted in
                   overstating the rates of improvement. Our recalculation of the metrics
                   after correcting for the methodology errors still showed positive—although
                   less dramatic—improvement trends.

                   We could not verify the accuracy of specific improvement percentages
                   reported for payment recording errors and commercial payment
                   delinquencies, in large part because of DOD’s archaic and nonintegrated
                   systems. Either the transaction-level detail supporting the completeness
                   and accuracy of the reported metrics was no longer available or it would
                   have been extremely onerous and time consuming for DOD staff to gather
                   and reconcile the transaction-level data. However, DOD was able to
                   provide us with summary-level data that matched the amounts reported for
                   these metrics and we did verify that DOD has made numerous policy,
                   procedural, and systems changes that would support an overall trend
                   toward improved performance in these areas. In contrast, we were able to
                   obtain corroborating data from an independent source that properly
                   supported the reductions in travel card payment delinquencies cited by the
                   Comptroller. In addition, the reduction in delinquency rates for individual
                   travel card payments is supported by our recent findings that the military
                   services, in particular the Air Force,4 have begun to give delinquencies
                   greater attention and have used travel card audits to identify problems and
                   needed corrective actions.


                   3
                     Intransit transactions include payments that have not yet been received by the DFAS
                   accounting office for recording and matching against the corresponding obligation. DOD
                   began reporting intransits in response to a recommendation in a report titled Financial
                   Management: Status of Defense Efforts to Correct Disbursement Problems, GAO/AIMD-95-
                   7 (Washington, D.C.: Oct. 5, 1994).
                   4
                     U.S. General Accounting Office, Travel Cards: Air Force Management Focus Has
                   Reduced Delinquencies, but Improvements in Controls Are Needed, GAO-03-298
                   (Washington, D.C.: Dec. 20, 2002).




                   Page 4                                               GAO-03-457 DOD's Metrics Program
For metrics to be effective, they must be properly defined, correctly
measured, and able to be verified. However, none of the DOD
Comptroller’s cited metrics meet all of these criteria. In addition, some of
the metrics may not be good indicators of financial management
improvement if considered separately. For example, focusing only on
delinquency rates for centrally billed travel cards and commercial
payments may place too much emphasis on paying bills promptly. As a
result, DOD staff may be tempted to shortcut important internal control
mechanisms that are meant to ensure that the goods and services being
paid for were properly authorized and actually received. We have
previously reported on problems related to DOD individually billed travel
card purchases and contract payments5 that indicate the need for increased
attention to the propriety as well as promptness of such payments.
Another limitation in the reported metrics for payment recording errors
and commercial payment backlogs is that only DFAS performance is being
measured and reported. Even though the military services and other
defense organizations are key contributors to preventing and resolving
these problems, these organizations do not have complementary
measurement programs.

Despite their shortcomings, the cited metrics have served an important
purpose by focusing DOD’s attention on highly visible financial
management problems, setting challenging goals, and encouraging staff to
be diligent and innovative in their attempts to attain those goals. In
general, the improvements cited by the Comptroller demonstrate what can
be accomplished in the short term as a result of the focus and intensive
effort day after day of DOD management and staff. According to recent
information, DFAS is continuing to report improvements in the measured
areas.

• Payment recording errors and commercial pay backlogs for September
  2002 show reductions of 26 percent and 35 percent, respectively, from
  the balances reported at October 2001.



5
 GAO-03-298; U.S. General Accounting Office, Travel Cards: Control Weaknesses Leave
Army Vulnerable to Potential Fraud and Abuse, GAO-02-863T (Washington, D.C.: July 17,
2002); U.S. General Accounting Office, Travel Cards: Control Weaknesses Leave Navy
Vulnerable to Potential Fraud and Abuse, GAO-03-148T (Washington, D.C.: Oct. 8, 2002);
U.S. General Accounting Office, Financial Management: Improvements Needed in Air
Force Vendor Payment Systems and Controls, GAO/AIMD-98-274 (Washington, D.C.: Sept.
28, 1998).




Page 5                                               GAO-03-457 DOD's Metrics Program
• Centrally billed travel card delinquencies declined from 2 percent at
  December 2001 to 1.5 percent at December 2002—1.5 percent is equal to
  the delinquency rate for other federal agencies.

• Individual travel card delinquencies were reduced from 12.2 percent at
  December 2001 to 8.3 percent at December 2002.

However, DFAS has cautioned that, without modern integrated systems
and the streamlined processes they engender, reported progress for
payment recording errors and commercial payment delinquencies may not
be sustainable if its workload is significantly increased or its staffing
significantly decreased.

As discussed in our January 2003 high-risk report, the keys to effective
reform of DOD financial management include an integrated approach,
sustained leadership, results-oriented performance measures, and
appropriate incentives and consequences. In line with this, DOD is
currently developing a departmentwide, balanced program of metrics that
is intended to align with its strategic goals, focus on results, and achieve
auditable reports. DFAS, the military services, and other defense agencies
will all be supporting players in this program. From the individual
performance measurement programs of the military services, defense
agencies, and DFAS, certain metrics will be selected and reported to the
top levels of DOD management for evaluation and comparison. In this
scenario, it is important that DOD properly and consistently report the
selected metrics and that the services, agencies, and DFAS develop
complementary metrics programs to assist in identifying, measuring, and
resolving crosscutting issues. We are making specific recommendations in
this report to address weaknesses where (1) the DOD Comptroller’s office
used methodologies that overstated improvements, (2) the military
services were critical partners with DFAS in making improvements or
resolving problems but were not being measured on their performance in
these areas, and (3) changes in travel card metrics would improve
performance evaluation.

In comments on a draft of this report, DOD agreed with our
recommendations and explained actions it is taking to implement them.




Page 6                                        GAO-03-457 DOD's Metrics Program
Background                 For years, auditors have reported long-standing weaknesses in DOD’s
                           ability to promptly pay its bills and accurately account for and record its
                           disbursements. Numerous of our and DOD Inspector General audit reports
                           have cited deficiencies in management oversight, a weak internal control
                           environment, flawed financial management systems, complex payment
                           processes, delinquent and inaccurate commercial and vendor payments,
                           and lax management of DOD’s travel card programs.6 Those deficiencies
                           have resulted in billions of dollars in unrecorded or improperly recorded
                           disbursements, over- and underpayments or late payments to contractors,
                           and fraudulent or unpaid travel card transactions.



Payment Recording Errors   DOD’s disbursement processes are complex and error-prone. Although
                           DFAS is responsible for providing accounting services for DOD, military
                           service and other defense agency personnel play a key role in DOD’s
                           disbursement process. In general, military service and defense agency
                           personnel obligate funds for the procurement of goods and services,
                           receive those goods and services, and forward obligation information and
                           receiving reports to DFAS. Separate DFAS disbursing offices and
                           accounting offices then pay the bills and match the payments to obligation
                           information. Several military services and DOD agencies can be involved
                           in a single disbursement and each has differing financial policies,
                           processes, and stand-alone, nonstandard systems. As a result, millions of
                           disbursement transactions must be keyed and rekeyed into the vast
                           number of systems involved in any given DOD business process. Also,
                           transactions must be recorded using an account coding structure that can
                           exceed 75 digits and this coding structure often differs—in terms of the
                           type, quantity, and format of data required—by military service. DFAS’s
                           ability to match disbursements to obligation records is complicated by the
                           fact that DOD’s numerous financial systems may contain inconsistent or
                           missing information about the same transaction. Input errors by DFAS or
                           service personnel and erroneous or missing obligation documents are two
                           of the major causes of inconsistent information.




                           6
                             See the “Related Audit Reports and Testimonies” section for a list of GAO and DOD
                           Inspector General reports related to payment recording errors, commercial payment
                           problems, and travel card program deficiencies.




                           Page 7                                                 GAO-03-457 DOD's Metrics Program
For calculating and reporting performance metrics related to payment
recording errors, officials from the Comptroller’s office included the
following categories.

• Unmatched disbursements—Payments that were made by a DFAS
  disbursing office and received by a DFAS accounting office but have not
  yet been matched to the proper obligation.

• Negative unliquidated obligations—Payments that have been matched
  to and recorded against the cited obligations but which exceed the
  amount of those obligations.

• Intransits—Payments that have not yet been received by the DFAS
  accounting office for recording and matching against the corresponding
  obligation.

• Suspense account transactions—Payments that cannot be properly
  recorded because of errors or missing information (e.g., transactions
  that fail system edit controls because they lack proper account coding)
  and are therefore temporarily put in a holding account until corrections
  can be made.

For DOD to know how much it has spent and/or how much is still available
for needed items, all transactions must be promptly and properly recorded.
However, we reported as early as 1990 that DOD was unable to fully
identify and resolve substantial amounts of payment recording errors. We
also stated that DOD’s early reporting of these errors significantly
understated the problems. For example, DFAS excluded $14.8 billion of
intransits from its 1993 benchmark against which it measured and reported
its progress in reducing recording problems in later years. In addition,
DOD excluded suspense account transactions from its reporting of
payment recording errors until as late as 1999. Finally, when negative
unliquidated obligations, intransits, and suspense account transactions
were reported, they were reported using net rather than absolute values.7




7
  Collections, reimbursements, and adjustments are offset against disbursements when net
amounts are reported. When absolute amounts are reported, collections, reimbursements,
and adjustments are added to disbursements. Reporting net amounts can significantly
understate the magnitude and impact of payment recording errors.




Page 8                                                GAO-03-457 DOD's Metrics Program
Commercial Payments   DFAS has overall responsibility for the payment of invoices related to
                      goods and services supplied by commercial vendors. As part of a
                      reorganization effort in April 2001, DFAS separated its commercial
                      payment services into two efforts—contract pay and vendor pay.

                      Contract pay handles invoices for formal, long-term contract instruments
                      that are typically administered by the Defense Contract Management
                      Agency (DCMA). These contracts tend to cover complex, multiyear
                      purchases with high dollar values, such as major weapon systems.
                      Payments for contracts are made from a single DFAS system—
                      Mechanization of Contract Administration Service (MOCAS). For fiscal
                      year 2001, DFAS disbursed about $78 billion for over 300,000 contracts
                      managed in MOCAS.

                      The vendor pay product line handles invoices for contracts not
                      administered by DCMA, plus miscellaneous noncontractual payments such
                      as utilities, uniforms/clothing, fuels, and food. Vendor pay is handled by 15
                      different systems throughout DFAS and, annually, DFAS personnel pay
                      nearly 10 million vendor invoices in excess of $70 billion. In general, DOD
                      makes vendor payments only after matching (1) a signed contractual
                      document, such as a purchase order, (2) an obligation, (3) an invoice, and
                      (4) a receiving report. If any one of these components is missing, such as
                      an obligation not being entered into the payment system, payment of the
                      invoice will be delayed. According to DOD officials, approximately 80
                      percent of payment delinquencies are due to the delayed receipt of
                      receiving reports by DFAS from the military service activities.



Travel Cards          DOD implemented the current travel card program in November 1998,
                      through a DOD task order with Bank of America. This was in response to
                      the Travel and Transportation Reform Act of 1998 (P.L. 105-264), which
                      modified the existing DOD Travel Card Program by mandating that all
                      government personnel must use the government travel card to pay official
                      travel costs (for example, hotels, rental cars, and airfare) unless
                      specifically exempted. The travel card can also be used for meals and
                      incidental expenses or to obtain cash from an automatic teller machine.
                      The intent of the travel card program was to provide increased
                      convenience to the traveler and lower the government’s cost of travel by
                      reducing the need for cash advances to the traveler and the administrative
                      workload associated with processing/reconciling travel advances.




                      Page 9                                         GAO-03-457 DOD's Metrics Program
DOD’s travel card program, which is serviced through Bank of America,
includes both individually billed accounts and centrally billed accounts.
When the travel card is submitted to a merchant, the merchant will process
the charge through its banking institution, which in turn charges Bank of
America. At the end of each banking cycle (once each month), Bank of
America prepares a billing statement that is mailed to the cardholder (or
account holder) for the amounts charged to the card. The statement also
reflects all payments and credits made to the account. For both individual
and centrally billed accounts, Bank of America requires that the cardholder
make payment on the account in full within 30 days of the statement
closing date. If the cardholder—individual or agency—does not pay the
monthly billing statement in full and does not dispute the charges within 60
days of the statement closing date, the account is considered delinquent.

For individually billed accounts, within 5 business days of return from
travel, the cardholder is required to submit a travel voucher claiming
legitimate and allowable expenses, which must be reviewed and approved
by a supervisor. DOD then has 30 days in which to make reimbursement.
Although DOD, like other agencies, relies on its employees to promptly pay
their individually billed accounts, DOD does have some tools to monitor
travel card activity and related delinquencies, including Bank of America’s
Web-based Electronic Account Government Ledger System (EAGLS).
Using EAGLS, supervisors can obtain reports on their cardholders’
transaction activity and related payment histories. For the centrally billed
accounts, the travel office at each military installation or defense agency
must first reconcile the charges shown on the centrally billed travel charge
card account with the office’s internal records of transportation requests.
After reconciliation has been completed, the voucher is sent to DFAS for
payment.

Because the travel card program is fairly new, DOD does not have a long
history of reporting statistics for delinquencies. However, in our previous
reports and testimonies, we have reported that DOD’s individually billed
delinquency rate is higher than that of other federal agencies. As of
September 2002, DOD’s delinquency rate was approximately 7.3 percent,
about 3 percent higher than other federal agencies. Among the military
services, however, the Air Force had the lowest delinquency rate. As of
September 2002, the Air Force delinquency rate was 4.8 percent,
significantly lower than the rest of DOD. Even though the Air Force had
lower numbers of delinquent accounts, we found that control environment
weaknesses and breakdowns in key controls were departmentwide and




Page 10                                       GAO-03-457 DOD's Metrics Program
                       that these deficiencies led to instances of potential fraud and abuse with
                       the use of travel cards in all the military services.



Performance Metrics    In 1998, DFAS developed its Performance Contract to focus on continued
Programs               achievement of its mission to provide responsive, professional finance and
                       accounting services to DOD. As part of this contract with DOD, DFAS
                       defined its performance objectives and identified specific performance
                       measurement indicators. DFAS managers—and sometimes staff—are
                       rated and rewarded based on their ability to reach annual reduction goals
                       for each indicator. Performance metrics are now calculated monthly and
                       the DFAS Director and the DOD Comptroller regularly review the results.

                       Section 1008 of the National Defense Authorization Act for Fiscal Year 1998
                       (P.L. 105-85) directed the Secretary of Defense to submit a biennial
                       strategic plan for the improvement of financial management to the
                       Congress. In conjunction with the plan, the DOD Comptroller decided to
                       develop a performance measurement system—a set of departmentwide
                       metrics that will provide clear-cut goals for financial managers to monitor
                       their progress in achieving reform. To begin this effort, the Comptroller
                       adopted many of the DFAS performance measurement indicators because
                       the DFAS metrics program had been underway for some time and was
                       reporting successes. For payment recording errors and commercial
                       payment backlogs in particular, the Comptroller’s metrics used information
                       gathered and tracked by DFAS for its performance management contract.

                       The metrics cited in the Comptroller’s testimony represent only a few of the
                       financial management performance metrics developed to date. From a
                       comprehensive set, the detailed metrics will be rolled up into “dashboard”
                       metrics that will provide the Secretary of Defense and the Congress with a
                       quick measure of DOD’s status in relation to critical financial management
                       goals. This effort is part of an even larger effort by DOD to develop
                       programmatic metrics for all of its operations.



Cited Metrics Were     In general, the definitions and methodologies for gathering the data used by
                       DOD Comptroller officials to calculate the cited improvement percentages
Generally Based on     at the ending measurement date were either consistent with or better than
Improved Definitions   those used at the beginning measurement date or for prior reporting on
                       payment recording errors, commercial payment backlogs, and travel card
and Methodologies      payment delinquencies. We did find that the reported metrics overstated



                       Page 11                                       GAO-03-457 DOD's Metrics Program
                           the rate of improvement in some areas because Comptroller officials
                           included transactions that DFAS would not consider to be payment errors
                           or because they chose an inappropriate comparison to measure travel card
                           delinquencies. However, recalculation of the metrics after correcting for
                           these factors still showed positive—although less dramatic—improvement
                           trends.



Payment Recording Errors   DOD has gradually improved its reporting of payment recording errors over
                           the years. DOD is now including all known categories of payment errors—
                           unmatched disbursements, negative unliquidated obligations, intransits,
                           and suspense account transactions—in its definition and, except in the
                           case of intransits, is using absolute rather than net amounts in its
                           calculations. However, the reporting of payment recording errors may not
                           be complete. For example, work that we have performed on closed DOD
                           accounts and on unliquidated obligations8 indicates that recording errors
                           are not always identified or resolved appropriately. DFAS agrees that to
                           properly manage and improve its payment processes, it must have a
                           complete universe of payment recording errors. Therefore, DFAS
                           personnel are currently working to determine whether the error categories
                           identified to date contain all of the relevant transactions and whether other
                           error categories exist.

                           While the same basic methodologies were used for calculating the cited
                           metrics at the beginning and ending measurement dates, Comptroller
                           officials overstated DOD’s improvement percentages because the October
                           2000 calculation included transactions that did not meet the DFAS criteria
                           for being considered payment errors while the October 2001 calculation did
                           not include them. First, the October 2000 calculation for payment
                           recording errors included all transactions that were being held in DFAS
                           suspense accounts; however, DFAS uses certain suspense accounts to
                           record collection transactions, such as accrued payroll taxes and receipts
                           for the sale of military property, that are held temporarily before being
                           distributed to the proper government agency or DOD entity. The
                           transactions in these accounts, which DFAS labels as “exempt suspense
                           accounts,” do not represent payment recording errors. In fiscal year 2001,


                           8
                             U.S. General Accounting Office, Defense Budget: Improved Reviews Needed to Ensure
                           Better Management of Obligated Funds, GAO-03-275 (Washington, D.C.: Jan. 30, 2003); U.S.
                           General Accounting Office, Canceled DOD Appropriations: $615 Million of Illegal or
                           Otherwise Improper Adjustments, GAO-01-697 (Washington, D.C.: July 26, 2001).




                           Page 12                                               GAO-03-457 DOD's Metrics Program
DFAS Cleveland changed its practice of charging payroll taxes to suspense
accounts and began appropriately accruing taxes in an accrued payroll tax
account. As a result, payment recording errors as calculated by
Comptroller officials at October 2001 were reduced by an estimated $7.5
billion—the amount of DFAS Cleveland’s accrued payroll taxes—even
though payment processes were not improved at all.

Second, in fiscal year 2001, DFAS Indianapolis corrected a reporting error
by a defense agency that had been double-counting transactions in its
suspense accounts. This resulted in an estimated $1.1 billion reduction
from amounts reported in October 2000, even though no payment
recording errors were corrected or resolved.

In addition, Comptroller officials measured intransits using net rather than
absolute values and did not adopt DFAS criteria for aging intransit and
suspense account transactions. These practices affected the balances used
to calculate the metrics at both the beginning and ending measurement
dates. First, net rather than absolute values were used to calculate
intransits at October 2000 and October 2001, which understated both
balances by approximately $4 billion. When net amounts are reported,
collections, reimbursements, and adjustments are offset against
disbursements, thus reducing the balance of intransit transactions.
Second, the reported metrics included all intransit and suspense account
transactions at October 2000 and October 2001 regardless of their age.
However, DOD allows 60 days to 180 days9 for the normal processing of
various payment transactions because of systems limitations and the
complexity of the department’s processes and, in line with these criteria,
DFAS’s metrics related to payment errors only consider aged intransit and
suspense account transactions. By not using DFAS’s criteria for aged
intransit and suspense account transactions, the Comptroller officials
overstated the balances of payment recording errors by approximately $6
billion at the beginning and $5 billion at the ending measurement dates.

Figure 1 illustrates the effect on improvement rates of (1) eliminating
exempt suspense accounts and double counting, (2) using DFAS’s criteria
for aged intransits and suspense amounts, and (3) using absolute rather

9
  According to DOD’s Financial Management Regulation (FMR), intraservice transactions
should not remain in transit for more than 60 days and interservice and interfund
transactions for more than 120 days. For suspense accounts, the DOD FMR allows 60 days
for clearing most transactions and 180 days for interfund transfers. DOD has stated that the
normal processing time for most transactions is 60 days.




Page 13                                                 GAO-03-457 DOD's Metrics Program
than net amounts for intransits. Our recalculation shows an overall 46
percent reduction in payment recording errors between October 2000 and
October 2001 rather than the 57 percent reduction reported by the
Comptroller; however, the reductions are still significant and the trend is
still overwhelmingly positive. Between October 2001 and September 2002,
DOD continued to report that it had reduced payment recording errors.
Comptroller officials calculated a 26 percent reduction during that period
while our recalculation shows a 22 percent reduction.



Figure 1: Reductions in Payment Recording Errors Between October 2000 and
September 2002
Dollars in billions
30
       27.20

25


20


15
                                                                  11.78

      8.74                                                                                              8.66
10

                                                                    4.68
                                                                                                       3.63
 5


 0

     Oct. 2000                                                   Oct. 2001                           Sept. 2002
               Comptroller’s calculation
               GAO’s calculation
Source: DOD Comptroller officials’ and GAO’s calculations using DFAS data.




Page 14                                                                      GAO-03-457 DOD's Metrics Program
Commercial Payment          The metrics for commercial payment backlogs (delinquent unpaid
Backlogs                    invoices) at April 2001 and October 2001 were calculated using consistent
                            definitions and methodologies. An invoice was considered delinquent if
                            payment was not made within the time frame established by the contract
                            terms (e.g., by the 15th day after the invoice date) or, if no time frame was
                            specified, on or before the 30th day after a proper invoice was received.10
                            DFAS reported information on delinquent invoices to Comptroller officials
                            monthly using standardized input sheets. The total backlog percentages
                            were then calculated by dividing the number of delinquent invoices
                            outstanding by the total number of invoices on hand.

                            According to the DOD Comptroller’s metrics, delinquent invoices for
                            vendor pay decreased by 41 percent from April 2001 through October 2001
                            while delinquent invoices for contract pay decreased by 32 percent during
                            that same period. Because DFAS officials stated that the decrease cited in
                            the Comptroller’s metrics was primarily due to intensive focus placed on
                            decreasing the backlog of delinquent vendor invoices, our review
                            concentrated on vendor pay issues.



Travel Card Delinquencies   For the travel card metrics, consistent definitions and methodologies were
                            used to gather the data and calculate the improvement percentages cited by
                            the DOD Comptroller for January 2001 and December 2001. Travel card
                            payments were considered delinquent if they were not paid within 60 days
                            of the monthly statement closing date. Even though the terms of the travel
                            cardholder’s agreement with Bank of America requires payment of the
                            statement within 30 days of the statement closing date, it is industry
                            practice to allow 60 days before the invoice is considered delinquent and
                            interest is charged. Comptroller officials used a standard industry practice
                            to calculate the travel card delinquency rates— the total dollar amount
                            outstanding for 60 days or more was divided by the total balance
                            outstanding.

                            While the definitions and methodology were consistent with standard
                            practices, the metrics comparison of delinquencies for individually billed


                            10
                              The principal guidance used for making payment to vendors is the Prompt Payment Act,
                            as implemented by the Office of Management and Budget (OMB) in 5 Code of Federal
                            Regulations (C.F.R.) Part 1315, “ Prompt Payment: Final Rule,” September 29, 1999. The
                            Prompt Payment Act requires the government to pay interest if an invoice is not paid on
                            time.




                            Page 15                                               GAO-03-457 DOD's Metrics Program
                           accounts in January to those in December could be misleading. As our
                           recent work shows, individually billed travel card delinquencies have been
                           cyclical, with the highest delinquencies occurring in January and February.
                           Therefore, the most useful metrics would compare same month to same
                           month, for example, January to January or December to December. If the
                           Comptroller officials had compared individual travel card delinquencies at
                           January 2001 to those at January 2002, the reported decrease would have
                           been 16 percent as opposed to 34 percent.



Most Cited Metrics Are     DFAS only provided us with internally generated summary-level data that
                           reconciled to the totals reported for payment recording errors and
Not Verifiable             commercial pay backlogs. DFAS did not provide us with detailed
                           transaction-level data that supported those metrics. As a result, we were
                           unable to test whether (1) all payment recording errors and delinquent
                           commercial payments were properly included in the metrics and (2) the
                           actions taken to resolve or correct payment recording errors were
                           appropriate. For individual and centrally billed travel card delinquencies,
                           we were able to obtain independent verification from a source outside
                           DOD that supported the Comptroller’s metrics.

                           Although we could not audit the reported metrics for all of the measured
                           areas, we verified that DFAS and other DOD organizations have made
                           numerous policy, procedure, and systems changes that would support an
                           overall trend toward improved performance. For payment recording errors
                           and commercial payment backlogs, perhaps the most significant change
                           has been DOD’s inclusion of performance measures in its contracts with
                           DFAS. The performance contract and an accompanying data dictionary
                           provide specific, measurable reduction goals, which DFAS management—
                           and in some cases staff—are held accountable for reaching. The resulting
                           focus has fostered innovative process and systems improvements as well
                           as better communication among the parties involved in preventing or
                           resolving these problems. For example, DFAS holds monthly
                           videoconferences with its centers and field sites to discuss progress and
                           any impediments to reaching that period’s goals.



Payment Recording Errors   In general, DFAS centers did not maintain history files of all the
                           transactions that were not promptly matched with obligations, created
                           negative unliquidated obligations, were in transit longer than allowable, or
                           were in suspense accounts during the period October 2000 through



                           Page 16                                        GAO-03-457 DOD's Metrics Program
October 2001—information that is necessary in order to verify the
completeness and accuracy of the reported metrics. DFAS officials
explained that the detailed data supporting the reported monthly totals are
compiled by hundreds of DFAS field sites using numerous accounting
systems and there is no specific requirement for the field sites to save the
data. While some DFAS officials believe that it would be possible to
recreate transaction-level detail to support month-end totals, the task
would be extremely onerous and time consuming.

Although we were unable to verify through audit procedures the accuracy
of the reductions reported by the Comptroller, we did reconcile summary-
level information provided by the DFAS centers to the metric amounts. We
also verified that DFAS has made numerous policy and systems
improvements that support a continuing trend of reductions in payment
recording errors as illustrated by the metrics in figure 2.



Figure 2: Reductions in the Number of Payment Recording Errors
Dollars in billions
10

 9

 8

 7

 6

 5

 4

 3

 2

 1

 0
       Oct. 2000               Oct. 2001           Sept. 2002

               Suspense

               Intransits

               Negative unliquidated obligations

               Unmatched disbursements

Source: GAO calculations using DFAS data.




Page 17                                                         GAO-03-457 DOD's Metrics Program
DFAS has been working to reduce payment recording errors for more than
a decade. In the late 1990s, DFAS consolidated most of its disbursing and
accounting functions from 300 defense accounting offices into 5 centers, in
large part to help streamline the payment recording process. DFAS has
also been working with other DOD components to consolidate or replace
about 250 outdated and nonintegrated financial and accounting systems.
While the systems effort will take many years and must be accomplished
within DOD’s overall plan for systems development and integration,11 DFAS
has made, and continues to make, improvements in the policies and
systems tools available to DFAS personnel for preventing and correcting
payment recording errors.

Since October 2000, DFAS has made several policy changes that have
affected the payment recording process. In January 2001, DOD revised its
official guidance12 to clarify and strengthen policies related to the prompt
(1) recording of disbursements and obligations and (2) resolution of
payment recording errors. If the military services or DOD components
have not provided DFAS with accurate obligation information within
specified time frames, the revision gave DFAS the authority to record
obligations in order to resolve individual unmatched disbursements,
negative unliquidated obligations, and certain suspense account
transactions. DFAS also expanded its prevalidation policy, which it claims
has been key to reducing payment errors associated with commercial
contracts. Prevalidation requires that DFAS personnel ascertain that there
is a valid obligation recorded in the accounting records before making a
payment. Between November 2000 and October 2001, DFAS lowered the
dollar threshold amount for transactions requiring prevalidation from
$100,000 to $25,000.

DFAS developed new systems tools13 for communicating accounting
information among its centers and field locations that have reduced the
amount of time DFAS personnel need to match disbursements to


11
  Section 1004 of the National Defense Authorization Act for Fiscal Year 2003 (P.L.107-314,
Dec. 2, 2002) requires the Secretary of Defense to develop a comprehensive financial
management enterprise architecture plan no later than May 1, 2003. It also requires the
preparation of a Transition Plan that will guide DOD in implementing its architecture plan,
and must include a schedule for terminating “legacy systems.”
12
     DOD 7000.14-R, Financial Management Regulation, Volume 3, Chapter 11.
13
  Appendix I discusses the limited scope of our review of systems tools and systems
improvements.




Page 18                                                 GAO-03-457 DOD's Metrics Program
                          obligations. For example, since the late 1990s DFAS has implemented the
                          following.

                          • Electronic data access capability, which provides web access to
                            contract, billing, and other documents pertinent to the payment
                            recording process. Electronic access to these documents enables users
                            to obtain information more quickly than in the past, when many
                            documents were stored in hard-copy format.

                          • Phase 1 of the Defense Cash Accountability System (DCAS), which
                            provides a standardized, electronic means for DFAS centers to report
                            expenditure data for transactions involving more than one military
                            service (cross-disbursements). Prior to DCAS, the centers had different
                            systems and formats for reporting this information to one another and
                            to Treasury, a situation that increased the complexity of recording and
                            matching cross-disbursements. According to DFAS officials, DCAS
                            reduced the cross-disbursement cycle time from 60 days to 10 days.

                          • The Standard Contract Reconciliation Tool (SCRT), which provides
                            DFAS personnel a consolidated database for researching commercial
                            contract records. Prior to SCRT, locating and accessing these records
                            was difficult due to the variety of accounting, contracting, and
                            entitlement systems involved.

                          DFAS centers have also developed individual applications that have
                          improved payment processes. For example, DFAS Indianapolis
                          implemented an Access “Wizard” application to automate the process of
                          matching intragovernmental expenditure transactions to obligation
                          records. The program also enables center staff to identify transactions that
                          have not been processed within 30 days so they can follow up with field
                          accounting personnel.



Commercial Pay Backlogs   DFAS was unable to provide detailed transaction-level data that supported
                          the metrics related to vendor payment backlogs—the most significant
                          contributor to the reductions. DFAS only maintained summary-level data
                          that were generated by the 23 DFAS field sites. Using standard definitions
                          and standard summary spreadsheets, DFAS personnel collected the
                          summary information monthly through data calls to the more than 15
                          different systems that track DOD vendor pay backlog information. As a
                          result, we were only able to confirm that the summary information
                          provided by DFAS reconciled to the amounts reported by the Comptroller.



                          Page 19                                       GAO-03-457 DOD's Metrics Program
We were unable to verify by audit the accuracy or completeness of that
data.

DFAS management has focused on reducing commercial payment backlogs
since fiscal year 2000 and this focus is continuing through the present.
According to its performance contracts, DFAS’s goal was to reduce the
backlog by 15 percent per year beginning in fiscal year 2000 from a baseline
of 48,000 delinquent invoices. In April 2001, DFAS centralized operational
control of contract pay and vendor pay under one executive, who was
given ultimate responsibility for meeting these performance goals.

DFAS also made site-specific procedural changes to reduce the backlog of
vendor payments. These included

• hiring temporary contract and permanent staff in key sites, such as
  Limestone and Dayton;

• forecasting when civilian employees in Europe would be taking vacation
  and then staggering vacation leave and/or hiring temporary help (e.g., in
  Germany, every civilian employee has 6 weeks of annual leave, which is
  usually taken during the summer); and

• forming partnerships with the military services and defense agencies to
  improve their processing time for receiving reports, since DFAS must
  match the receiving report to the invoice before payment can be made.

DFAS credits these and other changes for the continued reduction of the
backlog of delinquent invoices. Figure 3 below illustrates the trend in the
reduction of outstanding delinquent vendor invoices compared to the total
number of invoices on-hand.




Page 20                                       GAO-03-457 DOD's Metrics Program
Figure 3: Decrease in Vendor Pay Backlogs for the Period April 2001 through September 2002
Percent
12



10                                    9.82                   9.61

                                              9.25
 8
      6.84           6.74                                                                                           7.84
                                                                           6.68                            7.35
                                                                    6.71
 6
                              4.96                                                  5.88       5.67
                                     4.17
 4                                             3.91          3.62                    3.67
                                                                    3.42    3.65


 2



 0
     Jan.            Feb.   Mar.     Apr.     May            June   July   Aug.     Sept.      Oct.      Nov.       Dec.
     Months

                                                      2001
                                                      2002

Source: DFAS data.




Travel Card Delinquencies                    We were able to verify the reductions cited by the Comptroller in individual
                                             and centrally billed travel card delinquencies. We obtained travel card
                                             delinquency information from an independent source, the General Services
                                             Administration (GSA), that supported the Comptroller’s metrics. GSA
                                             receives information from individual travel card vendors, such as Bank of
                                             America, and prepares a monthly summary report for DOD that documents
                                             individual and centrally billed travel card delinquencies by military service
                                             or defense agency. We compared the GSA data to the cited metrics and
                                             verified that the reported reductions in travel card delinquencies were
                                             accurate. As with the other problem areas, DOD credits the decrease in
                                             travel card delinquency rates in both individual and centrally billed
                                             accounts primarily to increased management attention.

                                             For the centrally billed accounts, DOD has attributed the initial high
                                             delinquency rates to problems in transferring the travel card contract from
                                             American Express to Bank of America. When Bank of America was given
                                             the contract, its on-line travel information system, EAGLS, was not fully
                                             operational and therefore was unable to accurately process all of the travel



                                             Page 21                                         GAO-03-457 DOD's Metrics Program
                                                        data being transferred by American Express. Because EAGLS contained
                                                        incorrect account numbers, invoice information, and billing addresses,
                                                        DOD agency program coordinators did not have the information necessary
                                                        to determine which accounts were delinquent, in suspense, or canceled.
                                                        While DOD and Bank of America officials were working jointly to identify
                                                        and resolve the problems, centrally billed invoices became backlogged.
                                                        Once the problems were resolved, DOD was able to reduce the backlog. As
                                                        of December 31, 2002, DOD’s centrally billed delinquency rate was 1.5
                                                        percent, well below fiscal year 2002’s proposed goal of 3.0 percent and
                                                        equal to the delinquency rate for other federal agencies. Figure 4 below
                                                        shows the centrally billed delinquency rates from January 2001 through
                                                        December 2002.



Figure 4: Decrease in Centrally Billed Travel Card Delinquencies for the Period January 2001 through December 2002
Percent
16
      14.3
14


12
                        9.9
10


 8
                                           6.2    6.2
 6                                                          5.5
       4.8                                                           4.7
                                                                              4.3      4.1       4.2
 4                      2.6                                                                                              3
                                                                                                              2.3
                                                                                       1.9                                          2
 2                                                          1.7
                                                                                                 1.3                     1
                                           0.4                       0.8                                                           1.5
                                                 0.2                          0.4                              1.5
 0
      Jan.            Feb.             Mar.      Apr.      May     June       July    Aug.      Sept.         Oct.      Nov.       Dec.
      Months

                                                         2001
                                                         2002
Source: General Services Administration.




                                                        Page 22                                         GAO-03-457 DOD's Metrics Program
For individual travel cards, our recent work also supports the improved
delinquency rates being reported by DOD. During the past year, we
reported on the travel card programs for all three military services.14 In
general, we found that the military services, in particular the Air Force,
have given delinquencies greater attention and have used travel card audits
to identify problems and needed corrective actions. We reported that all of
the services are now holding commanders responsible for managing the
delinquency rates of their subordinates. For example, Air Force
management holds monthly command meetings where individual travel
card delinquencies are monitored and briefed. The individual services have
also implemented new programs to help reduce delinquencies, including
the following.

• In January 2003, the Army established two goals of not more than 4.5
  percent of dollars delinquent and not more than 3 percent of accounts
  delinquent. The Navy has established a similar goal of no more than 4
  percent delinquent accounts.

• The Air Force is providing financial training to all inductees that
  includes developing a personal budget plan, balancing a checkbook,
  preparing a tax return, and understanding financial responsibility. The
  training also covers the disciplinary actions and other consequences of
  financial irresponsibility by service members.

• The Navy has developed a three-pronged approach to address travel
  card issues: (1) provide clear procedural guidance to agency program
  coordinators (APCs) and travelers that is available on the Internet,
  (2) provide regular training to APCs, and (3) enforce the proper use and
  oversight of the travel card by using data mining to identify problem
  areas and abuses.

• In January 2003, the Army issued two directives to its major
  commanders, which address a range of policy requirements, to include:
  (1) training for APCs and cardholders, (2) monthly review of cardholder
  transactions, (3) exempting and/or discouraging the use of the card for
  en route travel expenses associated with deployments, and
  (4) prohibiting use of the card for travel expenses associated with
  permanent change of station moves.



14
     GAO-03-298; GAO-02-863T; GAO-03-148T.




Page 23                                       GAO-03-457 DOD's Metrics Program
In addition, DOD has implemented a number of departmentwide programs
to improve the individually billed travel card program. Beginning in
November 2001, DOD began a salary and military retiree pay offset
program for delinquencies—similar to wage garnishment. In March 2002,
the Comptroller created a Credit Card Task Force to address management
issues related to the purchase and individually billed travel card programs.
On July 19, 2002, the DOD Comptroller directed the cancellation of
(1) inactive travel charge card accounts, (2) active travel card accounts not
used in the previous 12 months, and (3) travel card accounts for which the
bank cannot identify the cardholders’ organization. DOD is also
encouraging individual cardholders to elect to have all or part of their
travel reimbursement sent directly by DFAS to Bank of America—a
payment method that is standard practice for many private sector
employers. The Congress has recently addressed this issue in section
1008(a) and (b) of the National Defense Authorization Act for Fiscal Year
2003,15 which provides the Secretary of Defense the authority to require use
of this payment method. According to DOD, about 32 percent of its
individually billed cardholders elected this payment option for fiscal year
2002.

As a result of these and other actions, DOD has been able to sustain
reduced delinquency rates between October 2002 and December 2002, as
illustrated in figure 5 below. However, DOD still needs to do more to
address the underlying causes of the problems with its travel card program.
In a recent testimony, we concluded that actions to implement additional
“front-end” or preventative controls are critical if DOD is to effectively
address the high delinquency rates and charge-offs, as well as potentially
fraudulent and abusive activity. 16 As a result of our work on travel cards,17
the Congress included a provision in the Department of Defense
Appropriations Act for Fiscal Year 200318 requiring the Secretary of Defense
to evaluate whether an individual is creditworthy before authorizing the
issuance of any government charge card. If this requirement is effectively



15
     P.L. 107-314, Dec. 2, 2002.
16
     GAO-02-863T.
17
  See the “Related Audit Reports and Testimonies” section for a list of our reports and
testimonies related to DOD travel cards.
18
     P.L. 107-248, Oct. 23, 2002, Sec. 8149.




Page 24                                                 GAO-03-457 DOD's Metrics Program
                                                        implemented, DOD should continue to improve delinquency rates and
                                                        reduce potential fraud and abuse.



Figure 5: Decrease in Individually Billed Travel Card Delinquencies for the Period January 2001 through December 2002
Percent
20
    18.4
18

16 15.4

14                                                                                                                   12.7
                        12.4                                                                              12.2                  12.2
12
                 11.7
                                           8.7                                                9.4
10                                                                                    8.7
                                                                             8.2
                                     8.9                               7.8
 8                                                6.5    6.4                 8.1      8.5                             8.1       8.3
                                                                       7.8                                 7.9
                                                                                               7.3
 6                                               6.9       7

 4

 2

 0
     Jan.           Feb.             Mar.        Apr.     May         June   July     Aug.     Sept.       Oct.      Nov.       Dec.
     Month


                                                               2001
                                                               2002
Source: General Services Administration.




Cited Metrics Serve                                     The metrics that the DOD Comptroller highlighted in the March 2002
                                                        hearing relate to areas that have received considerable congressional and
Important Purpose But                                   audit attention. As discussed earlier, the metrics program increased
Further Steps Are                                       management focus on these problem areas and led to improvements in
                                                        policies, processes, and—in a limited way—systems. While some of the
Needed to Sustain                                       cited metrics could be effective indicators of short-term financial
Improvements                                            management progress, assuming they could be verified, others are not
                                                        necessarily good indicators, particularly if taken alone. In addition,
                                                        continued financial management progress will require additional actions.
                                                        For example, the military services and other defense agencies are key
                                                        contributors to preventing and resolving payment recording errors and
                                                        commercial payment delinquencies but they do not have the same
                                                        incentives to improve their performance in these areas. Also, because
                                                        DFAS lacks modern, integrated financial management systems, preventing



                                                        Page 25                                        GAO-03-457 DOD's Metrics Program
                            and resolving payment delinquencies and errors require intensive effort day
                            after day by DFAS and other DOD organizations, which could be difficult to
                            sustain.



Effectiveness of Measures   The cited metrics for individual travel card delinquencies and payment
                            recording errors could be effective indicators of financial management
                            improvement. For payment recording errors, continuing reductions would
                            indicate better controls over obligation, disbursement, and collection
                            processes and that, as a result, DOD is less prone to fraud, waste, or abuse
                            of appropriated funds. Monitoring the delinquency rates for individual
                            travel card payments would provide DOD with an early indication that
                            employees may be abusing their cards (i.e., using the cards for personal
                            purchases) or having credit problems.

                            However, improved delinquency rates do not necessarily indicate improved
                            financial management of centrally billed travel cards or commercial
                            payments. In fact, by placing too much emphasis on paying bills promptly,
                            DOD staff may be tempted to shortcut important internal control
                            mechanisms that are meant to ensure that the goods and services being
                            paid for were properly authorized and actually received. We and DOD
                            auditors have issued several reports on the improper use of individually
                            billed travel cards at DOD and on over- and underpayments to DOD
                            contractors but are just beginning work to identify and evaluate the
                            adequacy of DOD policies, procedures, and controls related to purchases
                            from vendors and centrally billed travel cards. As a result of these audits,
                            we will likely recommend additional metrics related to program
                            performance and internal controls for monitoring performance in these
                            areas.




                            Page 26                                       GAO-03-457 DOD's Metrics Program
Sustainability of   Measures such as the ones discussed in this report may be useful in the
Improvements        short term but may not be appropriate once DFAS has reengineered its
                    business processes and modernized its systems. As DFAS and the military
                    services develop integrated and/or interfaced financial management
                    systems, many of the problems related to transaction recording errors
                    should be eliminated. Based on the recent work we performed for your
                    committee related to DOD’s enterprise architecture,19 however, these new
                    systems are years away from implementation.

                    Because DFAS lacks modern, integrated financial management systems,
                    preventing and resolving payment delinquencies and errors require
                    intensive effort day after day by DFAS and military service staff. As a
                    result, DFAS has indicated that much of the reported progress to date is
                    sustainable only if its workload is not significantly increased or its staffing
                    significantly decreased.

                    Until new systems and reengineered processes are in place, DOD can take
                    a number of steps to help maintain improvements in these areas. First,
                    continued leadership and focus by top management will be a major factor
                    in the sustainability of progress made to date. Second, because DFAS alone
                    cannot resolve DOD’s payment recording problems or payment
                    delinquencies, integrated metrics programs across DOD will be important.
                    As noted earlier in this report, while the military services and other defense
                    agencies play key roles in obligating DOD funds, preparing obligation
                    documents, receiving and preparing billing documents, preparing receiving
                    reports, and recording transaction information into accounting systems,
                    these organizations do not currently have complementary metrics
                    programs. Thus the military services and defense agencies are not
                    measured on the accuracy and timeliness of their payment processes even
                    though their assistance is necessary for DFAS to make improvements and
                    resolve problems. For example, commercial payment backlogs were
                    largely due to failure by the military services in providing receiving reports
                    to DFAS, yet service delays were not being measured.

                    DOD is currently developing a departmentwide, balanced program of
                    metrics that is intended to align with its strategic goals, focus on results,
                    and achieve auditable reports. As contemplated, DFAS, the military

                    19
                      U.S. General Accounting Office, DOD Business Systems Modernization: Improvements
                    to Enterprise Architecture Development and Implementation Efforts Needed, GAO-03-458
                    (Washington, D.C.: Feb. 28, 2003).




                    Page 27                                            GAO-03-457 DOD's Metrics Program
                  services, and other defense agencies will all be supporting players in this
                  program. From the individual performance measurement programs of the
                  military services, defense agencies, and DFAS, certain metrics will be
                  selected and reported to the top levels of DOD management for evaluation
                  and comparison. In this scenario, it is important that DOD properly and
                  consistently calculate and report the selected metrics and that the military
                  services, other agencies, and DFAS develop integrated metrics programs to
                  assist in identifying, measuring, and resolving crosscutting issues.



Conclusion        As the cited metrics demonstrate, DOD can make meaningful, short-term
                  progress toward better financial management while waiting for long-term
                  solutions, such as integrated financial systems. Leadership, real incentives,
                  and accountability—hallmarks of a good performance measurement
                  program—have brought about improvements in DFAS policies and
                  processes. The cited metrics are also serving as important building blocks
                  for DOD’s current efforts to develop a departmentwide performance
                  measurement system for financial management. However, before the
                  payment recording error and commercial payment backlog metrics can be
                  relied upon for decision-making purposes, they must be

                  • verifiable,

                  • properly defined and correctly measured, and

                  • linked to the goals and performance measures of other relevant DOD
                    organizations.

                  In addition, because the reported improvements depend heavily on the day-
                  to-day effort of DFAS staff, sustaining the progress may be difficult if DFAS
                  has significant workload increases or staff decreases.



Recommendations   DOD systems do not provide the transaction-level support needed to verify
                  the accuracy and completeness of many of its selected metrics. However,
                  because DOD is currently working on developing an enterprisewide system
                  architecture to guide its future systems development and implementation
                  strategies, we are not making any recommendations in this report related
                  to improving the underlying business systems. We did identify several
                  steps that DOD could take now to improve the reported metrics. We are
                  recommending that the DOD Comptroller



                  Page 28                                        GAO-03-457 DOD's Metrics Program
                      • use definitions and criteria that are consistent with DFAS definitions
                        and criteria when calculating and reporting metrics related to payment
                        recording errors,

                      • measure improvements in individually billed travel card delinquencies
                        by using same month to same month comparisons, and

                      • work with the military service Assistant Secretaries for Financial
                        Management to develop performance measures for the military services
                        and other defense agencies in areas for which there is shared
                        responsibility, in order to complement the DFAS metrics program.



Agency Comments and   In written comments on a draft of this report (see appendix II), the Under
                      Secretary of Defense (Comptroller) stated that the department concurred
Our Evaluation        with our recommendations and described actions to address them. The
                      department also provided several technical comments, which we have
                      incorporated in the report as appropriate.


                      We are sending copies of this report to other interested congressional
                      committees; the Secretary of Defense; the Under Secretary of Defense
                      (Comptroller); the Director, Defense Finance and Accounting Service; and
                      the Assistant Secretaries for Financial Management (Comptroller) for the
                      Army, the Navy, and the Air Force. Copies will be made available to others
                      upon request.

                      Please contact me at (202) 512-9505 or kutzg@gao.gov if you or your staff
                      have any questions about this report. Other GAO contacts and key
                      contributors to this report are listed in appendix III.

                      Sincerely yours,




                      Gregory D. Kutz
                      Director
                      Financial Management and Assurance


                      Page 29                                       GAO-03-457 DOD's Metrics Program
Appendix I

Objectives, Scope, and Methodology                                                                 AA
                                                                                                    ppp
                                                                                                      ep
                                                                                                       ned
                                                                                                         n
                                                                                                         x
                                                                                                         id
                                                                                                          e
                                                                                                          x
                                                                                                          Iis




              As requested by the Chairman and Ranking Minority Member of the
              Subcommittee on Readiness and Management Support, Senate Committee
              on Armed Services, we undertook an assessment of the consistency,
              accuracy, and effectiveness of certain DOD-reported metrics related to
              payment recording errors, commercial payment backlogs, and delinquent
              travel card payments. Specifically, our objectives were to determine
              whether (1) the cited performance measures were applied and calculated
              in a manner consistent with previous reporting on payment delinquencies
              and recording errors, (2) the cited improvement data were properly
              supported and represent real improvements in performance, and (3) the
              metrics are effective indicators of short-term financial management
              progress.

              To complete this work, we visited DOD Comptroller offices and DFAS
              centers in Arlington, Cleveland, Columbus, Indianapolis, and Denver where
              we did the following.

              • Gathered, analyzed, and compared information on how payment
                recording errors, commercial payment backlogs, and travel card
                delinquencies were defined, calculated, and reported both in the past
                and for the cited metrics.

              • Reviewed GAO, DOD IG, and other service auditors’ reports for the past
                10 years.

              • Reviewed DOD consolidated financial statement reporting of payment
                recording errors over the last 10 years.

              • Reviewed DOD policy for maintaining financial control over
                disbursement, collection, and adjustment transactions. This policy1
                specifically describes the requirements for researching and correcting
                payment recording errors.

              • Obtained and analyzed the underlying summary spreadsheets from
                DFAS that were the information source for the Comptroller officials’
                calculations for payment recording errors and commercial pay
                backlogs. DFAS gathers this information monthly through data calls
                from numerous systems used to process and account for payments.
                Although we requested the underlying detailed transaction-level data


              1
                  DOD 7000.14-R, Financial Management Regulation, Volume 3, Chapter 11.




              Page 30                                               GAO-03-457 DOD's Metrics Program
Appendix I
Objectives, Scope, and Methodology




   supporting the spreadsheets so that we could perform audit tests, we
   were unable to obtain the detail-level data.

• Obtained and analyzed the underlying summary spreadsheets from
  DFAS that were the information source for the Comptroller officials’
  calculations for travel card delinquencies. Obtained independent
  summary data for travel card delinquencies from GSA and compared
  amounts to Comptroller-reported metrics.

• Interviewed center personnel about process and system improvements
  and gathered and analyzed relevant output that demonstrated the
  results of those changes. Our review of new systems tools and
  purported systems improvements was limited: we did not validate
  whether systems changes followed appropriate requirements or
  whether they resulted in the production of reliable financial
  information.

• Obtained explanations from officials from the Office of the Secretary of
  Defense regarding the metrics program and assessed whether the cited
  metrics are effective indicators of short-term financial management
  progress.

The data in this report are based on DFAS records. With the exception of
travel card delinquency rates, we were unable to independently verify or
audit the accuracy of these data. We performed our work from June 2002
to February 2003 in accordance with U.S. generally accepted government
auditing standards.

We received written comments on a draft of this report from the Under
Secretary of Defense (Comptroller). These comments are presented and
evaluated in the “Agency Comments and Our Evaluation” section and are
reprinted in appendix II. We considered technical comments from the
department and incorporated them as appropriate but did not reprint them.




Page 31                                      GAO-03-457 DOD's Metrics Program
Appendix II

Comments from the Under Secretary of
Defense                                                       Appendx
                                                                    Ii




              Page 32          GAO-03-457 DOD's Metrics Program
Appendix II
Comments from the Under Secretary of
Defense




Page 33                                GAO-03-457 DOD's Metrics Program
Appendix III

GAO Contact and Acknowledgments                                                               Appendx
                                                                                                    iI




GAO Contact       Molly B. Boyle (202) 512-9524



Acknowledgments   Staff making key contributions to this report were Rathi Bose, Steve
                  Donahue, Diane Handley, Fred Jimenez, and Carolyn Voltz.




                  Page 34                                      GAO-03-457 DOD's Metrics Program
Related Audit Reports and Testimonies



Payment Recording Errors   Defense Budget: Improved Reviews Needed to Ensure Better Management
                           of Obligated Funds. GAO-03-275. Washington, D.C.: January 30, 2003.

                           Canceled DOD Appropriations: Improvements Made but More Corrective
                           Actions Are Needed. GAO-02-747. Washington, D.C.: July 31, 2002.

                           Canceled DOD Appropriations: $615 Million of Illegal or Otherwise
                           Improper Adjustments. GAO-01-697. Washington, D.C.: July 26, 2001.

                           Department of Defense Office of the Inspector General. Trends and
                           Progress in Reducing Problem Disbursements and In-Transit
                           Distribution. Report No. 99-135. Arlington, Va.: April 16, 1999.

                           Financial Management: Problems in Accounting for Navy Transactions
                           Impair Funds Control and Financial Reporting. GAO/AIMD-99-19.
                           Washington, D.C.: January 19, 1999.

                           Financial Management: Improved Reporting Needed for DOD Problem
                           Disbursements. GAO/AIMD-97-59. Washington, D.C.: May 1, 1997.

                           DOD Problem Disbursements: Contract Modifications Not Properly
                           Recorded in Payment System. GAO/AIMD-97-69R. Washington, D.C.:
                           April 3, 1997.

                           Financial Management: Improved Management Needed for DOD
                           Disbursement Process Reforms. GAO/AIMD-97-45. Washington, D.C.:
                           March 31, 1997.

                           Financial Management: Status of Defense Efforts to Correct
                           Disbursement Problems. GAO/AIMD-95-7. Washington, D.C.: October 5,
                           1994.

                           Financial Management: Problems in Accounting for DOD
                           Disbursements. GAO/AFMD-91-9. Washington, D.C.: November 9, 1990.



Commercial Payment         Department of Defense Office of the Inspector General. Controls Over the
Backlogs                   Computerized Accounts Payable System at Defense Finance and
                           Accounting Service Columbus. Report No. D-2002-113. Arlington, Va.: June
                           21, 2002.




                           Page 35                                     GAO-03-457 DOD's Metrics Program
                            Related Audit Reports and Testimonies




                            Department of Defense Office of the Inspector General. Closing Overage
                            Contracts Prior to Fielding a New DOD Contractor Payment System.
                            Report No. D-2002-027. Arlington, Va.: December 19, 2001.

                            Department of Defense Office of the Inspector General. Controls Over
                            Vendor Payments Made for the Army and Defense Agencies Using the
                            Computerized Accounts Payable System. Report No. D-2002-056.
                            Arlington, Va.: March 6, 2002.

                            Contract Management: Excess Payments and Underpayments Continue
                            to Be a Problem at DOD. GAO-01-309. Washington, D.C.: February 22, 2001.

                            Internal Controls: Reporting Air Force Vendor Payment System
                            Weaknesses Under the Federal Managers’ Financial Integrity Act.
                            GAO/AIMD-99-33R. Washington, D.C.: December 21, 1998.

                            Financial Management: Improvements Needed in Air Force Vendor
                            Payment Systems and Controls. GAO/AIMD-98-274. Washington, D.C.:
                            September 28, 1998.

                            Contract Management: Fixing DOD’s Payment Problems is Imperative.
                            GAO/NSIAD-97-37. Washington, D.C.: April 10, 1997.

                            Department of Defense Office of the Inspector General. Vendor
                            Payments - Operation Mongoose Fort Belvoir Defense Accounting Office
                            and Rome Operating Location. Report No. 97-052. Arlington, Va.:
                            December 23, 1996.

                            High Risk Series: Defense Contract Management. GAO/HR-95-3.
                            Washington, D.C.: February 1995.



Travel Card Delinquencies   Travel Cards: Control Weaknesses Leave Navy Vulnerable to Fraud and
                            Abuse. GAO-03-147. Washington, D.C.: December 23, 2002.

                            Travel Cards: Air Force Management Focus Has Reduced Delinquencies,
                            but Improvements in Controls Are Needed. GAO-03-298. Washington, D.C.:
                            December 20, 2002.

                            Travel Cards: Control Weaknesses Leave Army Vulnerable to Potential
                            Fraud and Abuse. GAO-03-169. Washington, D.C.: October 11, 2002.




                            Page 36                                     GAO-03-457 DOD's Metrics Program
           Related Audit Reports and Testimonies




           Travel Cards: Control Weaknesses Leave Navy Vulnerable to Fraud and
           Abuse. GAO-03-148T. Washington, D.C.: October 8, 2002.

           Travel Cards: Control Weaknesses Leave Army Vulnerable to Potential
           Fraud and Abuse. GAO-02-863T. Washington, D.C.: July 17, 2002.

           Department of Defense Office of the Inspector General. Acquisition:
           Summary of DOD Travel Card Program Audit Coverage. Report No. D-
           2002-065. Arlington, Va.: March 18, 2002.




(192060)   Page 37                                    GAO-03-457 DOD's Metrics Program
GAO’s Mission            The General Accounting Office, the audit, evaluation and investigative arm of
                         Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal government
                         for the American people. GAO examines the use of public funds; evaluates federal
                         programs and policies; and provides analyses, recommendations, and other
                         assistance to help Congress make informed oversight, policy, and funding
                         decisions. GAO’s commitment to good government is reflected in its core values of
                         accountability, integrity, and reliability.


Obtaining Copies of      The fastest and easiest way to obtain copies of GAO documents at no cost is
                         through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail this
                         list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO
                         Mailing Lists” under “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A check
                         or money order should be made out to the Superintendent of Documents. GAO
                         also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single
                         address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice: (202) 512-6000
                                                TDD: (202) 512-2537
                                                Fax: (202) 512-6061


To Report Fraud,         Contact:
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470



Public Affairs           Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
                         U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548
United States                  Presorted Standard
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300
Address Service Requested