oversight

Federal Telework: Program Measurement Continues to Confront Data Reliability Issues

Published by the Government Accountability Office on 2012-04-19.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States Government Accountability Office

GAO          Report to the Subcommittee on Oversight
             of Government Management, the Federal
             Workforce, and the District of Columbia,
             Committee on Homeland Security and
             Governmental Affairs, U.S. Senate
April 2012
             FEDERAL
             TELEWORK
             Program Measurement
             Continues to Confront
             Data Reliability Issues




GAO-12-519
                                               April 2012

                                               FEDERAL TELEWORK
                                               Program Measurement Continues to Confront Data
                                               Reliability Issues
Highlights of GAO-12-519, a report to the
Subcommittee on Oversight of Government
Management, the Federal Workforce, and the
District of Columbia, Committee on Homeland
Security and Governmental Affairs, U.S.
Senate

Why GAO Did This Study                         What GAO Found
The Telework Enhancement Act of                To prepare for its reporting obligations under the Telework Enhancement Act of
2010 requires OPM to report to                 2010, the Office of Personnel Management (OPM) assembled the Interagency
Congress on the degree of telework             Telework Measurement Group, consisting of officials from several federal
participation at executive agencies. To        agencies, to assist in revising the telework data call—the survey OPM has used
meet this requirement, OPM collects            since 2002 to collect telework data from executive agencies. This group
information on agency telework                 standardized key terms such as telework, employee, and eligibility to promote a
programs through an annual survey,             common reporting methodology among the agencies. The revised telework data
which it refers to as the data call.           call also included changes to the time period for which OPM requested agencies
However, concerns exist about the              report telework data, and included more extensive training for respondents.
reliability of these data.
                                               Because of changes made to the data call to allow OPM to meet requirements of
GAO was asked to assess OPM’s:                 the act and assist agencies in responding to the data call, OPM officials believe
(1) actions in response to the                 they will be able to provide to Congress an improved report on telework in June
requirements of the act and                    2012. However, these changes also mean that OPM officials will not be able to
(2) handling of identified data reliability
                                               use participation and frequency data from the 2011 data call to compare to data
issues in the 2011 telework data call.
                                               from previous years and across agencies. OPM officials have noted that this
To address these objectives GAO
reviewed its previous reports
                                               could limit OPM’s ability to report agency progress in its first report to Congress.
addressing telework data reliability,          The ability to compare with previous years is affected by:
and used the Office of Management              •   agencies use of methods of varying reliability to collect telework data, and
and Budget’s guidance for federal                  some agencies made changes to their data collection systems for the 2011
surveys to review OPM’s (1) plans to               data call. Executive agencies provide telework participation and frequency
collect telework participation data from           data by relying on estimates, counting telework agreements, or using
agencies and (2) development of a
                                                   automated time and attendance records.
data collection instrument. GAO
interviewed key OPM officials about its        •   modifications to the data call instrument, including changes to terminology
implementation of the 2011 data call.              and the time period during which telework data was requested. OPM officials
                                                   said they expect these changes will improve the consistency of data. But if
What GAO Recommends                                OPM reports progress based on data collected using changing terminology
                                                   and from different time periods, the agency may reach erroneous
GAO recommends that OPM
                                                   conclusions.
(1) clearly report reliability limitations
with the 2011 telework data call in its        Participants at the two data call training sessions may not have received the
June 2012 report to Congress and               same reporting instructions, and uncertainty about whether all agency
(2) continue efforts to improve data           respondents attended training, created a risk that some respondents may be
collection and gather information to           unaware of important terms and instructions. While some of the information
allow for the appropriate qualification of     provided at the two training sessions was similar, each session contained some
year-to-year comparisons and inform            new information, usually in response to questions raised at a previous session.
users about the effects of data
collection changes going forward. OPM          Future data call improvement efforts could result in a trade-off between the
partially concurred with the first             desire for maintenance of a consistent data series over time for comparison with
recommendation. However, GAO                   previous data calls and a need to improve data collection. According to OPM,
believes it should report limitations in       agencies will begin piloting automated telework data collection during 2012 and
its annual report. OPM fully concurred         2013. OPM expects that this method of data collection will provide it more
with the second. OPM provided a                reliable data than other methods. However, these efforts to standardize methods
number of technical comments which             for tracking telework data may result in changes to agencies’ methods of data
GAO incorporated as appropriate.               collection. The 2011 data call, notwithstanding its limitations, will be useful to
                                               help OPM identify and understand major changes in reported participation data
View GAO-12-519. For more information,
contact Yvonne D. Jones at (202) 512-6806 or
                                               that could occur during a transition to automated data collection.
jonesy@gao.gov.

                                                                                        United States Government Accountability Office
Contents


Letter                                                                                             1
                       Background                                                                  3
                       OPM Revised the 2011 Telework Data Call to Prepare for Reporting
                         under the Telework Enhancement Act of 2010                                5
                       Various Methods of Data Collection and Ongoing Changes to the
                         Data Call Make It Difficult to Assure Data Reliability and to
                         Make Year-to-Year Comparisons                                            9
                       Conclusions                                                               13
                       Recommendations for Executive Action                                      14
                       Agency Comments and Our Evaluation                                        14

Appendix I             Objectives, Scope, and Methodology                                        16



Appendix II            Revisions to OPM’s Telework Data Call to Collect Information to
                       Prepare for Reporting under the Telework Enhancement Act of 2010          19



Appendix III           Comments from the Office of Personnel Management                          21



Appendix IV            GAO Contact and Staff Acknowledgments                                     23



Related GAO Products                                                                             24



Table
                       Table 1: Principles Used to Assess the Data Call                          17


Figure
                       Figure 1: Differences in Time Period of Data Requested and
                                Reported between 2010 and 2011 Telework Data Calls                 7




                       Page i                                    GAO-12-519 Federal Telework Programs
Abbreviations

EHRI              Employee Human Resources Integration
FAQ               frequently asked questions
ITMG              Interagency Telework Measurement Group
OMB               Office of Management and Budget
OPM               Office of Personnel Management



This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page ii                                             GAO-12-519 Federal Telework Programs
United States Government Accountability Office
Washington, DC 20548




                                   April 19, 2012

                                   The Honorable Daniel K. Akaka
                                   Chairman
                                   The Honorable Ron Johnson
                                   Ranking Member
                                   Subcommittee on Oversight of Government Management, the
                                    Federal Workforce, and the District of Columbia
                                   Committee on Homeland Security and Governmental Affairs
                                   United States Senate

                                   Congress has encouraged federal agencies to expand their use of
                                   telework by requiring that executive agencies establish telework policies,
                                   incorporate those policies into continuity of operations plans, and provide
                                   interactive telework training to employees and managers. The Office of
                                   Personnel Management (OPM) has reported telework can enhance the
                                   productivity and flexibility of federal employees; help recruit and retain a
                                   qualified workforce; and reduce real estate, overhead, energy, and
                                   environmental costs. The Telework Enhancement Act of 2010 established
                                   a statutory framework for implementing a comprehensive federal telework
                                   program. 1 The act requires executive agencies to establish and to
                                   implement a policy under which eligible employees may be authorized to
                                   telework. The act also requires OPM to report annually to Congress on
                                   the telework programs of each agency, including the level of participation
                                   in telework in each agency, each agency’s goal for increasing employee
                                   participation in telework, agency progress during the reporting period
                                   toward meeting telework goals, and reasons for significant increases or
                                   decreases from previous year telework participation.

                                   In 2010, OPM set a high-priority performance goal of increasing by 50
                                   percent the number of eligible federal workers who telework by fiscal year
                                   2011. To measure agency progress toward this and agency telework
                                   goals, OPM will use data it collects through an annual survey referred to
                                   as the data call. However, concerns have existed about the reliability of
                                   these data. According to OPM, in order to help agencies overcome
                                   barriers related to telework and raise federal telework participation rates,
                                   the agency needs reliable, systematically collected agency data.


                                   1
                                    Pub. L. No. 111-292, 124 Stat. 3165 (Dec. 9, 2010), codified at chapter 65 of title 5,
                                   United States Code.




                                   Page 1                                               GAO-12-519 Federal Telework Programs
Because of the new telework reporting requirements in the act and the
need for reliable data to assess the agencies’ progress in implementing
their individual telework programs, you asked us to assess OPM’s:

•   actions to respond to the reporting requirements of the Telework
    Enhancement Act of 2010, and
•   handling of identified data reliability issues in the 2011 telework data
    call.

To answer these questions, we reviewed relevant reports and guidance
published by OPM that describe the status of telework programs across
executive agencies, and previous telework data calls and their
instructions. We also reviewed our previous reports on telework and the
reliability of OPM’s telework data. We identified the relevant generally
accepted survey standards as promulgated by the Office of Management
and Budget and compared OPM’s design and implementation of the 2011
telework data call against these standards.

We interviewed the OPM officials responsible for the planning, design,
implementation, and analysis of the 2011 telework data call. This included
discussions on the role of the Interagency Telework Measurement Group
(ITMG), the process for developing the definitions and key terms used in
the data call, the training and assistance provided to executive agency
officials responsible for completing the data call, and agency plans to
address outstanding data reliability issues associated with the data call. 2
We were also able to simulate completion of the web-based data call that
allowed us to respond to the data call in the same manner as the other
agency respondents. This simulation gave us the opportunity to
determine the ease of use of the online data call instrument, and whether
there were any potential design or functionality issues that could affect
the validity and reliability of the data.

We conducted this performance audit from June 2011 through April 2012
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our


2
 The Interagency Telework Measurement Group consisted of officials from OPM, General
Services Administration, United States Patent and Trademark Office, Department of the
Navy, Defense Information Systems Agency, National Aeronautics and Space
Administration, and Department of Energy.




Page 2                                          GAO-12-519 Federal Telework Programs
             findings and conclusions based on our audit objectives. We believe that
             the evidence obtained provides a reasonable basis for our findings and
             conclusions based on our audit objectives.

             The Telework Enhancement Act of 2010, enacted in December 2010,
Background   requires each executive agency to designate a telework managing officer,
             develop training programs, establish a telework policy, and submit an
             annual report to the Chair and Vice Chair of the Chief Human Capital
             Officers Council on the agency’s efforts to promote telework. Under the
             act, OPM is to play a leading role in helping executive agencies
             implement the new telework provisions, which include setting telework
             goals and establishing qualitative and quantitative measures. The law
             requires OPM to provide policy and guidance for telework in several
             areas, including pay and leave, agency closure, performance
             management, official worksite information, recruitment and retention, and
             accommodations for employees with disabilities. In an annual report to
             Congress, OPM is to assess each agency’s progress toward goals for
             participation and other goals relating to telework, such as emergency
             readiness. The first of these reports under the act is due to Congress in
             June 2012.

             Since 2002, OPM has used a telework survey—the data call—to annually
             collect information from the executive agencies in order to provide
             Congress with a report on the status of telework across these agencies. 3
             OPM conducts this data call to determine the extent to which agency
             employees are teleworking and to gauge agency progress in various
             aspects of their telework programs, such as participation, policy, eligibility,
             cost savings, and technology, as well as to provide examples of barriers
             agencies face to implementing telework programs.

             However, throughout the past decade, OPM has been concerned about
             the reliability of the telework data it receives from executive agencies
             because, although data reported from agencies have improved, OPM
             continues to consider it an estimate of telework participation and
             frequency. 4 In its 2003 and 2007 telework reports to Congress, OPM


             3
              OPM, Report to Congress: Status of Telework in the Federal Government, 2010
             (Washington, D.C.: Feb. 2011).
             4
              OPM considers automated data collection through time and attendance to be a more
             reliable method of collecting telework data and the number of agencies collecting data this
             way has increased over the years.




             Page 3                                              GAO-12-519 Federal Telework Programs
raised concerns about the ability of agencies to track employee
participation in their telework programs. 5 In its 2008 report, OPM
identified weaknesses in the methodology most agencies used to collect
and report telework participation data and OPM stated that
inconsistencies within data systems and inaccuracies triggered by hand-
counting telework agreements could affect data reliability. 6 OPM cautions
that existing measures of telework participation are a barrier to measuring
any increase in telework as the measures vary widely in validity and
reliability and limit the capability of any federal body to track the actual
level and frequency of telework participation. 7

At the request of Congress, we have previously reported on telework
programs across the federal government and have made
recommendations related to the reliability of agency-reported data. In a
2005 report, we reviewed the telework data for five federal agencies and
found they had reported the total number of employees who were eligible
to telework, but had included individuals who were, in fact, excluded from
participation based on various criteria such as employee performance,
thereby raising concerns about the reliability of the telework data reported
by these agencies. In addition, none of the agencies could report the
actual number of employees who teleworked or how often they did so
because none had fully implemented the capability to track this through
their time and attendance systems. 8 Our 2007 testimony reiterated our
concern that agencies were measuring employee participation in telework
based on their potential to telework rather than their actual usage. 9




5
 OPM, Report to Congress, Status of Telework in the Federal Government, 2003
(Washington, D.C.: 2003) and OPM, Report to Congress, Status of Telework in the
Federal Government, 2007 (Washington, D.C.: Dec. 2007).
6
 OPM, Report to Congress, Status of Telework in the Federal Government, 2008
(Washington, D.C.: Dec. 2008).
7
 OPM, Congressional Budget Justification, Performance Budget Fiscal Year 2012
(Washington, D.C.: Feb. 2011).
8
 GAO, Agency Telework Methodologies: Departments of Commerce, Justice, State, the
Small Business Administration, and the Securities and Exchange Commission,
GAO-05-1055R (Washington, D.C.: Sept. 27, 2005).
9
 GAO, Human Capital: Telework Programs Need Clear Goals and Reliable Data,
GAO-08-261T (Washington, D.C.: Nov. 7, 2007).




Page 4                                           GAO-12-519 Federal Telework Programs
                        More recently, we reported that since the 2004 data call, OPM asked
                        agencies if they had integrated telework into agency emergency and
                        continuity of operations plans, but agencies had no guidance as to what
                        constitutes incorporating telework into continuity and emergency
                        planning. This lack of a definition or description raised concerns about the
                        reliability of reports on this matter. 10


                        In response to the Telework Enhancement Act of 2010, OPM revised the
OPM Revised the 2011    2011 data call and provided instructions to executive agency respondents
Telework Data Call to   that incorporated common definitions and standards to use in providing
                        OPM with their agency data. The revisions and additions to the 2011 data
Prepare for Reporting   call were developed in consultation with an Interagency Telework
under the Telework      Measurement Group (ITMG), which OPM formed in January 2011. (See
Enhancement Act of      app. II for a comparison of definitional and instructional changes from the
                        2010 and 2011 telework data calls.) The OPM official designated with
2010                    leading the ITMG said 10 agency officials from 7 agencies were selected
                        for the group because of their knowledge of federal telework programs.
                        According to the OPM official, the ITMG provided expertise in telework
                        program implementation, policy, and methodology development, work/life
                        balance programs, and expertise in research methods, such as surveys.
                        The ITMG met on biweekly from January 2011 until July 2011 and
                        resumed biweekly meetings in September 2011 with the goal of
                        addressing three primary topics:

                        •    Definitions of key terms, such as telework, eligibility, and employee, to
                             use in the 2011 data call. ITMG interpreted some requirements of the
                             act and developed additional instructions to encourage common
                             reporting methodology across the agencies. For example, according
                             to OPM officials, ITMG clarified the definition of eligibility in light of
                             agencies’ concerns that the act did not specifically define the
                             categories of employees that should be eligible to participate in their
                             agencies’ telework programs, and therefore notified about their
                             eligibility to telework. The group instructed respondents to ensure they
                             excluded military and contract personnel as employees when
                             reporting their telework data. The group also clarified that respondents




                        10
                          GAO, Emergency Preparedness: Agencies Need Coordinated Guidance on
                        Incorporating Telework into Emergency and Continuity Planning, GAO-11-628
                        (Washington, D.C.: July 22, 2011).




                        Page 5                                          GAO-12-519 Federal Telework Programs
    should include full-time, part-time and intermittent employees when
    responding to questions about telework participation and frequency.

•   Revision and/or addition of data call questions. For example, OPM
    officials stated that in collaboration with the ITMG, they clarified the
    definition of telework to specifically state that telework includes what is
    generally referred to as remote work, but excludes mobile work and
    work done on official travel. 11 OPM officials added a new question to
    capture the number of mobile workers. This addressed a reliability
    problem from the previous data call when some agencies included
    mobile work in reporting telework.

•   Revision and development of data collection instruments, in addition
    to the data call, to collect telework information. For example, ITMG
    worked to revise telework-related items in the Federal Employee
    Viewpoint Survey, an OPM data collection instrument that gauges
    employees’ perceptions of their agency. In the 2011 survey three out
    of 84 questions focused on telework. In the past, OPM has found this
    survey to provide complementary employee views on telework. The
    ITMG also assisted in developing focus groups of telework managing
    officers and telework coordinators to identify issues, challenges and
    strategies associated with implementing telework programs at the
    agency level, such as successful telework implementation strategies,
    as well as barriers to telework.

According to an OPM official, these revisions also included questions that
may enable them to better understand the differences between telework
programs across executive agencies, including differences in training on
telework, use of technology, and how agencies responded to the
requirements of the act. This official said such information will help inform
the development of telework programs.

OPM also made changes to the 2011 data call time period for which
employee telework participation and frequency was to be reported. This
change was made to allow agencies time to develop telework policies in
accordance with the act and to allow OPM time to meet its reporting
obligation under the act (see fig. 1). In previous data calls, OPM asked



11
   OPM defines mobile work as work characterized by routine and regular travel to conduct
work in customer or other worksites as opposed to a single authorized alternative worksite
(i.e., site audits).




Page 6                                             GAO-12-519 Federal Telework Programs
                                        agencies for telework data during the calendar year (12 months), if
                                        available. OPM reduced the time period for the 2011 data call to 4 weeks,
                                        as it decided this was the best methodology to meet its reporting
                                        requirements under the act. Agencies were to select a 4 week period
                                        during September and October on which to report. In addition to the
                                        change in time period for requested telework data, the 2011 data call
                                        asked for more detail on employee telework participation, frequency of
                                        employee telework participation, and additional information on telework
                                        policy and program implementation, and telework goals as required by
                                        the act.

Figure 1: Differences in Time Period of Data Requested and Reported between 2010 and 2011 Telework Data Calls




                                        Note: From April 2010 through January 2011, OPM verified, analyzed, and drafted its 2010 report to
                                        Congress on telework. From January 2012 through May 2012, OPM will be verifying, analyzing, and
                                        drafting its report to be submitted to Congress in June 2012.


                                        OPM officials believe the information they obtained from the 2011 data
                                        call will enable the agency to satisfy some of the act’s reporting
                                        requirements, but for this report to Congress, OPM cannot fulfill other



                                        Page 7                                                  GAO-12-519 Federal Telework Programs
requirements. OPM officials stated that it is not feasible for OPM to
measure agencies’ progress against their Telework Enhancement Act of
2010 goals, since these goals were established in June 2011, and the
time period for agencies’ actual participation data was
September/October 2011—just 3 to 4 months after agencies established
their goals. However according to OPM officials, based on the information
collected, OPM will be able to report the percentage and frequency of
telework at individual agencies.

To communicate changes to the 2011 data call, OPM officials increased
their training efforts to aid executive agency officials in developing a
common understanding of terms, key concepts, and the objectives of the
data call. According to OPM officials, in July 2011, OPM officials
responsible for the data call met with agency respondents to provide an
introduction and overview to the 2011 data call. The meeting covered the
new requirements under the act, and the planned timeframe for agency
reporting and OPM processing and analysis of the data collected through
the data call. OPM could not require agency officials to attend.
Nonetheless, OPM reminded agency officials responsible for the data
call, that it was important that they attend both September and October
training sessions being offered by OPM. The September training session
covered the data call questions and incorporated specific content of the
near final data call. The October training session reviewed specific
instructions on how to enter information into the online data call form, in
addition to reviewing instructions and questions from respondents. While
some of the information provided at the two training sessions was similar,
each session contained some new information usually in response to
questions raised at a previous session. OPM staff also maintained and
disseminated via email a list of the most frequently asked questions
posed by data call respondents.




Page 8                                     GAO-12-519 Federal Telework Programs
                          The act requires OPM to report, among other things, year-to-year
Various Methods of        executive agency progress on the number of employees who telework. To
Data Collection and       accomplish this, OPM needs reliable baseline data on telework
                          participation to be able to make year-to-year comparisons. However,
Ongoing Changes to        OPM officials expressed continuing concerns over the reliability of
the Data Call Make It     quantitative participation and frequency data submitted by the agencies
Difficult to Assure       through the 2011 telework data call. OPM officials explained that 2011 is
                          a transition year for telework programs across executive agencies, and
Data Reliability and to   some agencies made changes to their policies to bring their telework
Make Year-to-Year         programs into compliance with the requirements of the act. For example,
                          some agencies were implementing new data collection systems while
Comparisons               collecting data on telework. These agencies first needed to create and
                          implement new policies, and then consider and establish processes
                          related to the collection of telework data to report to OPM. OPM officials
                          responsible for preparing the report to Congress stated that because of
                          these changes, it would not be appropriate to compare the 2011 data to
                          data collected in prior years.

                          OPM will not be able to make comparisons to prior years due to the
                          previously discussed modifications to the data call, such as changes to
                          definitions. Definitional changes, including the change to specifically
                          exclude mobile workers, could result in agencies interpreting and
                          reporting telework participation and frequency of telework differently than
                          they did in the past. While these changes should improve the consistency
                          of data going forward, according to OPM officials there is no way that
                          OPM can assure that all agencies would provide comparable responses
                          to the same data call questions. Moreover, OPM officials have said that
                          key terms and definitions for the 2012 data call may continue to evolve. If
                          OPM reports agency telework progress based on data collected using
                          definitions revised year to year, OPM may reach erroneous conclusions,
                          although OPM officials have said they are taking steps to try and prevent
                          this possibility.

                          Another reason why comparisons to prior data calls would be invalid is
                          because of the changes OPM made to the time period when agencies
                          reported telework data. Maintaining consistent data series over time
                          necessitates using consistent data collection procedures for ongoing data
                          collections. OPM said that changing the reporting period from 1 calendar
                          year to 4 weeks may result in greater data consistency because in
                          previous data calls, agencies may have reported data from different time
                          periods within a calendar year. Now all agencies will report data from a
                          narrower and more similar time frame. However, there are no available
                          studies to support that the new time period is representative or “typical” of


                          Page 9                                      GAO-12-519 Federal Telework Programs
other months in total or of the experience of particular agencies. OPM
officials said that they will need to indicate this in their report to Congress.

While OPM made changes to the 2011 data call to allow it to meet some
of its reporting requirements under the act and better assist agencies in
responding to the data call, our analysis found the 2011 data call did not
fully meet some generally accepted survey standards. According to these
standards, to be valid, survey questions must adequately represent the
concept or behavior in question and consistently predict outcomes.
Moreover, questions must be designed and asked so that each recipient
will understand and answer the same question in the same way. But for
the 2011 data call, there can be no assurance that all respondents were
aware of associated definitions and instructions provided by the training
sessions and the frequently asked questions (FAQ). Although OPM
invested more in providing training in preparation for the 2011 data call
than in previous data calls and it disseminated training slides to all
invitees of its final training session, attendance lists were not recoverable
for all training sessions so there can be no assurance that all respondents
received training or reviewed the slides. Consequently, some data call
respondents may not have been aware of the definitions and instructions
provided in the training sessions or in the FAQ.

Additionally, some of the information provided in the training sessions
was inconsistent. For example, OPM officials said that during the last
training session they instructed respondents to report an employee
“telework day” if the employee teleworked for any portion of a work day. 12
However, this clarification of “telework day” was not given in either the
July overview meeting or in the other training session, and not clearly
included in the instructions in the survey instrument or distributed in the
FAQ. This information may have been important to agencies reporting on
situational telework through automated systems intended to capture more
precise data. OPM officials also explained that responses to certain
questions should reference the same time period, and this information
was not available in the data call instructions. 13 During one training


12
  Under this definition, if an employee teleworked 2 hours, it should be reported as a
“telework day”. Therefore, a “telework day” is not related to the number of hours of
telework but rather the number of work days during which any telework occurred.
13
  In the October training session, OPM explained that questions asking about the total
number of agency employees and the total number of employees who teleworked (for
participation data) should come from the same time period. However, the data call
questions asking for these two totals are not drawn from the same time period.




Page 10                                             GAO-12-519 Federal Telework Programs
session, an OPM official said she instructed those participants who
determined employee participation by counting telework agreements to
limit the agreements counted to those in effect after the agency
implemented its telework program under the act; however, this
information was not available in the online data call instructions.
Uncertainty about whether data call respondents attended both the
September and October training sessions, and variations in training
sessions, could cause agencies to have different understandings of data
call concepts and terms. OPM recognizes that the existing measures of
federal telework participation vary in validity and reliability, which affects
agencies’ ability to report accurate data, and it is taking steps to verify
data submitted by respondents to provide a more accurate picture of
telework in the federal government based on current definitions and
collection methodologies. However, as a result of issues raised above,
respondents may have provided inconsistent or inaccurate data on topics
required by the act.

OPM officials anticipate that telework data will be more reliable next year
because of the expected governmentwide implementation of automated
data collection based on time and attendance records. As we have
reported, OPM has concluded from research that the most reliable
telework data are collected through time and attendance tracking
systems. 14 Data collected through automated systems eliminate the need
to track telework data by counting telework agreements or relying on
estimates. Since 2003, OPM has consistently expressed concerns about
the methods agencies use to collect telework data. In its previous
telework reports to Congress, OPM has advocated for the development of
an automated data collection system. OPM officials noted that OPM does
not control what telework data responding agencies maintain, or their
methods of data collection. Executive agencies provide telework
participation and frequency data using a variety of methods, such as
relying on estimates, counting telework agreements, and using automated
time and attendance records to track telework participation.

In an effort to collect more uniform data across agencies, OPM officials
are standardizing definitions and data elements for use in automated time
and attendance systems. For example, OPM has identified routine



14
 GAO, Human Capital: Further Guidance, Assistance, and Coordination Can Improve
Federal Telework Efforts, GAO-03-679 (Washington, D.C.: July 18, 2003).




Page 11                                        GAO-12-519 Federal Telework Programs
telework hours in a pay period as a data element for automated data
collection and provided a standardized definition that will be used by all
agencies using the Enterprise Human Resources Integration (EHRI)
system. 15 OPM has introduced a timeline for modifying the existing EHRI
system to allow OPM to collect telework data from executive agencies.
According OPM, agencies will begin piloting these automated data
collection systems for the 2012 telework data call. OPM began to discuss
automated governmentwide data collection with the ITMG in July 2011.
According to OPM’s timeline, OPM began to communicate internal
requirements for automated telework data collection to telework
managing officers in March 2012, but an OPM official has stated they do
not expect full automation of telework data until 2013. This official also
noted that different agencies have varying abilities to implement this new
type of data collection and reporting mechanism, and considering
different levels of comfort with new systems, it will take time to adjust to
this method. However, continuous improvement efforts sometimes result
in a trade-off between the desire for data consistency and a need to
improve data collection and maintenance of a consistent data series over
time.

Because of the eventual planned move to automation, OPM may not be
able to use 2011 data as a baseline. With the planned change to the
method of data collection, it may not be possible to compare the 2011
data to future data. The 2011 data call requested data for a 2-month
period, and some data call respondents relied upon estimates. Planned
automation can provide a more uniform and accurate method for
collecting telework data, however it may make comparisons using the
2011 telework data as a baseline difficult. However, OPM officials believe
the 2011 data will provide an improved report of telework status because
of standardized definitions and the more uniform time period of data
collection. According to OPM officials, some executive agencies will need
time to become comfortable with automated reporting systems and,
during a period of transition to a new system, there could be initial
reliability issues. However, these officials said that the 2011 data,
notwithstanding its limitations, will be useful in identifying and
understanding any major agency changes in reported participation that
could occur during a transition to automated data collection. Such



15
  According to the OPM website, the EHRI system offers a standardized approach for
human resources record conversion, storage, access, sharing, and archiving.




Page 12                                          GAO-12-519 Federal Telework Programs
              changes could alert OPM to possible transition related issues in agencies’
              conversion to automated governmentwide data call collection efforts in
              2012 and 2013. In addition, for those agencies already responding on the
              basis of time and attendance reporting, major changes in agency
              responses provide OPM the opportunity to confirm with agencies that the
              uniform definitions are being consistently applied.


              The Telework Enhancement Act of 2010 requires OPM to collect telework
Conclusions   data and report annually to Congress, which emphasizes the need for
              telework data to be valid and reliable. OPM revised the 2011 telework
              data call in order to allow it to meet some of the act’s reporting
              requirements and assist agencies in responding to the data call. This
              revision resulted in changes to terminology, as well as changes in the
              collection time period of requested telework data. OPM provided greater
              assistance to agencies through training on changes to the 2011 data call
              to improve accuracy of agency reporting of telework participation and
              frequency for the data call reporting period of September and October
              2011. However, agencies use various methods, which OPM does not
              control, to report, collect, and maintain telework data, and this could affect
              the reliability of the telework data submitted. In addition, variation in
              training sessions and OPM’s uncertainty as to whether all respondents
              attended training, could lead to respondents’ potential misunderstanding
              of important terms and instructions. The validity and reliability of the
              reported 2011 telework data for some of the responding agencies may be
              questionable, and therefore agency telework participation and frequency
              data will not likely be comparable with previous data calls because of
              differences in definitions used, time periods of reporting, and individual
              agency tracking methods.

              With the revised 2011 data call, OPM establishes a baseline it could use
              to conduct a limited crosscheck of data collected through a
              governmentwide automated telework data collection system, which OPM
              plans to implement over the course of 2012 and 2013. OPM expects that
              automated data collection will provide it increasingly more reliable data on
              which to report progress. However, these efforts to improve future
              automated data collection may result in changes to agencies’ methods of
              data collection and a trade-off between the desire for consistency with
              previous data calls for comparison purposes and a need to improve
              overall data collection.




              Page 13                                     GAO-12-519 Federal Telework Programs
                      To improve OPM’s annual reporting of telework to Congress, we
Recommendations for   recommend that the OPM Director take the following two actions:
Executive Action
                      •   Ensure that the reliability limitations related to the 2011 telework data
                          call are clearly reported in its June 2012 report to Congress by fully
                          describing how existing measures of telework participation vary widely
                          in validity and reliability and limit the capability of OPM to reliably
                          report the actual level and frequency of telework participation.

                      •   Continue efforts to improve data collection and gather information that
                          allows for the appropriate qualification of year-to-year comparisons
                          and informs users about the effects of data collection changes going
                          forward.

                      We provided a draft of this report to the Director of OPM for review and
Agency Comments       comment. The Associate Director of OPM provided written comments,
and Our Evaluation    which we have reprinted in appendix III. In summary, OPM partially
                      concurred with our first recommendation and fully concurred with the
                      second. OPM highlighted a number of actions the agency has under way
                      or plans to undertake in response. For the first recommendation, OPM
                      noted that inadequate methods of data collection exist at the agency level
                      and OPM continues to address this data reliability issue through training
                      on evaluation and measurement. While this is an important step in
                      addressing data reliability issues, OPM should ensure that telework data
                      reliability limitations are clearly reported in their annual reports to
                      Congress. For the second recommendation, OPM noted its continued
                      plans to automate collection of telework data, and to regularly meet with
                      telework managing officers and telework coordinators to keep them
                      updated on changes to telework policy and data collection. OPM also
                      provided a number of technical comments, which we incorporated as
                      appropriate.


                      We are sending copies of this report to the Chairman and Ranking
                      Member of the Subcommittee on the Federal Workforce, U.S. Postal
                      Service and Labor Policy, Committee on Oversight and Government
                      Reform, House of Representatives; and the Director of OPM. In addition,
                      this report will be available at no charge on the GAO website at
                      www.gao.gov. If you have any questions about this report, please contact
                      me at 202-512-6806 or jonesy@gao.gov. Contact points for our Offices of




                      Page 14                                    GAO-12-519 Federal Telework Programs
Congressional Relations and Public Affairs may be found on the last page
of this report. Key contributions to this report are listed in appendix IV.




Yvonne D. Jones
Director, Strategic Issues




Page 15                                   GAO-12-519 Federal Telework Programs
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              This report (1) describes the Office of Personnel Management’s (OPM)
              actions to respond to the requirements of the Telework Enhancement Act
              of 2010 and (2) assesses how OPM is handling and addressing identified
              data reliability issues in the 2011 telework data call.

              To address these two objectives, we reviewed relevant reports and
              guidance published by OPM that describe the status of telework
              programs across executive agencies and previous telework data calls and
              their instructions. We also reviewed previous GAO reports on telework
              and the reliability of OPM’s telework data. Lastly, we interviewed the OPM
              officials responsible for the planning, design, implementation, and
              analysis of the 2011 telework data call. This included discussions on the
              role of the Interagency Telework Measurement Group (ITMG), the
              process for developing the definitions and key terms used in the data call,
              the training and assistance provided to executive agency officials
              responsible for completing the data call, and agency plans to address
              outstanding data reliability issues associated with the data call. We
              conducted additional analysis to answer selected objectives as described
              below.

              To assess the extent to which the 2011 telework data call met generally
              accepted survey methodology, GAO internal experts in survey research
              identified principles from the Office of Management and Budget’s (OMB)
              Standards and Guidelines for Statistical Surveys relevant to assessing
              the 2011 telework data call. 1 We also used relevant aspects of GAO’s
              guide to Developing and Using Questionnaires. 2 Using the OMB
              principles, two analysts independently reviewed the data call, supporting
              documentation, and clarifying information provided in interviews to assess
              the extent to which the data call methodology met research practices. The
              initial rate of agreement across 15 rated practices, including those rated
              as having insufficient information to judge, was 12 of 15, or 80 percent
              agreement. The rating for the three practices on which there was initial
              disagreement were reconciled by the two analysts conducting the review
              and the reconciled rating was then reviewed by a third analyst with survey




              1
               OMB, Standards and Guidelines for Statistical Surveys (Washington, D.C.: Sept. 2006).
              See www.whitehouse.gov/omb (last accessed Nov. 10, 2011).
              2
               GAO, Developing and Using Questionnaires, GAO/PEMD-10.1.7 (Washington, D.C.: Oct.
              1993).




              Page 16                                           GAO-12-519 Federal Telework Programs
                                           Appendix I: Objectives, Scope, and
                                           Methodology




                                           expertise. The third analyst did not recommend any changes to the
                                           reconciled ratings.

                                           We also simulated completion of the Web-based data call, accessing and
                                           responding to the data call in the same manner as executive agency
                                           respondents.

                                           Table 1 outlines the generally accepted survey research principles,
                                           derived from OMB’s guidelines, which we used in our assessment. This is
                                           not an exhaustive list of all OMB guidelines. 3 When we completed this
                                           review in April of 2012, OPM had not yet completed analyzing and
                                           reporting on the results of telework data call. Based on this, we could not
                                           yet assess whether the data call met all of the OMB principles related to
                                           data analysis and reporting. Some OMB principles, such as those related
                                           to sample design, were not appropriate to apply to the telework data call.
                                           These principles were therefore excluded as not relevant to our review.

Table 1: Principles Used to Assess the Data Call

Development of Concepts,         1.   The survey had a clear rationale
Methods, and Design              2.   Potential users were consulted to identify their requirements and expectations
                                 3.   A review of existing, surveys, studies, and reports, or other literature informed the survey
                                 4.   The scope of survey data items are defined and justified
                                 5.   To ensure data remain consistent over time, key variable and survey procedures remained
                                      consistent
                                 6.   A clear, logical, and easy-to-follow flow of questions from a respondents’ point of view
                                 7.   The questionnaire was pretested and all components of the final survey system were field
                                      tested
Collection of data               1.   The sample frame was appropriate
                                 2.   The respondents were provided appropriate informational materials
                                 3.   Questions are clearly written
                                 4.   The questionnaire design minimized respondent burden and maximized data quality
                                 5.   Response maximization efforts were appropriate
                                 6.   Agency staff developed proper protocols for handling respondent questions to help ensure
                                      data quality
                                 7.   Protocols to monitor data collection activities were developed
Processing and editing of data 1.     Appropriate checks and edits on the data collection system mitigated errors
                                           Source: GAO and OMB, Standards and Guidelines for Statistical Surveys (Washington, D.C.: Sept. 2006).




                                           3
                                            OMB standards and guidelines are also not designed to be completely exhaustive of all
                                           efforts that an agency may undertake to ensure the quality of its data. OMB encourages
                                           agencies to develop additional standards focused on their specific activities.




                                           Page 17                                                                  GAO-12-519 Federal Telework Programs
Appendix I: Objectives, Scope, and
Methodology




We conducted this performance audit from June 2011 through April 2012
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.




Page 18                                  GAO-12-519 Federal Telework Programs
Appendix II: Revisions to OPM’s Telework Data
                                            Appendix II: Revisions to OPM’s Telework Data
                                            Call to Collect Information to Prepare for
                                            Reporting under the Telework Enhancement

Call to Collect Information to Prepare for
                                            Act of 2010




Reporting under the Telework Enhancement Act of
2010
               2010 Telework data call                                  2011 Telework data call
Instructions   Time period: Based on the agency’s calendar year         Time period: Based on the months of Sept. and Oct. 2011.a
               (Jan. 1 to Dec. 31, 2009).
               Participation: agencies invited to participate in data   Participation: agencies required to submit telework data to
               call.                                                    OPM.
               Reporting deadline: March 2010                           Reporting deadline: Dec. 2011
Definitions    Telework:                                                Telework:
               “Telework refers to any arrangement in which an          “Telework is a work arrangement that allows an employee to
               employee regularly performs officially assigned duties   perform work, during any part of regular, paid hours, at an
               at home or other work sites geographically               approved alternative worksite (e.g., home, telework center).
               convenient to the residence of the employee.”            This definition of telework includes what is generally referred
                                                                        to as remote work but does not include any part of work done
                                                                        while on official travel or mobile work. See the following
                                                                        clarifications on remote and mobile work as telework.”
               Employee:                                                Employee:
               Not defined.                                             Employee refers to federal civilian employees excluding
                                                                        military personnel and contractors. Employee may also
                                                                        include full-time, part-time, and/or intermittent employees.
               Eligibility:                                             Eligibility:
               Agencies have the discretion to determine the            An employee is eligible to participate in telework if all of the
               telework eligibility requirements for their employees.   following parameters are true:
               For reporting purposes, in this Call for data, ALL       •    The employee has not been officially disciplined for being
               employees are considered eligible EXCEPT those                absent without permission for more than 5 days in any
               employees whose:                                              calendar year.
               •    positions require, on a daily basis (every work     •    The employee has not been officially disciplined for
                    day), direct handling of secure materials, or on-        violations of subpart G of the Standards of Ethical
                    site activity that cannot possibly be handled            Conduct for Employees of the Executive Branch.
                    remotely or at an alternate worksite;
                                                                        •    Teleworking does not diminish the employee’s
               •    most recent federal government performance               performance or agency operations.
                    rating of record (or its equivalent) is below fully
                    successful or conduct has resulted in disciplinary  •    Participation and performance complies with the
                    action within the last year.                             requirements and expectations of his/her telework
                                                                             agreement.
                                                                        •    The employee’s official duties do not require on a FULL
                                                                             daily basis (ALL DAY, every work day):
                                                                                       •    direct handling of secure materials
                                                                                            determined to be inappropriate for telework
                                                                                            by the agency head; or
                                                                                       •    on-site activity that cannot be handled
                                                                                            remotely or at an alternate worksite.
                                                                        •    The employee and/or the employee’s position are not
                                                                             disqualified based on additional criteria established by
                                                                             the organization.




                                            Page 19                                             GAO-12-519 Federal Telework Programs
                          Appendix II: Revisions to OPM’s Telework Data
                          Call to Collect Information to Prepare for
                          Reporting under the Telework Enhancement
                          Act of 2010




2010 Telework data call                                  2011 Telework data call
Types of telework:                                       Types of telework:
None provided                                            •   Routine: telework that occurs as part of an ongoing,
                                                             regular schedule, and
                                                         •   Situational: telework that is approved on a case-by-case
                                                             basis, where the hours worked were not part of a
                                                             previously approved, ongoing and regular telework
                                                             schedule (e.g., telework as a result of special work
                                                             assignments or doctor appointment.)b
                          Source: OPM.

                          Notes: This table is intended to illustrate the differences between the 2010 and 2011 data calls. OPM
                          made changes to the 2011 data call in response to the Telework Enhancement Act of 2010.
                          a
                           Under the Telework Enhancement Act of 2010, agencies were required to establish telework policies
                          by June 2011.
                          b
                           Situational telework is sometimes also referred to as episodic, intermittent, unscheduled or ad-hoc
                          telework.




                          Page 20                                                   GAO-12-519 Federal Telework Programs
Appendix III: Comments from the Office of
              Appendix III: Comments from the Office of
              Personnel Management



Personnel Management




              Page 21                                     GAO-12-519 Federal Telework Programs
Appendix III: Comments from the Office of
Personnel Management




Page 22                                     GAO-12-519 Federal Telework Programs
Appendix IV: GAO Contact and Staff
                  Appendix IV: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Yvonne D. Jones, (202) 512-6806 or jonesy@gao.gov
GAO Contact
                  In addition to the contact named above, William Doherty, Assistant
Staff             Director, and Keith O’Brien, analyst-in-charge, led the development of this
Acknowledgments   report. Virginia Chanley, Patricia Donahue, Robert Gebhart, Jill Lacey,
                  and Joseph Santiago made significant contributions to this report. Karin
                  Fangman provided legal counsel. Shirley Hwang, Jessica Nierenberg,
                  and Kathleen Padulchick verified the information in the report.




                  Page 23                                   GAO-12-519 Federal Telework Programs
Related GAO Products
             Related GAO Products




             Emergency Preparedness: Agencies Need Coordinated Guidance on
             Incorporating Telework into Emergency and Continuity Planning.
             GAO-11-628. Washington, D.C.: July 22, 2011.

             Human Capital: Telework Programs Need Clear Goals and Reliable Data.
             GAO-08-261T. Washington, D.C.: November 6, 2007.

             Human Capital: Greater Focus on Results in Telework Programs Needed.
             GAO-07-1002T. Washington, D.C.: June 12, 2007.

             Agency Telework Methodologies: Departments of Commerce, Justice,
             State, the Small Business Administration, and the Securities and
             Exchange Commission. GAO-05-1055R. Washington, D.C.: September
             27, 2005.

             Human Capital: Key Practices to Increasing Federal Telework.
             GAO-04-950T. Washington, D.C.: July 8, 2004.

             Human Capital: Further Guidance, Assistance, and Coordination Can
             Improve Federal Telework Efforts. GAO-03-679. Washington, D.C.:
             July 18, 2003.




(450930)
             Page 24                                 GAO-12-519 Federal Telework Programs
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.