United States Government Accountability Office GAO Report to the Subcommittee on Oversight of Government Management, the Federal Workforce, and the District of Columbia, Committee on Homeland Security and Governmental Affairs, U.S. Senate April 2012 FEDERAL TELEWORK Program Measurement Continues to Confront Data Reliability Issues GAO-12-519 April 2012 FEDERAL TELEWORK Program Measurement Continues to Confront Data Reliability Issues Highlights of GAO-12-519, a report to the Subcommittee on Oversight of Government Management, the Federal Workforce, and the District of Columbia, Committee on Homeland Security and Governmental Affairs, U.S. Senate Why GAO Did This Study What GAO Found The Telework Enhancement Act of To prepare for its reporting obligations under the Telework Enhancement Act of 2010 requires OPM to report to 2010, the Office of Personnel Management (OPM) assembled the Interagency Congress on the degree of telework Telework Measurement Group, consisting of officials from several federal participation at executive agencies. To agencies, to assist in revising the telework data call—the survey OPM has used meet this requirement, OPM collects since 2002 to collect telework data from executive agencies. This group information on agency telework standardized key terms such as telework, employee, and eligibility to promote a programs through an annual survey, common reporting methodology among the agencies. The revised telework data which it refers to as the data call. call also included changes to the time period for which OPM requested agencies However, concerns exist about the report telework data, and included more extensive training for respondents. reliability of these data. Because of changes made to the data call to allow OPM to meet requirements of GAO was asked to assess OPM’s: the act and assist agencies in responding to the data call, OPM officials believe (1) actions in response to the they will be able to provide to Congress an improved report on telework in June requirements of the act and 2012. However, these changes also mean that OPM officials will not be able to (2) handling of identified data reliability use participation and frequency data from the 2011 data call to compare to data issues in the 2011 telework data call. from previous years and across agencies. OPM officials have noted that this To address these objectives GAO reviewed its previous reports could limit OPM’s ability to report agency progress in its first report to Congress. addressing telework data reliability, The ability to compare with previous years is affected by: and used the Office of Management • agencies use of methods of varying reliability to collect telework data, and and Budget’s guidance for federal some agencies made changes to their data collection systems for the 2011 surveys to review OPM’s (1) plans to data call. Executive agencies provide telework participation and frequency collect telework participation data from data by relying on estimates, counting telework agreements, or using agencies and (2) development of a automated time and attendance records. data collection instrument. GAO interviewed key OPM officials about its • modifications to the data call instrument, including changes to terminology implementation of the 2011 data call. and the time period during which telework data was requested. OPM officials said they expect these changes will improve the consistency of data. But if What GAO Recommends OPM reports progress based on data collected using changing terminology and from different time periods, the agency may reach erroneous GAO recommends that OPM conclusions. (1) clearly report reliability limitations with the 2011 telework data call in its Participants at the two data call training sessions may not have received the June 2012 report to Congress and same reporting instructions, and uncertainty about whether all agency (2) continue efforts to improve data respondents attended training, created a risk that some respondents may be collection and gather information to unaware of important terms and instructions. While some of the information allow for the appropriate qualification of provided at the two training sessions was similar, each session contained some year-to-year comparisons and inform new information, usually in response to questions raised at a previous session. users about the effects of data collection changes going forward. OPM Future data call improvement efforts could result in a trade-off between the partially concurred with the first desire for maintenance of a consistent data series over time for comparison with recommendation. However, GAO previous data calls and a need to improve data collection. According to OPM, believes it should report limitations in agencies will begin piloting automated telework data collection during 2012 and its annual report. OPM fully concurred 2013. OPM expects that this method of data collection will provide it more with the second. OPM provided a reliable data than other methods. However, these efforts to standardize methods number of technical comments which for tracking telework data may result in changes to agencies’ methods of data GAO incorporated as appropriate. collection. The 2011 data call, notwithstanding its limitations, will be useful to help OPM identify and understand major changes in reported participation data View GAO-12-519. For more information, contact Yvonne D. Jones at (202) 512-6806 or that could occur during a transition to automated data collection. email@example.com. United States Government Accountability Office Contents Letter 1 Background 3 OPM Revised the 2011 Telework Data Call to Prepare for Reporting under the Telework Enhancement Act of 2010 5 Various Methods of Data Collection and Ongoing Changes to the Data Call Make It Difficult to Assure Data Reliability and to Make Year-to-Year Comparisons 9 Conclusions 13 Recommendations for Executive Action 14 Agency Comments and Our Evaluation 14 Appendix I Objectives, Scope, and Methodology 16 Appendix II Revisions to OPM’s Telework Data Call to Collect Information to Prepare for Reporting under the Telework Enhancement Act of 2010 19 Appendix III Comments from the Office of Personnel Management 21 Appendix IV GAO Contact and Staff Acknowledgments 23 Related GAO Products 24 Table Table 1: Principles Used to Assess the Data Call 17 Figure Figure 1: Differences in Time Period of Data Requested and Reported between 2010 and 2011 Telework Data Calls 7 Page i GAO-12-519 Federal Telework Programs Abbreviations EHRI Employee Human Resources Integration FAQ frequently asked questions ITMG Interagency Telework Measurement Group OMB Office of Management and Budget OPM Office of Personnel Management This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page ii GAO-12-519 Federal Telework Programs United States Government Accountability Office Washington, DC 20548 April 19, 2012 The Honorable Daniel K. Akaka Chairman The Honorable Ron Johnson Ranking Member Subcommittee on Oversight of Government Management, the Federal Workforce, and the District of Columbia Committee on Homeland Security and Governmental Affairs United States Senate Congress has encouraged federal agencies to expand their use of telework by requiring that executive agencies establish telework policies, incorporate those policies into continuity of operations plans, and provide interactive telework training to employees and managers. The Office of Personnel Management (OPM) has reported telework can enhance the productivity and flexibility of federal employees; help recruit and retain a qualified workforce; and reduce real estate, overhead, energy, and environmental costs. The Telework Enhancement Act of 2010 established a statutory framework for implementing a comprehensive federal telework program. 1 The act requires executive agencies to establish and to implement a policy under which eligible employees may be authorized to telework. The act also requires OPM to report annually to Congress on the telework programs of each agency, including the level of participation in telework in each agency, each agency’s goal for increasing employee participation in telework, agency progress during the reporting period toward meeting telework goals, and reasons for significant increases or decreases from previous year telework participation. In 2010, OPM set a high-priority performance goal of increasing by 50 percent the number of eligible federal workers who telework by fiscal year 2011. To measure agency progress toward this and agency telework goals, OPM will use data it collects through an annual survey referred to as the data call. However, concerns have existed about the reliability of these data. According to OPM, in order to help agencies overcome barriers related to telework and raise federal telework participation rates, the agency needs reliable, systematically collected agency data. 1 Pub. L. No. 111-292, 124 Stat. 3165 (Dec. 9, 2010), codified at chapter 65 of title 5, United States Code. Page 1 GAO-12-519 Federal Telework Programs Because of the new telework reporting requirements in the act and the need for reliable data to assess the agencies’ progress in implementing their individual telework programs, you asked us to assess OPM’s: • actions to respond to the reporting requirements of the Telework Enhancement Act of 2010, and • handling of identified data reliability issues in the 2011 telework data call. To answer these questions, we reviewed relevant reports and guidance published by OPM that describe the status of telework programs across executive agencies, and previous telework data calls and their instructions. We also reviewed our previous reports on telework and the reliability of OPM’s telework data. We identified the relevant generally accepted survey standards as promulgated by the Office of Management and Budget and compared OPM’s design and implementation of the 2011 telework data call against these standards. We interviewed the OPM officials responsible for the planning, design, implementation, and analysis of the 2011 telework data call. This included discussions on the role of the Interagency Telework Measurement Group (ITMG), the process for developing the definitions and key terms used in the data call, the training and assistance provided to executive agency officials responsible for completing the data call, and agency plans to address outstanding data reliability issues associated with the data call. 2 We were also able to simulate completion of the web-based data call that allowed us to respond to the data call in the same manner as the other agency respondents. This simulation gave us the opportunity to determine the ease of use of the online data call instrument, and whether there were any potential design or functionality issues that could affect the validity and reliability of the data. We conducted this performance audit from June 2011 through April 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our 2 The Interagency Telework Measurement Group consisted of officials from OPM, General Services Administration, United States Patent and Trademark Office, Department of the Navy, Defense Information Systems Agency, National Aeronautics and Space Administration, and Department of Energy. Page 2 GAO-12-519 Federal Telework Programs findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. The Telework Enhancement Act of 2010, enacted in December 2010, Background requires each executive agency to designate a telework managing officer, develop training programs, establish a telework policy, and submit an annual report to the Chair and Vice Chair of the Chief Human Capital Officers Council on the agency’s efforts to promote telework. Under the act, OPM is to play a leading role in helping executive agencies implement the new telework provisions, which include setting telework goals and establishing qualitative and quantitative measures. The law requires OPM to provide policy and guidance for telework in several areas, including pay and leave, agency closure, performance management, official worksite information, recruitment and retention, and accommodations for employees with disabilities. In an annual report to Congress, OPM is to assess each agency’s progress toward goals for participation and other goals relating to telework, such as emergency readiness. The first of these reports under the act is due to Congress in June 2012. Since 2002, OPM has used a telework survey—the data call—to annually collect information from the executive agencies in order to provide Congress with a report on the status of telework across these agencies. 3 OPM conducts this data call to determine the extent to which agency employees are teleworking and to gauge agency progress in various aspects of their telework programs, such as participation, policy, eligibility, cost savings, and technology, as well as to provide examples of barriers agencies face to implementing telework programs. However, throughout the past decade, OPM has been concerned about the reliability of the telework data it receives from executive agencies because, although data reported from agencies have improved, OPM continues to consider it an estimate of telework participation and frequency. 4 In its 2003 and 2007 telework reports to Congress, OPM 3 OPM, Report to Congress: Status of Telework in the Federal Government, 2010 (Washington, D.C.: Feb. 2011). 4 OPM considers automated data collection through time and attendance to be a more reliable method of collecting telework data and the number of agencies collecting data this way has increased over the years. Page 3 GAO-12-519 Federal Telework Programs raised concerns about the ability of agencies to track employee participation in their telework programs. 5 In its 2008 report, OPM identified weaknesses in the methodology most agencies used to collect and report telework participation data and OPM stated that inconsistencies within data systems and inaccuracies triggered by hand- counting telework agreements could affect data reliability. 6 OPM cautions that existing measures of telework participation are a barrier to measuring any increase in telework as the measures vary widely in validity and reliability and limit the capability of any federal body to track the actual level and frequency of telework participation. 7 At the request of Congress, we have previously reported on telework programs across the federal government and have made recommendations related to the reliability of agency-reported data. In a 2005 report, we reviewed the telework data for five federal agencies and found they had reported the total number of employees who were eligible to telework, but had included individuals who were, in fact, excluded from participation based on various criteria such as employee performance, thereby raising concerns about the reliability of the telework data reported by these agencies. In addition, none of the agencies could report the actual number of employees who teleworked or how often they did so because none had fully implemented the capability to track this through their time and attendance systems. 8 Our 2007 testimony reiterated our concern that agencies were measuring employee participation in telework based on their potential to telework rather than their actual usage. 9 5 OPM, Report to Congress, Status of Telework in the Federal Government, 2003 (Washington, D.C.: 2003) and OPM, Report to Congress, Status of Telework in the Federal Government, 2007 (Washington, D.C.: Dec. 2007). 6 OPM, Report to Congress, Status of Telework in the Federal Government, 2008 (Washington, D.C.: Dec. 2008). 7 OPM, Congressional Budget Justification, Performance Budget Fiscal Year 2012 (Washington, D.C.: Feb. 2011). 8 GAO, Agency Telework Methodologies: Departments of Commerce, Justice, State, the Small Business Administration, and the Securities and Exchange Commission, GAO-05-1055R (Washington, D.C.: Sept. 27, 2005). 9 GAO, Human Capital: Telework Programs Need Clear Goals and Reliable Data, GAO-08-261T (Washington, D.C.: Nov. 7, 2007). Page 4 GAO-12-519 Federal Telework Programs More recently, we reported that since the 2004 data call, OPM asked agencies if they had integrated telework into agency emergency and continuity of operations plans, but agencies had no guidance as to what constitutes incorporating telework into continuity and emergency planning. This lack of a definition or description raised concerns about the reliability of reports on this matter. 10 In response to the Telework Enhancement Act of 2010, OPM revised the OPM Revised the 2011 2011 data call and provided instructions to executive agency respondents Telework Data Call to that incorporated common definitions and standards to use in providing OPM with their agency data. The revisions and additions to the 2011 data Prepare for Reporting call were developed in consultation with an Interagency Telework under the Telework Measurement Group (ITMG), which OPM formed in January 2011. (See Enhancement Act of app. II for a comparison of definitional and instructional changes from the 2010 and 2011 telework data calls.) The OPM official designated with 2010 leading the ITMG said 10 agency officials from 7 agencies were selected for the group because of their knowledge of federal telework programs. According to the OPM official, the ITMG provided expertise in telework program implementation, policy, and methodology development, work/life balance programs, and expertise in research methods, such as surveys. The ITMG met on biweekly from January 2011 until July 2011 and resumed biweekly meetings in September 2011 with the goal of addressing three primary topics: • Definitions of key terms, such as telework, eligibility, and employee, to use in the 2011 data call. ITMG interpreted some requirements of the act and developed additional instructions to encourage common reporting methodology across the agencies. For example, according to OPM officials, ITMG clarified the definition of eligibility in light of agencies’ concerns that the act did not specifically define the categories of employees that should be eligible to participate in their agencies’ telework programs, and therefore notified about their eligibility to telework. The group instructed respondents to ensure they excluded military and contract personnel as employees when reporting their telework data. The group also clarified that respondents 10 GAO, Emergency Preparedness: Agencies Need Coordinated Guidance on Incorporating Telework into Emergency and Continuity Planning, GAO-11-628 (Washington, D.C.: July 22, 2011). Page 5 GAO-12-519 Federal Telework Programs should include full-time, part-time and intermittent employees when responding to questions about telework participation and frequency. • Revision and/or addition of data call questions. For example, OPM officials stated that in collaboration with the ITMG, they clarified the definition of telework to specifically state that telework includes what is generally referred to as remote work, but excludes mobile work and work done on official travel. 11 OPM officials added a new question to capture the number of mobile workers. This addressed a reliability problem from the previous data call when some agencies included mobile work in reporting telework. • Revision and development of data collection instruments, in addition to the data call, to collect telework information. For example, ITMG worked to revise telework-related items in the Federal Employee Viewpoint Survey, an OPM data collection instrument that gauges employees’ perceptions of their agency. In the 2011 survey three out of 84 questions focused on telework. In the past, OPM has found this survey to provide complementary employee views on telework. The ITMG also assisted in developing focus groups of telework managing officers and telework coordinators to identify issues, challenges and strategies associated with implementing telework programs at the agency level, such as successful telework implementation strategies, as well as barriers to telework. According to an OPM official, these revisions also included questions that may enable them to better understand the differences between telework programs across executive agencies, including differences in training on telework, use of technology, and how agencies responded to the requirements of the act. This official said such information will help inform the development of telework programs. OPM also made changes to the 2011 data call time period for which employee telework participation and frequency was to be reported. This change was made to allow agencies time to develop telework policies in accordance with the act and to allow OPM time to meet its reporting obligation under the act (see fig. 1). In previous data calls, OPM asked 11 OPM defines mobile work as work characterized by routine and regular travel to conduct work in customer or other worksites as opposed to a single authorized alternative worksite (i.e., site audits). Page 6 GAO-12-519 Federal Telework Programs agencies for telework data during the calendar year (12 months), if available. OPM reduced the time period for the 2011 data call to 4 weeks, as it decided this was the best methodology to meet its reporting requirements under the act. Agencies were to select a 4 week period during September and October on which to report. In addition to the change in time period for requested telework data, the 2011 data call asked for more detail on employee telework participation, frequency of employee telework participation, and additional information on telework policy and program implementation, and telework goals as required by the act. Figure 1: Differences in Time Period of Data Requested and Reported between 2010 and 2011 Telework Data Calls Note: From April 2010 through January 2011, OPM verified, analyzed, and drafted its 2010 report to Congress on telework. From January 2012 through May 2012, OPM will be verifying, analyzing, and drafting its report to be submitted to Congress in June 2012. OPM officials believe the information they obtained from the 2011 data call will enable the agency to satisfy some of the act’s reporting requirements, but for this report to Congress, OPM cannot fulfill other Page 7 GAO-12-519 Federal Telework Programs requirements. OPM officials stated that it is not feasible for OPM to measure agencies’ progress against their Telework Enhancement Act of 2010 goals, since these goals were established in June 2011, and the time period for agencies’ actual participation data was September/October 2011—just 3 to 4 months after agencies established their goals. However according to OPM officials, based on the information collected, OPM will be able to report the percentage and frequency of telework at individual agencies. To communicate changes to the 2011 data call, OPM officials increased their training efforts to aid executive agency officials in developing a common understanding of terms, key concepts, and the objectives of the data call. According to OPM officials, in July 2011, OPM officials responsible for the data call met with agency respondents to provide an introduction and overview to the 2011 data call. The meeting covered the new requirements under the act, and the planned timeframe for agency reporting and OPM processing and analysis of the data collected through the data call. OPM could not require agency officials to attend. Nonetheless, OPM reminded agency officials responsible for the data call, that it was important that they attend both September and October training sessions being offered by OPM. The September training session covered the data call questions and incorporated specific content of the near final data call. The October training session reviewed specific instructions on how to enter information into the online data call form, in addition to reviewing instructions and questions from respondents. While some of the information provided at the two training sessions was similar, each session contained some new information usually in response to questions raised at a previous session. OPM staff also maintained and disseminated via email a list of the most frequently asked questions posed by data call respondents. Page 8 GAO-12-519 Federal Telework Programs The act requires OPM to report, among other things, year-to-year Various Methods of executive agency progress on the number of employees who telework. To Data Collection and accomplish this, OPM needs reliable baseline data on telework participation to be able to make year-to-year comparisons. However, Ongoing Changes to OPM officials expressed continuing concerns over the reliability of the Data Call Make It quantitative participation and frequency data submitted by the agencies Difficult to Assure through the 2011 telework data call. OPM officials explained that 2011 is a transition year for telework programs across executive agencies, and Data Reliability and to some agencies made changes to their policies to bring their telework Make Year-to-Year programs into compliance with the requirements of the act. For example, some agencies were implementing new data collection systems while Comparisons collecting data on telework. These agencies first needed to create and implement new policies, and then consider and establish processes related to the collection of telework data to report to OPM. OPM officials responsible for preparing the report to Congress stated that because of these changes, it would not be appropriate to compare the 2011 data to data collected in prior years. OPM will not be able to make comparisons to prior years due to the previously discussed modifications to the data call, such as changes to definitions. Definitional changes, including the change to specifically exclude mobile workers, could result in agencies interpreting and reporting telework participation and frequency of telework differently than they did in the past. While these changes should improve the consistency of data going forward, according to OPM officials there is no way that OPM can assure that all agencies would provide comparable responses to the same data call questions. Moreover, OPM officials have said that key terms and definitions for the 2012 data call may continue to evolve. If OPM reports agency telework progress based on data collected using definitions revised year to year, OPM may reach erroneous conclusions, although OPM officials have said they are taking steps to try and prevent this possibility. Another reason why comparisons to prior data calls would be invalid is because of the changes OPM made to the time period when agencies reported telework data. Maintaining consistent data series over time necessitates using consistent data collection procedures for ongoing data collections. OPM said that changing the reporting period from 1 calendar year to 4 weeks may result in greater data consistency because in previous data calls, agencies may have reported data from different time periods within a calendar year. Now all agencies will report data from a narrower and more similar time frame. However, there are no available studies to support that the new time period is representative or “typical” of Page 9 GAO-12-519 Federal Telework Programs other months in total or of the experience of particular agencies. OPM officials said that they will need to indicate this in their report to Congress. While OPM made changes to the 2011 data call to allow it to meet some of its reporting requirements under the act and better assist agencies in responding to the data call, our analysis found the 2011 data call did not fully meet some generally accepted survey standards. According to these standards, to be valid, survey questions must adequately represent the concept or behavior in question and consistently predict outcomes. Moreover, questions must be designed and asked so that each recipient will understand and answer the same question in the same way. But for the 2011 data call, there can be no assurance that all respondents were aware of associated definitions and instructions provided by the training sessions and the frequently asked questions (FAQ). Although OPM invested more in providing training in preparation for the 2011 data call than in previous data calls and it disseminated training slides to all invitees of its final training session, attendance lists were not recoverable for all training sessions so there can be no assurance that all respondents received training or reviewed the slides. Consequently, some data call respondents may not have been aware of the definitions and instructions provided in the training sessions or in the FAQ. Additionally, some of the information provided in the training sessions was inconsistent. For example, OPM officials said that during the last training session they instructed respondents to report an employee “telework day” if the employee teleworked for any portion of a work day. 12 However, this clarification of “telework day” was not given in either the July overview meeting or in the other training session, and not clearly included in the instructions in the survey instrument or distributed in the FAQ. This information may have been important to agencies reporting on situational telework through automated systems intended to capture more precise data. OPM officials also explained that responses to certain questions should reference the same time period, and this information was not available in the data call instructions. 13 During one training 12 Under this definition, if an employee teleworked 2 hours, it should be reported as a “telework day”. Therefore, a “telework day” is not related to the number of hours of telework but rather the number of work days during which any telework occurred. 13 In the October training session, OPM explained that questions asking about the total number of agency employees and the total number of employees who teleworked (for participation data) should come from the same time period. However, the data call questions asking for these two totals are not drawn from the same time period. Page 10 GAO-12-519 Federal Telework Programs session, an OPM official said she instructed those participants who determined employee participation by counting telework agreements to limit the agreements counted to those in effect after the agency implemented its telework program under the act; however, this information was not available in the online data call instructions. Uncertainty about whether data call respondents attended both the September and October training sessions, and variations in training sessions, could cause agencies to have different understandings of data call concepts and terms. OPM recognizes that the existing measures of federal telework participation vary in validity and reliability, which affects agencies’ ability to report accurate data, and it is taking steps to verify data submitted by respondents to provide a more accurate picture of telework in the federal government based on current definitions and collection methodologies. However, as a result of issues raised above, respondents may have provided inconsistent or inaccurate data on topics required by the act. OPM officials anticipate that telework data will be more reliable next year because of the expected governmentwide implementation of automated data collection based on time and attendance records. As we have reported, OPM has concluded from research that the most reliable telework data are collected through time and attendance tracking systems. 14 Data collected through automated systems eliminate the need to track telework data by counting telework agreements or relying on estimates. Since 2003, OPM has consistently expressed concerns about the methods agencies use to collect telework data. In its previous telework reports to Congress, OPM has advocated for the development of an automated data collection system. OPM officials noted that OPM does not control what telework data responding agencies maintain, or their methods of data collection. Executive agencies provide telework participation and frequency data using a variety of methods, such as relying on estimates, counting telework agreements, and using automated time and attendance records to track telework participation. In an effort to collect more uniform data across agencies, OPM officials are standardizing definitions and data elements for use in automated time and attendance systems. For example, OPM has identified routine 14 GAO, Human Capital: Further Guidance, Assistance, and Coordination Can Improve Federal Telework Efforts, GAO-03-679 (Washington, D.C.: July 18, 2003). Page 11 GAO-12-519 Federal Telework Programs telework hours in a pay period as a data element for automated data collection and provided a standardized definition that will be used by all agencies using the Enterprise Human Resources Integration (EHRI) system. 15 OPM has introduced a timeline for modifying the existing EHRI system to allow OPM to collect telework data from executive agencies. According OPM, agencies will begin piloting these automated data collection systems for the 2012 telework data call. OPM began to discuss automated governmentwide data collection with the ITMG in July 2011. According to OPM’s timeline, OPM began to communicate internal requirements for automated telework data collection to telework managing officers in March 2012, but an OPM official has stated they do not expect full automation of telework data until 2013. This official also noted that different agencies have varying abilities to implement this new type of data collection and reporting mechanism, and considering different levels of comfort with new systems, it will take time to adjust to this method. However, continuous improvement efforts sometimes result in a trade-off between the desire for data consistency and a need to improve data collection and maintenance of a consistent data series over time. Because of the eventual planned move to automation, OPM may not be able to use 2011 data as a baseline. With the planned change to the method of data collection, it may not be possible to compare the 2011 data to future data. The 2011 data call requested data for a 2-month period, and some data call respondents relied upon estimates. Planned automation can provide a more uniform and accurate method for collecting telework data, however it may make comparisons using the 2011 telework data as a baseline difficult. However, OPM officials believe the 2011 data will provide an improved report of telework status because of standardized definitions and the more uniform time period of data collection. According to OPM officials, some executive agencies will need time to become comfortable with automated reporting systems and, during a period of transition to a new system, there could be initial reliability issues. However, these officials said that the 2011 data, notwithstanding its limitations, will be useful in identifying and understanding any major agency changes in reported participation that could occur during a transition to automated data collection. Such 15 According to the OPM website, the EHRI system offers a standardized approach for human resources record conversion, storage, access, sharing, and archiving. Page 12 GAO-12-519 Federal Telework Programs changes could alert OPM to possible transition related issues in agencies’ conversion to automated governmentwide data call collection efforts in 2012 and 2013. In addition, for those agencies already responding on the basis of time and attendance reporting, major changes in agency responses provide OPM the opportunity to confirm with agencies that the uniform definitions are being consistently applied. The Telework Enhancement Act of 2010 requires OPM to collect telework Conclusions data and report annually to Congress, which emphasizes the need for telework data to be valid and reliable. OPM revised the 2011 telework data call in order to allow it to meet some of the act’s reporting requirements and assist agencies in responding to the data call. This revision resulted in changes to terminology, as well as changes in the collection time period of requested telework data. OPM provided greater assistance to agencies through training on changes to the 2011 data call to improve accuracy of agency reporting of telework participation and frequency for the data call reporting period of September and October 2011. However, agencies use various methods, which OPM does not control, to report, collect, and maintain telework data, and this could affect the reliability of the telework data submitted. In addition, variation in training sessions and OPM’s uncertainty as to whether all respondents attended training, could lead to respondents’ potential misunderstanding of important terms and instructions. The validity and reliability of the reported 2011 telework data for some of the responding agencies may be questionable, and therefore agency telework participation and frequency data will not likely be comparable with previous data calls because of differences in definitions used, time periods of reporting, and individual agency tracking methods. With the revised 2011 data call, OPM establishes a baseline it could use to conduct a limited crosscheck of data collected through a governmentwide automated telework data collection system, which OPM plans to implement over the course of 2012 and 2013. OPM expects that automated data collection will provide it increasingly more reliable data on which to report progress. However, these efforts to improve future automated data collection may result in changes to agencies’ methods of data collection and a trade-off between the desire for consistency with previous data calls for comparison purposes and a need to improve overall data collection. Page 13 GAO-12-519 Federal Telework Programs To improve OPM’s annual reporting of telework to Congress, we Recommendations for recommend that the OPM Director take the following two actions: Executive Action • Ensure that the reliability limitations related to the 2011 telework data call are clearly reported in its June 2012 report to Congress by fully describing how existing measures of telework participation vary widely in validity and reliability and limit the capability of OPM to reliably report the actual level and frequency of telework participation. • Continue efforts to improve data collection and gather information that allows for the appropriate qualification of year-to-year comparisons and informs users about the effects of data collection changes going forward. We provided a draft of this report to the Director of OPM for review and Agency Comments comment. The Associate Director of OPM provided written comments, and Our Evaluation which we have reprinted in appendix III. In summary, OPM partially concurred with our first recommendation and fully concurred with the second. OPM highlighted a number of actions the agency has under way or plans to undertake in response. For the first recommendation, OPM noted that inadequate methods of data collection exist at the agency level and OPM continues to address this data reliability issue through training on evaluation and measurement. While this is an important step in addressing data reliability issues, OPM should ensure that telework data reliability limitations are clearly reported in their annual reports to Congress. For the second recommendation, OPM noted its continued plans to automate collection of telework data, and to regularly meet with telework managing officers and telework coordinators to keep them updated on changes to telework policy and data collection. OPM also provided a number of technical comments, which we incorporated as appropriate. We are sending copies of this report to the Chairman and Ranking Member of the Subcommittee on the Federal Workforce, U.S. Postal Service and Labor Policy, Committee on Oversight and Government Reform, House of Representatives; and the Director of OPM. In addition, this report will be available at no charge on the GAO website at www.gao.gov. If you have any questions about this report, please contact me at 202-512-6806 or firstname.lastname@example.org. Contact points for our Offices of Page 14 GAO-12-519 Federal Telework Programs Congressional Relations and Public Affairs may be found on the last page of this report. Key contributions to this report are listed in appendix IV. Yvonne D. Jones Director, Strategic Issues Page 15 GAO-12-519 Federal Telework Programs Appendix I: Objectives, Scope, and Appendix I: Objectives, Scope, and Methodology Methodology This report (1) describes the Office of Personnel Management’s (OPM) actions to respond to the requirements of the Telework Enhancement Act of 2010 and (2) assesses how OPM is handling and addressing identified data reliability issues in the 2011 telework data call. To address these two objectives, we reviewed relevant reports and guidance published by OPM that describe the status of telework programs across executive agencies and previous telework data calls and their instructions. We also reviewed previous GAO reports on telework and the reliability of OPM’s telework data. Lastly, we interviewed the OPM officials responsible for the planning, design, implementation, and analysis of the 2011 telework data call. This included discussions on the role of the Interagency Telework Measurement Group (ITMG), the process for developing the definitions and key terms used in the data call, the training and assistance provided to executive agency officials responsible for completing the data call, and agency plans to address outstanding data reliability issues associated with the data call. We conducted additional analysis to answer selected objectives as described below. To assess the extent to which the 2011 telework data call met generally accepted survey methodology, GAO internal experts in survey research identified principles from the Office of Management and Budget’s (OMB) Standards and Guidelines for Statistical Surveys relevant to assessing the 2011 telework data call. 1 We also used relevant aspects of GAO’s guide to Developing and Using Questionnaires. 2 Using the OMB principles, two analysts independently reviewed the data call, supporting documentation, and clarifying information provided in interviews to assess the extent to which the data call methodology met research practices. The initial rate of agreement across 15 rated practices, including those rated as having insufficient information to judge, was 12 of 15, or 80 percent agreement. The rating for the three practices on which there was initial disagreement were reconciled by the two analysts conducting the review and the reconciled rating was then reviewed by a third analyst with survey 1 OMB, Standards and Guidelines for Statistical Surveys (Washington, D.C.: Sept. 2006). See www.whitehouse.gov/omb (last accessed Nov. 10, 2011). 2 GAO, Developing and Using Questionnaires, GAO/PEMD-10.1.7 (Washington, D.C.: Oct. 1993). Page 16 GAO-12-519 Federal Telework Programs Appendix I: Objectives, Scope, and Methodology expertise. The third analyst did not recommend any changes to the reconciled ratings. We also simulated completion of the Web-based data call, accessing and responding to the data call in the same manner as executive agency respondents. Table 1 outlines the generally accepted survey research principles, derived from OMB’s guidelines, which we used in our assessment. This is not an exhaustive list of all OMB guidelines. 3 When we completed this review in April of 2012, OPM had not yet completed analyzing and reporting on the results of telework data call. Based on this, we could not yet assess whether the data call met all of the OMB principles related to data analysis and reporting. Some OMB principles, such as those related to sample design, were not appropriate to apply to the telework data call. These principles were therefore excluded as not relevant to our review. Table 1: Principles Used to Assess the Data Call Development of Concepts, 1. The survey had a clear rationale Methods, and Design 2. Potential users were consulted to identify their requirements and expectations 3. A review of existing, surveys, studies, and reports, or other literature informed the survey 4. The scope of survey data items are defined and justified 5. To ensure data remain consistent over time, key variable and survey procedures remained consistent 6. A clear, logical, and easy-to-follow flow of questions from a respondents’ point of view 7. The questionnaire was pretested and all components of the final survey system were field tested Collection of data 1. The sample frame was appropriate 2. The respondents were provided appropriate informational materials 3. Questions are clearly written 4. The questionnaire design minimized respondent burden and maximized data quality 5. Response maximization efforts were appropriate 6. Agency staff developed proper protocols for handling respondent questions to help ensure data quality 7. Protocols to monitor data collection activities were developed Processing and editing of data 1. Appropriate checks and edits on the data collection system mitigated errors Source: GAO and OMB, Standards and Guidelines for Statistical Surveys (Washington, D.C.: Sept. 2006). 3 OMB standards and guidelines are also not designed to be completely exhaustive of all efforts that an agency may undertake to ensure the quality of its data. OMB encourages agencies to develop additional standards focused on their specific activities. Page 17 GAO-12-519 Federal Telework Programs Appendix I: Objectives, Scope, and Methodology We conducted this performance audit from June 2011 through April 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Page 18 GAO-12-519 Federal Telework Programs Appendix II: Revisions to OPM’s Telework Data Appendix II: Revisions to OPM’s Telework Data Call to Collect Information to Prepare for Reporting under the Telework Enhancement Call to Collect Information to Prepare for Act of 2010 Reporting under the Telework Enhancement Act of 2010 2010 Telework data call 2011 Telework data call Instructions Time period: Based on the agency’s calendar year Time period: Based on the months of Sept. and Oct. 2011.a (Jan. 1 to Dec. 31, 2009). Participation: agencies invited to participate in data Participation: agencies required to submit telework data to call. OPM. Reporting deadline: March 2010 Reporting deadline: Dec. 2011 Definitions Telework: Telework: “Telework refers to any arrangement in which an “Telework is a work arrangement that allows an employee to employee regularly performs officially assigned duties perform work, during any part of regular, paid hours, at an at home or other work sites geographically approved alternative worksite (e.g., home, telework center). convenient to the residence of the employee.” This definition of telework includes what is generally referred to as remote work but does not include any part of work done while on official travel or mobile work. See the following clarifications on remote and mobile work as telework.” Employee: Employee: Not defined. Employee refers to federal civilian employees excluding military personnel and contractors. Employee may also include full-time, part-time, and/or intermittent employees. Eligibility: Eligibility: Agencies have the discretion to determine the An employee is eligible to participate in telework if all of the telework eligibility requirements for their employees. following parameters are true: For reporting purposes, in this Call for data, ALL • The employee has not been officially disciplined for being employees are considered eligible EXCEPT those absent without permission for more than 5 days in any employees whose: calendar year. • positions require, on a daily basis (every work • The employee has not been officially disciplined for day), direct handling of secure materials, or on- violations of subpart G of the Standards of Ethical site activity that cannot possibly be handled Conduct for Employees of the Executive Branch. remotely or at an alternate worksite; • Teleworking does not diminish the employee’s • most recent federal government performance performance or agency operations. rating of record (or its equivalent) is below fully successful or conduct has resulted in disciplinary • Participation and performance complies with the action within the last year. requirements and expectations of his/her telework agreement. • The employee’s official duties do not require on a FULL daily basis (ALL DAY, every work day): • direct handling of secure materials determined to be inappropriate for telework by the agency head; or • on-site activity that cannot be handled remotely or at an alternate worksite. • The employee and/or the employee’s position are not disqualified based on additional criteria established by the organization. Page 19 GAO-12-519 Federal Telework Programs Appendix II: Revisions to OPM’s Telework Data Call to Collect Information to Prepare for Reporting under the Telework Enhancement Act of 2010 2010 Telework data call 2011 Telework data call Types of telework: Types of telework: None provided • Routine: telework that occurs as part of an ongoing, regular schedule, and • Situational: telework that is approved on a case-by-case basis, where the hours worked were not part of a previously approved, ongoing and regular telework schedule (e.g., telework as a result of special work assignments or doctor appointment.)b Source: OPM. Notes: This table is intended to illustrate the differences between the 2010 and 2011 data calls. OPM made changes to the 2011 data call in response to the Telework Enhancement Act of 2010. a Under the Telework Enhancement Act of 2010, agencies were required to establish telework policies by June 2011. b Situational telework is sometimes also referred to as episodic, intermittent, unscheduled or ad-hoc telework. Page 20 GAO-12-519 Federal Telework Programs Appendix III: Comments from the Office of Appendix III: Comments from the Office of Personnel Management Personnel Management Page 21 GAO-12-519 Federal Telework Programs Appendix III: Comments from the Office of Personnel Management Page 22 GAO-12-519 Federal Telework Programs Appendix IV: GAO Contact and Staff Appendix IV: GAO Contact and Staff Acknowledgments Acknowledgments Yvonne D. Jones, (202) 512-6806 or email@example.com GAO Contact In addition to the contact named above, William Doherty, Assistant Staff Director, and Keith O’Brien, analyst-in-charge, led the development of this Acknowledgments report. Virginia Chanley, Patricia Donahue, Robert Gebhart, Jill Lacey, and Joseph Santiago made significant contributions to this report. Karin Fangman provided legal counsel. Shirley Hwang, Jessica Nierenberg, and Kathleen Padulchick verified the information in the report. Page 23 GAO-12-519 Federal Telework Programs Related GAO Products Related GAO Products Emergency Preparedness: Agencies Need Coordinated Guidance on Incorporating Telework into Emergency and Continuity Planning. GAO-11-628. Washington, D.C.: July 22, 2011. Human Capital: Telework Programs Need Clear Goals and Reliable Data. GAO-08-261T. Washington, D.C.: November 6, 2007. Human Capital: Greater Focus on Results in Telework Programs Needed. GAO-07-1002T. Washington, D.C.: June 12, 2007. Agency Telework Methodologies: Departments of Commerce, Justice, State, the Small Business Administration, and the Securities and Exchange Commission. GAO-05-1055R. Washington, D.C.: September 27, 2005. Human Capital: Key Practices to Increasing Federal Telework. GAO-04-950T. Washington, D.C.: July 8, 2004. Human Capital: Further Guidance, Assistance, and Coordination Can Improve Federal Telework Efforts. GAO-03-679. Washington, D.C.: July 18, 2003. (450930) Page 24 GAO-12-519 Federal Telework Programs GAO’s Mission The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no Obtaining Copies of cost is through GAO’s website (www.gao.gov). Each weekday afternoon, GAO Reports and GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, Testimony go to www.gao.gov and select “E-mail Updates.” Order by Phone The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, http://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Connect with GAO Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. Visit GAO on the web at www.gao.gov. Contact: To Report Fraud, Waste, and Abuse in Website: www.gao.gov/fraudnet/fraudnet.htm E-mail: firstname.lastname@example.org Federal Programs Automated answering system: (800) 424-5454 or (202) 512-7470 Katherine Siggerud, Managing Director, email@example.com, (202) 512- Congressional 4400, U.S. Government Accountability Office, 441 G Street NW, Room Relations 7125, Washington, DC 20548 Chuck Young, Managing Director, firstname.lastname@example.org, (202) 512-4800 Public Affairs U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Please Print on Recycled Paper.
Federal Telework: Program Measurement Continues to Confront Data Reliability Issues
Published by the Government Accountability Office on 2012-04-19.
Below is a raw (and likely hideous) rendition of the original report. (PDF)