oversight

Modernizing SSA Disability Programs: Progress Made, but Key Efforts Warrant More Management Focus

Published by the Government Accountability Office on 2012-06-19.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States Government Accountability Office

GAO          Report to the Chairman, Subcommittee
             on Social Security, Committee on Ways
             and Means, House of Representatives


June 2012
             MODERNIZING SSA
             DISABILITY
             PROGRAMS
             Progress Made, but
             Key Efforts Warrant
             More Management
             Focus




GAO-12-420
                                              June 2012

                                              MODERNIZING SSA DISABILITY PROGRAMS
                                              Progress Made, but Key Efforts Warrant More
                                              Management Focus
Highlights of GAO-12-420, a report to the
Chairman, Subcommittee on Social Security,
Committee on Ways and Means, House of
Representatives



Why GAO Did This Study                        What GAO Found
SSA administers two of the largest            The Social Security Administration (SSA) has taken steps that hold promise for
federal disability programs. GAO              improving the process for updating its medical criteria, but continues to face
designated federal disability programs        challenges ensuring timely updates. SSA now uses a two-tiered system for
as a high-risk area, in part because          ongoing revisions to its medical listings. First, it completes a comprehensive
eligibility criteria had not been updated     review of all medical conditions listed within each of 14 body systems, making
to reflect medical and technological          needed revisions. For subsequent updates for a body system, the agency uses a
advances and labor market changes.            targeted approach, selecting for review and revision only those medical
Given the size and cost of its disability     conditions most in need of change. To date, SSA has completed comprehensive
programs, SSA needs updated criteria
                                              revisions for 8 of the 14 body systems and now is reviewing conditions under
to appropriately determine who
                                              them to determine where targeted revisions are appropriate. However, some of
qualifies for benefits. GAO has been
asked to assess SSA’s efforts to
                                              these targeted revisions have experienced delays. Moreover, SSA has yet to
update its medical criteria and develop       complete comprehensive revisions for six body systems that have been ongoing
a new occupational information                for 19 to 33 years. SSA officials attributed delays to a lack of staff and expertise,
system, and to identify other steps           along with the complexity and unpredictability of the regulatory process.
taken to modernize disability                 SSA has embarked on an ambitious plan to design by 2016 an occupational
determination criteria. To do this, GAO       information system for use in its disability decision-making process, but has fallen
reviewed relevant publications and            short of best practices for estimating costs, maintaining a schedule, and
federal laws and regulations; assessed
                                              considering risks and alternatives. SSA currently relies on occupational
agency plans, cost estimates,
                                              information developed by the Department of Labor which has not had a major
schedules, and other documentation
against established project                   update since 1977. In 2008, SSA initiated a project to develop its own
management criteria; and interviewed          occupational information system (OIS), which SSA expects will provide up-to-
SSA officials, experts, and                   date information on the physical and mental demands of work to support its
stakeholders.                                 decision-making process. To guide the creation of its OIS, SSA established an
                                              advisory panel, collaborated with outside experts and other agencies, and in July
What GAO Recommends                           2011 issued a research and development plan detailing relevant activities
                                              through 2016. SSA has made progress on some baseline activities in the plan.
GAO recommends that SSA (1)
explicitly identify resources needed to       However, SSA’s cost estimate and schedule had key deficiencies, such as not
achieve its 5-year time frame for             including any estimate of the cost of producing, maintaining, and operating the
updating its medical listings; (2) follow     system, which can inform design options. SSA also did not adequately consider
best practices in its cost estimate,          inherent risks or potential alternatives, which could heighten the risk of additional
schedule, and risk assessment for the         costs or project failure.
occupational information system; and          Consistent with modern views of disability, SSA has taken some concrete steps
(3) conduct limited, focused studies on
                                              toward greater consideration of an individual’s ability to function with a disability
how to more fully consider assistive
                                              but faces constraints in fully modernizing. SSA has incorporated some criteria
devices and workplace
accommodations in its disability
                                              into its medical listings to determine whether a claimant’s impairments result in
determinations. SSA agreed with the           functional limitations that can prohibit the ability to work. SSA is also sponsoring
first two recommendations and                 research through the National Institutes of Health to evaluate how functional
disagreed with the third, stating that        abilities can further be considered in determining disability. One project aims to
such studies would be inconsistent            develop a computerized tool to assist adjudicators in evaluating how various
with Congress’ intentions. GAO                impairments affect an individual’s function and ability to work. However, SSA
continues to believe the                      officials maintain that other modern concepts of disability cannot be fully
recommendation has merit, as                  incorporated into SSA’s disability decisions. Specifically, SSA faces constraints
discussed more fully within the report.       considering the extent to which assistive devices and workplace
                                              accommodations can mitigate work disability, because these are not universally
View GAO-12-420. For more information,
contact Daniel Bertoni at (202) 512-7215 or   available and SSA lacks the resources to conduct individualized assessments.
bertonid@gao.gov.

                                                                                        United States Government Accountability Office
Contents


Letter                                                                                          1
               Background                                                                       2
               SSA Has Improved Its Process for Updating Medical Listings, but
                 Still Faces Challenges with Timely Updates                                     7
               SSA Has Begun an Ambitious Project to Develop Updated
                 Occupational Information, but Many Uncertainties Remain                      14
               SSA Has Taken Other Steps to Modernize Criteria but Faces
                 Limitations                                                                  25
               Conclusions                                                                    30
               Recommendations for Executive Action                                           31
               Agency Comments and Our Evaluation                                             31

Appendix I     Objectives, Scope, and Methodology                                             35



Appendix II    Progress Updating Six Body Systems since Last
               Comprehensive Revision                                                         38



Appendix III   Description of Cost Estimating Best Practices                                  39



Appendix IV    Description of Scheduling Best Practices                                       40



Appendix V     Comments from the Social Security Administration                               41



Appendix VI    GAO Contact and Staff Acknowledgments                                          44



Tables
               Table 1: Key Dates for Comprehensive Revisions to Body System
                        Listings                                                              11
               Table 2: Status of Select OIS Activities as of May 2012                        17
               Table 3: Assessment of OIS Cost Estimate                                       20
               Table 4: Assessment of OIS Schedule                                            21



               Page i                               GAO-12-420 Modernizing SSA Disability Criteria
          Table 5: Six Sound Project Management Practices Used to Evaluate
                   SSA’s Efforts                                                                    36


Figures
          Figure 1: SSA’s Five-Step Sequential Evaluation Process for
                   Determining Disability                                                           4
          Figure 2: Status of Comprehensive and Targeted Revisions for
                   SSA’s Body System Listings                                                       9




          Abbreviations

          ADA               Americans with Disabilities Act of 1990
          DI                Social Security Disability Insurance
          DOT               Dictionary of Occupational Titles
          Labor             Department of Labor
          ICF               International Classification of Functioning, Disability and
                            Health
          NIH               National Institutes of Health
          O*NET             Occupational Information Network
          OIS               Occupational Information System
          SSA               Social Security Administration
          SSI               Supplemental Security Income



          This is a work of the U.S. government and is not subject to copyright protection in the
          United States. The published product may be reproduced and distributed in its entirety
          without further permission from GAO. However, because this work may contain
          copyrighted images or other material, permission from the copyright holder may be
          necessary if you wish to reproduce this material separately.




          Page ii                                     GAO-12-420 Modernizing SSA Disability Criteria
United States Government Accountability Office
Washington, DC 20548




                                   June 19, 2012

                                   The Honorable Sam Johnson
                                   Chairman
                                   Subcommittee on Social Security
                                   Committee on Ways and Means
                                   House of Representatives


                                   Dear Mr. Chairman:

                                   The Social Security Administration (SSA) manages two of the largest
                                   federal disability benefit programs—Social Security Disability Insurance
                                   and Supplemental Security Income—which together resulted in payments
                                   of more than $165 billion to about 13.5 million people with disabilities and
                                   their families in fiscal year 2010. 1 Given the extensive size and cost of its
                                   disability programs, SSA must have current and appropriate criteria by
                                   which to assess whether an applicant’s medical conditions affect his or
                                   her ability to perform work in the national economy. However, we
                                   designated federal disability programs as high risk, in part because the
                                   medical criteria and occupational information that SSA relies on to make
                                   benefit decisions were found to be out of date. Moreover, we and others
                                   have found SSA’s disability programs—which historically have tended to
                                   equate severity of a medical condition with an inability to work—to be out
                                   of sync with a more modern concept of disability that considers the
                                   presence or lack of assistance that an individual with medical
                                   impairments might need to function more effectively in his or her
                                   environment.

                                   You recently asked us to assess SSA’s plans and efforts for revising its
                                   disability criteria. Accordingly, we examined the status and management
                                   of SSA’s efforts to update its medical listings to reflect current medical
                                   knowledge and develop a new occupational information system to reflect
                                   labor market changes, and we identified other steps taken by SSA to
                                   incorporate a modern view of disability into its eligibility criteria.




                                   1
                                    These data are from the fiscal year 2012 President’s Budget.




                                   Page 1                                     GAO-12-420 Modernizing SSA Disability Criteria
             To perform this work, we reviewed our prior reports and SSA Office of
             Inspector General reports; relevant federal laws and regulations; program
             documentation, including policies, procedures, strategic goals, and
             supporting project plans and cost estimates; relevant literature; and
             position papers and testimonies from disability groups and commissions.
             We interviewed SSA officials, key project contractors and stakeholders,
             disability experts, and representatives of other agencies that administer
             disability programs. We also evaluated SSA’s plans to update its criteria
             against sound project management and cost estimating practices. We
             narrowed our scope to criteria used for initial adult disability
             determinations. See appendix I for more on our scope and methodology.

             We conducted this performance audit from April 2011 through June 2012
             in accordance with generally accepted government auditing standards.
             Those standards require that we plan and perform the audit to obtain
             sufficient, appropriate evidence to provide a reasonable basis for our
             findings and conclusions based on our audit objectives. We believe that
             the evidence obtained provides a reasonable basis for our findings and
             conclusions based on our audit objectives.


             SSA administers two disability programs: the Social Security Disability
Background   Insurance (DI) program, enacted in 1956, and the Supplemental Security
             Income (SSI) program, enacted in 1972. 2 In order to be eligible for DI or
             SSI benefits based on a disability, an individual must meet the definition
             of disability for these programs—that is, they must have a medically
             determinable physical or mental impairment that (1) prevents the
             individual from engaging in any substantial gainful activity and (2) has
             lasted or is expected to last at least 1 year or result in death. 3

             To determine eligibility for both programs, SSA uses a five-step
             sequential process that is intended, in part, to expedite disability



             2
              The DI program provides monthly cash benefits to eligible applicants who have a
             sufficient work record. The SSI program provides monthly benefits to eligible applicants
             with limited income and resources who are disabled, blind, or age 65 or older.
             3
              42 U.S.C. §§ 423(d)(1)(A) and 1382c(a)(3)(A). Substantial gainful activity is generally
             work activity involving significant physical or mental activities that are done for pay or
             profit, whether or not a profit is realized. 20 C.F.R. §§ 404.1572 and 416.972. In 2012, the
             substantial gainful activity threshold was $1,690 per month for blind recipients and $1,010
             per month for individuals with other disabilities.




             Page 2                                       GAO-12-420 Modernizing SSA Disability Criteria
decisions when possible and limit administrative costs by conducting less
intensive assessments at earlier steps (see fig. 1). 4

•   At steps 1 and 2 of the process, SSA determines whether an applicant
    is working and meets income thresholds, as well as the medical
    severity of impairments. If not working (or not meeting income
    thresholds) and with the determination of a severe impairment, the
    applicant moves to step 3 of the process.

•   At step 3, SSA examiners assess the applicant’s medical impairments
    against the Listings of Impairments, also known as the medical
    listings, which are organized into 14 major body systems for adults
    and reflect medical conditions that have been determined by the
    agency to be severe enough to qualify an applicant for benefits. 5
    When using the listings to determine eligibility, SSA generally relies
    on information on the applicant’s diagnoses, including laboratory
    findings, diagnostic tests, and symptoms, as well as some limited
    consideration of the applicant’s functional limitations. If the individual’s
    impairment meets or is equal in severity to one or more of those in the
    listings, the individual is determined to have a disability at step 3. 6

•   If a disability determination is not made at step 3, SSA performs an
    assessment of the individual’s physical and mental residual functional
    capacity. 7 Based on this assessment, SSA determines whether the


4
See 20 C.F.R. §§ 404.1520 and 416.920.
5
  According to SSA officials, there are almost 300 medical impairments organized in 119
listings under the 14 major body systems for adults.
6
 SSA has also implemented Compassionate Allowances to quickly identify diseases and
other medical conditions that invariably qualify under the medical listings based on
minimal, objective medical information. According to SSA officials, many of the
Compassionate Allowances conditions are rare, and thus unfamiliar to examiners, such as
certain cancers and early-onset Alzheimer’s disease. In April 2012, SSA announced that
165 conditions would be on the list effective in August 2012, up from 113 conditions as of
the time this report issued in June 2012. The Compassionate Allowances initiative is one
of two parts of the agency’s fast-track system for certain disability claims. When combined
with the Quick Disability Determination process, which uses a predictive model to identify
cases with a high likelihood of being allowed, Social Security reported approving more
than 100,000 fast-track cases in 2009, usually in less than 2 weeks.
7
 SSA’s physical and mental residual functional capacity assessments establish the extent
to which an individual’s medically determinable impairments, including any related
symptoms, such as pain, may cause physical or mental limitations or restrictions that may
affect his or her capacity to do work-related physical and mental activities.




Page 3                                      GAO-12-420 Modernizing SSA Disability Criteria
                                             individual is able to perform past relevant work (step 4) or any work
                                             that is performed in the national economy (step 5). To inform
                                             determinations at steps 4 and 5, SSA uses a Department of Labor
                                             database—known as the Dictionary of Occupational Titles (DOT),
                                             which is an inventory of occupations performed in the national
                                             economy. At step 5, SSA also uses a set of rules and guidelines,
                                             referred to as the grid rules, to evaluate the combined effect of an
                                             individual’s physical residual functional capacity, age, education, and
                                             work experience. 8

Figure 1: SSA’s Five-Step Sequential Evaluation Process for Determining Disability




                                         8
                                          SSA relies on the grid rules to represent the total number of unskilled jobs in the nation at
                                         three physical strength levels as classified in the DOT, and to provide for consistent
                                         “rulemaking” or application of case facts to help ensure uniform decisions.




                                         Page 4                                        GAO-12-420 Modernizing SSA Disability Criteria
While originally created for expediency, 9 over time the medical listings used
at step 3 were relied on less to make program determinations as they
became increasingly outdated. In the early years of the program, more than
90 percent of cases were decided based on medical conditions specified in
the listings; in 2010 only 47 percent of allowances were made at step 3.
Experts attribute the decline in allowances based on the medical listings to
changes in the program, workplace, and medical treatment that the medical
listings had not kept pace with. Since the 1990s, we, along with SSA’s
Office of Inspector General and the Social Security Advisory Board, 10 have
expressed concerns that the medical listings being used no longer provide
current and relevant criteria to evaluate disability applicants’ inability to
work. In 2003, we deemed SSA’s and other federal disability programs as
high-risk areas, in part, because their programs continue to emphasize
medical conditions in assessing work capacity, without adequate
consideration of work opportunities afforded by advances in medicine,
technology, and changes in the labor market.

In 2008, we reported that SSA had recently established a new process—
referred to by SSA as the “business process”—for revising the listings to
better incorporate feedback into its continuous updates. 11 This process,
which has been in effect since 2003, incorporates feedback from multiple
parties, including medical experts and claims examiners, to update the
medical criteria. Under this process, SSA gathers external feedback from
comments associated with regulatory actions, such as the publication of
advanced notices of proposed rulemaking (advanced notices) and notices
of proposed rulemaking (notices) in the Federal Register. 12 In addition,


9
 Historically, SSA created the listings to have a clear set of medical conditions that, if
present, would preclude work. This in turn would limit the number of cases requiring a
more involved and individualized assessment of function.
10
  In 1994, when the Congress passed legislation establishing SSA as an independent
agency, it also created the seven-member bipartisan Social Security Advisory Board in
part to advise the President, Congress, and the Commissioner of Social Security on Social
Security and SSI policy. Pub. L. No. 103-296, § 103, 108 Stat. 1464, 1467.
11
 GAO, Federal Disability Programs: More Strategic Coordination Could Help Overcome
Challenges to Needed Transformation, GAO-08-635 (Washington, D.C.: May 20, 2008).
12
   Any changes to the medical listings have to proceed according to an established
process for rulemaking outlined in federal law. As such, when changes are made to the
listings, a notice of proposed rulemaking must generally be published in the Federal
Register with a public comment period before the final rule is issued. 5 U.S.C. § 553. Also,
the Office of Management and Budget must review and approve any notice that is
economically significant (i.e., has costs or savings greater than $100 million).




Page 5                                        GAO-12-420 Modernizing SSA Disability Criteria
this process includes conducting postimplementation reviews one year
after a revision is made to assess the impacts of a revised listing, areas to
improve, and whether expectations from the revisions have been
achieved. 13

With respect to information on jobs in the national economy that supports
SSA’s occupational criteria, we and others have reported that the DOT is
outdated, although SSA still relies on it to assess eligibility at steps 4 and
5 of the process. The DOT has not had a major update since 1977 14 and
the Department of Labor (Labor) replaced it with a new database in 1998
called the Occupational Information Network (O*NET). 15 However, SSA
determined that O*NET is not sufficiently detailed for evaluating DI and
SSI disability claims, so SSA has begun developing its own occupational
information system to better reflect the physical and mental demands of
work in the national economy.

Beyond dated medical criteria and vocational information, numerous
disability experts have expressed concern that SSA’s disability
programs—which statutorily require that an adult’s medical condition
prevents that person from engaging in substantial gainful activity—have
historically tended to equate the severity of medical conditions with
inability to work and thus are out of sync with modern concepts of
disability. Modern concepts focus on an individual’s functional abilities in
the workplace environment, including consideration of the presence or
lack of assistance, for example, per the requirements for reasonable
accommodation by the Americans with Disabilities Act of 1990. 16 These
modern views are reflected in the International Classification of
Functioning, Disability and Health (ICF), which is the World Health
Organization’s framework for assessing health and disability. This
framework takes into account the interaction of health conditions and


13
  Such reviews typically rely on reviews of decision data as well as surveys of field staff
responsible for applying the revised listings.
14
  The DOT was updated in 1991, but for less than 20 percent of the occupations.
15
  O*NET is a general purpose occupational information database used for workforce
development, economic development, career exploration, and academic and policy
research. Selected occupations within the O*NET system are updated each year.
According to Labor officials, by July 2012, 900 occupations will have been
comprehensively updated, and 413 will have more than one update.
16
  See, for example, 42 U.S.C. § 12112(b)(5).




Page 6                                        GAO-12-420 Modernizing SSA Disability Criteria
                             contextual factors, such as products and technology, attitudes, and
                             services, on an individual’s functional capacity, rather than viewing
                             disability solely as a medical or biological issue. Along these lines,
                             experts have recommended that SSA incorporate more consideration of
                             individual function in its medical listings when doing so can improve their
                             use as a screening tool for determining inability to work. Further, several
                             groups, such as the Social Security Advisory Board 17 and the Urban
                             Institute, 18 have reported that SSA’s disability programs should focus
                             more on whether an individual can work given appropriate environmental
                             or other supports, and that SSA—through its demonstration authority or
                             other means—could play a role in determining how. These suggestions
                             are consistent with SSA’s long-term targeted outcomes, which include not
                             only regularly updating regulations and policies to incorporate the most
                             recent medical advances, but also making it easier for individuals with
                             disabilities to return to work.



SSA Has Improved Its
Process for Updating
Medical Listings, but
Still Faces Challenges
with Timely Updates

SSA Has Improved Its         Since our last review in 2008, SSA has made several changes that hold
Strategy to Update Medical   promise for improving its medical listings updates. First, the agency is
Listings                     using a two-tiered system for ongoing revisions to the listings. Under this
                             system, SSA first completes a comprehensive listings update for a body
                             system, which entails reviewing all the diseases and disorders listed
                             within that system and making necessary revisions. Second, following a
                             comprehensive revision, SSA will pursue a more targeted approach—that
                             is, SSA will conduct ongoing reviews and updates of a smaller number of
                             medical diseases or disorders within that body system. Agency officials



                             17
                               See Social Security Advisory Board, The Social Security Definition of Disability
                             (Washington, D.C.: October 2003).
                             18
                              David Wittenburg and Pamela Loprest, A More Work Focused Disability Program?:
                             Challenges and Options (Washington, D.C.: Nov. 1, 2003).




                             Page 7                                       GAO-12-420 Modernizing SSA Disability Criteria
told us that targeted updates should be completed more quickly than
comprehensive updates, allowing them to focus on the most critical
changes needed. 19 As of early March 2012, SSA had begun the ongoing
review process to identify opportunities for targeted revisions for 8 of the
14 adult body systems that were recently comprehensively revised.

Another change, according to agency officials, is that in 2010, the SSA
Commissioner set a 5-year cycle for updating listings for each body
system. 20 Previously, SSA established cycles for periodically updating
listings under each body system, ranging from 3 to 8 years, but frequently
extended them. SSA officials believe that conducting targeted reviews will
generally allow the agency to conclude any necessary revisions prior to
the 5-year period. Additionally, they expect that using the “business
process,” which requires early public notification of changes and obtaining
necessary data and feedback from internal and external parties, should
help keep continuous reviews on track. See figure 2 for the status and
expiration date by which the listings should be reviewed and updated, if
needed, for the 14 body systems undergoing review for either
comprehensive or possible targeted revisions, as of early March 2012.




19
  Officials noted that these ongoing reviews could also result in major or even no
changes, as appropriate.
20
   The 5-year period will be applied to listings under a body system upon completion of
their current revision. SSA began applying the 5-year period in 2011 when it
comprehensively updated the endocrine body system listings. For other body system
listings updated prior to 2011, SSA generally assigned periods extending beyond 5 years.




Page 8                                      GAO-12-420 Modernizing SSA Disability Criteria
Figure 2: Status of Comprehensive and Targeted Revisions for SSA’s Body System
Listings




a
 For the two-tiered revision process, the beginning point is the date of the last comprehensive or
targeted revision.
b
 SSA officials told us they have grouped the disorders in the special senses and speech listings into
two broad sections—vision and hearing—and two standalone medical conditions. SSA has not yet
comprehensively revised the two standalone medical conditions that cover speech and disturbances
of labyrinthine-vestibular function. The two sections have undergone comprehensive revisions and
are undergoing reviews that began in 2007 for the vision section and in 2010 for the hearing section.
Special senses and speech listings are set to expire in 2015.
c
 According to SSA officials, SSA published limited revisions for the hematological disorders in 1988. It
also published final rules for limited revisions for the mental disorders listings in 2000 and the
musculoskeletal system listings in 2001.

SSA has made another change by more extensively engaging the
medical community to identify ways to improve the medical listings. For
example, SSA contracted with the Institute of Medicine to study its
medical criteria for determining disability and to make recommendations
for improving the timeliness and accuracy of its disability decisions,


Page 9                                            GAO-12-420 Modernizing SSA Disability Criteria
                           resulting in a 2007 report with recommendations 21 and a symposium of
                           experts in 2010. SSA has addressed some of the Institute of Medicine’s
                           recommendations, such as making better use of its administrative data to
                           update criteria and creating a standing committee through the Institute of
                           Medicine to provide recommendations for listings revisions. In addition,
                           through the Institute of Medicine, SSA created consensus committees to
                           conduct research and provide other assistance with updating SSA’s
                           cardiovascular listings and its Human Immunodeficiency Virus (HIV)
                           listing, which culminated in two reports with recommendations. 22


SSA Has Experienced        SSA has made progress, but continues to face delays in completing both
Delays with Its Revision   comprehensive and ongoing targeted updates. As of early March 2012,
Process                    SSA officials told us they had completed comprehensive revisions of
                           listings for eight body systems, some of which resulted in significant
                           changes. For example, in 2011, SSA removed the endocrine body system
                           listings for all the adult disorders, such as diabetes, because they found
                           that they were now generally diagnosed early and treated very
                           successfully. 23 Nevertheless, according to SSA, the agency still needs to
                           complete the comprehensive revisions for listings of the remaining six
                           body systems—a process that has been ongoing for the last 19 to 33
                           years, with numerous extensions beyond the original expiration periods
                           (see table 1). For example, it has been at least 27 years since SSA
                           finalized comprehensive revisions for two of the six body system
                           listings—mental and neurological disorders, which are among those SSA
                           uses most frequently in its eligibility determination process. 24 SSA has
                           made progress on four of the six body systems set to expire in 2012, but
                           will likely miss targeted time frames. SSA issued a notice of proposed
                           rulemaking to revise its listings of mental disorders in 2010 and has told


                           21
                             Institute of Medicine, Improving the Social Security Disability Decision Process
                           (Washington, D.C.: 2007).
                           22
                             For the reports, see Institute of Medicine, Cardiovascular Disability: Updating the Social
                           Security Listings (Washington, D.C.: 2010), and HIV and Disability: Updating the Social
                           Security Listings (Washington, D.C.: 2010).
                           23
                             SSA officials noted that while the most common endocrine disorder, diabetes, can be
                           disabling, this is due to its effects on other organs such as the heart or kidneys; as such,
                           they can be assessed under listings for these body systems or the effects on functionality
                           can be assessed at steps 4 or 5.
                           24
                            According to SSA officials, SSA made limited, but not comprehensive, revisions to the
                           mental disorders listings in 2000.




                           Page 10                                      GAO-12-420 Modernizing SSA Disability Criteria
                                       us that it plans to finalize this comprehensive revision by the end of 2012,
                                       after its current expiration date in July 2012. 25 SSA is still developing
                                       notices for the neurological and two other body system listings, which are
                                       also set to expire in 2012. Because SSA has generally taken more than a
                                       year to move from publishing the notice to final revision and may also
                                       need additional time for any internal revision and review, as well as a
                                       review by the Office of Management and Budget, it is also unlikely that
                                       SSA will meet its planned 2012 time frames for updating these listings.
                                       See appendix II for details on progress updating these listings.

Table 1: Key Dates for Comprehensive Revisions to Body System Listings

                                               Year of last            Years initially set                    Years since last
                                            comprehensive                 to review and                       comprehensive               Current
Body system and section                           revision                         revise                            revisiona   expiration dateb
Special senses and speechc                                1979                                  8                          33                2015
sections:
•    Speech
•    Disturbances of labyrinthine-
     vestibular function
Neurological disorders                                    1985                                  8                          27                2012
Mental disorders                                          1985                                  3                          27                2012
Hematological disorders                                   1985                                  8                          27                2012
Musculoskeletal system                                    1985                                  5                          27                2013
Respiratory system                                        1993                                  6                          19                2012
                                       Source: GAO analysis based on information provided by SSA officials.

                                       a
                                        According to SSA officials, SSA published limited revisions for the hematological disorders in 1988. It
                                       also published final rules for limited revisions for the mental disorders listings in 2000 and the
                                       musculoskeletal system listings in 2001.
                                       b
                                           SSA has continually extended the expiration dates of the listings.
                                       c
                                        The special senses and speech listings have two sections—vision and hearing—which have been
                                       comprehensively revised.

                                       Updates using SSA’s targeted approach seem to be moving at a faster
                                       pace than the comprehensive revisions, but some delays have occurred
                                       here as well, and more are expected. For example, SSA has already
                                       extended the expiration date for its targeted review of the cardiovascular
                                       system. According to SSA officials, it may also need to extend expiration



                                       25
                                            See 75 Fed. Reg. 51,336 (Aug. 19, 2010).




                                       Page 11                                                         GAO-12-420 Modernizing SSA Disability Criteria
dates for listings under two other body systems set to expire in 2012.
Extensions may be needed because SSA has not yet published the
notice of proposed rulemaking for those two body system listings and
may not have time to publish these notifications, respond to comments,
and complete the final updates by their current expiration date.

At the same time that SSA has been experiencing delays completing
timely revisions, agency officials reported challenges with other steps in
the business process. Whereas SSA’s business process includes a
postimplementation review of case data at the 1-year mark to determine
whether expectations from a revision were met, several did not undergo
this review, such as those for skin disorders, genitourinary impairments,
and impairments affecting multiple body systems. SSA officials told us
they only began conducting these reviews in 2010, and to date, SSA has
completed just one in 2011 that involved a targeted sample of 175 cases.
SSA officials told us they are conducting or planning to conduct two more
reviews at the 1-year mark in 2012. While disability experts we
interviewed spoke highly of SSA’s business and targeted review
processes to obtain feedback early on and update the listings more
promptly, ongoing delays raise questions about the agency’s ability to
fully follow its current business process while completing continuous and
timely revisions for all 14 body systems. 26

SSA officials offered two key reasons for the delays in updating the
listings: (1) limitations in the number and expertise of staff and (2) the
complexity and unpredictability of the regulatory process. According to
SSA officials, revising the medical listings requires research, deliberation,
testing, regulatory review, and consensus with many stakeholders, and
consequently is difficult and time-consuming to achieve. According to an
SSA official in the Office of Medical Listings Improvement, the office is
short-staffed and there is a lack of expertise needed to perform this work.
To address these constraints, SSA has contracted with the Institute of
Medicine to review and develop recommendations for revising two of the
body system listings. While not finalized as of March 2012, SSA officials
reported that the agency plans to renew this contract after it expires in
2012 and extend it to 2015. However, SSA has not yet determined how it




26
  According to agency officials, in addition to updating adult listings the agency must also
update listings for children, which adds to its workload and resource needs.




Page 12                                      GAO-12-420 Modernizing SSA Disability Criteria
will use the Institute of Medicine to revise the listings or the extent to
which the contract will address staffing shortfalls.

Also contributing to delays is the time required for internal review and
public comment under the regulatory process, which depends largely on
the number and the substance of comments received, according to an
SSA official. Obtaining public comment is one way SSA receives critical
information for identifying areas for revision and obtaining stakeholder
consensus. As such, SSA reviews and responds to each comment that
would result in a significant change, and the time required for doing so
varies depending on the number of comments and resources. For
example, SSA officials told us that the advanced notice of proposed
rulemaking to the listings of mental disorders in 2003 resulted in 500
comments, which took SSA 5 years to incorporate into a draft notice of
proposed changes for regulatory review. SSA officials reported they have
considered options for automating and thereby speeding up the process
of obtaining and reviewing public comment. For example, they told us that
they eventually plan to use a web-based tool to obtain early public
feedback on medical listing updates to help target their limited resources
and more quickly make changes. According to an SSA official, the agency
plans to retain its optional use of the advanced notice of proposed
rulemaking because it can also help to identify appropriate areas to focus
on to ultimately make timely updates. 27




27
  Under the Administrative Procedure Act, a notice of proposed rulemaking is generally
required for regulatory revision, but an advanced notice is not required. See 5 U.S.C. §
553.




Page 13                                     GAO-12-420 Modernizing SSA Disability Criteria
SSA Has Begun an
Ambitious Project to
Develop Updated
Occupational
Information, but Many
Uncertainties Remain

SSA is Designing a New     In 2008, SSA began a multiyear project to research and design a new
Occupational Information   source of occupational information that will replace the outdated
System, but Has Fallen     information currently being used to determine if claimants are able to do
                           their past work or any other work in the national economy. Since the
behind with Some           1960s, SSA has been using the DOT, which contains a list of job titles
Activities                 found in the national economy and had a last major update in 1977. 28 The
                           DOT provides SSA with descriptions of the physical demands of work—
                           such as climbing, balancing, and environmental requirements—for each
                           of the more than 12,000 occupations listed. 29 According to SSA, these
                           descriptions have been essential to its evaluations of how much a
                           claimant can do despite his or her impairment and whether this level of
                           functioning enables the claimant to do his or her past work or any other
                           work. After its last limited update, Labor decided to replace the DOT with
                           O*NET, which has far fewer occupational titles compared with the DOT
                           and has served Labor’s purposes more efficiently. According to an SSA
                           report, after investigating potential alternatives, SSA decided that O*NET
                           and other existing databases with occupational information were not
                           sufficiently detailed and not able to withstand legal challenges for use in
                           its decision-making process. SSA further decided to develop its own
                           occupational information system (OIS), which would contain detailed
                           information as in the DOT, but would also include additional information,



                           28
                             The DOT provides a wide range of occupational information that could be used for job
                           placement, occupational research, career guidance, labor-market information, curriculum
                           development, and long-range job planning.
                           29
                             In 1966, SSA contracted with Labor to produce a companion volume to the DOT entitled
                           the Selected Characteristics of Occupations that provides measures for additional physical
                           demands of work for DOT occupations, such as climbing, balancing, reaching, handling,
                           special senses requirements (visual acuity, hearing, etc.), and environmental
                           requirements (noise levels, exposure to cold, etc.).




                           Page 14                                     GAO-12-420 Modernizing SSA Disability Criteria
such as the mental demands of work. In addition, SSA has determined
that the OIS will (1) meet its legal, program, and data requirements; (2) be
flexible enough to incorporate changes in its policies and processes; and
(3) be able to be updated to reflect the evolving workplace environment.

In 2008, SSA began taking steps to guide the development of its OIS.
SSA created an internal office and working group, as well as an
Occupational Information Development Advisory Panel. While the number
of panel members has fluctuated over time, in April 2012, 14 external
experts were serving on the panel and represented various affiliations,
such as medicine, disability law, rehabilitation, and industrial
organizational issues. 30 The advisory panel holds quarterly public
meetings and has several subcommittees that review material and make
recommendations to SSA on developing various components of the OIS.
For example, in a 2009 report, the advisory panel supported the need for
SSA to develop a new source of occupational information, rather than
adapt O*NET, and recommended the type of data SSA should collect, as
well as approaches for classifying occupations.

To further inform its efforts, SSA has sought input from agencies or
organizations that either collect occupational information or also use the
DOT. For example, SSA officials held initial meetings with Labor and U.S.
Census Bureau officials to gain information on sampling methods used for
O*NET, the Occupational Employment Statistics program, and U.S.
Census Bureau’s household surveys. 31 Additionally, since February 2011,
SSA and Labor have been in the process of approving a memorandum of
understanding to formalize their collaboration efforts on the new OIS.
According to an SSA official, as the OIS project progresses, SSA plans to
convene ad hoc roundtables with experts and other agency officials to


30
  The Occupational Information Development Advisory Panel was formed in December
2008 under the Federal Advisory Committee Act, and broadly includes representatives of
organizations that have used the DOT or other occupational information systems, and
subject matter experts. The panel was originally comprised of 11 members, including an
SSA official. Over time, some members resigned and additional members were added.
31
  The Occupational Employment Statistics program produces employment and wage
estimates for approximately 800 occupations. The U.S. Census Bureau’s household
surveys include (1) the American Community Survey, which is an ongoing survey that
provides annual data on demographics such as age, education, and disabilities, and (2)
the Current Population Survey, which is primarily a labor force survey, conducted every
month by the U.S. Census Bureau for the Bureau of Labor Statistics and provides data
such as the national employment rate.




Page 15                                     GAO-12-420 Modernizing SSA Disability Criteria
explore specific subject areas, such as sampling issues. Besides working
with Labor and U.S. Census Bureau officials, SSA officials and panel
members have sought input from other experts and current users of the
DOT, such as SSA disability adjudicators and external rehabilitation
professionals. SSA officials conducted a user needs analysis in 2009 and
have presented on the OIS project at events and conferences.

In July 2011, SSA published a detailed research and development plan
outlining all activities related to researching, developing, and testing the
key components of the OIS in order to implement it by 2016 at an
estimated total cost of $108 million. For example, the plan includes
several baseline activities to identify and study other occupational
information systems and various approaches for analyzing occupations
that may inform or could be leveraged in SSA’s OIS data collection. The
plan also includes activities to identify the primary occupational,
functional, and vocational characteristics of current beneficiaries. Other
key components of the plan include developing descriptions of work
requirements, such as the physical and mental demands for jobs;
developing data collection and analysis strategies; and identifying the
occupations, categories, and definitions that will constitute the structure of
the new OIS. SSA also plans to develop a strategy for piloting how it
would ultimately collect data for the OIS nationwide within this time frame.

As of May 2012, SSA had made progress on many of the baseline
activities outlined in its research and development plan for the OIS. 32 For
example, according to an SSA official, its investigation of existing
occupational information systems, now complete, resulted in useful
information about design issues other organizations have confronted and
mitigated when creating their own system. Additionally, SSA’s preliminary
analysis of its own administrative data identified the most frequently cited
occupations and functional and vocational characteristics of disability
applicants. SSA officials told us the agency will target the occupations
identified in this analysis for its pilot studies of the OIS. Also in 2011, SSA
completed a comprehensive framework for assessing an individual’s
capacity to work—key to informing the OIS content, according to SSA
officials—which was based on recommendations of outside experts as
well as SSA’s policy and program requirements.



32
  Some of the activities that were part of the research and development plan were
completed in years prior to the plan’s issuance.




Page 16                                    GAO-12-420 Modernizing SSA Disability Criteria
                                              While SSA has made progress on several key activities, agency officials
                                              delayed 2011 completion dates for certain activities and anticipate making
                                              additional changes to its timeline as a result of not meeting its staffing
                                              goals for fiscal year 2011 (see table 2). Some activities that were delayed
                                              by several months included finalizing reports for the baseline studies and
                                              conducting a literature review that would inform how occupations might
                                              be analyzed. SSA officials told us that they did not have enough staffing
                                              to complete all of the 2012 planned activities within the estimated
                                              schedule. Further, SSA officials said they did not have the budget to hire
                                              new staff in September 2011. To address this challenge, SSA officials
                                              hired consultants to meet some of their needs. SSA officials also met with
                                              the Office of Personnel Management to explore the possibility of an
                                              interagency agreement that would allow SSA to use one or two of the
                                              office’s industrial organizational psychologists to help on a part-time
                                              basis.

Table 2: Status of Select OIS Activities as of May 2012

Activity and objective                                                                    Fiscal year target    Status
Investigate existing OISs. To identify lessons learned about features, requirements, 2010-2011                  Completed in fiscal year
processes, and options through the examination of the development and operation                                 2012 (late)
of existing domestic and international OISs.
Occupational and medical-vocational study. To identify and record the primary             2010-2011             In progress
occupational, functional, and vocational characteristics of adult disability applicants                         (late)
at steps 4 or 5 at the initial and hearings levels.
Job analysis methodologies (performed by contractor). To identify existing job            2011                  Completed on time
analysis methods, their features, and their potential usefulness for OIS
development.
Business processes for recruiting, training, and certifying job analysts (performed       2011                  Completed on time
by contractor). To identify existing business processes used to recruit, train, and
certify job analysts and potential usefulness of these processes for OIS
development.
Identify OIS usability standards. To identify standards and criteria for assessing the    2009-2012             Completed 3 of 4
operational usability of the new OIS and to incorporate those standards into OIS                                activities (one completed
research and development activities.                                                                            late)
Identify OIS scientific standards. To identify relevant scientific standards,             2011                  Completed
guidelines, and best practices that enable SSA to meet its responsibilities under
applicable federal requirements, and to incorporate those standards into OIS
research and development activities.
Identify OIS legal requirements. To identify standards for evaluating and ensuring        2011                  Completed
that the new OIS accurately reflects the language and the intent of the vocational
requirements of the Social Security Act.
Identify OIS elements. To establish basic design parameters and requirements to           2012                  In progress
ensure that OIS meets SSA’s legal, scientific, and usability requirements for
occupational information.




                                              Page 17                                        GAO-12-420 Modernizing SSA Disability Criteria
Activity and objective                                                                                Fiscal year target     Status
OIS work taxonomy. To identify a comprehensive set of constructs that may form                        2011-2012              In progress
the basis of OIS. These constructs must reflect the physical and mental-cognitive
requirements of work and context of work that can be objectively measured or
delineated to develop an occupational classification.
                                            Source: GAO based on information provided by SSA officials.




SSA’s Cost Estimate and                     Our Cost Estimating and Assessment Guide 33 identifies a number of best
Schedule Fell Short of Best                 practices for effective cost estimating and scheduling that should result in
Practices in Key Areas                      reliable and valid cost estimates that management can use for making
                                            informed decisions. Per these criteria, the success of any program
                                            depends in part on having reliable cost estimates and a reliable schedule.
                                            A reliable cost estimate provides the basis for informed investment
                                            decision making, realistic budget formulation and program resourcing,
                                            meaningful progress measurement, proactive course correction when
                                            warranted, and accountability for results. A reliable schedule defines,
                                            among other things, when work activities will occur, how long they will
                                            take, and how they are related to one another. As such, the schedule not
                                            only provides a road map for systematic execution of a program, but also
                                            provides a means by which to gauge progress, identify and address
                                            potential problems, and promote accountability.

                                            We compared SSA’s cost estimate to three best practices for assuring a
                                            reliable cost estimate 34 and found SSA only minimally or partially met
                                            each of these (see table 3). First, we found SSA’s estimated cost of the
                                            OIS research and development phase—$108 million—was not well-
                                            documented. For example, after reviewing all of SSA’s documentation
                                            supporting this cost estimate as of December 2011, we did not find step
                                            by step documentation showing how the estimate was derived so that
                                            someone unfamiliar with the project could use the documentation to
                                            recreate the estimate and get the same results. Well-documented cost
                                            estimates are considered a best practice and without good
                                            documentation, SSA is not in a position to defend the reliability of its
                                            estimate. Additionally, SSA only partially met the best practice of having


                                            33
                                              GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and
                                            Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009).
                                            34
                                              We did not assess SSA’s estimate on a fourth criterion—credibility—which evaluates
                                            any limitations of the analysis because of uncertainty or biases surrounding data or
                                            assumptions, as this criterion is most applicable to major capital acquisitions.




                                            Page 18                                                       GAO-12-420 Modernizing SSA Disability Criteria
an accurate cost estimate. While SSA provided documentation showing
how some of their estimates compared to actual costs and the reasons
for any variances, SSA did not provide adequate documentation for us to
determine, for example, if the estimate took inflation into account or if
there were any calculation errors. Generally speaking, in the absence of a
detailed cost model, third party reviewers cannot be certain that cost
estimate calculations are accurate and account for all costs.

We also found the cost estimate was not comprehensive as it does not
include any costs beyond the research and development phase, such as
the costs of producing, maintaining, and updating the final data system,
which could be significant. SSA officials told us that they are still in the
process of determining what information will be included in the data
system, how it will be collected, and how many occupations will be
covered, each of which will influence the cost of developing and
maintaining the OIS. As such, they maintain that it is too early to estimate
the costs of the final system or the costs for maintaining the system.
According to industry best practices, cost estimates should be
comprehensive and include all costs necessary to achieve agency
objectives and should be updated as the agency proceeds with the
project and gains more information. At this point in the project, estimating
total costs of options under consideration could enhance decision making
by evaluating the potential tradeoffs of different designs. Without any
estimate for the cost of producing the OIS, SSA risks designing a system
that would not be a viable or affordable option to complete. Additionally,
without maintenance cost estimates, SSA is at risk of designing a system
that would be too costly to maintain on a regular basis resulting in
outdated information. Other federal information systems provide some
basis for estimating the cost of producing or maintaining an OIS. For
example, Labor officials indicated that the cost of maintaining O*NET—an
occupational information system that includes approximately 1,000
occupations and uses a paper or web survey to collect data—is roughly
$6 million a year. 35



35
  O*NET’s annual maintenance cost may constitute a lower bound estimate for the OIS,
since SSA has considered more intensive data collections than a paper- or web-based
survey questionnaire and a larger number of occupations. Another occupational
information system maintained by the U.S. Bureau of Labor Statistics—the Occupational
Employment Statistics program—costs around $30 million annually to maintain. This
program also covers about 800 occupations predominantly using a mail survey, but may
involve additional surveys, telephone interviews, or site visits to reach nonrespondents.




Page 19                                     GAO-12-420 Modernizing SSA Disability Criteria
Table 3: Assessment of OIS Cost Estimate

 Best practice                                                Extent best practice met
 Well-documented                                              Minimally met
 Accurate                                                     Partially met
 Comprehensive                                                Partially met
Source: GAO analysis of SSA’s cost estimate for the OIS.


Note: Not met—SSA provided no evidence that satisfies any of the criteria; minimally met—SSA
provided evidence that satisfies a small portion of the criteria; partially met—SSA provided evidence
that satisfies about half of the criteria; substantially met—SSA provided evidence that satisfies a large
portion of the criteria; and met—SSA provided complete evidence that satisfies the entire criteria. See
appendix III for more information on these best practices.


We also reviewed SSA’s master schedule for the OIS against nine best
practices for a reliable schedule, and found that SSA did not meet four
and minimally met five of these practices (see table 4). For example,
among those minimally met, we found a significant number of activities
within SSA’s schedule were not logically sequenced in the order that they
were going to be carried out (best practice 2) and activities that were
dependent on completion of a prior activity were not identified (best
practice 5, 6, and 7). For example, SSA officials and experts
acknowledged that activities associated with the “OIS work taxonomy”
part of the research and development plan—activities that together
determine which information should be included in the OIS—are an
essential building block that will inform other OIS activities, such as
developing the “OIS work analysis instrument,” which will determine how
OIS information will ultimately be collected.

Nevertheless, when we tested SSA’s schedule for sequencing and
linkages associated with the OIS work taxonomy activity, we found a
significant delay of almost 1,000 days that should have significantly
delayed the project actually barely affected the final completion date in
the OIS project. Such missing links between key activities in the schedule
represent broken logic that reduces the reliability of the forecasted dates.
The OIS schedule also did not meet the best practice of conducting a
schedule risk analysis (best practice 8), which is an essential tool for
project managers to understand the most important risks to the project
and focus on mitigating them. Based on the schedule provided to us, we
do not have any indication that SSA has considered the vulnerability
within its schedule for meeting time frames for individual activities which
would in turn impact the time frames for the entire project. Without
explicitly identifying risks to its schedule, SSA either does not know or is



Page 20                                                    GAO-12-420 Modernizing SSA Disability Criteria
                            not conveying the probability of completing research and development
                            activities on time, and its relationship to overall costs for this phase.
                            However, in order to conduct a schedule risk analysis the schedule has to
                            be properly sequenced and networked with all logic links in place for the
                            analysis to produce credible results.

                            Table 4: Assessment of OIS Schedule

                                                                                                        Extent best practice
                             Best practice                                                              met
                             1.    Capturing all activities                                             Minimally met
                             2.    Sequencing all activities                                            Minimally met
                             3.    Assigning resources to all activities                                Minimally met
                             4.    Establishing the duration of all activities                          Minimally met
                             5.    Integrating schedule activities horizontally and vertically          Not met
                             6.    Establishing the critical path for all activities                    Not met
                             7.    Identifying float between activities                                 Minimally met
                             8.     Conducting a schedule risk analysis                                 Not met
                             9.    Updating the schedule using logic and durations to                   Not met
                                   determine dates
                            Source: GAO analysis of SSA’s schedule for the OIS.


                            Note: Not met—SSA provided no evidence that satisfies any of the criteria; minimally met—SSA
                            provided evidence that satisfies a small portion of the criteria; partially met—SSA provided evidence
                            that satisfies about half of the criteria; substantially met—SSA provided evidence that satisfies a large
                            portion of the criteria; and met—SSA provided complete evidence that satisfies the entire criteria. See
                            appendix IV for more information on these best practices.



Other Challenges with OIS   Beyond gaps in their cost estimate and schedule of specific activities,
Development Could           SSA faces broader challenges that could impede the success of the OIS
Undermine the Project’s     design and implementation; however, SSA has not done a formal risk
                            analysis of these challenges. 36 While SSA officials said they had recently
Success
                            begun discussing various risks and how they might address them, they
                            said it was premature to provide us with this information. Examples of
                            potential challenges for OIS that experts and stakeholders we spoke with
                            cited include:



                            36
                              Risk assessment helps decision makers identify and evaluate potential risks to an
                            agency’s mission, so that countermeasures can be designed and implemented to prevent
                            or mitigate the risks. As part of a risk assessment, agencies could identify alternatives to
                            alter the likelihood or outcome of a high risk situation.




                            Page 21                                               GAO-12-420 Modernizing SSA Disability Criteria
•   SSA’s lack of expertise with designing an OIS. SSA does not have
    prior experience with designing a complex and, in some respects,
    unprecedented occupational information system. As such, SSA will
    need to depend on many outside experts and contractors to complete
    the system. While the Occupational Information Development
    Advisory Panel members can provide some expert counsel, most of
    its members lack the technical and scientific background essential to
    informing this complicated effort. To help design the OIS, SSA
    recently hired a research psychologist and plans to hire two more
    individuals in the near future, but may need additional resources. In
    March 2012, SSA put out a request for information and, according to
    an agency official, received ideas from knowledgeable experts about
    how to move forward with technical aspects of the project. Until the
    agency secures required and sufficient expertise, SSA may not be
    able to move ahead with key technical decisions, such as defining the
    number of occupations to include and the data collection
    methodology, which may cause further delays.

•   Cost of maintaining an OIS. SSA has not yet made design decisions
    on the OIS that will ultimately define the overall cost of producing,
    implementing, and maintaining the OIS system. For example, SSA
    officials told us that they were not sure how many occupations they
    would include in the OIS, although they believe the total number will
    be somewhere between the number of occupations in the O*NET and
    the DOT, or between approximately 1,000 and 12,000 occupations. A
    2010 study conducted by the National Academy of Sciences
    reviewing Labor’s O*NET cited the three major cost drivers of an
    occupational information system as (1) size of sample, (2) number of
    occupations, and (3) frequency of updates. This study noted that as
    the agency increases any one of these cost drivers, it does so at the
    expense of the others or of the overall cost. For example, if SSA
    chooses to frequently update the entire OIS, the agency may need to
    make tradeoffs with the number of occupations or sample size it
    chooses if it wishes to contain costs.

•   Managing large, multiyear projects. In the past, SSA has experienced
    difficulties managing complex, multiyear efforts. In prior work, we
    reported that SSA has cancelled numerous demonstration projects
    due to limitations or weaknesses in design or implementation and
    lacked sufficient controls to ensure effective management of its




Page 22                             GAO-12-420 Modernizing SSA Disability Criteria
     demonstration projects. 37 We have also reported on challenges that
     SSA faced managing two separate redesigns of its disability decision-
     making process. 38 While the OIS project alone represents a complex
     effort, stakeholders have emphasized the importance of this project
     progressing in close concert with other ongoing research managed by
     another part of SSA, placing greater emphasis on broad and effective
     project planning. 39 In light of past experiences and as SSA faces a
     potential change in leadership in 2013, the management
     shortcomings we identified increase the vulnerability of this complex
     project.

Given the range of challenges and potential risks they pose to the
success of the OIS project, it is important to identify and carefully explore
feasible alternatives that may mitigate these risks. Examples of potential
alternatives suggested by experts and other observers in the disability
field include:

•    Leveraging O*NET. In a report issued in 2010, SSA’s advisory panel
     found that it did not believe that O*NET would meet SSA’s needs, in
     part, because it is not detailed enough. However, the National
     Academy of Sciences also issued a report in 2010 and concluded that
     O*NET could be altered to better meet SSA’s needs for disability
     adjudication. SSA officials have told us that modifying O*NET would
     not result in savings for the agency, but they have not conducted any
     analysis to determine this. While most disability experts agree that
     O*NET in its current form would not be a suitable source of
     occupational information for SSA, the National Academy of Sciences
     study noted that there are potential linkages between O*NET and an



37
 GAO, Social Security Disability: Management Controls Needed to Strengthen
Demonstration Projects, GAO-08-1053 (Washington, D.C.: Sept. 26, 2008).
38
  SSA’s most recent redesign effort—the Disability Service Improvement initiative—was
suspended in large part in 2007 by the new Commissioner, due to budgetary and other
considerations.
39
  Specifically, while the OIS is identifying functional requirements in the workplace,
another SSA office is working through an interagency agreement with the National
Institutes of Health to identify methods to more accurately capture the functional capacity
of individuals. Both SSA and its advisory panel recognize that SSA’s disability criteria
must reflect both the capabilities of people and work demands, and that the OIS must be
designed and developed to support SSA’s process for assessing individual function. As
such, some stakeholders believe it is critical that these two efforts be fully aligned for the
overall success of the disability determination process.




Page 23                                       GAO-12-420 Modernizing SSA Disability Criteria
    SSA OIS that are worthy of more careful exploration on SSA’s part
    and that leveraging one system to serve broader purposes could be
    most cost-effective overall.

•   Adjusting the scope of the OIS. SSA could reduce costs by limiting the
    scope of the OIS. For example, the OIS could be designed to capture
    fewer occupations. SSA officials have told us that since they are early
    in their design, they do not know how many occupations will be
    included in their OIS. Since this decision is fundamental to informing
    the OIS data collection methods, feasibility, and cost, consideration of
    alternative scope should be given high priority attention.

•   Limiting data collection methods. Another key factor that SSA must
    consider is the method they will use to collect data for their OIS as
    some methods are more costly than others. For example, among the
    methods SSA is considering, conducting on-site job analyses of
    occupations would be more expensive than surveys, which are also
    more resource and time intensive than collecting information through
    telephone interviews.

•   Leveraging resources from other agencies and OIS users. Other
    federal agencies with experience in data collection and occupational
    information have resources that SSA could leverage. SSA officials
    told us that they have been considering how they could benefit from
    the infrastructure that other agencies such as Labor and the U.S.
    Census Bureau have for data collection, but that they have not
    identified specific ways to leverage those resources. Additionally,
    while SSA needs to assure the OIS serves its most basic needs, it
    may be appropriate for SSA to explore the possibility of cost sharing
    or applying users’ fees with organizations that currently rely on the
    outdated DOT and could ultimately benefit from using a more modern
    and comprehensive OIS database. 40




40
  Current users of the DOT include organizations in the private and public sectors, such
as companies providing life insurance and short and long term disability insurance, and
organizations providing state and federal vocational rehabilitation services, among others.




Page 24                                     GAO-12-420 Modernizing SSA Disability Criteria
SSA Has Taken Other
Steps to Modernize
Criteria but Faces
Limitations

SSA Has Taken Steps to        Although SSA’s adult disability programs were initially built upon the
Incorporate Modern            assumption that certain severe medical conditions equate to work
Concepts of Disability into   incapacity, through its medical listings updates and ongoing research, SSA
                              has taken steps to modify its eligibility criteria. SSA now is taking a more
Eligibility Criteria          modern view of disability that looks beyond the claimant’s medical condition
                              by giving greater consideration to his or her functional capacity, that reflects
                              the International Classification of Functioning, Disability and Health (ICF)
                              framework. 41 According to the 2007 Institute of Medicine report, a modern
                              concept of disability should recognize that disability is not just inherent in
                              the individual and his or her medical condition, but is the result of complex
                              interactions between the person, the person’s medical impairments,
                              assistive devices to which they have access, and features of his or her
                              socioeconomic environment, such as the presence or lack of accessible
                              transportation and workplace accommodations. Under this concept, as
                              described in this report, two people with the same impairment might have
                              different degrees of work disability for a variety of reasons.

                              Updates to the medical listings have been one vehicle through which SSA
                              can include an assessment of an individual’s functional abilities to
                              determine whether an impairment prevents work. For example, as part of
                              SSA’s comprehensive revision to the listings for the immune system, the
                              agency included several functional criteria, such as performing activities
                              of daily living, maintaining social functioning, and completing tasks in a


                              41
                                As noted earlier, the ICF focuses on ability to function despite a medical impairment,
                              including taking into account the impact of environmental factors, such as products and
                              technology, attitudes, and services. The World Health Organization developed the ICF as
                              a universal classification of disability and health for use in health and health-related
                              sectors, especially as a planning and policy tool for decision makers. All 191 World Health
                              Organization member states endorsed the use of the ICF as the international standard to
                              describe and measure health and disability. As a result, the international community now
                              has classification schema that provides both a common language and underlying code for
                              disability terminology. There are four basic ICF domains, with associated codes, in the ICF
                              classification: Body Functions, Body Structures, Activities and Participation, and
                              Environmental Factors.




                              Page 25                                     GAO-12-420 Modernizing SSA Disability Criteria
timely manner despite deficiencies in concentration or persistence.
Generally, SSA officials, adjudicators, and disability experts we spoke
with support incorporating appropriate functional criteria into the medical
listings to facilitate a more reliable assessment of an individual’s ability to
work. However, some have also noted that as SSA continues to more
broadly incorporate functional criteria into listings updates, such efforts
may result in a more subjective assessment by adjudicators that could
increase the difficulty of making step 3 disability determinations.

Since 2008, SSA has had an ongoing interagency agreement with the
National Institutes of Health (NIH) to conduct short- and long-term
research that has informed SSA’s efforts to incorporate functional
information into the disability criteria. For example, in its 2011 annual
report to SSA, NIH presented its findings on the use of functional criteria,
as defined by the ICF, in the adult listings. The objective of the project
was to quantify the comprehensiveness and consistency of functional
criteria among the adult listings and any influence this has in
determination of outcomes. NIH found that while the use of functional
terms were particularly apparent in five body systems, nearly one-half of
the 14 body systems did not capture the influence of health conditions
and impairment on human functioning. NIH is currently working on
another project, which examines the presence and consistency of
functional terms in the listings criteria. It is estimated the project will be
completed by 2013.

Besides exploring how medical listings could further incorporate
functional considerations, SSA is also sponsoring longer term research
through the NIH to develop an automated method to more quickly and
comprehensively evaluate a claimant’s functional abilities for its disability
determinations. Specifically, project researchers are developing a
computer-based tool to rapidly and reliably assess the functional abilities
of individual claimants considering their medical conditions. 42 As
envisioned, the claimant, a medical provider, or both would respond to a
series of questions through the computer-based tool, which would likely


42
   Boston University’s Health and Disability Research Institute is developing this tool under
a subcontract with NIH. The tool is based on Item Response Theory using Computer
Adaptive Testing. The tool is adaptive in the sense that each item that is administered is
tailored to the unique level of functioning of an individual. The items are selected based on
the individual’s prior responses assessing that individual’s level of function, while precision
is gained regarding a person’s placement along a continuum of functioning by asking
more questions at the level that is appropriate for that individual.




Page 26                                       GAO-12-420 Modernizing SSA Disability Criteria
take an hour to complete the entire battery, perhaps less if a more limited
assessment is warranted. The project’s intended scope encompasses six
areas of function consistent with the ICF activity domain: (1) mobility, (2)
interpersonal/social interaction and relationships, (3) self-care, (4)
communication, (5) general tasks and demands (such as multitasking and
carrying out daily routines), and (6) learning and applying knowledge. NIH
researchers said they plan to consider the use of common personal
assistive devices, such as wheelchairs, in developing the tool. As of April
2012, NIH had completed preliminary testing of instruments in two of the
six areas—mobility and interpersonal/social interaction and
relationships—with a sample of claimants and providers and had begun
analyzing the data. SSA and NIH officials anticipate several benefits from
the functional assessment tool, such as providing information on the
impact of impairments more consistently, comprehensively, and early in
the disability determination process. This information would help
adjudicators more accurately assess whether a person can perform
certain kinds of work given his or her functional and occupational
capabilities.

While this research is promising, there are several unknown variables.
SSA officials said they have not yet determined when or how the tool will
be integrated into the disability determination process. NIH officials
indicated they believe that the tool may be most useful early in the
determination process. SSA officials said they expect to pilot the
functional assessment tool after all relevant domains have been tested
and validated, which will likely be by 2016. Additionally, NIH researchers
told us that, ultimately, this tool, which collects information on the
applicant, would need to be aligned with the OIS, which collects
information on the demands of work. SSA and NIH officials reported they
have held meetings to share their progress to date, but both projects are
in the early stages and additional coordination will be needed going
forward.

SSA is in the preliminary stages of taking additional steps to broaden its
use of functional criteria throughout the disability determination process in
response to other findings by NIH. Specifically, NIH evaluated a number
of forms that SSA uses during the disability application and determination
processes to determine how well these forms captured information on
claimant functional activity as described by the ICF. The researchers
found that current SSA assessment processes had major gaps in
covering the ICF concepts. For example, SSA captured only limited
information relative to sensory experiences, such as watching or listening,
learning and applying knowledge, communication, and interpersonal


Page 27                              GAO-12-420 Modernizing SSA Disability Criteria
                           interactions and relationships, all of which NIH considered particularly
                           relevant to work. NIH concluded that the gaps need to be addressed in
                           order to characterize individual functioning more comprehensively in
                           relationship to the demands of the workplace. Another step the agency
                           reports that it plans to take is the issuance of a Federal Notice of
                           Solicitation of Collaboration from federal agencies in developing a
                           standard for coding functional capacity in federal disability programs
                           based on the ICF. The notice will explain that SSA believes using the ICF
                           would help standardize how agencies describe and measure different
                           aspects of disability, improve the clarity and comparability of research
                           findings, and strengthen the base of scientific knowledge that guides
                           public policies and health practices. In addition, SSA has asked the
                           Institute of Medicine to plan an international symposium focused on how
                           best to use and assess function in the disability determination process.
                           SSA officials also stated that the agency may ultimately revise the
                           residual functional capacity forms based on criteria from the ICF.


SSA Faces Constraints to   Consistent with modern views of disability, a key consideration in
Incorporating Modern       assessing disability is that the environment can hinder or enhance an
Concepts                   individual’s ability to function. While assistive devices and workplace
                           accommodations can play a critical role in an individual’s ability to
                           function in the work environment, SSA does not always consider them in
                           its assessment of disability. Regarding the incorporation of assistive
                           devices into the medical listings, SSA officials and experts we spoke with
                           expressed concern about the extent to which specific technologies should
                           be incorporated if they are not widely available. SSA officials told us that
                           they currently incorporate assistive devices into the medical listings once
                           these devices become standard in the medical community—a threshold
                           that SSA officials described as generally involving some combination of
                           availability, accessibility, and insurance coverage. After an assistive
                           device, such as a prosthetic device for walking, is incorporated into a
                           listing, adjudicators must evaluate the individual’s ability to walk with the
                           device being used. For example, evaluations of people who have had
                           amputations involving a lower extremity or extremities are to be done with
                           the prosthetic device in place. When we asked if wheelchairs are
                           considered standard in the medical community, and whether SSA
                           considers how individuals with wheelchairs might function in today’s
                           knowledge-based labor market, given their age and education, we heard
                           conflicting information from SSA officials. Some explained that individuals
                           in wheelchairs are generally allowed at step 3 on the basis of their
                           underlying medical condition, without evaluating how their disability might
                           be assessed at steps 4 or 5. Other officials maintained that according to


                           Page 28                              GAO-12-420 Modernizing SSA Disability Criteria
SSA’s policy, an individual in a wheelchair would be evaluated for upper
body strength or other medical issues. However, SSA officials we spoke
with said they have not evaluated these types of allowances specifically.

Regarding workplace accommodations, SSA officials said their policy is to
not consider them for several reasons. First, officials cited SSA’s inability
to ensure that workplace accommodations are provided by employers—a
concern shared by other disability experts we interviewed. SSA officials
also indicated the agency would be unable to assess the effectiveness of
workplace accommodations for claimants. Further, officials noted that
SSA already faces resource constraints managing its disability claims
workload and expanding the scope of individualized assessments would
exacerbate those constraints. Finally, they noted that data on the
availability and use of workplace accommodations are lacking.

SSA’s policy notwithstanding, some opportunities exist for SSA to learn
more about the availability of workplace accommodations. For example,
developing the OIS may provide an opportunity to obtain some, albeit
limited, information on workplace accommodations for the disability
determination process. Specifically, while SSA officials reported that they
do not plan to collect specific information on workplace accommodations
as they develop the system, they may collect more specific information on
the physical requirements of different jobs, such as options for sitting or
standing. Some experts we spoke with agreed that while information on
workplace accommodations would be immensely useful to include in the
OIS, given the current scope of that project, they agreed collecting this
information via the OIS would likely be too great a task for SSA to
accomplish with existing budget constraints and time frames. However,
another potentially viable option for collecting more information about
workplace accommodations might be the Disability Research Consortium,
which SSA is creating pursuant to section 1110 of the Social Security Act,
as amended. 43 SSA envisions the consortium as a 5-year cooperative
agreement that will serve as a national resource for fostering high quality
research, communication, and education on matters related to disability
policy, such as identifying or eliminating barriers encountered by people
with disabilities in returning to or maintaining work.




 See 42 U.S.C. § 1310.
43




Page 29                              GAO-12-420 Modernizing SSA Disability Criteria
              Researchers and disability experts have commented on the limitations of
              SSA’s current disability program eligibility criteria to fully consider whether
              an individual can function in the workplace due to limited consideration of
              assistive devices, including those in the workplace—limitations that we
              have also noted. 44 Moreover, officials we spoke with from an organization
              of vocational examiners expressed frustration when they see young
              individuals who could work with minor accommodations being provided
              disability benefits likely throughout their working life, rather than receiving
              support to pursue work. Representatives of the organization added that
              minor accommodations can include a stool for sitting or devices to assist
              with vision impairments. Although giving broad consideration of assistive
              devices and workplace accommodations may be difficult to incorporate
              into the current disability criteria and process, considering how common
              and inexpensive workplace supports may affect work disability seems
              feasible and reasonable.


              SSA has taken important and concrete steps toward modernizing its
Conclusions   disability criteria, but varied challenges may prevent the agency from
              achieving its goals. Specifically, SSA has a better process in place for
              updating medical listings but delays in meeting its own goals will likely
              continue unless the agency explicitly identifies and assigns the resources
              needed to achieve them. SSA is also taking bold and needed steps to
              replace outdated occupational information with a new OIS. However,
              especially in an era of governmentwide fiscal constraints, success of its
              OIS may hinge on SSA prioritizing its analysis of risks and feasible
              alternatives to address potential funding shortfalls and other significant
              challenges. Regardless of the shape and scope of the OIS, absent a
              complete, reliable, and transparent cost estimate and schedule, SSA risks
              schedule and performance shortfalls. Finally, SSA is sponsoring
              promising research to increase consideration of functional ability in its
              medical listings and other aspects of eligibility criteria—research that
              experts believe must be aligned with the design of SSA’s new OIS system
              at the earliest opportunity to best serve the disability determination
              process. On the other hand, SSA is not considering the full range of
              assistive devices and workplace accommodations available today. While
              SSA and others raise valid concerns as to their universal availability and



              44
                GAO, SSA Disability Programs: Fully Updating Disability Criteria Has Implications for
              Program Design, GAO-02-919T (Washington, D.C.: July 11, 2002).




              Page 30                                     GAO-12-420 Modernizing SSA Disability Criteria
                      other considerations, in the absence of studies on how certain assistive
                      devices and workplace accommodations are playing a role in helping
                      individuals with impairments to stay at work or return to work, and their
                      costs in comparison with potentially many years of disability benefit
                      payments, SSA may be missing an opportunity to potentially assist
                      individuals with disabilities to re-engage with the workforce and to target
                      finite resources efficiently and effectively.


                      1. To achieve the goal of updating listings for each body system within
Recommendations for      SSA’s 5-year time frame, we recommend that the Commissioner of
Executive Action         Social Security explicitly identify the resources needed to achieve this
                         goal, such as staff, contractors, and technology aids, and its plans to
                         overcome any resource limitations.

                      2. To ensure that its work to revise occupational information is feasible
                         and cost effective, and to improve its chance for success, we
                         recommend that the Commissioner of Social Security:

                          •     formally assess risks to the success of the OIS—addressing such
                                challenges as related to controlling cost, acquiring expertise,
                                managing project complexity, and coordinating with ongoing and
                                related SSA research—and develop appropriate mitigation
                                strategies, and

                          •     develop a comprehensive and reliable cost estimate and schedule
                                for the life cycle of the project, in accordance with best practices.

                      3. To help ensure that SSA’s disability decisions are as equitable and
                         consistent with modern views of disability as possible, we recommend
                         the Commissioner of Social Security conduct limited and focused
                         studies on the availability and effects of considering more fully
                         assistive devices and workplace accommodations in its disability
                         determinations.


                      We provided a draft copy of this report to SSA and the Departments of
Agency Comments       Commerce, Health and Human Services, and Labor for review and
and Our Evaluation    comment. The Department of Commerce did not provide comments.
                      SSA, Labor, and the Department of Health and Human Services provided
                      technical comments, which we incorporated as appropriate. SSA also
                      provided written comments, which are reproduced in appendix V.




                      Page 31                                 GAO-12-420 Modernizing SSA Disability Criteria
SSA agreed with the first two recommendations and disagreed with the
third recommendation that the agency should conduct limited and focused
studies on the availability and effects of considering more fully assistive
devices and workplace accommodations in its disability determinations.
SSA stated that conducting such studies is inconsistent with Congress’
intentions. Specifically, SSA noted that the Americans with Disabilities
Act of 1990 (ADA), as amended, addresses requirements for workplace
accommodations. SSA referred to a 1999 Supreme Court case which
acknowledged the complexity of SSA involvement in determining the
availability of reasonable accommodations and also noted that ADA
determinations are separate from disability determinations under the
Social Security Act. SSA asserts that because Congress has made no
effort to change the balance between its disability programs and the ADA
in the past 22 years, it would be inappropriate for SSA to spend its limited
administrative resources on “an initiative that would undermine the
balance Congress chose to strike.”

Notwithstanding SSA’s response, we continue to believe that SSA should
conduct limited and focused studies on the availability and effects of
considering more fully assistive devices and workplace accommodations
in its disability determination process. Although SSA asserts that
workplace accommodations are addressed by the ADA, this act serves a
very different purpose than SSA’s disability programs. We do not think
that the fact that workplace accommodations are addressed by the ADA
would necessarily preclude SSA from potentially considering them in
making disability determinations. Further, although we agree with SSA
that Congress has not explicitly directed the agency to consider
incorporating assistive devices and workplace accommodations into its
disability determinations process, we also note that Congress has not
explicitly prohibited SSA from making such considerations. Because the
language of the Social Security Act is silent on this issue, and in the
absence of clear Congressional intent, we think it would be reasonable for
SSA to conduct limited studies on the feasibility of considering such
factors.

SSA further cites the 1999 Supreme Court decision in Cleveland v. Policy
Management Systems Corporation to point out that assessing reasonable
accommodation may turn on highly disputed workplace-specific matters,
and that an SSA misjudgment about that detailed matter could deprive a
disabled person of the financial support the statute provides. While this is
true, SSA fails to acknowledge that by not incorporating such factors, it
may be providing benefits to individuals who are capable of working with
accommodations, thereby potentially expending scarce government


Page 32                              GAO-12-420 Modernizing SSA Disability Criteria
resources in a manner that may not be most appropriate. SSA’s response
acknowledges that the agency’s resources are limited—a concern that we
share. By conducting limited and focused studies on this issue, SSA
would be in a better position to thoughtfully weigh the costs and benefits
of these various policy options before deciding on an appropriate course
of action.

As we discussed in the report, SSA has avenues to research the
availability of select devices or accommodations and the impact of their
inclusion on disability determinations, such as through the SSA funded
Disability Research Consortium, which is charged to conduct research to
identify and eliminate barriers encountered by people with disabilities in
returning to work or maintaining their ability to work. SSA could also
incorporate discussion of assistive devices and workplace
accommodations into its planned symposium with the Institute of
Medicine that will assess the inclusion of functional measures in the
disability determination process. Through these efforts, SSA may be able
to identify certain common and widespread workplace accommodations
that could be incorporated into the disability criteria, just as it incorporates
certain medical treatments and assistive devices into the medical listings
when they become a medical standard. Despite SSA’s efforts to better
address an individual’s functioning in its disability criteria, without at least
evaluating the costs and benefits of considering more assistive devices
and workplace accommodations, SSA is likely to face ongoing barriers in
its efforts to integrate a more modern concept of disability into its
determination process.

As agreed with your office, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from the
report date. At that time, we will send copies of this report to the
Commissioner of the Social Security Administration and the Secretaries
of Commerce, Health and Human Services, and Labor; relevant
congressional committees; and other interested parties. In addition, this
report will be available at no charge on GAO’s website at
http://www.gao.gov.




Page 33                                GAO-12-420 Modernizing SSA Disability Criteria
If you or your staff have any questions about this report, please contact
me at (202) 512-7215 or bertonid@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. GAO staff who made key contributions to this
report are listed in appendix VI.

Sincerely yours,




Daniel Bertoni, Director
Education, Workforce, and Income Security Issues




Page 34                             GAO-12-420 Modernizing SSA Disability Criteria
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              We were asked to assess the Social Security Administration’s (SSA)
              plans and efforts to revise its disability criteria. Specifically, we examined
              the status and management of efforts to update its medical listings to
              reflect current medical knowledge and develop a new occupational
              information system (OIS) to reflect labor market changes, and we
              identified other steps taken by SSA to incorporate a modern view of
              disability into its eligibility criteria. For this review, we narrowed our scope
              to criteria used for initial adult disability determinations that involve the
              medical listings at step 3 or a medical-vocational assessment at steps 4
              and 5 of SSA’s five-step sequential process.

              To determine the status of activities to revise the medical listings and to
              develop the OIS, we reviewed our prior reports on the subject and SSA
              Office of Inspector General reports, relevant federal laws and regulations,
              and program documentation. This documentation included policies, such
              as those listed in SSA’s Program Operations Manual System for
              examiners; strategic goals for fiscal years 2011 and 2012 presented in
              the Agency Performance Plan for 2012; and other guidance. We also
              interviewed SSA officials from relevant offices with direct responsibility for
              revising disability criteria, as well as those offices that provide support for
              these efforts, and key project contractors and stakeholders. To obtain
              contextual information on modernization, we reviewed relevant literature,
              including studies, position papers, and testimonies from disability groups
              and commissions and interviewed disability experts. We assessed the
              reliability of data used in this report and found it to be sufficiently reliable
              for our purposes.

              To determine the extent to which SSA’s efforts to revise its medical
              listings and develop a new OIS were anchored in sound project
              management practices, we first identified sound project management
              practices by reviewing A Guide to the Project Management Body of
              Knowledge, our guidance on internal controls, and the Government
              Performance and Results Act of 1993. We also identified our recent work
              (for fiscal years 2008, 2009, and 2010) that evaluated federal planning
              efforts. Through these efforts, we identified six practices for sound project
              management which we used as a framework for evaluating SSA’s efforts
              to revise the medical listings and develop the OIS. Although there is no
              established set of requirements for all plans, we determined that these
              practices help implementing parties and decision makers effectively
              shape policies, programs, priorities, and resource allocations so that they
              can achieve desired results while ensuring accountability. While these
              practices may be organized in a variety of ways or use different terms, for
              the purposes of this report, we grouped them into six categories, from


              Page 35                                GAO-12-420 Modernizing SSA Disability Criteria
                                             Appendix I: Objectives, Scope, and
                                             Methodology




                                             plan conception to implementation. Given the differences among SSA’s
                                             efforts to update the medical listings and develop the OIS, we selected
                                             broad, higher-level criteria that may apply to a wide variety of projects or
                                             plans (see table 5).

Table 5: Six Sound Project Management Practices Used to Evaluate SSA’s Efforts

Project management
practice                    Description
Plan to identify problem,   In establishing the problem, goals, and methodology, the agency identifies the problem to be addressed
goals, and methodology      and the causes of the problem, as well as the strategy, including goal and objectives, and the methodology
                            for achieving these goals and objectives.
Activities and timeline     An agency should identify and document the specific activities that must be performed in order to complete
                            the project. The agency should develop a schedule that defines, among other things, when work activities
                            will occur, how long they will take, and how they are related to one another, as well as interim milestones
                            and checkpoints to gauge the completion of the project.
Resources                   The agency should identify the sources and types of resources or investments needed (e.g., budgetary,
                            human capital, information technology, research and development, contracts) to complete project activities.
                            If resource availability cannot be assured, the agency will need to establish alternate plans. The agency
                            should develop a reliable estimate of the costs of these resources.
Coordination                The agency should identify stakeholders—individuals and organizations that are involved in or may be
                            affected by project activities—and ensure that they are included in developing and executing the project
                            plan, allowing them to contribute appropriately. In addition to internal communications, management
                            should ensure there are adequate means of communicating with, and obtaining information from, external
                            stakeholders that may have a significant impact on the agency achieving its goals.
Risk                        Risk assessment generally includes estimating the significance of risks from both external and internal
                            sources, assessing the likelihood of its occurrence, and deciding how to manage the risk.
Performance evaluation      The agency should describe how goals will be achieved and establish performance indicators to be used in
                            measuring or assessing the relevant outputs, service levels, and outcomes of each program activity, and
                            identify a process to monitor and report on progress.
                                             Source: GAO analysis based on Project Management Institute information, the Government Performance and Results Act of 1993, and
                                             GAO reports.


                                             Additionally, we compared the cost estimate and schedule for completing
                                             SSA’s OIS and related documents with best practices in our Cost
                                             Estimating and Assessment Guide. We compared these practices to the
                                             OIS project because it requires a significant commitment of resources
                                             and time by SSA to complete and will result in an end product. For SSA’s
                                             OIS cost estimate and schedule, we scored each best practice as follows:

                                             •     Not met—SSA provided no evidence that satisfies any of the criteria.

                                             •     Minimally met—SSA provided evidence that satisfies a small portion
                                                   of the criteria.




                                             Page 36                                                       GAO-12-420 Modernizing SSA Disability Criteria
Appendix I: Objectives, Scope, and
Methodology




•   Partially met—SSA provided evidence that satisfies about half of the
    criteria.

•   Substantially met—SSA provided evidence that satisfies a large
    portion of the criteria.

•   Met—SSA provided complete evidence that satisfies the entire
    criteria.

We provided the results of our schedule and cost analyses to SSA
officials and met with them to confirm the results. Based on the interviews
and additional documentation provided by SSA officials, we updated the
results of our analyses, as needed.




Page 37                              GAO-12-420 Modernizing SSA Disability Criteria
Appendix II: Progress Updating Six Body
              Appendix II: Progress Updating Six Body
              Systems since Last Comprehensive Revision



Systems since Last Comprehensive Revision




              Note: SSA’s special senses and speech listings include impairments associated with speech and
              balance (i.e., disorders of the labyrinthine-vestibular function).




              Page 38                                        GAO-12-420 Modernizing SSA Disability Criteria
Appendix III: Description of Cost Estimating
                                        Appendix III: Description of Cost Estimating
                                        Best Practices



Best Practices


Characteristica   Characteristic description
Comprehensive     The cost estimate should include all costs necessary to achieve agency objectives including government and
                  contractor labor costs as well as any necessary material or equipment costs. Comprehensive cost estimates
                  should be structured in sufficient detail to ensure that cost elements are neither omitted nor double counted.
                  Specifically, the cost estimate should be based on a standardized structure that allows a program to track cost
                  and schedule consistently over time. Finally, where information is limited and judgments must be made, the cost
                  estimate should document all cost-influencing ground rules and assumptions.
Well documented   The cost estimate should be supported by detailed documentation that describes the purpose of the estimate,
                  the program background and system description, the scope of the estimate, the ground rules and assumptions,
                  all data sources, estimating methodology and rationale, and the results of the risk analysis. Moreover, this
                  information should be captured in such a way that the data used to derive the estimate can be traced back to,
                  and verified against, the sources.
Accurate          The cost estimate should be based on an assessment of most likely costs (adjusted for inflation), documented
                  assumptions, historical cost estimates, and actual experiences on other comparable programs. Estimates should
                  be cross-checked against an independent cost estimate for accuracy, double counting, and omissions. In
                  addition, the estimate should be updated to reflect any changes.
                                        Source: GAO analysis.

                                        a
                                          We did not assess SSA’s estimate on a fourth characteristic—credibility—which evaluates any
                                        limitations of the analysis because of uncertainty or biases surrounding data or assumptions, as this
                                        criterion is most applicable to major capital acquisitions.




                                        Page 39                                           GAO-12-420 Modernizing SSA Disability Criteria
Appendix IV: Description of Scheduling Best
                                               Appendix IV: Description of Scheduling Best
                                               Practices



Practices


Best practice                      Description
Capturing all activities           A schedule should reflect all activities defined in the program’s work breakdown structure and include
                                   all activities to be performed by the government and contractor.
Sequencing all activities          The schedule should be planned so that all activities are logically sequenced in the order they are to
                                   be carried out.
Assigning resources to all         The schedule should realistically reflect the resources (i.e., labor material and overhead) needed to
activities                         do the work, whether all required resources will be available when needed, and whether any funding
                                   or time constraints exist.
Establishing the duration of all   The schedule should reflect how long each activity will take to execute.
activities
Integrating schedule activities    The schedule should be horizontally and vertically integrated—that is, it should link already
horizontally and vertically        sequenced activities with outcomes while also delineating the relation of supporting tasks and
                                   subtasks to upper-level milestones. Such mapping among levels enables different groups to work to
                                   the same master schedule.
Establishing the critical path for The schedule should identify the critical path, or those activities that, if delayed, will negatively impact
all activities                     the overall project completion date. The critical path enables analysis of the effect delays may have
                                   on the overall schedule.
Identifying float between          The schedule should identify float—the amount of time an activity can slip in the schedule before it
activities                         affects other activities—so that flexibility in the schedule can be determined. As a general rule,
                                   activities along the critical path typically have the least amount of float.
Conducting a schedule risk         The schedule should include a schedule risk analysis that uses statistical techniques to predict the
analysis                           probability of meeting a completion date. A schedule risk analysis can help management identify and
                                   understand the most important risks and focus on mitigating them.
Updating the schedule using        The schedule should use realistic durations for activities and be monitored to determine when
logic and durations to             forecasted completion dates differ from the planned dates. This analysis can be used to assess
determine dates                    whether schedule variances will affect future work.
                                               Source: GAO analysis.




                                               Page 40                                        GAO-12-420 Modernizing SSA Disability Criteria
Appendix V: Comments from the Social
             Appendix V: Comments from the Social
             Security Administration



Security Administration




             Page 41                                GAO-12-420 Modernizing SSA Disability Criteria
Appendix V: Comments from the Social
Security Administration




Page 42                                GAO-12-420 Modernizing SSA Disability Criteria
Appendix V: Comments from the Social
Security Administration




Page 43                                GAO-12-420 Modernizing SSA Disability Criteria
Appendix VI: GAO Contact and Staff
                  Appendix VI: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  Daniel Bertoni, (202) 512-7215 or bertonid@gao.gov
GAO Contact
                  In addition to the contact named above, Michele Grgich, Assistant
Staff             Director; James Bennett; Kate Blumenreich; Tisha Derricotte; Jennifer
Acknowledgments   Echard; Julie DeVault; Alex Galuten; Sheila McCoy; Patricia M. Owens;
                  Carol Petersen; Karen Richey; Anjali Tekchandani; Kathleen Van Gelder;
                  and Walter Vance made key contributions to this report. Also contributing
                  to this report were Jaime Allentuck, Susan Bernstein, and Amy Frazier.




(131069)
                  Page 44                              GAO-12-420 Modernizing SSA Disability Criteria
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.