oversight

Issues in Managing Applied Social Research

Published by the Government Accountability Office on 1977-03-03.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                         DOCUMENT RESUSE
03315 - [A1051741]
Issues in Managing Applied Social Research. March 3,   1977.   8 pp.
Staff study Conference of the Council for Applied Social
Research, Washington, DC; by Keith I. Marvin, Associate
Director, Program Analysis Div.
Insue Area: Accounting and Financial Reporting (2800); Internal
    Auditing Systems: Sufficiency of Federal Auditors and
    Coverage (201).
Contact: Program Analysis Div.
Budget Function: General Government: Central Fiscal Operations
     (803).
          The General Accounting Office (GAO) has become
increasingly involved in the review and audit of evaluative and
social experimental efforts. The primary concern of GAO in this
 Lea has been to recognize the utility and effectiveness of the
experimental methods employed while simultaneously assuring
Congress that it will fulfill its role to invedtigate all
matters relating to the receipt, disbursement, and application
of public funds. While carrying out its responsibilities to
Congress, GAO must, at the same time, minimize any problems
created by audits of social experiments. GAO has identified 14
issues pertaining to the movement of the methods of social
research into the policy process. GAO believes it should have
access to all aspects of a contracted experimental program. The
benefits of a more constructive public involvement in public
policymaking is probably worth the cost of answering the
resulting technical questions and issues. Answers to these
questions need to be carefully considered by all those
organizations and citizens who have an interest in building
public accountability into the social oxperimentation process.
(LDR)
                 'ISSUES IN MANAGING APPLIED SOCIAL RESEARCH
                                         BY
                                KEITH E. MARVIN*
                              ASSOCIATE DIRECTOR
                           PROGRAM ANALYSIS DIVISION
                         U.S. GENERAL ACCOUNTING OFFICE
           AT THE CONFERENCE OF THE COUNCIL FOR APPLIED SOCIAL RESEARCH
                      WASHINGTON, D.C., MARCH 3, 1977

     As a direct result of its historical mission to audit and review
programs of the Federal Government, the General Accounting Office has
increasingly become involved in the review and audit of evaluative and
social experimental efforts.     The perceived role of the GAO in this area
of Governmental activity has been slowly building.     Beginning with it's
work in the New Jersey Work Incentive Experiment, and extended into the
Performance Incentive Remedial Education Experiment, and most recently
in the Experimental Housing Allowance Program, GAO has come to realize
that it's participation in the experimental process involves it in some

long-standing issues, issues previously thought to be solely the province
of the experimental community.
     Early on, in the New Jersey experiment, it became apparent to the
General Accounting Office, that the preliminary release of data from
that experiment by the contracting agency raised questions     about the
right to access experimental data prior to the completion and the
publication of experimental work done for the Federal Government.


* The assistance of Dr. Stephen Baratz in this paper is acknowledged and
  appreciated, particularly his help in placing the issues and ideas
  in a timely science policy context.
  More recently, in the Experimental Housing Allowance
                                                       Experiment, the
  General Accounting Office has become concerned and
                                                     interested in the
  issue the audit and privacy of individually identified
                                                         data gathered   in
  support of the policy process in social experiments.
       The General Accounting Office has entered this area
                                                           with a large
  measure of deliberation and care. It's major overriding
                                                           concern has been
  to assure all parts of the interested public that
                                                    it recognizes the utility
 and effectiveness of the experimental methods employed
                                                         while it
 simultaneously assures the Congress of the maintenance
                                                         of GAO's role to
 investigate all matters, such as accuracy and completer
                                                          's,relating to
 the receipt, disbursement, and application of public
                                                      funds.
      Recognizing the significance of GAO's responsibility
                                                           in this area,
 the Comptroller General, Elmer B. Staats, called together
                                                           a meeting of
 expert and concerned citizens to explore with GAO
                                                   people, how GAO can
 both meet it's responsibilities to the Congress
                                                  while assuring that
 any problems created by audits of socidl experiments
                                                       are minimized. Some
 of the Directors of the Council for Applied Social
                                                     Research were among
those who met with the Comptroller General on May
                                                   18, 1976.
      In anticipation of that meeting, the GAO conducted
                                                         a considerable
search of the literature on the experimental method
                                                     and it's application
to the public policy context. The results of that
                                                    literature search
revealed that there were at least fourteen separate
                                                     sets of issues and
concerns which could be identified as resulting
                                                from the movement of the
methods of social research into the policy process
                                                   (from bibliogrpahy
listed at end).



                                      -2-
     For the purpose of my brief discussion with you today, I have
distilled those fourteen sets of issues into three conceptual clusters.       They are:
     1) Issues relating to the ethical and legal constraints and
trade-offs when social research moves into the policy process.     Included
ini this cluster of issues is the right to individual citizen privacy and
the need for confidentiality of individually identified data, versus
societies' right to know about that part of the universe covered by the
ope ration of Government programs, particularly where the responsibility
for public accountability is involved.    This issue, recognized by
researchers, contains elements of the precipitating issue which led to
the May 18th meeting.
     2) -Issues relating to the justification for, and the nature of
design review procedures, as well as other technical problems of implementa-
tion and management of social experiments.    Incluaed in this cluster is the
set of issues dealing with the need to consider alternative hypotheses and
experimental procedures.
     3) issues relating to the political, organizational and other
practical problems of implementation and management of social experiments.
Included here are issues relating to the need to consider alternative
methods for the maintenance of quality control of social experiments.
     The May 18th meeting at the GAO identified an additional cluster of
Issues on a Dar with the already identified fourteen issues.     The May 18th
meeting addressed but did not resolve all of the issues of how the society builds
public accountability (as defined by GAO's mandate) into the social
experimentation process.




                                    -3-
    GAO's opinion is that its mandate requires     its   access to all

aspects of a contracted experimental program.     The importance of this is

indicated by the fact that our individual credentials which we carry con-
t;in the key wow,' I referred to earlier "the Comptrollcr :'General shall
investigate, at the seat of Government or elsewhere, all macters relating
to the receipt, disbursement, and application of public funds.
     Perhaps Wilbur 'ampbell, Associate Director, in charge of the GAO
review of the Housing Allowance Experiment, has put more thought than
anyone else into how this applies to the audit of a large experiment.        At

the May 1976 nee+ing, he made the following statement, "I think the type
of experiment we are talking about needs to be considered; and that is
one where Fedcral funds are tUsed in experiments, paid to recipients for
their use, and it impacts on public policy in that--depending upon the
results of this experiment--it may result in a national program.         It seems

that this type of experiment is a little different from the experiment
where an individual researcher goes out on his own and does research for
some given purpose.   Here you have Federal funds and public policy and
it seems to us that the public has a right to know."
     This type of accountability, which implies much more thdn t:: 1t,

research profession policing itself, must be considered of at least ce.. 1
importance to the other 14 sets of issues.      This type of accountability

extends concerns and issues relating to the development of procedures for
methodological accountability, and the accountability for professionally
competent treatments contained in an experiment.
     The literature reviewed is diverse with respect to how one resolves
many of the issues raised.   Ina sense, all of these issues apply to
experiments on human subjects regardless of the size of the experiment.
The methods of peer review and other institutional review of human
experimentation have appeared.to be generally accepted and appear to have
worked reasonably well when used in the way they were intended to be
applied.
     No attempt is made in these remarks to propose specific refinements
in those well established methods of managing human experimentation.
     This paper addresses the large social experiment and social research
which is intended, by government contract, to be generalizable for public
decisionmaking about the nature of future Government social programs and
it is to that set of considerations tne paper now turns.
     In line with the topic of the present panel, Policies for Managing
Applied Social Research, all important finding from an analysis of the
identified issue clusters described previously Isthat a good deal of
thought has gone into them; yet much or this thinking has been isolated
in the various disciplines and sub-disciplines in the social and behavioral
sciences without much cross-communication or cross-fertilization.
     For example, proper protection of detailed computerized con-
fidential data involves the Information and the Computer Sciences.

Cluarly, a number of different disciplines need to be involved in issues
about design review procedures. The issues of accountability overlap
the other issues and involve the Audit profession.
     Thus, one could conclude that it is vital that management of social
experiments include a proper mix of appropriate disciplines. To properly

                                    -5-
 balance many of the issues identified above, it appears that the membership
 of these interdisciplinary teams, particularly those involved in the
 planning process of an experiment, should be publicly available.    Also,
 the process by which they arrive at an acceptable, implementable, design
for experimentation should be well documented as well as made available
to the public concurrently.
     Auditing represents the public interest and thus the participation
of the audit discipline symbolizes the nee. for broader public in-
volvement in social experimentation intended to int!uence public policy.
If policymaking is to be expanded to include the results of social
experimentation then appropriate citizens need to be involved in a
public process starting in the early planning stages of a social
experiment; and throughout the different phases of the experiment itself.
The Washington Post for today March 3, 1977, has an article describing
a survey of 300,000 people, selected by the Census Bureau, to obtain their
views relevant to alleviating the energy shortage, for consideration by
the Administration in formulating energy policy.    Opinions are to be obtained
also from a large number of officials.     I mention this only to indicate
its feasibility, at modest cost relative to the cost of a major program.
Could a more targeted, smaller survey approach be helpful, for example,
in deciding the hypotheses which should be tested in major social experiments?
     Some may argue that it may be impossible to implement a major
social experiment with this kind of visibility and openness.    I do not
agree.   While there is a chance that it might delay or terminate some




                                     -6-
 proposals, I believe that this will be a temporary effect.   The public
education gained by such dissemination would benefit the planning and
implementation process in futu:'e experiments, es well as provide a
guarantee that the results would be used to the extent to which they are
applicable in the decisionmaking process. With the type of openness
described, I suspect that the Public would insist on no less.
     Questions relating to the above proposal come to mind immediately
that need consideration by all those involved.   Many questions have no
answers that are immediately available.   For example, would wide publicity
given to an experiment in advance of its implementation invalidate the
experimental design? In a more bureaucratic sense, would there be conflicts
with existing Federal procurement regulations in selecting who is going
to perform an experiment?
     The major conclusion of this paper is that the benefits of more
constructive public involvement in this important area of public policy-
making probably is well worth the cost of answering the kind of technical
questions raised by this analysis. Answers to these questions need to be
carefully considered and debated by all of those organizations and citizens
who have interest and responsibilities in the public policy use of social
experiments.




                                   -7-
                                Bibliography

 1. Adams, Stuart, Evaluative Reseagch in Corrections: A Practical
       Guide, Washington, D.C., U.S. Department cf Justice, 1975
 2.   Bernstein, Ilene Nagel and Howard E. Freeman, Academic and
         Entrepreneurial Research: The Consenuences of DiverSitv in
         Federal EvalJatiDn Studies, New York, Russell Sage, 1975
 3. Committee on Federal Agency Evaluation Research, Assembly of
       Behavioral and Social Sciences, National Acaaemy of Sciences,
       Protecting Individual Privacy in Evaluation Research, Weshington,
            T.C.5,
               195
 4. Gotlieb, Allan, Charles Dalfen and Kenneth Katz, The Transborder
       Transfer .f Information bv Communications and Computer Systems:
         Issues rd Approacres to Guidina Prirciples, American Journal
       of Inter:national Law, Vol. 68, No. 2, April 1974 (pp. 227-257)
 5. Hatry, Harry P., Richard E. Winnie and Donald M. Fisk, Practical
       Program Evaluation for State and local Government Officials,
       Washington, D.C., The Urban Institute, 1973 (pp. 56-6-2)
 6. Horowitz, Irving Louis and James Everett Katz, Social Science and
       Public Policy in the United States, flew York, Praeger, 1975
 7. Katz, Jay, Experimentation With Human Beings: The Authority of the
       Investigator, Subject, Professions, and State in the Human
       Experimentation Process, riew York, Russell Sage, 1972
 8. Levin, Henry M., Some Methodoloaical Problems in Economic Polic
       Research: Determinin^ Hcw r:uch Should be SDent on Compensatory
       Education, Education and Urban Society, Vol. 7, N;o. 3, May 1975
         (pp. 303-349)
 9. Nagle, Stuart S., Ed. Policy Studies and The Social Sciences,
       Lexington, Mass., Lexington Books, 1975
10. Noble, John H., Jr., Peer Review: Quality Control of Applied
       Social Research, Science, Vol. 185, September 13, 1974
         (pp.   lb-92i)
11. Reicken, Henry W., and Robert F. Boruch, Ed. Social Experimentation:
       A Method for Planning and Evaluating Socia Intervention, New York,
       AEademic Press, 1974

12.   Rivlin, Alice M. and P. Michael Timpane, eds., Ethical and Legal
         Issues of Social Experimentation, The Brookings Institution, 1975.

13.   Struening, Elmer L., and Marcia Guttentag, Ed. Handbook of
         Evaluation Research, Vol. I and II, Beverly Hills, California,
         Sage Publications, 1975
14.   Technology Assessment, Office of, Drug Bioequivalence, Washington,
         D.C., 1974