Control of the National Science Foundation Peer Review Study

Published by the Government Accountability Office on 1977-03-04.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                                                                  --------                  ----
                      COMPTROLLER     GENERAL     OF   THE      UNITED   SATES
                                    WASHINGTON.    D3.C.     20548

B-133183                                                                         llnlllllllllllllllllllllllllll

                                                                     March 4, 1977

The Honorable Ray Thornton
Chairman, Subcommittee    on Science,
  Research,  and Technology           1
Committee on Science and
House o'f Representatives
Dear Mr. Chairman:
        In accordance with the November 7, 1975 request of former
Subcommittee     Chairman James W. Symington and subsequent agree-
ments with the Subcommittee        office,     we monitored   the study
jointly    commissioned   by the Subcommittee         and the National       / I                                   ,:
Science Hoard to-obtain      information       through questionnaires
about the scientific      community's      views of the National       Science
Foundation    peer review process;
       In November 1975, the Foundation                 mailed questionnaires
to a random sample of 1,552 individuals,                    selected    from its
file   of about 31,000 individuals,               who had reviewed a research
proposal     for the Foundation           in fiscal     year 1974.      Question-
naires were also mailed to 3,256 individuals                      who were randomly
selected     from the Foundation's           file     of applicants     who submit-
ted about 20,000 proposals              which were awarded, declined,             or
withdrawn      in fiscal     year 1975.        The questionnaires         asked the
reviewers      and applicants        their   experiences       with the Founda-
tion's    proposal     review process,         their    opinions    of the review
procedures,       and  their    feelings     regarding      various    possible
modifications.         Dr. Deborah R. Hensler L/ was employed by the
Foundation       as a private      consultant       to assist     in the survey
design by analyzing          the responses to the questionnaires                and
reporting      the results      to the Subcommittee           and the National
Science Hoard.

&/Dr. Hensler‘has       a Ph.D in         Political     Science from the
   Massachusetts    Institute   of         Technology      and is an Associate
   Head of Rand Corporation's              Social Science Department with
   primary  responsibility     for         coordinating      survey research.


            The scope of our work consisted             of (1) maintaining        the
    confidentiality        of survey respondents'          names by directly
    receiving       the returned    questionnaires        and destroying     the
    envelopes with the respondents'             names, (2) determining          that
    only those individuals          selected     to receive     questionnaires
    were included       in the study results          by checking the returned
    questionnaires        against   the names included        in the sample, (3)             s
    verifying       the accuracy of processing          the original     questionnaire
    responses into a computerized             data file     (master data file),
    and (4) verifying         ,the statistics      produced from the data as
    reported      by Dr. Hensler.       L/
           Returned questionnaires      were accepted through March 31,
    1976,   for inclusion     in the study.     The completed questionnaires
    for the reviewer      and applicant    surveys number 1,068 (69 per-
    cent response rate) and 2,684 (82 percent          response rate),
    respectively.       The responses were processed      by a Foundation
    contractor     (TeleSec)    into a master data file    containing   the
    simple tabulated      results.
           Prior to releasing          the completed questionnaires             to the
    Foundation,     we reproduced         responses to-questions         at random
    from every second reviewer              questionnaire      and every third
    applicant    questionnaire.           The reviewer     questionnaire       had 34
    questions,     while the applicant           questionnaire      had 26 questions.
    The responses to questions             which asked the respondent             to pro-
    vide an explanation          for his answer were not included               in our
    sample for verification.              We used this sample to check the
    accuracy of TeleSec's           transferring       the questionnaire        responses
    to a computerized         data file.        We  found   no  errors   in   the   trans-
    fer of sampled responses for 33 of the 34 questions                       asked of
    reviewers     and for 22 of the 26 questions               asked of applicants.
    The following       table shows the questions            for which we found
    transfer    errors,     the projected        number of questions        in the
    universe    for which errors          were likely     to exist,    and the esti-
    mated error      rates.

    L/Dr. Hensler reported            the analysis   of the questionnaire        re-
       sponses in a December 1976 report             entitled  "Perceptions        of
       the National       Science Foundation       Peer Review Process:        A
       Report on a Survey of NSF Reviewers and Applicants."                    The
       report    consists     largely    of 24 tables which present       statistics
       created     by analyzing       the master data file.     We independently
       verified      the statistics,      but did not evaluate     Dr. Hensler's
       interpretation        of the statistics.



                                                 Projected      no. of         Estimated       % of
    Questionnaire/             Universe           questions      with            universe      with
     Question    No.             size             transfer      errors         transfer       errors
              31                  1,068                    63                          5.88


                2                 2,684                    69                          2.56
                8                 2,684                    69                          2.56

              10                  2,684                    96                          3.57
              21                  2,684                    84                        .3.12

             The Foundation       created      additional        data from the responses
    to questions        31 and 32 on the reviewers'                  questionnaire,       and
    questions       20 and 21 on the applicants'                 questionnaire.          These
    questions       concern the institutions              which awarded the respon-
    dents their        highest    academic degrees,            and the institutions
    with which respondents             are currently          affiliated.          The Founda-
    tion categorized          the reviewers'         and applicants'            degree-awarding
    institutions        and their      current     affiliation           by type,    using an
    American Association            of University         Professors         code-l/    Our ran-
    dom sample of the data base created                     from this process showed
    no errors       for the coding of reviewers'                  institutions.         However,
    for the coding of applicants'                institutions,             our sample showed
    estimated       error   rates of 1.02 percent              for institutions          award-
    ing the respondents           their    highest       academic degrees and 2.04
    percent      for institutions         with which respondents                are currently
           We believe    that none of the errors        in the master data
    file   of questionnaire      responses would greatly       affect     the tabu-
    lated    resu,lts.   In addition,     it is highly    probable     that the
    errors    are randomly distributed       throughout    the alternative
    choices of responses       to a question     which further      reduces the
    chances to affect       the tabulated    results.
            We independently  verified  the statistics contained                         in
    Dr.    Hensler's  December 1976 report  which were produced                        from

    L/Generally   ranks participating institutions by categories,
       such as type and number of degrees awarded.

    .   c


        the master- data file           of questionnaire        responses.       We did not
        verify   report      tables     8, 13, 17, and 24 which contain              respon-
        dents'   explanatory        data, and parts of tables             2 and 3 which
        contain    data compiled by the National                Science Foundation
        separate      from the jointly-commissioned               study.     With these
        exceptions,       we believe       the statistics       in the tables accurately
        present     the master data file          of questionnaire          responses except
        for insignificant          differences.         Furthermore,      the errors     con-
        tained in the data base, as previously                    discussed,     do not appear
        to greatly       affect    the statistics        presented      in the tables.
               We are available  to discuss          our findings   and to provide
        any further   assistance   you might         need in studying   the Founda.-
        tion's   peer review process.
                                                 Sincerely     yours,

                                                 Comptroller  General
                                                 of the United States

                                               - 4 -'