National Research Council Doctoral Surveys
Update September 10, 2007
by Linda Putnam, Chair, Sub-Committee on the NRC
Report to the Council of Communication Association Meeting
Progress and Timetable
The NRC has completed its data collection with the exception of receiving some corrected Program Questionnaires. From the 230 participating institutions, the response rates were very high—Institutional Survey—100%, Program Questionnaires—97%; Faculty Surveys—86%; Student—72%. Charlotte Kuh, Duputy Executive Director, felt the response rate for the student surveys was very high (sampled only students in English, Physics, Economics, Neuroscience, and Chemical Engineering). Most student surveys yield about a 40% response rate. Students seemed interested in the survey and wanted to share their impressions of their graduate program experiences. When the survey is conducted again, Dr. Kuh felt that NRC might include student surveys for all fields.
Timetables:
November 27, 2007—Draft of Report and Submission to NRC Committee for review
February 15, 2008—Publication of Final Report
The Survey of Program Quality or Faculty Rating Survey was conducted in May/June and received, in general, about a 50% return rate, but return rates varied per field. When a faculty member failed to respond by designated deadline, NRC selected another faculty member from the stratified random faculty list (based on completion of 1st questionnaire, willingness to do a rating, faculty size, geographic location (5 regions), etc. Eventually they sampled 200 raters per field (50 respondents for each program). Respondents were asked to consider the “universe” of programs in their field and reflect on the criteria considered important in assessing quality doctoral programs. Raters used a scale from 1 to 6 to assess the overall program quality for 15 different programs (1=not adequate for doctoral education, 2=marginal, 3=adequate, 4=good, 5=strong, and 6=distinguished). They also indicated their degree of familiarity with each program on a three-point scale and could choose not to rate a particular program due to lack of familiarity. So respondents rated each school’s program on two scales: familiarity and quality. Respondents also had links to a program’s information page that provided names of faculty working with doctoral students (divided into categories of core, new and associated), # of PhD 2001-2006, % of PhD in academic positions, % who completed in eight years or less, median time to degree, # of female faculty and # of non-white faculty, and a program URL.
Corrections on Program Questionnaires. NRC identified errors on program questionnaires and sent them back for corrections (e.g., % reported above 100%). These errors stemmed from misunderstandings in calculations, e.g., completion or attrition rates, etc. The goal was to get these calculations consistent across programs based on similar years and ratios. Corrections on the questionnaires are due on Sept. 17, 2007. Then the data needs to be organized into a useable database. The NRC Committee will write its analytic essay in Sept./Oct. The essay and database will undergo NRC review.
Input Wanted from CCA, Deans, and Fields of Study
What “standard summaries” about doctoral programs would be useful for a field? or for each program? NRC seeks our input in answering this question. Here are some suggestions that NRC has made:
1. Program Characteristics
Faculty size, # of doctoral students in Fall 2005, current student/faculty ratio, doctoral degrees awarded (2001-2006), median time to degree in past three years, % of doc. students advance to candidates.
2. Faculty Characteristics—based on percentage
Gender, citizenship status, race/ethnicity, rank.
3. Student Characteristics—gender; citizenship; race-ethnicity; full-time or part-time; full, partial or no financial support
4. What other descriptive data would we like to see in the summary report about the field’s doctoral programs? What would be useful information for the standard summaries across fields? Across individual institutions?
Ratings and Rankings
The report on calculating ratings and rankings per program is similar to what I presented on March 5, 2007. Basically, NRC plans to use explicit and implicit measures to compute quartile ranges and rankings of programs. This approach captures the uncertainty in ratings by emphasizing ranges and overlap of ratings. The rating is based on overall program quality, not scholarly quality of faculty as in previous studies.
Explicit weights stem from faculty assessment of the criteria of quality programs in a field. These key variables will be determined empirically either through factor analysis or judgmentally. Faculty in a particular field rated the relative importance of different factors in determining program quality and these ratings will be used as explicit measures.
Intrinsic weights is the anchoring or benchmark study and will be based on faculty responses to the Survey of Program Quality. NRC will regress the quantitative variables on program ratings for a sample of programs. The resulting weights will be used to construct a range of ratings for all programs in a field.
Additional Analyses and Subsidiary Data
The NRC Committee will produce subsidiary ratings to highlight components of quality programs. These include: 1) Researcher impact—possible metrics on number of publications, citations, impact of faculty members, honors and awards; 2) Student support and outcomes- metrics might include number of students with full support, time to degree, attrition rate, placement in relevant academic position, etc.; 3) Diversity—% of students and faculty who are female and/or minority.
Overall Range of Ratings.
The NRC will combine explicit and implicit ratings to calculate a range of ratings for each program and a range of rankings for overall program quality. Three additional rankings and ratings will be constructed for research impact, student support and outcomes, and diversity of academic environment. Data will be available on the web, disseminated through professional societies, and users will be given the capability to construct their own customized comparisons based on choices of different criteria. NRC will sponsor a conference on this study in September 2008 and publish papers from the conference.
