Annual Review of Faculty

I will once again be doing annual reviews of our faculty. Here I share my views on the process; and, describe the metrics I use to make evaluations.

There are three general categories; (1) teaching, (2) research and (3) service to the university and the profession. The annual review is meant to give the faculty feedback about their performance for a particular year, and is one factor in determining raises. However, given budget cuts, there will be no raises in the foreseeable future.

While a faculty member may have a long and distinguished career, it is possible to have up and down years based on the simple fact that there are fluctuations in the metrics; one year a faculty member may have 10 excellent papers followed by a year with none. However, the annual reviews will not fluctuate as much as the metrics given that fluctuations are inevitable and usually not a sign of a problem.

Teaching is evaluated based on several pieces of information. The syllabus shows how the course is organized and faculty usually provide a narrative on new ideas that they explored to enhance learning outcomes as well as student feedback in the form of course evaluations.

I am strictly opposed to standardized teaching evaluation forms because they often lead to meaningless comparisons between professors in the form of numbers that each students assign using very different criteria. I marvel at the well-reasoned forms that each of our professors produce and the unique information that they solicit. Since each faculty member may be using different approaches, a single number assigned by a student does not tell a meaningful story. Thus, I usually read every comment made by each student - even in classes with 250 students. It takes lots of time, but provides quality information with which I can get a better sense of how well our courses are working.

As a case in point, several years ago, I scanned through a bunch of evaluations of a young professor who was teaching a large section with hundreds of students and noticed two things. The mean was slightly below average (i.e. the typical student choose average or below average when comparing this professor to others at our university) and the distribution was bimodal with a quarter of the students giving him the highest marks and three quarters giving him below average ratings.

I gained great insights from the correlations between the numbers and the narratives. The typical student who gave a low ranking said the course was "unfair" because it was too hard. The students giving him higher marks stated that the course was challenging but the professor spent lots of time explaining complex concepts to the students during class as well as during office hours. In addition, this professor scheduled non-mandatory help sessions so that students could ask questions and get extra help.

A picture emerged of a dedicated teacher who expected excellence but was willing to expend a great deal of additional effort to make sure that the students learned. While his average course evaluation was lower than many of the other faculty in our department, I judged him to be a more dedicated teacher than those whose evaluations were high and the comments uniformly positive about the fact that the class was "fun" and "easy."

Research is more difficult to judge. The fact that a funding source is willing to pay top dollar to a given faculty member is a good indication that this individual and his/her work is considered useful or interesting. Similarly, papers that make it through the peer review process at good journals have convinced an editor and a few reviewers of correctness and importance. If a piece of work gets many citations, then it shows that people are reading the papers and finding them useful. Thus, a blend of these factors provides a good indicator of research productivity and quality.

However, these indicators can have the opposite meaning. For example, a paper may be cited many times as an example of bad science. Or, an individual with lots of citations may be a technician who provides specialized samples to many research groups. While such a person is contributing to the science by providing samples, the number of citations may not be a sign that the work is particularly interesting or creative.

Lots of funding is not always a sign of good science. A company may grant big bucks to a researcher to test a trivial property of a product. In contrast, a small grant from the National Science Foundation for work by a single PI who is challenging our perceptions of the nature of space-time would carry much more weight in my mind.

I try to consider all these factors together when evaluating the research productivity of a faculty member, which includes learning a little bit about their research. I may look at the h-index and or the numbers of publications or citations per dollar spent, or other ratios to develop an impression of the research quality. Faculty members who have attained Fellow status in a professional society get such honors from significant lifetime contributions to their fields, so fellowship in a society factors into the annual review. In the end, I make a value judgement that may or may not be in line with what others may think. I accept the fact that the process is far from perfect.

Finally, there is service. Every faculty member is expected to contribute to the operations of the department and the university. Faculty members sit on tenure committees, thesis examination committees, search committees, and do all kinds of thankless tasks such as recruiting students, writing reports for the administration, and doing self studies that support our claims that we are a high quality department relative to our peers. I look for not only quantity of service, but the impact that it has on our department and university.

Service to the profession takes the form of membership on program committees for conferences, being on editorial boards of journals, reviewing papers for journals or proposals for funding agencies, organizing conferences, acting as an external examiner at other universities, and serving on panels that take advantage of an individual's scientific expertise. An active scientist does this kind of service as a matter of daily routine. I thus expect to see a substantial professional service component from each faculty member.

To make a final assessment, all of these factors are taken into account. A positive annual review requires significant accomplishments in at least two of the three major areas, with emphasis on research and education. What makes my job especially difficult is that our department is very strong. All faculty members are doing high-quality research, are well known in their fields, are well funded and are advising undergraduate/graduate students.

I have a few weeks break until I need to get around to this unpleasant task, which takes 5 full days to complete. For now, I am enjoying my break doing some new physics, which includes having completed an interesting calculation that was motivated by my proofing of one of Shoresh's manuscripts. Before the intensity of the new semester begins, I need to proof and submit a few more manuscripts for publication, as well as review a few more papers.

This past year has been a local maximum with a record 12 publications in refereed journals (almost 10% of my lifetime 123 publications), not to mention a bunch of invited and contributed talks. Now I will need to get back to work to maintain this momentum. But not until I take a day off to enjoy family, friends, and this wonderful place we call home.