Even as a senior professor, I still cringe at negative reviews of my work. It's nice to occasionally get a good one, as I did for a recent paper. Below is the review:
Review of “A path to ultralarge nonlinear‐optical susceptibilities” by Mark G. Kuzyk
This paper is essentially a quantitative ‘stream of consciousness’ of its author as he ruminates about the origins of the nonlinear polarization in molecular‐scale objects. Not every author could get away with this type of discourse, but this author is the pioneer of explorations of the fundamental limits of nonlinear optical phenomena and also the founder of the fundamental understanding of scaling phenomena in nonlinear optical molecules. Those two achievements enable the author to precisely reveal the essential elements of what constitutes a good nonlinear optical structure in this most original of works.
The present paper brings together the two concepts of limits and scaling to attempt to understand how to build a quantum unit (a molecule, in the author’s nomenclature) whose absolute nonlinearity gets large with size when its intrinsic nonlinearity is constant with molecule size. Many of the scaling concepts are familiar to readers of his papers, but this paper appears to be the first to step back and take a close look at the origin of nonlinearities as a function of the number of participating electrons. The author carefully examines the quantum to classical transition and shows that the ideal system is made of microscopic units whose size falls just below the classical limit, provided they are properly arranged in the bulk form. The author examines scaling in 1D and 3D, showing that the existence of transverse modes in 3D essentially removes electrons from participating in the nonlinear response, whereas in 1D, all electrons are longitudinal and can participate, provided they aren't otherwise tied up keeping the molecule bound together or living in shells close to the nucleus. The emphasis on ‘participating’ electrons isn't new, but their use in scaling arguments appears to be original.
The author emphasizes the need to optimize the so‐called figure of merit (FOM) of a material, a macroscopic quantity related to the physics of a device or phenomena exploited in a device. Recent work by Mossman et al. has shown how this works using resonant enhancement in a nonlinear material. The fact that the present paper raises the issue of FOM optimization indicates an attempt by the author to shift the collective discussion of how to make better molecules from a focus on better nonlinearities to a focus on better materials. The reviewer believes this paper may be referenced in future years as a turning point for the field of nonlinear optics materials research, much as the author’s 2000 Phys. Rev. Letter on the fundamental limits led to discoveries of the origins of large intrinsic nonlinearities, delineation of the required spectral properties of good molecules, and the invention of new materials.
In brief, it is an excellent, original piece of work and should be published.
My original pleasure induced by these strokes to my ego dissipated quickly. I detest authority and would not want to become one. Everyone should make judgments based on the strength of the evidence, not on the source. I don't want to be given a free pass, nor do I want to impose my will on others.
Accepting the title of "authority" admits to stagnation. I would rather be exploring new territories, filled with confusion but fueled by a drive to seek an understanding of the mysterious novelties that are just out of reach. Answers always lead to more questions, like a series of stepping stones that stretch out in all directions with new branches emanating from each one. The stepping stones in the distance are an alluring draw from the familiar ones that I have visited. I yearn for them.
My many years as an academic have brought me great fulfillment in making discoveries and learning new things in the companionship of students and colleagues. It keeps me young. On the flip side, I am burdened by responsibilities that are neither intellectually stimulating, nor produce any tangible benefits to my constituents. I stand at a crossroads where I am bursting with ideas that are squelched by inane responsibilities. Should I abrogate my duties and dive into my research, whose benefits I know will surpass my selfish desire to learn, or should I continue to be a loyal soldier, obeying orders that restrain intellectual creativity? It is a decision that I alone need to make, and nobody can lesson this burden. The sense of responsibility yanks me away from my passions.
I dream of becoming a recluse for a summer or a semester, surrounded by my books and doing calculations on huge sheets of paper, without interruption. I would have to ignore funding agency reporting requirements that ask the same questions in 14 different ways, as if to excise every bit of creative energy. I would have to ignore annual reviews that force me to waste a day filling in boxes in a form that neither makes the provided information clearer or more assessable. I would have to ignore calls for proposals, which require researchers to come up with ideas that are not so novel as to not be appreciated, but beyond trivial so that reviewers can be impressed. I would have to ignore nagging emails that incessantly interrupt my train of thought. Ignoring these activities would surely result in a loss of funding, something that the university expects, but that I wouldn't need. Is it ethical to use the security of my tenure to ignore expectations that come from above even though I believe the university would potentially benefit more in the long run?
The guilt strings pull at my gut as I spend time writing this post, even though I'm at the keyboard at 9:00pm on a Saturday night -- a time that should be my own. Airing out these frustrations, I hope, will be an exercise that will help me sleep, which has not come easily these days. The obligations that are pulling me in multiple directions prevent me from making any meaningful progress in my most important activities, and I lay in bed in panic that I will miss important deadline.
The check list on my desk has many items that need attention. I just finished grading and need to prepare for lectures, make up homework problems, and solve them -- a process that I find deeply fulfilling. Next I need to reply to the reviewers' comments and revise my manuscript. Rereading my manuscript, which is necessary when making revisions, is like visiting with old friends and rehashing the past, a pleasant pastime where ideas are batted around and reformed. Since my colleagues were kind enough to review my manuscripts, I am ethically bound to act as a reviewer when asked. Obligations of this sort are made many times each day, so I must accept the fact that I will never be able to withdraw fully.
Now it's time to get back to responding to the reviewers and revising my paper. The details in the reviews suggest that both reviewers took the time to critically read manuscript and understood the nuances of my paper. For this I am grateful and excuse them for implying that I am an authority. Since it's almost 10:00pm, it's time for me to complete this post. No time to proof it. No apologies.
Showing posts with label annual review. Show all posts
Showing posts with label annual review. Show all posts
Saturday, September 24, 2016
Sunday, February 12, 2012
A sampling of everything that is possible
It is spine tingling to view a two-dimensional representation of everything that is possible given the laws of physics. I had previously written about our work, where Shoresh used Monte Carlo simulations to let the computer randomly sample all possible quantum systems that are compatible with the Schrodinger equation. In my recent report to the National Science Foundation, my awe in Physics and its ability to make such grand and sweeping statements was rekindled. Above is a summary of our work in graphical form. It is aesthetically pleasing in its visual appeal, but more so in its intellectual content.
I cannot explain in the small amount of space here the beauty of these results, nor what they mean in terms of the basic science. Nature is clearly hinting at some deep structure of the quantum world when we probe molecules with light. My summary to NSF states it best:
"The uniqueness of our work is that it provides a new way of thinking, which in the words of a Spotlight on Optics article describes our approach as "reminiscent of the cutting of the Gordian knot by Alexander the Great. (see article)" Rather than focusing on one application, our work provides a new paradigm for understanding the critical material properties that need to be controlled for making better materials for any application that is based upon the interaction of light with matter.
"Technologies that may benefit from this work include high-contrast medical imaging, laser-based cancer therapies that target malignant cells without damaging surrounding cells, all-optical computers that are ultrafast by virtue of new architectures enabled by light, high-speed telecommunications through technologies that eliminate the electronics bottleneck, and high-density reconfigurable optical information storage."
Now I need to get back to the painfully boring task of updating my CV and preparing my annual review materials. I can't wait to finish these inane tasks so that I can get back to thinking about the things that I find most meaningful in life.
Wednesday, December 28, 2011
Annual Review of Faculty
I will once again be doing annual reviews of our faculty. Here I share my views on the process; and, describe the metrics I use to make evaluations.There are three general categories; (1) teaching, (2) research and (3) service to the university and the profession. The annual review is meant to give the faculty feedback about their performance for a particular year, and is one factor in determining raises. However, given budget cuts, there will be no raises in the foreseeable future.
While a faculty member may have a long and distinguished career, it is possible to have up and down years based on the simple fact that there are fluctuations in the metrics; one year a faculty member may have 10 excellent papers followed by a year with none. However, the annual reviews will not fluctuate as much as the metrics given that fluctuations are inevitable and usually not a sign of a problem.
Teaching is evaluated based on several pieces of information. The syllabus shows how the course is organized and faculty usually provide a narrative on new ideas that they explored to enhance learning outcomes as well as student feedback in the form of course evaluations.
I am strictly opposed to standardized teaching evaluation forms because they often lead to meaningless comparisons between professors in the form of numbers that each students assign using very different criteria. I marvel at the well-reasoned forms that each of our professors produce and the unique information that they solicit. Since each faculty member may be using different approaches, a single number assigned by a student does not tell a meaningful story. Thus, I usually read every comment made by each student - even in classes with 250 students. It takes lots of time, but provides quality information with which I can get a better sense of how well our courses are working.
As a case in point, several years ago, I scanned through a bunch of evaluations of a young professor who was teaching a large section with hundreds of students and noticed two things. The mean was slightly below average (i.e. the typical student choose average or below average when comparing this professor to others at our university) and the distribution was bimodal with a quarter of the students giving him the highest marks and three quarters giving him below average ratings.
I gained great insights from the correlations between the numbers and the narratives. The typical student who gave a low ranking said the course was "unfair" because it was too hard. The students giving him higher marks stated that the course was challenging but the professor spent lots of time explaining complex concepts to the students during class as well as during office hours. In addition, this professor scheduled non-mandatory help sessions so that students could ask questions and get extra help.
A picture emerged of a dedicated teacher who expected excellence but was willing to expend a great deal of additional effort to make sure that the students learned. While his average course evaluation was lower than many of the other faculty in our department, I judged him to be a more dedicated teacher than those whose evaluations were high and the comments uniformly positive about the fact that the class was "fun" and "easy."
Research is more difficult to judge. The fact that a funding source is willing to pay top dollar to a given faculty member is a good indication that this individual and his/her work is considered useful or interesting. Similarly, papers that make it through the peer review process at good journals have convinced an editor and a few reviewers of correctness and importance. If a piece of work gets many citations, then it shows that people are reading the papers and finding them useful. Thus, a blend of these factors provides a good indicator of research productivity and quality.
However, these indicators can have the opposite meaning. For example, a paper may be cited many times as an example of bad science. Or, an individual with lots of citations may be a technician who provides specialized samples to many research groups. While such a person is contributing to the science by providing samples, the number of citations may not be a sign that the work is particularly interesting or creative.
Lots of funding is not always a sign of good science. A company may grant big bucks to a researcher to test a trivial property of a product. In contrast, a small grant from the National Science Foundation for work by a single PI who is challenging our perceptions of the nature of space-time would carry much more weight in my mind.
I try to consider all these factors together when evaluating the research productivity of a faculty member, which includes learning a little bit about their research. I may look at the h-index and or the numbers of publications or citations per dollar spent, or other ratios to develop an impression of the research quality. Faculty members who have attained Fellow status in a professional society get such honors from significant lifetime contributions to their fields, so fellowship in a society factors into the annual review. In the end, I make a value judgement that may or may not be in line with what others may think. I accept the fact that the process is far from perfect.
Finally, there is service. Every faculty member is expected to contribute to the operations of the department and the university. Faculty members sit on tenure committees, thesis examination committees, search committees, and do all kinds of thankless tasks such as recruiting students, writing reports for the administration, and doing self studies that support our claims that we are a high quality department relative to our peers. I look for not only quantity of service, but the impact that it has on our department and university.
Service to the profession takes the form of membership on program committees for conferences, being on editorial boards of journals, reviewing papers for journals or proposals for funding agencies, organizing conferences, acting as an external examiner at other universities, and serving on panels that take advantage of an individual's scientific expertise. An active scientist does this kind of service as a matter of daily routine. I thus expect to see a substantial professional service component from each faculty member.
To make a final assessment, all of these factors are taken into account. A positive annual review requires significant accomplishments in at least two of the three major areas, with emphasis on research and education. What makes my job especially difficult is that our department is very strong. All faculty members are doing high-quality research, are well known in their fields, are well funded and are advising undergraduate/graduate students.
I have a few weeks break until I need to get around to this unpleasant task, which takes 5 full days to complete. For now, I am enjoying my break doing some new physics, which includes having completed an interesting calculation that was motivated by my proofing of one of Shoresh's manuscripts. Before the intensity of the new semester begins, I need to proof and submit a few more manuscripts for publication, as well as review a few more papers.
This past year has been a local maximum with a record 12 publications in refereed journals (almost 10% of my lifetime 123 publications), not to mention a bunch of invited and contributed talks. Now I will need to get back to work to maintain this momentum. But not until I take a day off to enjoy family, friends, and this wonderful place we call home.
Saturday, February 12, 2011
Cutting life in half
When our son was born in 1985, we purchased a Sony Betamax camcorder. Back then, the Sony Beta format was king. Its quality was clearly superior to the competing VHS format. However, Sony made the mistake of keeping their technology proprietary, so the VHS format eventually won out. These days, VHS tapes are passe, as is its 8mm successor.
While I was in graduate school, our Egyptian friend introduced us to tabouli, a Lebanese dish that my wife and I loved from the minute we took our first taste. Our friend Saleem graciously volunteered to teach us to make this wonderful but labor intensive dish at our home, and that's how our Friday-night tabouli and pasta dinner ritual started.
I introduced my new Betamax camcorder to our guests at one of these dinners by taping the event. Later, while we excitedly watched ourselves on a tiny TV, Saleem commented, "This camera cuts life in half. We spend half our time living and the other half watching ourselves on TV." I was reminded of these past good times while preparing my annual review materials. It made me realize how much time we waste documenting what we do.
To waste more of our time, our university implemented an ill-conceived notion to computerize the whole process with a clunky system. The system, called WORQS, was designed to save time; but, it becomes more cumbersome when the reported activities become nuanced and diverse, as is often the case for productive faculty members. My extreme dissatisfaction with this system lead me to coin the phrase WORQS SUQS.
I have already spent a large chunk of my Saturday afternoon updating my CV so that I can then spend another day entering it again into WORQS. My aggravation with the process reminds me of all sorts of new well-intentioned but time-wasting demands on researchers. For example, in an effort to prevent scientific fraud, the National Science Foundation now requires a data management plan. I agree with transparency as well as forcing overly-protective scientists to publicly share their data, especially when the work is paid for by our citizens. However, such rules enforce undue additional time demands on all the scientists.
I have many complaints about regulations that waste time with little in the way of overall returns, but I will leave that for another day. For now, I am attaching, below, an exchange that I had with the administration a few years ago. You may wonder why I wasted time doing so. My assumption was that a university, populated by members of the intelligentsia, would respond positively to a logical argument, resulting in increased efficiency. Instead, I was perceived a spoiled brat who refused to do what little was asked of me. Here is the exchange, with names removed:
Dear Administrator,
I am surprised at the university's attitude that we are to fill out our annual reviews online because WSU pays us and that's what we are expected to do - case closed. This goes against what I understand to be the principle of shared governance. The purpose of my email to you and the WORQS committee was to point out how some parts of the system are poorly designed from the point of view of the faculty member and the chair who does the annual reviews. I was not attempting to shirk responsibility, but rather, was bringing to your attention the features of WORQS that are inefficient, inferior to the present system, and do not fulfill - as far as I can see - the needs of the administration and the mission of the university.
I wholeheartedly embrace the idea of a computerized system that both collects important data for the university; and saves time of faculty members and chairs when preparing annual reviews. In particular, the data mining features are excellent. Not having to enter my lists of grants and proposals; but rather having WORQS lift this information from OGRD saves lots of time and effort. And having WORKS download a list of the classes that we teach by pulling data from the registrar is another good idea.
Unfortunately, there are other aspects of WORQS that are cumbersome and poorly designed. Is the point of WORQS to make it less time-consuming to enter information? Is it to provide better information for the chair in evaluating performance? Are there certain key metrics that the university needs to collect for the state legislature for purposes of accountability or to give us ammunition for showing how great we are? If so, portions of WORQS need a serious overhaul.
I focus my criticism on the scholarship menu as an example of bad design.
Scholarship Menu
Time Savings?
Since the WORQS system is purported to save time - or at least to not take much extra effort, I timed the publication-entry process in the "scholarship" menu item. It took me, a fast typist and computer geek, 50 minutes to enter my 2006 publications (had I not timed myself, I would have guessed 10 minutes) compared with 55 seconds to cut and paste the same ones. Net time wasted: 49 minutes.
The major problems from the faculty member's perspective:
a) Having to add journals in one menu area first then pulling down a journal name from the publications area takes time. Note: One of the new journals in which I publish was not on the list, which I had to add. After adding it (I needed to first find its ISSN), I got an email a few minutes later that it was on the system.
b) The method of adding coauthors is perhaps the worst feature. In one case, I had a paper with 13 coauthors, most of whom will probably never coauthor another paper with me in the future. Your system erroneously assumes that collaborators remain static. Since the coauthor field was not required, I just left it blank. Going through this process would have wasted lots more time. In fact, I left most of the non-required fields blank.
c) There is no field to add the year of a publication. There are cases where a paper appears in print in the year following the "publication date" (the volume number, etc. is not known until after annual reviews), so those papers fall through the cracks.
d) The present system allows for double counting by allowing a talk and a conference proceedings paper to be entered individually.
Usefulness to Chair?
With regards to publications, the information that is not required (like the coauthor list) is perhaps one of the most important since it gives a snapshot of collaborations and the students involved in the research. As chair, I would refuse to force my faculty to waste their time making such entries since for my purposes, seeing a CV is enough to judge their work.
Usefulness to WSU
Getting the names of the journals entered correctly and free from typos appears to be a priority to the administration and WORQS is clearly designed in a way that gets this information accurately. Other parts of the form leave plenty of room for inaccuracies.
This one menu item illustrates the vast number of the kinds of scholarly works produced by our faculty and the unique regard for journal articles. On the one hand, entry of journal publications enforces strict compliance on getting the journal name right, while we can add whatever we wish in all the other fields. This suggests that journal names are held in high esteem while other fields are not, so these other sections could and should therefore be left to free-form entry.
If WSU truly is interested in the most efficient system, then the following entry method would be superior:
1. Set up WORQS to import from online databases (such as Web of Science, for example) an individual's publications that appear in refereed journals.
2. Allow the faculty member to enter missing journal articles by wholesale cutting and pasting. Units whose research is published in journals that are not covered by databases should be permitted to use free-form entry.
3. Have WORQS scan the list of WSU coauthors and mark the ones that are students (grad or undergrad) from WSU's personnel/student database.
4. Since WORQS does not seem to care about the accuracy of conference proceedings, etc, allow faculty to cut and paste such "other" scholarly work freestyle.
The above would be a smart use of technology and would better embrace what I assume to be in the true spirit of WORQS.
Other Menu items
There are many other areas that can be improved. Similar to the method of getting grant activity directly from OGRD, why not do the same for committee service? At the level of department, college, and university, there are lists of committee memberships in place, so lift that info electronically in one swoop. The same thing can be done with M.S. and Ph.D. committees from lists compiled by the grad school. Is the WORQS committee analyzing the system and implementing such features to save time? If there are difficulties with implementing these kinds of recommendations, the response should not be to stick the faculty with the chore. Rather, you should call upon the faculty for ideas to improve the system. We all have a common vested interest in making WSU a better place.
As a general observation, it does not make sense to design a uniform online system when there are so many permutations in the diversity of work that we do. Even if you were to take into account all contingencies, there are just too many variables to enter efficiently. In fact, I would argue that if a single efficient form could be designed that takes into account all that we do, it would be an indicator that we are a mediocre institution. WORQS should be a system that is designed to save time and to augment annual reviews, not to drive a process that encumbers us in a web of minutia and forces us into conformity. Given the present implementation of WORQS, it emphasizes the latter.
In your email, you state, "we are relying on a faculty committee to work on this form and they simply don't agree with your argument." The issue is not open to opinion. Rather, we should turn to factual data to support our conclusions. In your trials of WORQS, have there been objective studies that compare it with the old-fashioned CV in terms of time and flexibility? Have you timed faculty who are using WORQS to determine the time it takes to fill out various parts of the form? Would a totally automated system that gathers the info needed by WSU, augmented with free form entry for the chair's evaluation, not make the most sense? If such a system is going to be required of all faculty, a cost-benefit analysis needs to be done; and, if any unit or group takes the brunt of the cost, then the system should not be imposed on anyone. I find it unacceptable that we are being told that WORQS is great when there is evidence to the contrary. Faculty should have more input before use of the system is required.
A university's top priority is to foster an atmosphere that nurtures a dynamic interplay of education and the generation of new knowledge. The administration's responsibility is to remove burdens on faculty so that we can fulfill our mission. I believe that this whole business with WORQS and how it is being imposed on us sends the opposite message. Perhaps I am naive in believing that our faculty is so diverse and creative that one form can not do justice without becoming cumbersome.
On a closing note, we need to ask ourselves the question, "How are we to think outside the box if we spend so much of our time filling out boxes?" Let's use technology to save time and make us more productive.
Sincerely,
Mark G. Kuzyk
Professor and Associate Chair of Physics
and past Chair of the Materials Science Program
P.S. Most faculty in our department with whom I have discussed WORQS also found it quite cumbersome. I am attaching an email that I got from Faculty X. Also, you need to consider the reactions of the staff to this new system. They do not have the luxury of speaking up in the same way as faculty. Based on several conversations, my sense is that WORQS is demoralizing to the staff when most of the form is devoted to information that has no bearing on their work. Have you had broader input from staff beyond committee members?
My Original Email:
Thanks for your detailed response. I have been thinking about annual reviews from the point of view of a faculty member and as associate chair (I will be doing annual reviews this year in our department). I applaud the time saving features such as grants showing up automatically. I was credited with a $6 million dollar grant which I never got. So, that will make me look good! On the other hand, adding publications in the piecemeal way requested is very cumbersome. Even when the journal names are on a pull-down menu, it still takes lots of time to add the title, volume number, pages, type of pub, etc. - especially for productive faculty. Other parts of the form suffer from the same issues. The best solution is to give a choice of cutting and pasting items wholesale OR using the form.
The bottom line is that annual reviews are used to assess performance; and, the chair is best suited to do so. I can tell when one publication is of sufficient high quality that it trumps 10 others. So statistics are useless in my analysis of faculty performance. If WSU wants to gather statistics, then I would be happy to provide my CV for statistical analysis. But, as faculty members, we should not be REQUIRED to waste time entering information in a form that really has nothing to do with annual review, nor does it enhance the process in any way. In your email, you state, "…we can't do that if the system is to be useful." Useful to whom? It's certainly not useful to faculty nor is it to those of us who are doing evaluations.
Our time is chipped away to the extent that I am finding it impossible to do any research, so I do it in my times of "leisure." Yeh, this way of doing annual reviews may take only an extra ½ hour, but there are countless numbers of other such things that we are already doing that also don't make sense. Enough is enough!
My previous communication was sent to you through so and so because I was not aware of the committee working on this form. I hope that my comments are passed along to whatever group is working on this process and to whoever is making the decision to impose this online system.
At this rate, next we'll be requiring all teaching evaluations to be standardized. Though I would be a big beneficiary of such a system, this would be a huge mistake.
Your Response:
I understand your frustration about the many activities that faculty members are asked to do. Unfortunately, your comments have already been passed on to the faculty committee, and they disagree with you. WSU pays your salary all year. For that, the University only asks you to explain what you're doing once per year. Even if you have many, many, many publications and you are a completely incompetent typist, using the pulldown list will take you only minutes more than cutting and pasting. Your use of the pulldown list will allow the University to collect much better data. (So, for example, you won't get credited with the publication equivalent of a $6M grant that you didn't earn.)
I'm sorry that this issue is frustrating to you, but we are relying on a faculty committee to work on this form and they simply don't agree with your argument. So far, approximately half the colleges have decided to use WORQS for annual review this year even though it is still optional. To date, at least some of your colleagues believe that WORQS is a much needed effort saver for the entire University.
While I was in graduate school, our Egyptian friend introduced us to tabouli, a Lebanese dish that my wife and I loved from the minute we took our first taste. Our friend Saleem graciously volunteered to teach us to make this wonderful but labor intensive dish at our home, and that's how our Friday-night tabouli and pasta dinner ritual started.
I introduced my new Betamax camcorder to our guests at one of these dinners by taping the event. Later, while we excitedly watched ourselves on a tiny TV, Saleem commented, "This camera cuts life in half. We spend half our time living and the other half watching ourselves on TV." I was reminded of these past good times while preparing my annual review materials. It made me realize how much time we waste documenting what we do.
To waste more of our time, our university implemented an ill-conceived notion to computerize the whole process with a clunky system. The system, called WORQS, was designed to save time; but, it becomes more cumbersome when the reported activities become nuanced and diverse, as is often the case for productive faculty members. My extreme dissatisfaction with this system lead me to coin the phrase WORQS SUQS.
I have already spent a large chunk of my Saturday afternoon updating my CV so that I can then spend another day entering it again into WORQS. My aggravation with the process reminds me of all sorts of new well-intentioned but time-wasting demands on researchers. For example, in an effort to prevent scientific fraud, the National Science Foundation now requires a data management plan. I agree with transparency as well as forcing overly-protective scientists to publicly share their data, especially when the work is paid for by our citizens. However, such rules enforce undue additional time demands on all the scientists.
I have many complaints about regulations that waste time with little in the way of overall returns, but I will leave that for another day. For now, I am attaching, below, an exchange that I had with the administration a few years ago. You may wonder why I wasted time doing so. My assumption was that a university, populated by members of the intelligentsia, would respond positively to a logical argument, resulting in increased efficiency. Instead, I was perceived a spoiled brat who refused to do what little was asked of me. Here is the exchange, with names removed:
Dear Administrator,
I am surprised at the university's attitude that we are to fill out our annual reviews online because WSU pays us and that's what we are expected to do - case closed. This goes against what I understand to be the principle of shared governance. The purpose of my email to you and the WORQS committee was to point out how some parts of the system are poorly designed from the point of view of the faculty member and the chair who does the annual reviews. I was not attempting to shirk responsibility, but rather, was bringing to your attention the features of WORQS that are inefficient, inferior to the present system, and do not fulfill - as far as I can see - the needs of the administration and the mission of the university.
I wholeheartedly embrace the idea of a computerized system that both collects important data for the university; and saves time of faculty members and chairs when preparing annual reviews. In particular, the data mining features are excellent. Not having to enter my lists of grants and proposals; but rather having WORQS lift this information from OGRD saves lots of time and effort. And having WORKS download a list of the classes that we teach by pulling data from the registrar is another good idea.
Unfortunately, there are other aspects of WORQS that are cumbersome and poorly designed. Is the point of WORQS to make it less time-consuming to enter information? Is it to provide better information for the chair in evaluating performance? Are there certain key metrics that the university needs to collect for the state legislature for purposes of accountability or to give us ammunition for showing how great we are? If so, portions of WORQS need a serious overhaul.
I focus my criticism on the scholarship menu as an example of bad design.
Scholarship Menu
Time Savings?
Since the WORQS system is purported to save time - or at least to not take much extra effort, I timed the publication-entry process in the "scholarship" menu item. It took me, a fast typist and computer geek, 50 minutes to enter my 2006 publications (had I not timed myself, I would have guessed 10 minutes) compared with 55 seconds to cut and paste the same ones. Net time wasted: 49 minutes.
The major problems from the faculty member's perspective:
a) Having to add journals in one menu area first then pulling down a journal name from the publications area takes time. Note: One of the new journals in which I publish was not on the list, which I had to add. After adding it (I needed to first find its ISSN), I got an email a few minutes later that it was on the system.
b) The method of adding coauthors is perhaps the worst feature. In one case, I had a paper with 13 coauthors, most of whom will probably never coauthor another paper with me in the future. Your system erroneously assumes that collaborators remain static. Since the coauthor field was not required, I just left it blank. Going through this process would have wasted lots more time. In fact, I left most of the non-required fields blank.
c) There is no field to add the year of a publication. There are cases where a paper appears in print in the year following the "publication date" (the volume number, etc. is not known until after annual reviews), so those papers fall through the cracks.
d) The present system allows for double counting by allowing a talk and a conference proceedings paper to be entered individually.
Usefulness to Chair?
With regards to publications, the information that is not required (like the coauthor list) is perhaps one of the most important since it gives a snapshot of collaborations and the students involved in the research. As chair, I would refuse to force my faculty to waste their time making such entries since for my purposes, seeing a CV is enough to judge their work.
Usefulness to WSU
Getting the names of the journals entered correctly and free from typos appears to be a priority to the administration and WORQS is clearly designed in a way that gets this information accurately. Other parts of the form leave plenty of room for inaccuracies.
This one menu item illustrates the vast number of the kinds of scholarly works produced by our faculty and the unique regard for journal articles. On the one hand, entry of journal publications enforces strict compliance on getting the journal name right, while we can add whatever we wish in all the other fields. This suggests that journal names are held in high esteem while other fields are not, so these other sections could and should therefore be left to free-form entry.
If WSU truly is interested in the most efficient system, then the following entry method would be superior:
1. Set up WORQS to import from online databases (such as Web of Science, for example) an individual's publications that appear in refereed journals.
2. Allow the faculty member to enter missing journal articles by wholesale cutting and pasting. Units whose research is published in journals that are not covered by databases should be permitted to use free-form entry.
3. Have WORQS scan the list of WSU coauthors and mark the ones that are students (grad or undergrad) from WSU's personnel/student database.
4. Since WORQS does not seem to care about the accuracy of conference proceedings, etc, allow faculty to cut and paste such "other" scholarly work freestyle.
The above would be a smart use of technology and would better embrace what I assume to be in the true spirit of WORQS.
Other Menu items
There are many other areas that can be improved. Similar to the method of getting grant activity directly from OGRD, why not do the same for committee service? At the level of department, college, and university, there are lists of committee memberships in place, so lift that info electronically in one swoop. The same thing can be done with M.S. and Ph.D. committees from lists compiled by the grad school. Is the WORQS committee analyzing the system and implementing such features to save time? If there are difficulties with implementing these kinds of recommendations, the response should not be to stick the faculty with the chore. Rather, you should call upon the faculty for ideas to improve the system. We all have a common vested interest in making WSU a better place.
As a general observation, it does not make sense to design a uniform online system when there are so many permutations in the diversity of work that we do. Even if you were to take into account all contingencies, there are just too many variables to enter efficiently. In fact, I would argue that if a single efficient form could be designed that takes into account all that we do, it would be an indicator that we are a mediocre institution. WORQS should be a system that is designed to save time and to augment annual reviews, not to drive a process that encumbers us in a web of minutia and forces us into conformity. Given the present implementation of WORQS, it emphasizes the latter.
In your email, you state, "we are relying on a faculty committee to work on this form and they simply don't agree with your argument." The issue is not open to opinion. Rather, we should turn to factual data to support our conclusions. In your trials of WORQS, have there been objective studies that compare it with the old-fashioned CV in terms of time and flexibility? Have you timed faculty who are using WORQS to determine the time it takes to fill out various parts of the form? Would a totally automated system that gathers the info needed by WSU, augmented with free form entry for the chair's evaluation, not make the most sense? If such a system is going to be required of all faculty, a cost-benefit analysis needs to be done; and, if any unit or group takes the brunt of the cost, then the system should not be imposed on anyone. I find it unacceptable that we are being told that WORQS is great when there is evidence to the contrary. Faculty should have more input before use of the system is required.
A university's top priority is to foster an atmosphere that nurtures a dynamic interplay of education and the generation of new knowledge. The administration's responsibility is to remove burdens on faculty so that we can fulfill our mission. I believe that this whole business with WORQS and how it is being imposed on us sends the opposite message. Perhaps I am naive in believing that our faculty is so diverse and creative that one form can not do justice without becoming cumbersome.
On a closing note, we need to ask ourselves the question, "How are we to think outside the box if we spend so much of our time filling out boxes?" Let's use technology to save time and make us more productive.
Sincerely,
Mark G. Kuzyk
Professor and Associate Chair of Physics
and past Chair of the Materials Science Program
P.S. Most faculty in our department with whom I have discussed WORQS also found it quite cumbersome. I am attaching an email that I got from Faculty X. Also, you need to consider the reactions of the staff to this new system. They do not have the luxury of speaking up in the same way as faculty. Based on several conversations, my sense is that WORQS is demoralizing to the staff when most of the form is devoted to information that has no bearing on their work. Have you had broader input from staff beyond committee members?
My Original Email:
Thanks for your detailed response. I have been thinking about annual reviews from the point of view of a faculty member and as associate chair (I will be doing annual reviews this year in our department). I applaud the time saving features such as grants showing up automatically. I was credited with a $6 million dollar grant which I never got. So, that will make me look good! On the other hand, adding publications in the piecemeal way requested is very cumbersome. Even when the journal names are on a pull-down menu, it still takes lots of time to add the title, volume number, pages, type of pub, etc. - especially for productive faculty. Other parts of the form suffer from the same issues. The best solution is to give a choice of cutting and pasting items wholesale OR using the form.
The bottom line is that annual reviews are used to assess performance; and, the chair is best suited to do so. I can tell when one publication is of sufficient high quality that it trumps 10 others. So statistics are useless in my analysis of faculty performance. If WSU wants to gather statistics, then I would be happy to provide my CV for statistical analysis. But, as faculty members, we should not be REQUIRED to waste time entering information in a form that really has nothing to do with annual review, nor does it enhance the process in any way. In your email, you state, "…we can't do that if the system is to be useful." Useful to whom? It's certainly not useful to faculty nor is it to those of us who are doing evaluations.
Our time is chipped away to the extent that I am finding it impossible to do any research, so I do it in my times of "leisure." Yeh, this way of doing annual reviews may take only an extra ½ hour, but there are countless numbers of other such things that we are already doing that also don't make sense. Enough is enough!
My previous communication was sent to you through so and so because I was not aware of the committee working on this form. I hope that my comments are passed along to whatever group is working on this process and to whoever is making the decision to impose this online system.
At this rate, next we'll be requiring all teaching evaluations to be standardized. Though I would be a big beneficiary of such a system, this would be a huge mistake.
Your Response:
I understand your frustration about the many activities that faculty members are asked to do. Unfortunately, your comments have already been passed on to the faculty committee, and they disagree with you. WSU pays your salary all year. For that, the University only asks you to explain what you're doing once per year. Even if you have many, many, many publications and you are a completely incompetent typist, using the pulldown list will take you only minutes more than cutting and pasting. Your use of the pulldown list will allow the University to collect much better data. (So, for example, you won't get credited with the publication equivalent of a $6M grant that you didn't earn.)
I'm sorry that this issue is frustrating to you, but we are relying on a faculty committee to work on this form and they simply don't agree with your argument. So far, approximately half the colleges have decided to use WORQS for annual review this year even though it is still optional. To date, at least some of your colleagues believe that WORQS is a much needed effort saver for the entire University.
Subscribe to:
Posts (Atom)