Confirmation Bias and the Scientific Method

Marc Hauser, a primate psychologist at Harvard and expert on the evolution of morality, was recently found guilty of eight counts of scientific misconduct. At issue was his interpretation of video tapes of rhesus monkeys performing tasks that test their ability to learn sound patterns. Hauser "saw" the behavior that supported his hypothesis and was convinced that he was right even when other members of his research group could not. Being an eminent scientist, his group members deferred to his authority, and his interpretation prevailed in the publications that followed.

Physicist Robert Park wrote in a web column that Hauser fudged the data, implying that it was premeditated and deliberate. Scott Lilienfeld, a psychologist, feels that this might be a simple case of confirmation bias, a psychological response of the brain to reinterpret the world by distorting the data to favor the believer's expectations - a phenomena that is commonly at play in strengthening religious faith.

I just returned from a trip to Wright Patterson Air Force Base, where I gave a seminar about our work on self-healing materials and fundamental limits. My visits to the materials lab are always gratifying because the interdisciplinary team of researchers there understand our work from a broad range of angles. There are chemists who understand structural subtleties of chromophores and how they aggregate as well as the role of the host polymer on the properties of the embedded chromphores. The physicists and quantum chemists, on the other hand appreciate the beauty and utility of our models of light-matter interactions. Each individual brings a unique perspective that enriches my understanding of materials and potential mechanisms of a variety of interesting phenomena that we can apply to interpreting our data.

The initial response of people who have just learned about our observations of self healing conclude that diffusion is responsible. Some of the air force scientists shared this concern. The idea is simple; the laser heats the molecules in the polymer, and the added kinetic energy causes them to move away from the laser. When the laser is turned off, the random walk associated with thermal jiggling causes the molecules to return. Thus, rather than the molecule burning (i.e. breaking into pieces) and then recovering (i.e. resembling), they simply move away and return.

When we first observed this phenomena, diffusion was the first hypothesis that we tested using optical absorption spectroscopy. All molecules absorb light at a set of discrete characteristic wavelengths. The DO11 molecule, our model system, has a big absorption peak centered in the middle of the visible part of the electromagnetic spectrum. If molecules move away from the beam, then the height of the peak will drop, but the shape will remain unchanged. If the molecular structure changes, as should happen in the photo-decomposition process, then a new peak forms that is characteristic of the "burnt" molecule. One can show that as one set of molecules is being converted into another set, then the spectrum evolves in a way where all the spectra cross at one point. This point is called an isobestic point.

When I am asked if self healing might be due to diffusion, I can confidently respond that we see an isobestic point in the linear spectrum. Everyone in the audience then usually thoughtfully shake their heads and acknowledge that this is strong evidence against the diffusion hypothesis.

However, upon reading the article about the Hauser case, I began wondering if I am not being deluded by confirmation bias. While we do see an isobestic point, the process of aligning the probe light (used to measure the spectrum) with the pump is difficult, so the measured absorption changes are not always clean. So, I started wondering if we were not being fooled by subconsciously dismissing data that does not meet our expectations. Upon my return, I met with my students and suggested that they try all sorts of other experiments that differentiate between the two mechanisms. I regret this extra burden that I place on my graduate students, but I would hate to be wrong.

On the computational front, we are also seeing weird results that may render some of our ideas invalid. In particular, we normally observe that when a quantum system is at the fundamental limit, only two excited states contribute to the nonlinear response. We call this the three-level ansatz. In recent Monte Carlo calculations, we are seeing rare outliers where more states contribute. Even more disturbing is that in these rare cases, the limit appears to be broken by a tad.

There are two interpretations to these results. First, there may be a flaw in the fundamental limit calculations. In our work using variational approaches of potential energy functions and vector potentials, we never see a system with a nonlinear response larger than 0.709 times the fundamental limit. In these cases, the three-level ansatz seems to always hold. That gives me some degree of confidence that the calculated limits are correct. The resolution to the problem might lie in the fact that the Monte Carlo work uses truncated sum rules. However, such truncated sum rules are also used to calculate the limits. Are we dismissing the effects of truncation when it suits our purposes?

It is likely that there are many subtle issues that we will need to consider to resolve our new observations. I fear that my brain may be driven by confirmation bias into believing in the fundamental limits and to blame counterexamples on the problem of truncation. We must consider the possibility that the limit calculation may be flawed, in which case it needs to be fixed. In the end, getting to the truth should be our top priority.

I learned about the quirky Monte Carlo results just before my trip to Dayton, Ohio. In addition to my preoccupation with my possible affliction with confirmation bias, I am overwhelmed with all sorts of other work, such as doing the annual reviews of our faculty, refereeing papers, reading dissertations, writing letters of recommendation, and writing new papers, as well as struggling to keep up with my lectures and homework solutions - all while fighting a nasty cold.

I look forward to weekends as a time to catch up with my work. Given my workload, I cannot realistically climb out of this hole; but, I take pleasure in my expectations that in the process of preparing for class and pondering our problems, I will learn new physics. Perhaps these activities will lead to the next new breakthrough on our understanding of the universe. I'll keep you posted.