While reading Scientific American a few days ago, I came upon a quote by Christopher Hitchens, "What can be asserted without proof can be dismissed without proof." This quote is simple yet powerful. If someone makes an assertion that is not supported by evidence, we have the right to reject it without providing evidence.
This idea is related to Bertrand Russell's teapot, a concept first coined by the philosopher Bertrand Russell to illustrate the fact that the burden of proof lies upon the person making scientifically unfalsifiable claims. Russell argued that if he claimed that a teapot were orbiting the sun, it would be nonsense for him to expect others to believe him just because they could not prove him wrong.
In a piece titled, "The Neurology of the Spiritual Experience," an essay in hypertext by Scott Bidstrup, he argues that, "If no metaphysical explanation is required to explain your experience, your 'evidence' is no longer evidence of anything metaphysical." In other words, if a vase falls over in another room, this is not proof that a ghost was responsible if we see a fluttering curtain over an open window that is within range of the vase. Since a fluttering curtain is a known cause for such an event, there is no need to call upon the supernatural for an explanation.
In his essay, Scott Bidstrup summarizes several neurological studies that show how the brain can be artificially stimulated to experience a feeling of spirituality and divine presence. Since such experiences can be produced in the laboratory, again, there is no need to call upon the divine. Some may argue that a deity acts on the brain to create this sensation. Making that claim is logically equivalent to asserting that the flip of a light switch does not release current that powers a light (a known scientific process), but that the phenomena is magic (an unfalsifiable assertion).
As I have written in other posts, I have a strong desire in learning the "truth" without succumbing to the common pitfalls associated with wishful thinking and confirmation bias. The above principles provide examples of simple ways of assessing claims. The scientific method provides an additional arsenal of tools to test assertions that are falsifiable.
I spend lots of time reading articles that illustrate examples of debunked claims, (See, for example, The Skeptical Inquirer, which is an excellent source of information on studies of the paranormal, medical quackery, scientific fraud, etc.) Not only are there many excellent articles that show how logic and controlled experiments can be used to get the real story, there is also a large scientific knowledge base on how the mind can fool itself.
After reading articles on such topics over many years, a picture of the truth slowly comes into focus. For example, there are many well-designed experiments that show that acupuncture does not work as claimed. There are two main issues. First, there are no physiological structures that connect the purported cause with the effect. In other words, there are no mechanisms nor physical pathways that would explain why placing a needle in the foot would relieve the pain of a headache. We understand the network of nerves that transmit information about pain, and there is no conceivable way that acupuncture can work. This alone, however, is not proof against the efficacy of acupuncture.
A series of elegant experiments were performed that fooled patients into believing that they had been administered acupuncture, but, the skin was never punctured. The researchers used trick needles that were in reality dull collapsing rods. Even so, patients reported feeling relief from pain - clearly, a placebo effect. All studies that are carefully controlled for both researcher and patient bias always show a negative result. All of the studies that have shown positive effects were later found to be flawed experiments.
The preponderance of evidence shows that acupuncture is ineffective. When people try to convince me otherwise by alerting me to studies that show a positive correlation, these studies are invariably the ones that have already been found to be flawed. At this point in my life, I do not have the time nor energy to argue with such people. I usually point out that all of the evidence discredits the efficacy of acupuncture and that I prefer to discus a more interesting topic. Accusations follow that I am closed-minded and arrogant.
There are many wacky things that are believed by even intelligent people. It may sound arrogant when scientists appear to pompously reject such ideas, but at some point, enough is enough. Though we may be arrogant to some extent, I do not believe that we are closed minded. When new evidence is produced, our profession demands that we accept it on its merits. When ideas that I believe to be wacky are proven right, I will gladly change my mind. Until then, pardon me for appearing arrogant...
Showing posts with label confirmation bias. Show all posts
Showing posts with label confirmation bias. Show all posts
Monday, March 14, 2011
Friday, February 25, 2011
Confirmation Bias and the Scientific Method
Marc Hauser, a primate psychologist at Harvard and expert on the evolution of morality, was recently found guilty of eight counts of scientific misconduct. At issue was his interpretation of video tapes of rhesus monkeys performing tasks that test their ability to learn sound patterns. Hauser "saw" the behavior that supported his hypothesis and was convinced that he was right even when other members of his research group could not. Being an eminent scientist, his group members deferred to his authority, and his interpretation prevailed in the publications that followed.
Physicist Robert Park wrote in a web column that Hauser fudged the data, implying that it was premeditated and deliberate. Scott Lilienfeld, a psychologist, feels that this might be a simple case of confirmation bias, a psychological response of the brain to reinterpret the world by distorting the data to favor the believer's expectations - a phenomena that is commonly at play in strengthening religious faith.
I just returned from a trip to Wright Patterson Air Force Base, where I gave a seminar about our work on self-healing materials and fundamental limits. My visits to the materials lab are always gratifying because the interdisciplinary team of researchers there understand our work from a broad range of angles. There are chemists who understand structural subtleties of chromophores and how they aggregate as well as the role of the host polymer on the properties of the embedded chromphores. The physicists and quantum chemists, on the other hand appreciate the beauty and utility of our models of light-matter interactions. Each individual brings a unique perspective that enriches my understanding of materials and potential mechanisms of a variety of interesting phenomena that we can apply to interpreting our data.
The initial response of people who have just learned about our observations of self healing conclude that diffusion is responsible. Some of the air force scientists shared this concern. The idea is simple; the laser heats the molecules in the polymer, and the added kinetic energy causes them to move away from the laser. When the laser is turned off, the random walk associated with thermal jiggling causes the molecules to return. Thus, rather than the molecule burning (i.e. breaking into pieces) and then recovering (i.e. resembling), they simply move away and return.
When we first observed this phenomena, diffusion was the first hypothesis that we tested using optical absorption spectroscopy. All molecules absorb light at a set of discrete characteristic wavelengths. The DO11 molecule, our model system, has a big absorption peak centered in the middle of the visible part of the electromagnetic spectrum. If molecules move away from the beam, then the height of the peak will drop, but the shape will remain unchanged. If the molecular structure changes, as should happen in the photo-decomposition process, then a new peak forms that is characteristic of the "burnt" molecule. One can show that as one set of molecules is being converted into another set, then the spectrum evolves in a way where all the spectra cross at one point. This point is called an isobestic point.
When I am asked if self healing might be due to diffusion, I can confidently respond that we see an isobestic point in the linear spectrum. Everyone in the audience then usually thoughtfully shake their heads and acknowledge that this is strong evidence against the diffusion hypothesis.
However, upon reading the article about the Hauser case, I began wondering if I am not being deluded by confirmation bias. While we do see an isobestic point, the process of aligning the probe light (used to measure the spectrum) with the pump is difficult, so the measured absorption changes are not always clean. So, I started wondering if we were not being fooled by subconsciously dismissing data that does not meet our expectations. Upon my return, I met with my students and suggested that they try all sorts of other experiments that differentiate between the two mechanisms. I regret this extra burden that I place on my graduate students, but I would hate to be wrong.
On the computational front, we are also seeing weird results that may render some of our ideas invalid. In particular, we normally observe that when a quantum system is at the fundamental limit, only two excited states contribute to the nonlinear response. We call this the three-level ansatz. In recent Monte Carlo calculations, we are seeing rare outliers where more states contribute. Even more disturbing is that in these rare cases, the limit appears to be broken by a tad.
There are two interpretations to these results. First, there may be a flaw in the fundamental limit calculations. In our work using variational approaches of potential energy functions and vector potentials, we never see a system with a nonlinear response larger than 0.709 times the fundamental limit. In these cases, the three-level ansatz seems to always hold. That gives me some degree of confidence that the calculated limits are correct. The resolution to the problem might lie in the fact that the Monte Carlo work uses truncated sum rules. However, such truncated sum rules are also used to calculate the limits. Are we dismissing the effects of truncation when it suits our purposes?
It is likely that there are many subtle issues that we will need to consider to resolve our new observations. I fear that my brain may be driven by confirmation bias into believing in the fundamental limits and to blame counterexamples on the problem of truncation. We must consider the possibility that the limit calculation may be flawed, in which case it needs to be fixed. In the end, getting to the truth should be our top priority.
I learned about the quirky Monte Carlo results just before my trip to Dayton, Ohio. In addition to my preoccupation with my possible affliction with confirmation bias, I am overwhelmed with all sorts of other work, such as doing the annual reviews of our faculty, refereeing papers, reading dissertations, writing letters of recommendation, and writing new papers, as well as struggling to keep up with my lectures and homework solutions - all while fighting a nasty cold.
I look forward to weekends as a time to catch up with my work. Given my workload, I cannot realistically climb out of this hole; but, I take pleasure in my expectations that in the process of preparing for class and pondering our problems, I will learn new physics. Perhaps these activities will lead to the next new breakthrough on our understanding of the universe. I'll keep you posted.
Physicist Robert Park wrote in a web column that Hauser fudged the data, implying that it was premeditated and deliberate. Scott Lilienfeld, a psychologist, feels that this might be a simple case of confirmation bias, a psychological response of the brain to reinterpret the world by distorting the data to favor the believer's expectations - a phenomena that is commonly at play in strengthening religious faith.
I just returned from a trip to Wright Patterson Air Force Base, where I gave a seminar about our work on self-healing materials and fundamental limits. My visits to the materials lab are always gratifying because the interdisciplinary team of researchers there understand our work from a broad range of angles. There are chemists who understand structural subtleties of chromophores and how they aggregate as well as the role of the host polymer on the properties of the embedded chromphores. The physicists and quantum chemists, on the other hand appreciate the beauty and utility of our models of light-matter interactions. Each individual brings a unique perspective that enriches my understanding of materials and potential mechanisms of a variety of interesting phenomena that we can apply to interpreting our data.
The initial response of people who have just learned about our observations of self healing conclude that diffusion is responsible. Some of the air force scientists shared this concern. The idea is simple; the laser heats the molecules in the polymer, and the added kinetic energy causes them to move away from the laser. When the laser is turned off, the random walk associated with thermal jiggling causes the molecules to return. Thus, rather than the molecule burning (i.e. breaking into pieces) and then recovering (i.e. resembling), they simply move away and return.
When we first observed this phenomena, diffusion was the first hypothesis that we tested using optical absorption spectroscopy. All molecules absorb light at a set of discrete characteristic wavelengths. The DO11 molecule, our model system, has a big absorption peak centered in the middle of the visible part of the electromagnetic spectrum. If molecules move away from the beam, then the height of the peak will drop, but the shape will remain unchanged. If the molecular structure changes, as should happen in the photo-decomposition process, then a new peak forms that is characteristic of the "burnt" molecule. One can show that as one set of molecules is being converted into another set, then the spectrum evolves in a way where all the spectra cross at one point. This point is called an isobestic point.
When I am asked if self healing might be due to diffusion, I can confidently respond that we see an isobestic point in the linear spectrum. Everyone in the audience then usually thoughtfully shake their heads and acknowledge that this is strong evidence against the diffusion hypothesis.
However, upon reading the article about the Hauser case, I began wondering if I am not being deluded by confirmation bias. While we do see an isobestic point, the process of aligning the probe light (used to measure the spectrum) with the pump is difficult, so the measured absorption changes are not always clean. So, I started wondering if we were not being fooled by subconsciously dismissing data that does not meet our expectations. Upon my return, I met with my students and suggested that they try all sorts of other experiments that differentiate between the two mechanisms. I regret this extra burden that I place on my graduate students, but I would hate to be wrong.
On the computational front, we are also seeing weird results that may render some of our ideas invalid. In particular, we normally observe that when a quantum system is at the fundamental limit, only two excited states contribute to the nonlinear response. We call this the three-level ansatz. In recent Monte Carlo calculations, we are seeing rare outliers where more states contribute. Even more disturbing is that in these rare cases, the limit appears to be broken by a tad.
There are two interpretations to these results. First, there may be a flaw in the fundamental limit calculations. In our work using variational approaches of potential energy functions and vector potentials, we never see a system with a nonlinear response larger than 0.709 times the fundamental limit. In these cases, the three-level ansatz seems to always hold. That gives me some degree of confidence that the calculated limits are correct. The resolution to the problem might lie in the fact that the Monte Carlo work uses truncated sum rules. However, such truncated sum rules are also used to calculate the limits. Are we dismissing the effects of truncation when it suits our purposes?
It is likely that there are many subtle issues that we will need to consider to resolve our new observations. I fear that my brain may be driven by confirmation bias into believing in the fundamental limits and to blame counterexamples on the problem of truncation. We must consider the possibility that the limit calculation may be flawed, in which case it needs to be fixed. In the end, getting to the truth should be our top priority.
I learned about the quirky Monte Carlo results just before my trip to Dayton, Ohio. In addition to my preoccupation with my possible affliction with confirmation bias, I am overwhelmed with all sorts of other work, such as doing the annual reviews of our faculty, refereeing papers, reading dissertations, writing letters of recommendation, and writing new papers, as well as struggling to keep up with my lectures and homework solutions - all while fighting a nasty cold.
I look forward to weekends as a time to catch up with my work. Given my workload, I cannot realistically climb out of this hole; but, I take pleasure in my expectations that in the process of preparing for class and pondering our problems, I will learn new physics. Perhaps these activities will lead to the next new breakthrough on our understanding of the universe. I'll keep you posted.
Monday, July 26, 2010
The Supreme Court of Science
I recently read a short article in The Economist on the topic of fairness. The gist of the piece was that fairness depends on the perspective of the observer. For example, a conservative may argue that an entrepreneur is entitled to keep the proceeds resulting from hard work provided that the rules ensure a level playing field. If all individuals have an equal opportunity to participate, and the rewards are in proportion to the effort, then any redistribution would be unfair. Even under such idealized conditions, a liberal might argue that it is unfair for someone to keep all the profits because the resources would be better spent by a benevolent government on projects that are more noble than the greedy and wasteful accumulation of wealth by an individual.
Imagine a sporting event where two teams battle for a couple hours before throngs of cheering fans. At the end of the contest, the winning team is required to donate points to the opponents to force a tie. Now would this be fair? Clearly, this solution would anger all parties. Such analogies gloss over the nuances and complexities of the real issues, and are the staple of spin doctors that make for hyper-actively-delivered sound bites on Fox News. This is not to say that a Utopian society run by a benevolent government is a viable alternative to the unscrupulous industrialists portrayed by liberals.
Members of a free society are entitled to jabber nonsense and to exercise personal preferences in private or with like-minded (or mindless) individuals. However, when such nonsense finds its ways into politics and fills the halls of power, government becomes dysfunctional. The problems are compounded by politicians who are forced to pander to the masses. Many of us complain that politicians all lie. The root of the problem resides in the stupidity of the electorate, not in the politicians who must walk a fine line that avoids the minefield of our dogmatic sensitivities yet satisfies our contradictory demands.
This craziness originates in the way our neurons are hard-wired. Scientific evidence has documented all sorts of peculiar but fascinating traits of the human brain such as confirmation bias, which selectively reinforces beliefs based on dubious evidence and ignores floods of data to the contrary. We fool ourselves into believing all kinds of wacky ideas while easily detecting the folly in the conviction of others. We do not hesitate to accept contradictory concepts of truth. All this leads to vigorous ideological wars. The scientific method is designed to minimize these biases when seeking the truth.
Consider the irrational protest against childhood vaccinations. While nothing is totally risk free, the benefits of inoculations far exceed the risks. In addition to a growing swell in deniers of the efficacy of all vaccinations, some individuals believe that some vaccines cause autism. There is no scientific evidence that shows autism to result from the mercury compound used to preserve vaccinations (the rate of autism is no different with and without the mercury compound), but large numbers of people don't buy it and are applying political pressure to remove the important preservative. Though it is one of the most powerful theories in all of science, there are large numbers of people that do not believe in evolution. Similarly, many people can't seem to shake the belief that power lines and cell phones cause brain tumors.
False beliefs are innocent enough, but when these beliefs lead to action - such as forcing teachers to give equal weight to evolution and creationism/intelligent design or withholding inoculations from children - humanity suffers. Real children die and the power of ignorance fuels further disinformation that leads to a decrease in the quality of life.
The supreme court weighs in on legal matters, so why not assemble a group of unbiased and open-minded scientists to rule on the veracity of issues that can be resolved with evidence-based methods? The patent office automatically rejects perpetual motion machine filings, so why not automatically disallow a bogus settlement against a defendant who has science on her side? Empower science court to strike down a proposed curriculum that teaches religion under the guise of science, to veto consumer pressure to force insurance payments to alternative medical practices whose methods are baseless, and to disallow lawsuits that break the laws of physics.
Legal courts arbitrate disputes between parties using the law of the land as the basis of a decision. When science becomes an integral part of a case, the courts rely on expert witnesses. Unfortunately, it is always possible to find an "expert" with seemingly acceptable credentials who will contradict the scientific consensus. In these cases, jurors may end up returning verdicts based on the way the testimony is extracted rather than on the weight of the evidence. This method of interrogation may be the best available when deciding whose side of the story to accept, but scientific truth is best gleaned by a careful analysis of the evidence in the absence of theatrics and emotions.
For example, there is no evidence linking brain tumors to electromagnetic radiation from power lines and cell phones. Not only is there no statistically significant correlation between the two, but all plausible mechanisms of electromagnetically-induced tumor growth are inconsistent with the laws of physics describing the interactions of electromagnetic waves with matter. Suits against power companies for causing brain tumors should not be permitted to be brought to trial when the best scientific evidence shows the accusation baseless. That such cases are even heard implies ignorance of the facts; or, perhaps the courts believe that they can determine the veracity of the underlying science in lieu of the well-established body of literature.
In the absence of evidence, a typical vacuous argument posits that there might be a connection based on unknown science. This is equivalent to stating that unicorns cause sexual dysfunction. We would chuckle if an individual brought a case against a supermarket for allowing invisible unicorns to roam its isles, endangering the reproductive ability of its shoppers. The electromagnetic suits are scientifically equally ridiculous, but are not met with the incredulity they deserve.
A science court would make rulings based on the best available scientific evidence - the statutes being the laws of physics, chemistry, biology, etc. If an electromagnetic tumor case were to be filed, it would be dismissed before wasting society's resources. A suit against a pharmaceutical company for causing autism would also be dismissed as would a suit to force feed creationism in public schools under the guise of science.
Science evolves to accommodate new evidence. The guiding principles used by the judges of science court would be adjusted to take into account new discoveries. In short, science court would provide the most current and up-to-date information to ensure that judgments are made on the basis of the best available evidence rather than emotional bias or political philosophy. Judges, who are experts in the law, are well qualified to deliver verdicts pertaining to legal matters. Individuals who are well versed in the scientific method should rule on issues that are based on scientific principles, and to set precedents that would be used by the legal system. It is time that we recognize that science is not legislated, but discovered as a result of carefully-designed experiments that minimize bias.
Bias colors the perceived fairness of an outcome, even when each party draws on the same facts. To be ignorant of the established scientific consensus when deliberating a settlement is unconscionable. To purposefully ignore or obfuscate the facts to obtain a favorable outcome is criminal, especially when championing an ideology . To me, fairness starts with a process of evaluating and accepting the scientific consensus and using it as the bedrock of the laws. The establishment of an independent judicial branch that codifies the best science into a practical form is a necessary step for moving forward into a future that is both dependent upon and threatened by the technologies that science has enabled.
Imagine a sporting event where two teams battle for a couple hours before throngs of cheering fans. At the end of the contest, the winning team is required to donate points to the opponents to force a tie. Now would this be fair? Clearly, this solution would anger all parties. Such analogies gloss over the nuances and complexities of the real issues, and are the staple of spin doctors that make for hyper-actively-delivered sound bites on Fox News. This is not to say that a Utopian society run by a benevolent government is a viable alternative to the unscrupulous industrialists portrayed by liberals.
Members of a free society are entitled to jabber nonsense and to exercise personal preferences in private or with like-minded (or mindless) individuals. However, when such nonsense finds its ways into politics and fills the halls of power, government becomes dysfunctional. The problems are compounded by politicians who are forced to pander to the masses. Many of us complain that politicians all lie. The root of the problem resides in the stupidity of the electorate, not in the politicians who must walk a fine line that avoids the minefield of our dogmatic sensitivities yet satisfies our contradictory demands.
This craziness originates in the way our neurons are hard-wired. Scientific evidence has documented all sorts of peculiar but fascinating traits of the human brain such as confirmation bias, which selectively reinforces beliefs based on dubious evidence and ignores floods of data to the contrary. We fool ourselves into believing all kinds of wacky ideas while easily detecting the folly in the conviction of others. We do not hesitate to accept contradictory concepts of truth. All this leads to vigorous ideological wars. The scientific method is designed to minimize these biases when seeking the truth.
Consider the irrational protest against childhood vaccinations. While nothing is totally risk free, the benefits of inoculations far exceed the risks. In addition to a growing swell in deniers of the efficacy of all vaccinations, some individuals believe that some vaccines cause autism. There is no scientific evidence that shows autism to result from the mercury compound used to preserve vaccinations (the rate of autism is no different with and without the mercury compound), but large numbers of people don't buy it and are applying political pressure to remove the important preservative. Though it is one of the most powerful theories in all of science, there are large numbers of people that do not believe in evolution. Similarly, many people can't seem to shake the belief that power lines and cell phones cause brain tumors.
False beliefs are innocent enough, but when these beliefs lead to action - such as forcing teachers to give equal weight to evolution and creationism/intelligent design or withholding inoculations from children - humanity suffers. Real children die and the power of ignorance fuels further disinformation that leads to a decrease in the quality of life.
The supreme court weighs in on legal matters, so why not assemble a group of unbiased and open-minded scientists to rule on the veracity of issues that can be resolved with evidence-based methods? The patent office automatically rejects perpetual motion machine filings, so why not automatically disallow a bogus settlement against a defendant who has science on her side? Empower science court to strike down a proposed curriculum that teaches religion under the guise of science, to veto consumer pressure to force insurance payments to alternative medical practices whose methods are baseless, and to disallow lawsuits that break the laws of physics.
Legal courts arbitrate disputes between parties using the law of the land as the basis of a decision. When science becomes an integral part of a case, the courts rely on expert witnesses. Unfortunately, it is always possible to find an "expert" with seemingly acceptable credentials who will contradict the scientific consensus. In these cases, jurors may end up returning verdicts based on the way the testimony is extracted rather than on the weight of the evidence. This method of interrogation may be the best available when deciding whose side of the story to accept, but scientific truth is best gleaned by a careful analysis of the evidence in the absence of theatrics and emotions.
For example, there is no evidence linking brain tumors to electromagnetic radiation from power lines and cell phones. Not only is there no statistically significant correlation between the two, but all plausible mechanisms of electromagnetically-induced tumor growth are inconsistent with the laws of physics describing the interactions of electromagnetic waves with matter. Suits against power companies for causing brain tumors should not be permitted to be brought to trial when the best scientific evidence shows the accusation baseless. That such cases are even heard implies ignorance of the facts; or, perhaps the courts believe that they can determine the veracity of the underlying science in lieu of the well-established body of literature.
In the absence of evidence, a typical vacuous argument posits that there might be a connection based on unknown science. This is equivalent to stating that unicorns cause sexual dysfunction. We would chuckle if an individual brought a case against a supermarket for allowing invisible unicorns to roam its isles, endangering the reproductive ability of its shoppers. The electromagnetic suits are scientifically equally ridiculous, but are not met with the incredulity they deserve.
A science court would make rulings based on the best available scientific evidence - the statutes being the laws of physics, chemistry, biology, etc. If an electromagnetic tumor case were to be filed, it would be dismissed before wasting society's resources. A suit against a pharmaceutical company for causing autism would also be dismissed as would a suit to force feed creationism in public schools under the guise of science.
Science evolves to accommodate new evidence. The guiding principles used by the judges of science court would be adjusted to take into account new discoveries. In short, science court would provide the most current and up-to-date information to ensure that judgments are made on the basis of the best available evidence rather than emotional bias or political philosophy. Judges, who are experts in the law, are well qualified to deliver verdicts pertaining to legal matters. Individuals who are well versed in the scientific method should rule on issues that are based on scientific principles, and to set precedents that would be used by the legal system. It is time that we recognize that science is not legislated, but discovered as a result of carefully-designed experiments that minimize bias.
Bias colors the perceived fairness of an outcome, even when each party draws on the same facts. To be ignorant of the established scientific consensus when deliberating a settlement is unconscionable. To purposefully ignore or obfuscate the facts to obtain a favorable outcome is criminal, especially when championing an ideology . To me, fairness starts with a process of evaluating and accepting the scientific consensus and using it as the bedrock of the laws. The establishment of an independent judicial branch that codifies the best science into a practical form is a necessary step for moving forward into a future that is both dependent upon and threatened by the technologies that science has enabled.
Subscribe to:
Posts (Atom)