Program / Course 23: Research Methodology / Lesson 5.4
What you will learn
- Define confirmation bias and describe how it affects searching, reading, remembering and writing.
- Use concrete practices to test your own conclusions, such as stating disconfirming evidence in advance.
- Represent an opposing view in a form its holders would accept.
- Recognize the limits of these practices and the value of other readers.
The big idea
Confirmation bias is the habit of looking for, noticing, remembering and favoring evidence that supports what you already think, and of discounting evidence against it. It affects everyone, and it is hardest to see in oneself, especially on subjects one cares about. It cannot be removed by good intentions, but it can be reduced by practice: stating in advance what would change your mind, searching for the best opposing scholarship, keeping a record of evidence that does not fit, and asking others to challenge the draft.
Key terms
- Confirmation bias
- The tendency to seek, notice and remember information that confirms existing beliefs and to overlook or downplay information that does not.
- Motivated reasoning
- Reasoning guided by a wish for a particular conclusion, which makes weak support for that conclusion seem strong.
- Disconfirming evidence
- Evidence that tells against a claim. A researcher should seek it as actively as supporting evidence.
- Falsifiability
- The quality of a claim that can be shown wrong by some possible evidence. A claim that no evidence could ever count against is not testable.
- Steelman
- A statement of an opposing view in its strongest and most persuasive form, the opposite of a straw man.
- Devil’s advocate
- A person who argues against a position in order to test it, whether or not they hold the opposing view.
- Hindsight bias
- The tendency, after learning an outcome, to think it was obvious or inevitable.
- Pre-registration of expectations
- Writing down what you expect to find and what would surprise you before you look, so that you can compare afterward.
Background
Psychologists have documented confirmation bias in many settings. People search for evidence that fits their view, interpret ambiguous evidence in its favor, and recall supporting examples more readily than opposing ones. Researchers in every field have to guard against it. In religion and history, where convictions are strong and the evidence is often sparse and open to several readings, the danger is especially real. A believer may favor sources that support the tradition, and a skeptic may favor sources that undermine it. Neither is exempt.
The Bible speaks to the problem in its own way. Proverbs says that the first to present a case seems right until another comes and examines him (Proverbs 18:17), and that the wise listen to counsel (Proverbs 12:15). Paul warns against thinking more highly of oneself than one ought, and asks for sober judgment (Romans 12:3). Such texts do not use the phrase confirmation bias, but they point to the same weakness and to the same remedy: another person who tests what you say.
Reducing bias is a matter of method. You cannot will yourself to be unbiased. You can, however, build habits that make bias harder to act on. These include writing down expectations, searching for the opposing case on purpose, reading in the other direction, keeping a record of what does not fit, and having others examine the work. They do not guarantee fairness, but they improve it. A researcher who uses them will often find that the evidence is more complicated than expected, and that is a sign that the method is working.
Step-by-step explanation
Step 1: Write down what you expect and what would change your mind
At the start, state your working thesis and write: ‘I expect to find X. I would be surprised by Y. If I found Z, I would have to revise my thesis.’ Date the note. Later you can compare the result with your expectation and see where it moved.
If you cannot say what evidence would count against your view, ask whether the view is open to testing at all.
Step 2: Search for the opposing case on purpose
Search for the strongest scholarship that disagrees with you, using the methods of Module 3. Read it as if you were to be examined on it. Include sources from other traditions or schools. If you cannot find any opposing work, the search is probably incomplete.
Read the opposing side’s best book, not its weakest pamphlet. Winning against a weak version proves little.
Step 3: State the opposing view so that its holders would accept it
Write a paragraph presenting the opposing view as persuasively as you can. This is sometimes called steelmanning. Show it to someone who holds the view, if possible, or compare it with the way its best advocates state it. If they would say ‘Yes, that is what I think,’ you have it right.
Only after that, state your objections. Disagreement based on understanding carries more weight.
Step 4: Keep a record of evidence that does not fit
Keep a running list headed ‘Evidence that does not fit.’ Add every passage, date or argument that complicates your thesis, with its source. Do not delete entries because they are inconvenient.
Use the list in the draft. Either revise the thesis, qualify it, or give a reasoned reply to each item. A paper that has dealt with its strongest objection is both more honest and more persuasive.
Step 5: Ask others to challenge the draft
Ask a careful reader, ideally one who disagrees with you, to read the draft and tell you where the argument is weakest, what they would reply, and what you have left out. Give them permission to be blunt. ‘Faithful are the wounds of a friend’ (Proverbs 27:6) describes the spirit.
Receive criticism without defending yourself at once. Write down what they say, wait a day and then decide which points need a response. Their criticism is evidence, too.
- What do I hope to find, and how might that hope shape my search?
- What evidence would make me change my thesis?
- Have I read the best case against my view?
- Would a supporter of the opposing view accept my description of it?
- What have I left out of my notes because it was inconvenient?
- Who could read my draft and tell me honestly where it is weak?
Examples
Biased searching
A student who wants to show that a doctrine is ancient searches for ‘early evidence for [the doctrine]’ and finds many results. She does not search for ‘development of [the doctrine]’ or ‘objections to an early date’, where she would find scholars who disagree. Her search terms decided the result. (A constructed example.)
Selective reading
A student reads a Father’s long work looking for a particular phrase and underlines it each time. He misses three passages that qualify the phrase because he was not looking for them. A second reading with the question ‘What complicates this?’ finds them. (A constructed example.)
Steelmanning
A Protestant student writing on the veneration of icons begins by stating the case of the Seventh Ecumenical Council and the Orthodox and Catholic defenders of images, using their own texts and categories, and only then states the objections of the Reformers. A reader from either side would recognize the account of its own position.
A changed conclusion
A student expected to show that a writer had no interest in a certain doctrine. After keeping a list of evidence that did not fit, she found five passages that discussed it. She revised the thesis to say that the writer treated it in a different way than later writers did. (A constructed example.)
Different perspectives
Psychology and the sciences
Research on reasoning shows that confirmation bias is widespread and that intelligence does not remove it. Scientists use practices like blind testing, replication and peer criticism to reduce its effects. The same ideas apply to historical and textual work, with the caution that the evidence in these fields is more often indirect.
Catholic
The Catholic scholastic method of stating the strongest objections before answering them is a model of tested reasoning. Catholic teaching also sees the examination of conscience as a discipline that includes honesty about one’s own motives, and scholars may apply this to the study of evidence.
Orthodox
Orthodox spiritual writers warn about self-deception and the passions that cloud judgment, and value the guidance of an elder or confessor. A scholar may find in this tradition the habit of subjecting one’s thoughts to another’s counsel, which agrees with the practice of asking others to read a draft.
Protestant
Protestant traditions stress the fallibility of human judgment and the need to test teachers against Scripture, as the Bereans did (Acts 17:11). They have also seen strong, and sometimes bitter, debates among themselves. A fair Protestant researcher takes account of arguments from other Protestant groups and from other traditions.
Why it matters
A paper that confirms the writer’s views without testing them adds nothing to what the writer already knew, and a reader will sense it. A paper that has met its strongest objections honestly is worth reading even by those who disagree. Beyond the paper, the habits of this lesson make you a better thinker in every area where you care about the answer, which is where you are most at risk of fooling yourself.
Common misunderstandings
Misunderstanding: I am objective, so confirmation bias does not affect me.
Correction: The bias is largely unconscious and affects careful people as well as careless ones. Feeling objective is not evidence of being so. Use methods, not confidence.
Misunderstanding: To avoid bias I must have no opinions.
Correction: Holding a view is not the problem. The problem is protecting it from testing. A researcher can hold a view firmly and still test it fairly.
Misunderstanding: Presenting the other side fairly weakens my own argument.
Correction: It strengthens it, because readers trust a writer who is fair and see that the conclusion survived real opposition.
Misunderstanding: Asking others to criticize my work means I lack confidence.
Correction: It is a mark of mature scholarship. Good scholars seek criticism before publication so that errors are found early.
Knowledge check
What is confirmation bias?
It is the tendency to seek, notice, interpret and remember evidence in ways that support what one already believes, and to overlook or downplay evidence against it. It affects searches, reading, memory and writing, and is hard to see in oneself, especially on subjects one cares about.
Name three practices that reduce confirmation bias in research.
Examples: write down in advance what you expect and what would change your mind; search for the strongest opposing scholarship; state the opposing view in a form its holders would accept; keep a list of evidence that does not fit; ask a critical reader to challenge the draft. Any three of these are correct.
What is steelmanning, and why use it?
Steelmanning means stating an opposing view in its strongest and most persuasive form. It is the opposite of a straw man. It ensures that you understand the view and that your replies address what its best defenders actually say, which makes the argument fairer and stronger.
Why keep a list of evidence that does not fit?
Inconvenient evidence is easy to forget or discard. A written list keeps it in view and makes the draft deal with it, either by revising the thesis, qualifying it or replying to the point. A paper that faces its strongest objections is more honest and more persuasive.
Think it through
- On what subject are you most likely to be affected by confirmation bias? How would you know?
- Who in your life would tell you the truth about a draft, and have you asked?
- How is it possible to hold strong convictions and still test them fairly?
- What does it mean to be willing to be wrong about something you care about deeply?
Independent assignment
Testing my own thesis
In your own notebook, take your working thesis. Write about 400 words in four parts: what you expect to find and what would surprise you; a list of at least three pieces of evidence that do not fit or complicate your thesis; a paragraph stating the best opposing view so that its holders would accept it; and a plan for who will read your draft critically. Do not send this to anyone unless you choose to ask a reader for comments.
Further study
- Stanford Encyclopedia of Philosophy, Fallacies Read the sections on cherry-picking and begging the question as tests for your own draft.
- Stanford Encyclopedia of Philosophy, Philosophy of History Read selectively on objectivity and perspective in historical writing.
- Bruce M. Metzger, The Canon of the New Testament Observe how a scholar reasons from limited evidence and states what cannot be known.
- Proverbs 12:15 The wise listen to counsel.
- Proverbs 27:6 Faithful are the wounds of a friend.
- Romans 12:3 Think with sober judgment.
- Acts 17:11 The Bereans examined the Scriptures daily to see whether these things were so.