Skip to content

Course 23 · Lesson 3.1

How peer review works and what it does not guarantee

10 minute read

Lessons in this course

Course 23 outline

Module 1: How Research Works

  1. What research is and what a scholarly paper is for
  2. From topic to question: writing a research question that can be answered
  3. The working thesis and how it changes as research develops
  4. Taking notes, recording sources and organizing a project
  5. Academic integrity: quotation, paraphrase and plagiarism

Module 2: Primary and Secondary Sources

  1. Primary and secondary sources in theology and church history
  2. Source criticism: author, date, audience, purpose and transmission
  3. Translations, editions and the problem of reading ancient texts in English
  4. Judging the reliability of a source, including websites and popular books

Module 3: Finding and Reading Scholarship

  1. How peer review works and what it does not guarantee
  2. Finding articles, books and reference works with free and library tools
  3. Reading a scholarly work efficiently and critically
  4. Writing a literature review: summarizing a debate

Module 4: Citation

  1. Chicago notes and bibliography: books, articles and chapters
  2. Citing the Fathers, councils, creeds and online sources
  3. Biblical citation conventions: books, abbreviations, versions and numbering
  4. Managing a bibliography and checking citations

Module 5: Evaluating Arguments and Bias

  1. The anatomy of an argument: claim, evidence, assumption and conclusion
  2. Common fallacies and weak arguments in theological writing
  3. Identifying bias in sources: confessional, ideological and commercial
  4. Confirmation bias in your own work, and ways to reduce it

Module 6: Constructing Arguments and Writing

  1. Building a theological argument from Scripture, tradition and reason
  2. Outlining and drafting a research paper
  3. Clear analytical prose: paragraphs, transitions and plain language
  4. Revising, editing and checking quotations and facts

All courses

Program / Course 23: Research Methodology / Lesson 3.1

What you will learn

  • Describe the steps by which an article or book passes peer review.
  • Explain what peer review is good at and what it cannot do.
  • Tell peer-reviewed work from editor-reviewed, self-published and unreviewed work.
  • Treat a peer-reviewed source as a strong starting point while still reading it critically.

The big idea

Peer review is the process by which experts judge a manuscript before it is published. The editor sends it to specialists, who report on its competence, originality and argument, and the author revises or is rejected. The process improves quality and screens out much weak work, but it does not guarantee truth. Reviewers share assumptions, journals have commitments and errors pass through. A peer-reviewed source is a good place to start, not a reason to stop thinking.

Key terms

Peer review
Evaluation of a manuscript by independent experts in the field before publication.
Referee or reviewer
An expert who reads a submitted work and reports on whether it should be published, revised or rejected.
Double-blind review
A system in which neither author nor reviewer knows the other’s identity, intended to reduce bias.
Editor
The person responsible for a journal or book series who decides, with reviewers’ advice, what is published.
Monograph
A scholarly book on a single subject by one or a few authors, often published by a university press.
Edited volume
A book of chapters by different authors, gathered by an editor. Review standards vary.
Retraction
A published notice that an article has been withdrawn because of serious error or misconduct.
Predatory journal
A publication that takes fees from authors and offers little or no genuine review.

Background

When a scholar finishes an article, she sends it to a journal. The editor reads it and decides whether it fits the journal and is worth reviewing. If so, it goes to two or three experts, often anonymously. They read it, judge the argument, the use of sources and the contribution, and recommend acceptance, revision or rejection. The author usually must revise in response, sometimes several times. Only then is it published. Book publishers at university presses use a similar process, sending proposals and manuscripts to outside readers.

The system has real strengths. It catches many errors, forces authors to answer objections, and gives readers some assurance that a competent person has checked the work. It is why a scholarly journal is usually a better source than a blog. Reviewers can also suggest sources the author has missed. For a student, it is a useful first filter when choosing which sources to read closely.

It also has limits, which a researcher must know. Reviewers are busy and unpaid and cannot recheck every source. They share the field’s assumptions, so a view that most experts hold may pass without challenge, and a view that challenges the field may be treated harshly. Journals have their own commitments: some are confessional, some are secular, some favor a school of thought. Errors and even fraud occasionally pass through, and are later corrected or retracted. Peer review in theology also varies, since the field includes both university-based and church-based publishing, with different traditions of review.

Step-by-step explanation

Step 1: Recognize the signs of peer review

Journals usually state their review process on their web pages under headings such as ‘About’ or ‘Information for authors’. University presses and other academic publishers describe their review for books. Look for a statement that submissions are reviewed by outside experts.

Library databases often let you limit a search to peer-reviewed or scholarly journals. This is helpful, though the filter is not perfect.

Step 2: Know the kinds of publication

Peer-reviewed journals and university press monographs are the core of scholarship. Chapters in edited volumes may be reviewed in various degrees. Book reviews are short evaluations by a scholar and are not peer-reviewed articles, though they are useful for judging a book. Encyclopedia entries by specialists are edited and often reviewed, but are tertiary.

Self-published works, blogs, magazine essays and many church publications are not peer-reviewed, though some are excellent. Treat them according to the lessons on reliability.

Step 3: Be alert to weak or false review

Some publications call themselves scholarly and charge authors fees but do not review seriously. Warning signs include no named editors, vague claims of fast review, an odd range of subjects, and aggressive requests for submissions. If a journal is unfamiliar, search for it and see whether established libraries and scholars treat it as legitimate.

A journal’s name can be confusing, since some deliberately resemble respected titles. Check the publisher.

Step 4: Read peer-reviewed work critically anyway

Treat a peer-reviewed article as an argument by a person, checked by others. Ask the questions of Module 5: what is the claim, what is the evidence, what is assumed, what are the alternatives? Check whether later scholars have responded.

Read book reviews of important monographs. Experts often point out strengths and weaknesses, and a review may give you a more balanced view than the book’s own introduction.

Step 5: Consider the journal’s character

Find out who publishes the journal and what its aims are. A journal of a theological society, a university press or a church may have a stated confession, and its reviewers may share it. This is not disqualifying, but it affects which questions are likely to be raised and which are not.

To balance this, read widely across journals with different commitments, and include voices that disagree.

  1. Does the journal or publisher describe how it reviews work?
  2. Is this an article, a book chapter, a review or an essay, and how are they reviewed?
  3. Has this work been answered or criticized by later scholars?
  4. What commitments does the journal or publisher hold?
  5. Is the claim one that most experts accept, or one that is contested?
  6. What would I look for if I disagreed with the article?

Examples

A strong source

An article in an established journal published by a university or a learned society, written by a scholar at a recognized institution, with notes and a bibliography, and a statement that it was reviewed by outside experts. It is a good starting point for evidence.

A source that looks scholarly but is not

A website with the word ‘Journal’ in its title, a long list of editors without institutions, and a promise of publication within a week for a fee. It resembles a journal but offers no real review. (A constructed example of the pattern.)

A reviewed work that was later challenged

A monograph published by a respected press that argues a bold thesis. Several reviewers praise its learning, but others point out that the central argument is not supported by the evidence. The reviews are part of the scholarly conversation, and a researcher should read them. (A constructed example.)

Different perspectives

Academic scholarship

Most scholars regard peer review as the best available system for maintaining quality, while recognizing its flaws. Discussions in many fields have raised questions about bias, delays and reviewers’ workloads, and have led to experiments with open review and with post-publication comment. Few suggest abandoning it altogether.

Catholic

Catholic theology is published in university presses and in journals of Catholic universities and religious orders. Some works carry ecclesiastical approvals such as the nihil obstat and imprimatur, which indicate that the Church found nothing contrary to faith and morals in them. These are statements about doctrine, not scholarly peer review, and the two should not be confused.

Orthodox

Orthodox scholarship appears in the journals of Orthodox seminaries and in general university presses. Because the Orthodox community in the English-speaking world is relatively small, some fields have fewer specialist reviewers. Orthodox authors also publish widely in non-Orthodox journals, where their work meets outside review.

Protestant

Protestant scholars publish in a very wide range of journals, from those of university religion departments to those of confessional seminaries and societies. Journals differ in doctrinal commitments, which may affect what is submitted and accepted. A researcher can benefit from reading both confessional and non-confessional journals on the same question.

Why it matters

Understanding peer review lets you use it wisely. If you treat it as a guarantee, you may accept weak claims because they appear in a good journal. If you dismiss it, you lose a major source of reliable work. Knowing what it does and does not do lets you give scholarship the trust it has earned and no more, which is how scholars themselves treat it.

Common misunderstandings

Misunderstanding: Peer-reviewed means proven.

Correction: It means that competent people judged the work to be worth publishing. The conclusions may still be disputed or later corrected. Peer review improves reliability but does not produce certainty.

Misunderstanding: Anything not peer-reviewed is worthless.

Correction: Many valuable works, such as books from trade presses, public lectures and careful blogs by experts, are not peer-reviewed. Judge them by the earlier criteria, and use scholarly sources to confirm important claims.

Misunderstanding: Peer review removes bias.

Correction: Reviewers have views and the field has fashions. Review reduces some mistakes but not shared assumptions. Reading across schools of thought helps.

Misunderstanding: A church’s approval of a book is a scholarly review.

Correction: A church’s approval concerns doctrine or suitability for its members. It is a different judgment from a scholarly review of the quality of research.

Knowledge check

Briefly describe how peer review works.

An author submits a manuscript to a journal or press. The editor decides whether it is suitable and sends it to independent experts, often anonymously. They judge its competence, originality and argument and recommend acceptance, revision or rejection. The author revises, and the editor makes the final decision. Only then is the work published.

Name two things peer review does well and two things it does not do.

It catches many errors and weak arguments, and it makes authors answer objections and use sources properly. It does not guarantee that the conclusion is true, and it does not remove the shared assumptions of the field. Errors, bias and even fraud can pass through. Other correct answers are possible.

How can you spot a predatory journal?

Warning signs include promises of very fast publication, fees without clear review, no named or verifiable editors, a very broad range of subjects, a name that mimics a respected journal, and aggressive email invitations. Check whether libraries and established scholars recognize the journal and who publishes it.

Is a church’s imprimatur the same as peer review?

No. An imprimatur is an ecclesiastical statement that a work contains nothing contrary to faith and morals. Peer review is an evaluation of the quality of research by independent experts. A book may have one, both or neither, and each tells the reader something different.

Think it through

  1. When have you accepted a claim because it appeared in a respectable place? What would checking have involved?
  2. How might a confessional journal both help and limit a field of study?
  3. What would a fair way to read a journal from a tradition not your own look like?
  4. Why might it be wise to read reviews of an important book before relying on it?

Independent assignment

Tracing a journal article

In your own notebook, write about 300 words. Find a journal article on a subject from this program. Find the journal’s page that explains its review process, and find at least one book review or later article that discusses the article or its author. Record what you learn about the journal’s publisher and aims. Then write a short judgment of how far you would rely on the article and for what purpose. Do not send this to anyone.

Further study