RESEARCH / BLOG

Research,
Explained.

Read the research question, follow the authors’ methods and explore the evidence behind the article.

Research Blog

Research methodology · Research insights

Compare Research Papers With AI: Build a Defensible Evidence Matrix

Compare research papers using verified methods, outcomes and limitations. Build an AI-assisted evidence matrix without turning missing information into a novelty claim.

By NPT Research7 min read
The novelty toolkit · S207 / 09
Make every comparison traceable.A clear matrix reveals uncertainty as well as agreement.
NPT Research

The Novelty Toolkit · Series 2 · Part 7 of 9

Five research papers lie open on your desk. All describe 'sensor accuracy', but one measured laboratory precision, another evaluated outdoor drift, and a third compared instruments after recalibration. Their abstracts sound similar; their evidence is not. A useful AI assistant can help arrange the information, but only the researcher can decide whether the studies are comparable and what they actually establish.

Think of a comparison matrix as a set of transparent glass drawers. Each piece of evidence goes in a labelled place. If a drawer is empty, you can see the absence without inventing its contents.

Select papers before asking AI to summarize

Begin with a human-reviewed set of papers: landmark studies, the closest methodological competitor, recent follow-ups and any credible contradictory findings. Record full citation details and where the relevant passages appear. An AI-generated list of papers should not be the starting dataset unless every item is verified.

For the running example, select studies that report temperature behaviour, ageing, reference-measurement procedures, recalibration and field duration. If a paper does not address those questions, that may be a reason to exclude it from the close comparison—not proof that the entire field has a gap.

Create a fixed extraction schema

Before summarization, define the columns. A practical set is: citation; system or participants; research question; design; sample or number of devices; operating conditions; reference/comparator; key findings; uncertainty; stated limitations; evidence location (page, table or figure).

A crucial extra column is 'not reported / cannot determine'. AI systems often fill gaps with plausible-sounding assumptions. If the source does not report an item, the cell must stay uncertain.

Example prompt

You are an evidence-extraction assistant, not a novelty judge. Use only the full text I supply. Extract: study question, design, sample, conditions, comparison method, key results, limitations and page/section evidence. Quote no more than a short necessary phrase per claim. If information is missing, write 'not reported'. If the paper is outside my scope, say why. Do not infer results from the title or abstract alone.

The novelty toolkit · Series 2Extract first. Compare only after checking.The quality of a matrix depends on the evidence in each cell.
  1. 01
    SelectChoose relevant, verified papers.
  2. 02
    DefineSet extraction columns before summarising.
  3. 03
    ExtractRecord methods, findings and evidence locations.
  4. 04
    VerifyCheck each important cell against the full text.
  5. 05
    CompareSeparate comparable findings from unresolved differences.
A learning framework, not a measured result. Verify evidence before making a research claim.
Figure 1. Evidence moves from selected sources through human-checked extraction to a comparison.

Verify a small sample before scaling up

Start with a single paper. Compare the AI's extracted design, numbers and limitations with the original document. Check especially whether percentages refer to the right denominators, whether a result was statistically significant, and whether limitations come from the paper or AI speculation.

If errors appear, improve the instructions, shorten the source sections, or extract manually. Do not turn a flawed extraction procedure into a large batch merely because automation makes it fast.

Build the research-gap matrix

A simple illustrative matrix can be written in words:

  • Study A: laboratory temperature effects measured; field ageing not assessed.
  • Study B: several months of field observations; no independent reference for every time point.
  • Study C: recalibration method compared; held-out evaluation not reported.
  • Proposed study: combined temperature and calibration-age assessment with a predefined external reference and held-out validation.

These are fictional teaching summaries, not statements about actual publications. They demonstrate how to expose overlap and limitations. In real work, every cell requires a verified citation or an explicit 'not assessed' label.

Notice that a matrix can reveal more than one possible gap. It can also reveal that your study is redundant. A good matrix is valuable precisely because it can challenge the idea you hoped to defend.

Separate three things AI is likely to confuse

A paper's limitation: a restriction explicitly acknowledged or demonstrated in that study. A researcher's inferred limitation: a defensible observation after reviewing design and evidence. A field-wide research gap: an uncertainty supported by synthesis across appropriate literature.

A paper saying 'our work did not test long-term conditions' does not prove no one has done so. Search later studies and neighboring methods before turning that sentence into a novelty claim.

The novelty toolkit · Series 2Four outcomes of an honest comparison.A blank cell is not automatically a field-wide research gap.
  1. 01EstablishedA bounded claim supported by a source.
  2. 02UnclearMissing detail or insufficient evidence.
  3. 03ContradictoryComparable findings that require further examination.
  4. 04Candidate gapAn uncertainty that needs wider literature checking.
A learning framework, not a measured result. Verify evidence before making a research claim.
Figure 2. A four-part matrix tracks known results, ambiguity, contradictions and candidate gaps.

Ask AI to compare claims, then ask it to argue against the gap

Example prompt

Using only the verified extraction table below, compare studies on their outcomes, operating conditions, comparators and limitations. Identify which findings can be compared directly and which cannot. List candidate uncertainties without calling them novel. Then argue how a relevant paper not in the table might overturn each candidate gap, and propose targeted follow-up searches.

The second half of this prompt creates an explicit opportunity to challenge the preferred story; it does not guarantee an unbiased comparison. It also creates a final search plan before committing to a proposed contribution.

A quality-control checklist

Before citing the matrix, check whether results refer to the same outcome, whether the comparison has sufficient methodological detail, whether negative findings were included, whether each absence is correctly labelled, and whether the latest relevant work has been considered. PRISMA-S offers transparent search-reporting guidance for systematic reviews (the PRISMA-S search-reporting guideline), while Cochrane emphasizes systematic and supplementary evidence identification (Cochrane’s study-search guidance). These principles support auditability even when the current task is a preliminary topic exploration.

Reader exercise: two papers, one contradiction

Choose two genuine papers with different reported findings. Build a six-column extraction table manually first. Use AI to draft a comparison. Highlight every AI sentence that goes beyond the evidence. Revise the matrix and formulate one cautious gap statement. If you cannot support it, write 'insufficient evidence to claim a gap'—a useful and scientifically respectable outcome.

A two-paper comparison without overclaiming

Imagine a study reporting low laboratory variability and another reporting poor long-term field agreement. A careless summary says 'the papers contradict each other.' But both findings could be true: repeatability under stable laboratory conditions is not the same construct as accuracy over months of changing environmental exposure. AI can highlight different terms, yet the researcher must identify the underlying estimand and measurement context.

Try the reverse: two studies report opposite temperature effects. Before calling the results contradictory, inspect the sensor type, temperature range, reference method, sample medium and statistical model. The apparent discrepancy may disappear once like is compared with like—or it may become a genuine and interesting scientific puzzle.

Three ways to report a matrix honestly

Convergence: 'Several studies report similar findings under comparable conditions.' Heterogeneity: 'Findings differ, but the studies vary in methods or populations, limiting direct comparison.' Unresolved contradiction: 'Comparable studies report materially different findings despite broadly similar methods, warranting targeted investigation.' Each statement calls for different next steps.

A gap matrix should therefore include both the similarity of the question and the comparability of the evidence. A clever coloured table without this distinction can make a weak claim look stronger than it is.

The blind-paper challenge

Hide study titles and author names. Give a colleague only methods and results from two verified papers. Ask what questions they answer and which claims remain uncertain. Then compare the answers with your AI-generated matrix. This exercise reveals how much of your interpretation comes from reputation, attractive abstracts or AI phrasing rather than from the underlying design.

What to remember

AI is allowed to organize evidence; it is not allowed to manufacture what the evidence does not contain. A novelty matrix is strongest when it reveals the closest prior work, including inconvenient results. In the next tutorial, we ask AI to attack the reasoning behind the proposed study itself.

Methodological context and limits

Messeri and Crockett’s Nature Perspective explains why convincing AI output can coexist with limited understanding. The extraction and comparison exercises here are teaching methods, not a validated automated systematic-review pipeline.

Continue the research novelty series

Series 1: originality, evidence and scientific judgment

Begin with what makes research novel, then find the closest prior research and build a novelty matrix. Before using an assistant, examine the limits of AI novelty judgments and why the researcher owns the question. Develop the conversation with scientific prompting and six-round brainstorming, then challenge the idea against evidence. The complete worked research dialogue connects those decisions.

Series 2: from a problem to a defensible proposal

First separate a research gap from a research problem and identify a genuine problem worth investigating. Next search published literature for the gap and align the question, objectives and hypothesis. For AI-assisted work, clarify the problem without outsourcing judgment, verify the literature search and its references and compare papers in an evidence matrix. Then stress-test the research hypothesis before following the complete observation-to-proposal workflow.

Sources and further reading

  1. Rethlefsen et al. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Systematic Reviews (2021).
  2. Cochrane Handbook for Systematic Reviews of Interventions. Chapter 4: Searching for and selecting studies.
  3. Messeri & Crockett. Artificial intelligence and illusions of understanding in scientific research. Nature 627, 49–58 (2024), Perspective. Published 6 Mar 2024