The AI Research Assistant That Cites Its Sources — and Actually Means It

Most medical AI systems hallucinate citations. DeepER-Med retrieves references directly from PubMed and ClinicalTrials.gov, making every claim traceable to its source.

~6 min read · NIH / National Library of Medicine · April 21, 2026 · Agentic

TL;DR DeepER-Med is an agentic AI framework from NIH that structures medical literature research as an explicit, auditable workflow — decomposing questions, retrieving evidence from verified databases, and synthesizing grounded answers. Evaluated against OpenAI Deep Research, OpenEvidence, and Google AI Mode on 100 expert-level clinical questions, it outperformed all three on reference relevance, analytical quality, and accuracy. In 7 of 8 real oncology tumor board cases, clinicians judged its conclusions consistent with expert recommendations.

There's a dirty secret in AI-assisted medical research: the systems doctors are increasingly relying on frequently fabricate the citations that are supposed to justify their conclusions. A hallucinated PMID looks identical to a real one in a generated report. For a clinician deciding on a treatment, that difference is not trivial.

DeepER-Med, developed by a team spanning NIH, Johns Hopkins, and the University of Illinois, is built around one foundational principle: no reference gets into the output unless it was retrieved from an actual database. Period.

💡 The Core Idea Instead of letting a language model generate citations from memory (where hallucination is endemic), DeepER-Med queries PubMed, ClinicalTrials.gov, and other source APIs directly. Think of it as the difference between asking someone to recall a book from memory versus handing you the physical book off the shelf. The evidence is either there or it isn't.

How the Framework Works

DeepER-Med structures what it calls "evidence-based generation" (EBG) as a three-stage pipeline — an explicit alternative to the typical agent loop that iterates through queries until it has enough text to fill a report.

flowchart LR
    Q[Research Question] --> RP[Research Planning\nSub-question decomposition]
    RP --> AC[Agentic Collaboration\n13 APIs across 3 layers]
    AC --> KG[Knowledge Graph\nQuery Expansion]
    AC --> LR[Literature Retrieval\nPubMed / ClinicalTrials.gov]
    KG --> EA[Evidence Appraisal\nRelevance + Quality Screening]
    LR --> EA
    EA --> ES[Evidence Synthesis\nTraceable Report + References]

Stage 1 — Research Planning: Given a question, GPT-4o decomposes it into a hierarchy of sub-questions, moving from foundational concepts toward higher-order relationships. For a question like "What are the earliest imaging features of macular telangiectasia type 2?", the system first asks about relevant imaging modalities, then modality-specific findings, then infers early disease markers. This breadth-first decomposition is the key mechanism for expanding the evidence base without losing coherence.

Stage 2 — Agentic Collaboration: This is the most architecturally novel part. Thirteen APIs are organized into a three-layer hierarchy: a worker layer of individual tools (PubMed abstract retrieval, citation tracking, knowledge graph traversal, etc.); a manager layer that groups them into functional communities (query expansion, evidence retrieval, LLM response); and a director layer that applies explicit logical rules to decide which community handles each sub-question. If relational reasoning is needed, the knowledge graph is triggered first. If not, the system goes directly to literature search.

Stage 3 — Evidence Synthesis: Retrieved evidence is re-interpreted by GPT-4o against the user's specific constraints, then Gemini-3-Pro generates the final report. Critically, only references retrieved via API calls appear in the output — LLM-generated text is retained only when subsequently confirmed by database-retrieved sources.

The Benchmark: 100 Real Research Questions

Evaluating deep research systems is hard because most existing benchmarks test multiple-choice accuracy, which an LLM can ace largely from training data without touching a single reference. The authors address this by building DeepER-MedQA: 100 expert-level questions contributed by 11 biomedical researchers across NIH, Johns Hopkins, and UIUC.

The questions span 11 disease areas including melanoma, glioblastoma, and age-related macular degeneration. Fifty-four percent address basic biological mechanisms, 22% translational questions, and 24% clinical research — a realistic distribution for frontline biomedical inquiry.

Evaluation was blind: experts scored responses from all four systems without knowing which system produced which answer, across five dimensions: accuracy, analytical quality, reference relevance, novelty of insight, and comprehensiveness.

Results: Where DeepER-Med Wins

The largest advantages appeared in two areas that matter most for clinical trust: reference relevance (81 cases favoring DeepER-Med vs. 59 for the strongest baseline) and analytical quality (67 vs. 47). On accuracy, DeepER-Med produced highly accurate responses in 77 cases versus 69 for the best comparator.

When experts were asked to select the single best response per question, DeepER-Med was chosen in 60 cases, including 27 where it was the sole selection. The strongest baseline was chosen in 43 cases with 19 sole selections. OpenEvidence and Google AI Mode lagged further behind.

Perhaps the most practically significant finding: in 27 cases, DeepER-Med identified connections in the literature that reviewers had not previously recognized — grounded novel insights, not speculation.

Qualitative analysis of lower-quality responses from competing systems revealed a consistent pattern: retrieved information was "reformulated without being integrated into coherent analytical reasoning." In some cases, competing systems generated fabricated citations.

Mechanistic Validation

Beyond head-to-head comparison, the authors evaluated DeepER-Med's individual workflow stages on five public biomedical datasets.

On intent identification, sub-question decomposition improved accuracy by nearly 5 percentage points over using context alone on PubMedQA (79.2% with both, vs. 74.4% without sub-questions).

On literature retrieval, semantic similarity between retrieved literature and expert-curated reference publications exceeded 70% across all tested biomedical domains. Embedding visualizations showed that DeepER-Med's retrieved literature occupied a broader semantic region than the reference set while remaining within the same topic clusters — controlled expansion, not indiscriminate recall.

For hypothesis verification, retrieved literature covered 96.5% of ground-truth source studies, and 91.3% of true hypotheses were correctly verified.

On BioMaze open-ended QA, DeepER-Med achieved 90.6% accuracy. On PubMedQA, it reached 79.2%, comparable to MedReason (79.4%) and Flan-PaLM (79.0%), with a simpler inference pipeline than either.

The Clinical Reality Check

Eight oncology cases were drawn from actual Precision Oncology Tumor Board (POTB) discussions at the University of Illinois Cancer Center — real patients, real genomic variants, real clinical uncertainties. Three clinicians reviewed DeepER-Med's outputs and compared them against the original tumor board recommendations.

In seven of eight cases, DeepER-Med's conclusions were judged consistent with expert recommendations. The one discordant case involved "a clinically nuanced scenario with evolving evidence." Evidence reliability was judged fully satisfactory in five cases and partially satisfactory in the remaining three (insufficient coverage of prior studies), though even two of the partially reliable cases reached the right clinical conclusion.

⚠️ Watch out for

• DeepER-Med occasionally assembles multiple relevant studies without establishing biological relationships between them — parallel summaries rather than integrated explanations
• Performance can be skewed by highly cited older studies overshadowing more recent findings — a known hazard in fast-moving clinical fields
• The framework depends on the availability and quality of upstream biomedical resources, which vary across disease areas
• Processing complex queries takes substantial time, comparable to OpenAI Deep Research
• DeepER-MedQA contains 100 questions — large enough for expert evaluation but too small to cover the full breadth of biomedical research

Why It Matters

The citation hallucination problem in medical AI is not a minor inconvenience. A fabricated PMID in an oncology report can point a clinician toward a treatment approach that has no evidentiary basis — or worse, one that is explicitly contraindicated by the literature the system claims to be citing. Most existing systems treat this as an acceptable error rate. DeepER-Med treats it as a design constraint.

The broader architectural lesson here may be more important than the benchmark numbers: the authors find that modeling research intent explicitly and controlling evidence expansion contributed more to robust biomedical reasoning than complex prompt engineering. The hierarchy of sub-questions, the three-layer agentic network, the explicit inclusion criteria — these structural choices, not model scale or elaborate prompting, are what drove the quality gains.

For AI practitioners building research tools in regulated or high-stakes domains: the path forward appears to be less LLM autonomy in citation generation, not more.

Go Deeper

• MedCPT: Contrastive Pre-trained Transformers with Large-Scale PubMed Search Logs — The embedding model used for semantic similarity evaluation throughout DeepER-Med's mechanistic analysis
• DeepEvidence: Empowering Biomedical Discovery with Deep Knowledge Graph Research — A closely related system from the same research community, useful for comparing knowledge-graph-centric approaches
• Can 'Deep Research' agents perform systematic review and meta-analysis? — A direct assessment of current deep research systems in ophthalmology that contextualizes where the field stands

Source: DeepER-Med: Advancing Deep Evidence-Based Research in Medicine Through Agentic AI Authors: Zhizheng Wang, Chih-Hsuan Wei, Joey Chan, Robert Leaman, Chi-Ping Day, Chuan Wu, Mark A Knepper, Antolin Serrano Farias, Jordina Rincon-Torroella, Hasan Slika, Betty Tyler, Ryan Huu-Tuan Nguyen, Asmita Indurkar, Mélanie Hébert, Shubo Tian, Lauren He, Noor Naffakh, Aseem Aseem, Nicholas Wan, Emily Y Chew, Tiarnan D L Keenan, Zhiyong Lu Published: 2026-04-21 PDF: https://arxiv.org/pdf/2604.15456