The most interesting bio-agent path is the expensive one: live PubMed retrieval with
every LLM stage switched on. Planner, extractor, synthesis, advisory verifier, revision, and claim-logic
parser all get to help. None of them gets to be the final judge.
That is the design I want to show. The model is allowed into the chain at the places where language is genuinely hard: planning a biomedical search, extracting claims from abstracts, drafting a cited answer, parsing claim logic, checking the draft, and revising weak wording. The system then stores the modes, model names, prompt hashes, fallback reasons, trace steps, citation audit, and logic audit beside the answer.
The LLM chain is useful only because every LLM output becomes a typed, auditable artifact. If a stage fails, falls back, overclaims, or rewrites too aggressively, the trace should say so without asking a reviewer to trust the model's tone.
Use live PubMed as the source, but persist manifests and warnings before synthesis.
Let LLM stages plan, extract, draft, verify, revise, and parse logic frames.
Recheck the final claims with citation and deterministic logic audits.
The LLM can propose subquestions, but source policy and result caps still apply.
Evidence leaves the model as structured claims, entities, limitations, and spans.
The draft is parsed again instead of being trusted because it used a packet.
LLM-parsed frames feed deterministic entailment rules, not another vibes check.
The hard run starts with real literature
Mock literature is useful for tests, but the run I care about here uses live PubMed and a
real provider-backed LLM path. The request shape is intentionally explicit:
source="pubmed", use_llm_planner=true,
use_llm_extractor=true, use_llm_synthesis=true,
use_llm_verifier=true, use_llm_revision=true, and
use_llm_claim_logic=true. In the full live run I also turn on
execute_support_refute=true and export_logic_facts=true, because I
want the system to look for opposing evidence and leave symbolic audit facts behind.
That combination is where ordinary RAG starts to get slippery. There are many places for a model to overreach: it can broaden the search, extract a convenient claim, drop a limitation, draft too strongly, wave through its own answer, or revise away a caveat. So the live path treats each LLM call as a stage with metadata, not as one big hidden prompt.
PubMed search still produces normal retrieval artifacts: source, compiled query, API parameters, result counts, warnings, stored paper IDs, and returned paper IDs. The LLM does not get to smuggle search results into the answer by memory. It receives retrieved evidence through the service contract, and the run keeps enough trace data to replay what happened.
The dashboard screenshot shows the result of that design. The final answer can be accepted with caveats because "accepted" means the displayed claims survived the checks, not that the system is pretending the literature is complete or clinically actionable.
Live PubMed produces retrieval manifests, stored paper IDs, warnings, and abstract coverage before any answer is drafted.
Planner, extractor, synthesis, verifier, revision, and claim-logic parsing are enabled independently.
Each stage records mode, model, prompt hash, fallback reason, and audit metadata where it applies.
Each model call has a narrow job
The LLM planner does not answer the question. It decomposes the question into retrieval intents and subquestions. The LLM extractor does not summarize the literature. It converts retrieved papers into structured evidence items: claims, methods, entities, limitations, confidence, source scope, and spans. The LLM synthesizer does not get the whole search log. It drafts from the curated packet.
That separation is small, but it changes the failure mode. If retrieval is weak, the packet says so. If extraction falls back, the evidence item says so. If synthesis uses an unknown citation, the post-draft audit catches it. The model is useful at every stage, but each stage has less room to hide.
use_llm_planner Plans subquestions and retrieval intents, then records planner mode and validation.
use_llm_extractor Converts PubMed records into structured evidence, not free-form summary text.
use_llm_synthesis Drafts from packet evidence and must keep supplied citation labels intact.
use_llm_verifier Produces advisory disagreements, never the final audit verdict.
use_llm_revision Repairs unsupported or overstrong wording, then triggers another audit.
use_llm_claim_logic Parses claim and evidence frames for deterministic entailment checks.
The verifier and revision stages are intentionally advisory. The verifier can point out a mismatch between answer and evidence. The revision stage can soften, remove, limit, abstain, or refuse. But if the revised answer changes, the system audits the revised answer again. A nicer rewrite does not get a free pass.
The draft is parsed again after the model writes it
The citation audit path is blunt on purpose. It extracts atomic claims from the answer, reads the cited paper IDs, compares them with the evidence packet, and returns verdicts: supported, partial support, contradicted, insufficient evidence, irrelevant citation, or not cited. If a sentence says something biomedical, it has to carry a citation that can survive that check.
The important part is timing. The model can draft from an evidence packet, but the draft is not trusted because it used the packet. The post-draft audit re-parses the answer, checks the cited IDs, and stores a separate audit object. Retrieval produces evidence, synthesis produces text, audit decides what the text is allowed to claim.
The audit records claim support rate, citation precision, packet citation utilization, unsupported claim rate, overclaim rate, uncertainty calibration, failed claims, and a recommended action. That is why the dashboard can show "claim citation support 100%" and still warn that packet limitations are high. Those metrics answer different questions.
I like this separation because it kills a common trick in generated biomedical writing. The model can cite a real paper and still make a claim the paper does not support. The audit does not reward the presence of brackets. It asks whether the cited evidence actually grounds the claim on the page.
The same citation audit runs behind the dashboard, the service layer, tests, and release evaluation. It is not a paragraph of prompt instructions. It is a reusable contract.
The LLM parses logic, deterministic rules judge it
Citation audit catches a lot, but biomedical overclaiming often hides in the verb. A paper reports an association; the answer says it causes disease. A mouse result becomes a human claim. A mechanism becomes a treatment recommendation. A moderate trial result turns into a universal no-effect statement. Good lexical overlap will not catch those mistakes.
The claim-logic LLM stage is allowed to parse messy biomedical language into frames: predicate, polarity, modality, population, study design, scope terms, qualifiers, hedging, and source spans. Then the frame has to pass schema validation before deterministic rules run. Parser mode, model, prompt hash, parser warnings, and fallback reason stay attached to the audit trace.
That split is the clever bit. The LLM does semantic parsing, not final judging. Association does not entail causation. Animal and in-vitro evidence do not by themselves entail a human claim. Mechanistic evidence does not entail treatment. Inconclusive evidence cannot support an unhedged positive claim. Suggestive evidence does not support definitive wording. A biomarker association does not establish diagnostic utility.
- Parser layer: turn answer claims and evidence items into validated logical frames.
- Rule layer: run deterministic no-entailment checks over predicate, population, modality, scope, and study design.
- Trace layer: persist parser mode, triggered rules, mismatch axes, entailment score, and fallback warnings.
- Second audit: rerun citation and logic checks after revision changes the answer.
The part I trust
I want the strongest path to use the model heavily. Live PubMed plus LLM planner, extractor, synthesis, verifier, revision, and claim-logic parsing is more interesting than a cautious deterministic demo.
But the trust comes from the shape of the chain. Each LLM stage has a narrow job. Each important output becomes structured state. Each generated claim is audited after the model writes it. If revision changes the answer, the answer is audited again.
That is the design I care about: not avoiding LLMs, and not letting them float above the workflow either. Use them where language is hard. Keep the final authority in the trace, the citation audit, and the deterministic logic rules.