| Abstract: |
Large language models (LLMs) are increasingly used to draft scholarly text and to support evidence synthesis, but they often hallucinate bibliographic references that look legitimate. We contribute a structured evaluation framework for citation reliability under deployment constraints: a three-way label scheme (Existing/Unresolved/Fabricated), a deterministic scoring pipeline against Crossref and Semantic Scholar, human-audit calibration that characterizes the framework’s biases, and a released defensive tool (citecheck). Applying the framework to 144 claims across six domains, we evaluate two proprietary models (Claude Sonnet, GPT-4o) and two open-weight models (LLaMA 3.1–8B, Qwen 2.5–14B) across five regimes: Baseline, Temporal (publication-year window), Survey-style breadth, Non-Disclosure policy, and their combination. In the closed-book five-regime study, across 17,443 generated citations, no model exceeds a citation-level existence rate of 0.475; Temporal produces the steepest single-condition decline, and the stacked Combo condition yields the lowest overall verifiability, while outputs remain formatcompliant (well-formed bibliographic fields). Unresolved outcomes constitute a large share of outputs (36–61%); in a 200-citation stratified human audit, 44 of 87 pipeline-Unresolved citations (50.6%) were judged fabricated. To probe whether explicit grounding improves closed-book citation verification, we add a minimal retrieval-augmented baseline for Claude Sonnet that injects top-5 candidate records retrieved from Crossref (the DOI registration agency) into the same prompts; this raises Baseline existence from 0.381 to 0.699 (∆ = +0.318) and Temporal existence from 0.119 to 0.818 (∆ = +0.699). For these two cells, grounding substantially improves verifiability, though this minimal diagnostic baseline is not a full four-model RAG comparison. Because the framework has known label uncertainty, our conclusions rest on the relative pattern of effects across conditions and model families rather than on absolute fabrication rates. The pipeline is released alongside the paper to support post-hoc citation verification before LLM outputs enter scholarly literature, journalism, or policy briefings. |