Check and Trust with PAU: A Paper Analysis Utility for Automated Bibliographic Verification

Abstract

The correctness and appropriateness of scientific references is paramount for upholding the trust in and ethics of science. This standard has come under pressure with the advent of artificial intelligence (AI) and Large Language Models (LLMs), which facilitate academic writing and peerreviewing. Leading venues have become flooded with AI-enhanced manuscripts containing hallucinated references, endangering the general ethics and high standards of scientific practice. As researchers are under constant time pressure and cannot be expected to identify “HalluCitations” manually, automated bibliographic verification (ABV) becomes integral. Realizing the potentials of ABV, we introduce PAU (Paper Analysis Utility), an open-source tool for automating the validation of bibliographic information in scientific manuscripts. PAU extracts references from a given publication and compares them against major scholarly databases, while leaving the final judgment and thus agency to human reviewers. With hybrid verification, the tool detects and flags reference discrepancies both in terms of metadata validity (checked via pattern matching) and semantic alignment (leveraging LLMs). Both the design and performance of PAU are evaluated against researcher expectations and on real-world reference hallucinations. Our work contributes a socio-technical approach for establishing ABV in the research community to ensure scientific integrity and ethics in the age of AI-assisted academic publishing.