Bi-Level Preference Optimization for Retrieval-Augmented Generation (Student Abstract) - Cao, Sizhong (Proceedings of the AAAI Conference on Artificial Intelligence, 2026)
Retrieval-augmented generation (RAG) is the backbone ofknowledge-intensive NLP, yet its progress is hindered by along-standing asymmetry: Generators are refined whileretrievers remain static, and full end-to-end optimizationis prohibitively unstable. We present BPO-RAG, a bi-levelpreference-learning framework that redefines the trainingparadigm by jointly optimizing retrieval and generationwith a single supervision signal, pairwise preferences.Stage~1 (Retrieval Preference Optimization) learns toselect superior evidence sets, while Stage~2 (GenerationPreference Optimization) aligns answer generation with thesame evidence, closing the gap between what to read andwhat to write. This recipe without label requires no rewardmodel or online RL, integrates seamlessly with standard RAGpipelines, and transforms preferences into a unifyingtraining currency. Across open-domain QA benchmarks,BPO-RAG consistently advances retrieval quality and yieldsmore accurate, faithful answers, surpassing strong RAGbaselines with remarkable stability. By coupling retrievaland generation under a unified preference framework,BPO-RAG establishes a practical and principled path towardthe next generation of reliable, modular, and trustworthyknowledge-intensive language models.