Towards Fairer AI: Multi-Agent Debiasing of LLMs With Online Evidence Retrieval

Abstract

Large Language Models (LLMs) routinely reproduce the social biases embedded in their training data. Existing mitigation techniques such as data augmentation, RLHF, and post hoc filtering often blunt model capabilities or overlook biased reasoning steps. We introduce MADERA (Multi-Agent Debiasing with External Retrieval and Assessment), a self-contained multi-agent framework that (i) diagnoses biased chains of thought, (ii) retrieves relevant web evidence through a search agent, and (iii) iteratively rewrites reasoning until bias is eliminated. We evaluate MADERA on the BBQ–Hard benchmark with four backbones LLMs: DeepSeek-R1, GPT-3.5-Turbo, GPT-4, and Claude-3 Haiku. Across ambiguous prompts it lifts accuracy by an average of +8 percentage points and cuts directional bias by −0.08, with GPT-4 showing the largest gain (0.71 → 0.96 ACC; −0.29 → −0.04 BIAS). Across disambiguated prompts, where models already perform near ceiling, the search agent produces only marginal changes in accuracy and bias. These findings confirm that external web grounding is a key driver of reasoning-level debiasing.