Tracking AI Advisory Drift Before It Reaches the Farms: Measuring Contextual Reliability in LLM-Generated Advice Across the Agricultural Landscape

Abstract

Agricultural advice intended to protect the land is sensitive to local context because a best management practice (BMP) designed for one location might be risky elsewhere. This risk is addressed in the U.S. by Cooperative Extension experts stationed in different geographic regions, who can benefit from the ability of general-purpose large language models (LLMs) to produce plausible answers quickly at a low marginal cost; however, these answers must remain locally reliable. We evaluated local reliability by examining whether advisory behaviors in LLM responses repeatedly conformed to relevant BMPs. We analyzed this conformance in 288 LLM responses to eight publicly available Cooperative Extension inquiries. These inquiries were submitted by people in different locations who were concerned that local water could harm plants. Because the locations differed in plant types, water composition, soils, and how the inquiry was framed, responses from twelve LLMs at three low-temperature settings allowed us to examine the potential for unnecessary risk. We analyzed this risk using descriptive text metrics and an inductively developed advisory scorecard to identify BMP nonconformances, including unsupported claims, omissions, uncorrected naïve question framing, and contextually misframed solutions. To characterize the risks in context-sensitive advice, we introduce contextual advisory reliability as a scorecard-bound diagnostic metric for assessing whether LLM-generated land management advice consistently protects against risk. Risk is screened by evaluating repeated responses to calculate contextual advisory reliability. The ability to quantify reliability across different geographical regions supports responsible deployment in public advisory domains by helping institutions audit the suitability of AI-generated advice and communicate those results to institutional experts, land managers, and LLM designers.