Echo Systems and the Consequence Boundary: A Runnable Delegation Gate for High-Rapport AI Without Assuming Machine Consciousness - LaPosta, Paul (Proceedings of the AAAI Symposium Series, 2026)
High-rapport conversational systems can elicit attachment,over-trust, and implicit delegation while providing no stableevidence of accountable agency. This creates a governancefailure mode in which persuasive fluency becomes a decisionsurface, yet liability remains fully human when harm occurs.We propose the consequence boundary, a deployable delegationthreshold that does not depend on consciousness attribution.Crossing the boundary requires three properties. First,persistent identity, meaning a stable accountable unit acrossreset surfaces. Second, internal tension, meaning constraintstability under adversarial temptation across framings andreinstantiation. Third, internalized consequence, meaning anon-erasable binding consequence under a declared operatorthreat model. We provide the CB-3 Gate, a boxed evaluationprocedure that specifies inputs, steps, pass and fail criteria,stop-ship rules, outputs, and a falsifier. We ground the frameworkin observed deployment patterns that include liabilitysnapback in the Air Canada chatbot case, verification collapsein Mata v. Avianca, and reliance proxies in action paths viaCopilot-style suggestion acceptance. We close with a practicalposture. Below the consequence boundary, systems mayadvise, but they may not hold irreversible authority withouthuman-held keys.