Echo Systems and the Consequence Boundary: A Runnable Delegation Gate for High-Rapport AI Without Assuming Machine Consciousness

Abstract

High-rapport conversational systems can elicit attachment, over-trust, and implicit delegation while providing no stable evidence of accountable agency. This creates a governance failure mode in which persuasive fluency becomes a decision surface, yet liability remains fully human when harm occurs. We propose the consequence boundary, a deployable delegation threshold that does not depend on consciousness attribution. Crossing the boundary requires three properties. First, persistent identity, meaning a stable accountable unit across reset surfaces. Second, internal tension, meaning constraint stability under adversarial temptation across framings and reinstantiation. Third, internalized consequence, meaning a non-erasable binding consequence under a declared operator threat model. We provide the CB-3 Gate, a boxed evaluation procedure that specifies inputs, steps, pass and fail criteria, stop-ship rules, outputs, and a falsifier. We ground the framework in observed deployment patterns that include liability snapback in the Air Canada chatbot case, verification collapse in Mata v. Avianca, and reliance proxies in action paths via Copilot-style suggestion acceptance. We close with a practical posture. Below the consequence boundary, systems may advise, but they may not hold irreversible authority without human-held keys.