An Interactive Simulation Framework by Ensemble Imitation Learning Agents for Training Robust Trading Policies

Abstract

The reliable deployment of reinforcement learning (RL) for real-world algorithmic trading is critically hindered by the ``simulation-to-reality gap.'' Standard industry backtesting on static historical data ignores market impact—the feedback loop where an agent's trades influence price dynamics—leading to strategies that are fragile and untrustworthy in live markets. To solve this significant problem, we present a novel and emerging application of AI: a framework for building an interactive, responsive market simulator. Our system first uses imitation learning (IL) to automatically train an ensemble of agents, each learning a distinct trading strategy from a different historical market regime (e.g., bull, bear). This creates a data-driven proxy for a diverse population of real-world traders. We then deploy an innovative Action Synthesis Network to synthesize the actions of this ensemble, generating a realistic, synthetic price trajectory that endogenously models the market's reaction to trades. This interactive environment is then used to train a final RL policy. We evaluate our system on NASDAQ-100 (QQQ) data, and the results demonstrate strong potential for deployment. The RL policy trained in our responsive simulator achieves significantly more robust performance, exhibiting superior downside protection during market downturns compared to various traditional baselines. This application provides a scalable and technically sound methodology for building more realistic training environments, presenting a clear path toward the development and eventual deployment of more resilient and effective algorithmic trading strategies.