Efficient Continuous Control with Double Actors and Regularized Critics


  • Jiafei Lyu Tsinghua Unversity
  • Xiaoteng Ma Tsinghua University
  • Jiangpeng Yan Tsinghua University
  • Xiu Li Tsinghua University




Machine Learning (ML), Planning, Routing, And Scheduling (PRS)


How to obtain good value estimation is a critical problem in Reinforcement Learning (RL). Current value estimation methods in continuous control, such as DDPG and TD3, suffer from unnecessary over- or under- estimation. In this paper, we explore the potential of double actors, which has been neglected for a long time, for better value estimation in the continuous setting. First, we interestingly find that double actors improve the exploration ability of the agent. Next, we uncover the bias alleviation property of double actors in handling overestimation with single critic, and underestimation with double critics respectively. Finally, to mitigate the potentially pessimistic value estimate in double critics, we propose to regularize the critics under double actors architecture. Together, we present Double Actors Regularized Critics (DARC) algorithm. Extensive experiments on challenging continuous control benchmarks, MuJoCo and PyBullet, show that DARC significantly outperforms current baselines with higher average return and better sample efficiency.




How to Cite

Lyu, J., Ma, X., Yan, J., & Li, X. (2022). Efficient Continuous Control with Double Actors and Regularized Critics. Proceedings of the AAAI Conference on Artificial Intelligence, 36(7), 7655-7663. https://doi.org/10.1609/aaai.v36i7.20732



AAAI Technical Track on Machine Learning II