Deep Reinforcement Learning for a Multi-Objective Online Order Batching Problem


  • Martijn Beeks Eindhoven University of Technology
  • Reza Refaei Afshar Eindhoven University of Technology
  • Yingqian Zhang Eindhoven University of Technology
  • Remco Dijkman Eindhoven University of Technology
  • Claudy van Dorst Vanderlande Industries, Veghel, Netherlands
  • Stijn de Looijer Vanderlande Industries, Veghel, Netherlands



Deep Reinforcement Learning, Order Batching Problem, Explainable AI Planning, Bayesian Optimization


On-time delivery and low service costs are two important performance metrics in warehousing operations. This paper proposes a Deep Reinforcement Learning (DRL) based approach to solve the online Order Batching and Sequence Problem (OBSP) to optimize these two objectives. To learn how to balance the trade-off between two objectives, we introduce a Bayesian optimization framework to shape the reward function of the DRL agent, such that the influences of learning to these objectives are adjusted to different environments. We compare our approach with several heuristics using problem instances of real-world size where thousands of orders arrive dynamically per hour. We show the Proximal Policy Optimization (PPO) algorithm with Bayesian optimization outperforms the heuristics in all tested scenarios on both objectives. In addition, it finds different weights for the components in the reward function in different scenarios, indicating its capability of learning how to set the importance of two objectives under different environments. We also provide policy analysis on the learned DRL agent, where a decision tree is used to infer decision rules to enable the interpretability of the DRL approach.




How to Cite

Beeks, M., Refaei Afshar, R., Zhang, Y., Dijkman, R., van Dorst, C., & de Looijer, S. (2022). Deep Reinforcement Learning for a Multi-Objective Online Order Batching Problem. Proceedings of the International Conference on Automated Planning and Scheduling, 32(1), 435-443.