Understanding Reward Shaping in Planning: A Theoretical and Empirical Analysis

Abstract

Reward design is a critical yet often underexplored component of reinforcement learning for planning problems. In this work we analyze how different reward shaping strategies affect learning performance when using Proximal Policy Optimization (PPO). We evaluate several reward formulations derived from the objective function while keeping the learning algorithm and model architecture fixed. Experiments on benchmarks from the International Planning Competition 2023 show that commonly used shaping strategies do not consistently outperform sparse rewards. While some formulations provide competitive performance, our results highlight the difficulty of designing effective reward signals for policy gradient methods in planning domains.