Planning with Hidden Parameter Polynomial MDPs
DOI:
https://doi.org/10.1609/aaai.v37i10.26411Keywords:
PRS: Planning With Markov Models (MDPs, POMDPs), PRS: Model-Based Reasoning, PRS: Planning Under Uncertainty, RU: Sequential Decision Making, RU: Stochastic Models & Probabilistic InferenceAbstract
For many applications of Markov Decision Processes (MDPs), the transition function cannot be specified exactly. Bayes-Adaptive MDPs (BAMDPs) extend MDPs to consider transition probabilities governed by latent parameters. To act optimally in BAMDPs, one must maintain a belief distribution over the latent parameters. Typically, this distribution is described by a set of sample (particle) MDPs, and associated weights which represent the likelihood of a sample MDP being the true underlying MDP. However, as the number of dimensions of the latent parameter space increases, the number of sample MDPs required to sufficiently represent the belief distribution grows exponentially. Thus, maintaining an accurate belief in the form of a set of sample MDPs over complex latent spaces is computationally intensive, which in turn affects the performance of planning for these models. In this paper, we propose an alternative approach for maintaining the belief over the latent parameters. We consider a class of BAMDPs where the transition probabilities can be expressed in closed form as a polynomial of the latent parameters, and outline a method to maintain a closed-form belief distribution for the latent parameters which results in an accurate belief representation. Furthermore, the closed-form representation does away with the need to tune the number of sample MDPs required to represent the belief. We evaluate two domains and empirically show that the polynomial, closed-form, belief representation results in better plans than a sampling-based belief representation.Downloads
Published
2023-06-26
How to Cite
Costen, C., Rigter, M., Lacerda, B., & Hawes, N. (2023). Planning with Hidden Parameter Polynomial MDPs. Proceedings of the AAAI Conference on Artificial Intelligence, 37(10), 11963-11971. https://doi.org/10.1609/aaai.v37i10.26411
Issue
Section
AAAI Technical Track on Planning, Routing, and Scheduling