LTLf/LDLf Non-Markovian Rewards


  • Ronen Brafman Ben-Gurion University
  • Giuseppe De Giacomo Sapienza University of Rome
  • Fabio Patrizi Sapienza University of Rome


MDPs, non-Markovian Rewards, LTLf/LDLf


In Markov Decision Processes (MDPs), the reward obtained in a state is Markovian, i.e., depends on the last state and action. This dependency makes it difficult to reward more interesting long-term behaviors, such as always closing a door after it has been opened, or providing coffee only following a request. Extending MDPs to handle non-Markovian reward functions was the subject of two previous lines of work. Both use LTL variants to specify the reward function and then compile the new model back into a Markovian model. Building on recent progress in temporal logics over finite traces, we adopt LDLf for specifying non-Markovian rewards and provide an elegant automata construction for building a Markovian model, which extends that of previous work and offers strong minimality and compositionality guarantees.




How to Cite

Brafman, R., De Giacomo, G., & Patrizi, F. (2018). LTLf/LDLf Non-Markovian Rewards. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). Retrieved from



AAAI Technical Track: Knowledge Representation and Reasoning