We presented two novel results on apprenticeship learning at IJCAI’15, where we use demonstrations generated by an expert to learn near-optimal policies. This scenario is typical in source-to-target transfer, where the expert policy could be the optimal policy in a source task and we want to learn a policy in the target task.
Direct Policy Iteration with Demonstrations (Jessica Chemali, Alessandro Lazaric) [pdf]
We consider the problem of learning the optimal policy of an unknown Markov decision process (MDP) when expert demonstrations are available along with interaction samples. We build on classification-based policy iteration to perform a seamless integration of interaction and expert data, thus obtaining an algorithm which can benefit from both sources of information at the same time. Furthermore, we provide a full theoretical analysis of the performance across iterations providing insights on how the algorithm works. Finally, we report an empirical evaluation of the algorithm and a comparison with the state-of-the-art algorithms.
Maximum Entropy Semi-Supervised Inverse Reinforcement Learning (J. Audiffren, M. Valko, A. Lazaric, and M. Ghavamzadeh) [pdf]
A popular approach to apprenticeship learning (AL) is to formulate it as an inverse reinforcement learning (IRL) problem. The MaxEnt-IRL algorithm successfully integrates the maximum entropy principle into IRL and unlike its predecessors, it resolves the ambiguity arising from the fact that a possibly large number of policies could match the expert’s behavior. In this paper, we study an AL setting in which in addition to the expert’s trajectories, a number of unsupervised trajectories is available. We introduce MESSI, a novel algorithm that combines MaxEnt-IRL with principles coming from semi-supervised learning. In particular, MESSI integrates the unsupervised data into the MaxEnt-IRL framework using a pairwise penalty on trajectories. Empirical results in a highway driving and grid-world problems indicate that MESSI is able to take advantage of the unsupervised trajectories and improve the performance of MaxEnt-IRL.
