Indexed in

License and use

Citations

Altmetrics

Analysis of institutional authors

Share

MATRIX LOW-RANK TRUST REGION POLICY OPTIMIZATION

Publicated to:2023 Ieee 9th International Workshop On Computational Advances In Multi-Sensor Adaptive Processing, Camsap. 406-410 - 2023-01-01 (), DOI: 10.1109/CAMSAP58249.2023.10403480

Authors: Rozada, Sergio; Marques, Antonio G

Affiliations

Abstract

Most methods in reinforcement learning use a Policy Gradient (PG) approach to learn a parametric stochastic policy that maps states to actions. The standard approach is to implement such a mapping via a neural network (NN) whose parameters are optimized using stochastic gradient descent. However, PG methods are prone to large policy updates that can render learning inefficient. Trust region algorithms, like Trust Region Policy Optimization (TRPO), constrain the policy update step, ensuring monotonic improvements. This paper introduces low-rank matrix-based models as an efficient alternative for estimating the parameters of TRPO algorithms. By gathering the stochastic policy's parameters into a matrix and applying matrixcompletion techniques, we promote and enforce low rank. Our numerical studies demonstrate that low-rank matrix-based policy models effectively reduce both computational and sample complexities compared to NN models, while maintaining comparable aggregated rewards.

Keywords