What is policy gradient?

The Policy Gradient Theorem is central to many reinforcement learning algorithms that optimize policies directly. In contrast to value-based methods, policy gradient methods optimize the policy itself without estimating value functions.

Mathematically, the policy gradient is expressed as: $\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[ \sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t | s_t) R(\tau) ]$ where $\theta$ represents the policy parameters, $\tau$ is a trajectory, and $R(\tau)$ is the return from the trajectory.

The REINFORCE algorithm uses this theorem by sampling trajectories and updating the policy parameters using the gradient estimate: $\theta = \theta + \alpha \sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t | s_t) R_t$ where $R_t$ is the return from timestep $t$ onward and $\alpha$ is the learning rate.

Practical Applications: REINFORCE is used in environments where the policy must be learned from scratch, such as game playing or robotic control.

Mermaid Diagram of REINFORCE Algorithm Workflow:

graph TD
A[Start] --> B[Initialize Policy Parameters]
B --> C[Sample Trajectory]
C --> D[Compute Returns]
D --> E[Compute Policy Gradient]
E --> F[Update Policy Parameters]
F --> G[Converged?]
G -->|Yes| H[End]
G -->|No| C

Further Reading:

• Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.).
• Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4), 229-256.
• Spinning Up in Deep RL by OpenAI provides a practical introduction to policy gradient methods, including REINFORCE.