PPO Reinforcement Learning from Scratch

PPO implemented in pure NumPy, put to work tuning a real classifier.

Reinforcement LearningPPONumPyPolicy GradientsFrom scratch

Output

Training dynamics: episode reward climbs to a stable optimum, the learned threshold converges below the 0.5 default, F1 rises, and the before/after panel shows PPO lifting F1 from 0.52 to 0.58.

PPO vs grid search: the agent’s learned threshold (red) lands right on the F1 peak next to the exhaustive grid-search optimum (green), far from the suboptimal 0.5 default (grey).

Greedy rollout of the trained policy: the agent takes two coarse −0.10 steps to reach the good region, then fine −0.01 steps to settle on the optimal plateau, a learned coarse-to-fine search.

Problem

RL libraries hide the internals. Can you implement PPO end-to-end (policy gradients, GAE, the clipped surrogate) and point it at a problem that actually matters, like the decision threshold of an imbalanced classifier where the default 0.5 is badly suboptimal?

Approach

A from-scratch PPO agent in NumPy only: actor and critic MLPs with tanh activations and Xavier initialization, the PPO-clip surrogate objective, and Generalized Advantage Estimation, with backpropagation written by hand. It is applied to a threshold-optimization environment: given a trained decision tree on an imbalanced dataset, the agent adjusts the classification threshold step by step (state = current threshold + metrics, reward = F1 improvement over the 0.5 baseline) to maximize F1.

Result

On an imbalanced dataset where the default 0.5 threshold scores F1 0.52, the agent autonomously learns a threshold of ~0.18 that lifts F1 to 0.58 (+11%), matching an exhaustive grid search purely from reward feedback. The training curve shows reward climbing to a stable optimum, and the greedy rollout reveals a learned coarse-to-fine search.

Highlights

PPO-clip surrogate objective + GAE, all hand-written in NumPy
Actor–critic MLPs with backprop written by hand; no Gym, PyTorch, or Stable-Baselines
Applied to a real problem: threshold tuning for an imbalanced classifier
Matches grid-search optimum from reward alone (+11% F1 over the 0.5 default)

Next projectClinical RAG Assistant →