Goal-Space Planning, reproduced | Amir Ata Ghaffarian

Independent · 2026 · Code · Interactive report

A tabular reproduction of Goal-Space Planning with Subgoal Models (Lo et al., JMLR 2024) on the paper’s FourRooms domain. GSP plans over a small graph of subgoals instead of every state, then hands the resulting values to an ordinary TD learner as a potential-based shaping reward: learning gets faster, and the optimal policy cannot change. Using exact subgoal models isolates what planning and shaping do by themselves, a ceiling for learned models to be measured against.

Value iteration over 104 states next to goal-space planning over 5 subgoals

18 → 3value-iteration sweeps, 104 states vs. 5 subgoals

1 → 188state–action pairs updated by one episode, Sarsa(0) vs. GSP + Sarsa(λ)

35 → 18episodes to near-optimal, Sarsa(λ) vs. GSP

18 → 7episodes to re-route around new lava, Sarsa vs. GSP (consistent init.)

Learning curves for Sarsa with and without GSP — GSP halves the episodes Sarsa(λ) needs and rescues Sarsa(0), which is not near-optimal within 200 episodes on its own.

Highlights

One command regenerates all 8 experiments, their figures and the report in about 3 minutes on one CPU core; gsp check verifies 10 numerical invariants, and 34 tests cover the rest.
Measured rather than hypothetical: Dyna-Q with 30 planning updates per step still needs 2,732 environment steps for 50 episodes, while GSP needs 1,231 with 30 updates in total.
With 20% model noise, bootstrapping from the subgoal values collapses (670 steps per episode) while shaping degrades gracefully (27 steps).
Three open discrepancies with the paper are documented with candidate causes.