The recursive view
The Bellman optimality equation separates a decision into an immediate reward and the value of what follows:
That decomposition is useful whenever a long-horizon problem can be expressed as a sequence of smaller state transitions.
Why it matters for learned agents
In a learned system, the state is often a representation rather than a raw observation. The quality of planning therefore depends on both the transition model and the representation used to preserve information that matters for future decisions.
The practical lesson is simple: before tuning an optimizer, ask whether the state contains the variables that make the future predictable.
A working checklist
- Define the state and action spaces explicitly.
- Identify which information is lost by the representation.
- Compare one-step prediction quality with long-horizon planning quality.
A useful value function is a compressed account of consequences, not just a score attached to a state.