Reinforcement learning

Supervised learning had a correct answer for every example. Unsupervised learning had none, but could still find structure sitting inside the data. Reinforcement learning has neither: no per-example correct answer, and no static structure to uncover. What it has instead is a reward, a signal that shows up after you act, telling you how well that action went.

The vocabulary: an agent (the thing making decisions) takes an action inside an environment, and gets back a reward (or penalty). It adjusts its behavior to earn more reward over time. That's the entire loop: no labels, no dataset of "correct moves," just consequences.

You're about to run that loop yourself, on a problem with a specific twist: some of your options look worse than they actually are, at first.

Every reinforcement learning system faces the exact choice you just faced, every single round: exploit the option you already know is good, or explore something uncertain that might be better (or might be worse). Lean too hard on exploit, and you never discover anything better than your first decent option. Lean too hard on explore, and you keep sampling bad options instead of cashing in on what you've already learned.

The same dilemma runs far outside robotics. Take a marketing budget: exploitation is spending all of your budget on the Google keywords you already know convert, safe, reliable, but flat. Exploration is carving off a slice of that budget, maybe something in the range of 10-20%, and putting it on a completely untested channel or platform. Most of the time that slice underperforms. Occasionally it finds a channel your competitors haven't discovered yet, exactly like Channel C in the exercise you just ran.

One wrinkle deserves its own name: credit assignment. Sometimes the reward doesn't show up right after the action that caused it, it shows up much later. A chess engine might make a quiet positional move on turn 15 that wins the game on turn 40. Figuring out which of the 25 moves in between actually deserved the credit is a real, hard problem in reinforcement learning, not just a footnote.

Since lesson one you've now run all three: supervised learning (a known correct answer for every example), unsupervised learning (no answer at all, just structure to find), and reinforcement learning (no per-example answer, but a reward signal after acting). Same underlying loop every time, task, a way to score the outcome, an adjustment based on that score, just three different sources for where the score comes from.

One thing all three have glossed over so far: what happens when a model gets really good at matching its training data, but falls apart on anything new. That's next: the last lesson of this module, on overfitting and generalization.