Why a Coin Flip Beats This Learning Algorithm | Tanmay
A Humanitarians AI fellow works through the short-corridor puzzle from Sutton and Barto's reinforcement learning textbook. A fair coin scores -12 against Sarsa's -44, and REINFORCE learns the best bias of about 58.6 percent right.
Some problems have a best answer that is not any single choice but a random one. The video presents one such case from Sutton and Barto's textbook on reinforcement learning, a book about machines that learn by trial and error. Tanmay Kulkarni presents it for Humanitarians AI and reports his own runs.
The blindfolded corridor puzzle
The puzzle is four squares in a row. You start on the left, the goal is on the right, and every step costs a point. You have two buttons, left and right. The catch is that in the second square the buttons are swapped, and you are blindfolded: every square gives the same reading, and you cannot count your steps. Whatever you do in one square, you therefore do in all of them. The video's central observation is that any rule you could pick is really a coin, defined by how often it presses right, from never to always.
Why fixed rules fail
The two extremes are both disasters. Always pressing right moves you to square two, where the swap slides you back to square one, and the loop never ends, so the score never stops counting. Always pressing left walks you into the wall, and you stand there forever. No fixed button works.
Sarsa lands on minus 44
The video then hands the problem to Sarsa, a classic algorithm from the earlier chapters of the book. It learns one score per button, since every square reads the same, and mostly presses the higher-scoring one. One time in ten it presses a random button to keep exploring, which is what gets it out of the loops. The presenter ran it 30 times and all 30 runs settled on the same behavior: mostly right, worth about minus 44, the same number the book reports. The only other place it can land, mostly left, is worse at about minus 82. It is built to find a best button, and this corridor has none.
A fair coin scores minus 12
The next test was almost insulting: no learning at all, just a fair coin flipped at every step. That scores minus 12, which beats Sarsa by more than three and a half times, and it is the exact value rather than a lucky run. The reason is that the swapped square punishes commitment. Leaning hard on either button costs you through bouncing or getting stuck, and the coin never commits.
The best coin is biased
Is 50/50 the best possible coin? Plotting the score for every setting between never right and always right shows both ends never finish, the learner's two options sit far down, and the fair coin sits near the top but not at it. The best setting leans a little toward right: 2 minus the square root of 2, about 58.6 percent right, for a score of about minus 11.66. The book rounds it to 59 percent and leaves the exact form as an exercise. That form is irrational, so the best coin could never be minted exactly.
Learning the probabilities directly
Chapter 13 of the book introduces learners that do not score buttons at all. They learn the coin's bias directly, shifting the odds toward whatever the runs reward. The simplest is called REINFORCE. Starting from the worst setting, mostly left at minus 82, it averaged about minus 3 within 400 episodes in the video's run, and across 30 runs the average setting it settled on was 58.6 percent, so it found the irrational answer on its own. The book's point is that methods that score actions have no natural way of finding the best random policy, while methods that learn probabilities do.
When is randomness necessary?
The video closes with a test. Take off the blindfold and the same Sarsa learner, now keeping a separate score per square, learned right, left, right in all 30 runs for a score of minus 3. When the player knows exactly where it is, a fixed rule is always best, so the coin was covering for what it could not see. Randomness can also survive: in rock, paper, scissors against an opponent that learns your habits, always rock lost nearly every round and an even three-way split broke even, whereas against a random opponent always rock does as well as anything.
Key takeaways
- Any rule in the corridor is a coin, defined by how often it presses right.
- Sarsa settled on mostly right in all 30 runs, worth about minus 44.
- A fair coin scores minus 12, beating Sarsa by more than three and a half times.
- The best coin presses right about 58.6 percent of the time, for roughly minus 11.66.
- REINFORCE learns the bias directly and found that value on its own.
- If giving the player more information makes the randomness vanish, it was covering for hidden state.
- If the randomness survives, look for someone who would exploit a pattern.
Try it yourself
Apply the video's test to a goalkeeper facing a penalty and choosing which way to dive: does more information make the randomness disappear, or does it survive? A strong answer names the missing information or the person who would exploit the pattern. If you code, the video links the corridor script, so you can run it and watch the blindfold come off.
Chapters
Full transcript(auto-generated, with timestamps)
The Short-Corridor Blindfold Puzzle
[0:00]Here's a puzzle from a textbook. Four squares in a row. Start on the left, reach the goal on the right, and every step costs a point. Two buttons, left and right. Easy. Except in the second square, the buttons are swapped and you're blindfolded. Every square gives you the same reading and you can't keep count of your steps. So, whatever you do in one square, you do in all three. Which button do you press? Hi, this is
Why Fixed Rules & Sarsa Fail (-44 score)
[0:19]Tanme Kolc Carney in for humanitarians AI. This video is about one example from Sutton and Bardau's reinforcement learning, an introduction, a textbook about machines that learn by trial and error. Here's the key to the puzzle. Any rule you could pick is really a coin. How often it presses right from never to always. So, every answer gets a spot on this line and a score. We'll try them one at a time. By the end, you'll know
Why a Fair Coin Flip Beats Sarsa (-12 score)
[0:39]The best answer. It isn't a button. It's a biased coin set to one exact number. First answer, always press right, square one, right. You move to square two, square two, right? But it swapped, so you slide back to square one, right, square two, back again. You never reach the goal. Your score doesn't just get bad. It never stops counting.
Calculating the Optimal Biased Coin
[0:58]Always press left in the first square. Left walks you into the wall. You stand there forever. So fixed button works. Let's give the problem to a learning algorithm. A classic from the earlier chapters of the book called Sarsa. It learns a score for each button from experience. Just one score per button since every square reads the same. Then it mostly presses the higher one. One time in 10 it presses a random button
Action-Value Scoring vs. Policy Gradients (REINFORCE)
[1:18]Just to keep exploring. That's what gets it out of the loops. I ran it 30 times. All 30 settled on the same thing. Mostly right. and mostly right is worth about minus44. The book reports the same number. The only other place this learner can land mostly left is worse about minus 82. Those are its two options. It's built to find a best
Unobserved States vs. Adversarial Patterns (RPS & Penalty Kick Test)
[1:38]Button and in this corridor there isn't one. I tried something that felt almost insulting. No learning at all. Flip a fair coin every step. Heads right, tails left minus 12. A coin beat the learning algorithm by more than three and a half times. And that's not a lucky run. It's the exact value. Now, the coin isn't clever. It wins for a plain reason. The swapped square punishes commitment. Lean hard on either button and you pay for it bouncing or stuck. The coin doesn't commit at all, which raises a question. Is 50/50 actually the best you can do? So, let's bias the coin. At one end, it never says right. At the other, it always does. And here's the score for every setting in between. Both ends never finish. The learner's two options sit way down here. The fair coin is near the top, but it isn't the top. The best setting leans a little toward right. The best setting works out to
2 minus the<unk> of 2, about 58.6% right for a score of - 11.66. The book rounds it to 59% and leaves the exact form as an exercise for the reader. And that exact form is irrational. The best way through this corridor is a coin no one could ever mint exactly. This is what the book is really getting at. Almost everything before chapter 13 learns the way Sarsa did by scoring actions. Chapter 13 opens with, quote, "In this chapter, we consider something new. Learners that don't score buttons at all. They learn the coin's bias directly, shifting the odds toward whatever the runs reward. The simplest one is called reinforce. I started at the worst setting I had mostly left minus 82. Within 400 episodes, it was averaging about minus3. Across 30 runs, the average setting it settled on was 58.6%, it found 2 minus <unk>2 on its own." book says it plainly. Methods that score actions have no natural way of finding the
Best random policy. Methods that learn the probabilities can. But that leaves a fair question. Why did the best answer have to be random at all? Here's how to find out. Take off the blindfold. Same Sarsa learner as before, but now it keeps a separate score for each square. 30 runs out of 30. It learned right, left, right three steps minus three. Nearly four times better than the best coin. And the book says that's no accident. When the player knows exactly where it is, there's always a fixed rule that's best. The coin was covering for what it couldn't see. Randomness doesn't always disappear. The book's own example is poker, where the best play is often to bluff with a specific probability. Here's a simpler one. I ran rock paper scissors against an opponent that learns your habits. Nothing is hidden. Always rock lost essentially every round. Leaning toward rock still came out behind. Even three-way split broke even. Now swap in an opponent
That just plays at random. And always rock does as well as anything. That randomness is there because someone is watching for a pattern. So that's the test and you can carry it to other problems. When the best move is random, give the player more information. If the randomness disappears, it was covering for what the player couldn't see. If it survives, look for someone who'd exploit a pattern. The corridor is the first kind. Rock, paper, scissors is the second. Now try it on a goalkeeper facing a penalty, choosing which way to dive. Run the test. Does more information make the randomness disappear or does it survive? Put your answer in the comments. A strong answer names the missing information or the person who'd exploit the pattern. Saying it's random because it's uncertain doesn't count. And if you code, the corridor script is linked below. Run it and watch the blindfold come off to minus route two. Tanme Kulcarnney in for humanitarians AI signing off.
More from Humanitarians AI Fellows
3:27Google HOPE Explained: Fix for AI Memory or Illusion of Depth? | Uday Sonawane
2:24Deepa Shenoy explains: Brutalist Video Submission & QC Process
3:311 Model, 3 Benchmark Scores: Evaluating DeepSWE & Coding RL | Uday Sonawane
2:26Deepa Shenoy explains: How Training Data Shapes AI Decisions
6:19How Architecture Shapes Music: From Cathedrals to Streaming Intros | Tanmay Kulkarni
1:25