Google HOPE Explained: Fix for AI Memory or Illusion of Depth? | Uday Sonawane
Uday Sonawane examines the HOPE paper with three questions: what updates and how often, beaten against what, and did the authors say it is solved. The numbers show real but narrow gains, and the authors state forgetting is not solved.
Google published a paper called HOPE, and much of the coverage calls it a fix for AI memory. In this video, Uday Sonawane argues that the paper itself is more careful than the headlines, and that the careful reading is the more interesting one. He goes through the claims using three questions that can be applied to any paper saying a model learns continuously.
What HOPE is and is not
The video first sets out what HOPE is not. It is not a training technique you apply to a model you already have. It is an architecture. The paper's real claim is in its title: that deep learning, meaning stacking more layers, is an illusion hiding what matters. The important factors are how many levels of the network are learning and how fast each one updates, not how deep it is.
Three questions for any continuous learning claim
The speaker proposes three questions. First, what updates, and how often? Name the parts and their rates. Second, beaten against what? The comparison class is where most claims actually live. Third, did the authors say it is solved? The paper usually answers that in a section the summaries skip.
Question one: what updates and how often
A transformer has two memory speeds: weights frozen after training, and an attention window that forgets everything when the context ends. HOPE replaces that split with a spectrum called a continuum memory system, made of blocks that each update on their own clock. Fast blocks take the immediate material, slow ones consolidate, and the update rule itself is learned. The model modifies its own memory, which is where the self-modifying label comes from.
The numbers and the ablation
At 1.3 billion parameters trained on 100 billion tokens, HOPE reaches a WikiText perplexity of 14.39, against 15.60 for Titans, the architecture it builds on, and 17.92 for a tuned transformer, where lower is better. On Lambada accuracy it scores 51 percent against 49.1 and 42.6. The video calls these real but narrow gains. Removing the continuum memory system moves average perplexity from 12.24 to 13.04 and reasoning accuracy from 58.1 to 57.3, so it is load bearing by less than a point of accuracy.
Question two: beaten against what
On the hardest retrieval test, finding a random identifier buried in 16,000 tokens, HOPE scores 24.8. That beats Titans at 21.2 and leaves other recurrent models in single digits, but a plain transformer scores 40.8. HOPE wins its class, while attention still wins the task.
The forgetting experiment and the caveat
In the experiment the speaker highlights, a model is taught one new language in context, then a second, and asked to translate the first. Ordinary in-context learning collapses, because the second language overwrites the first. With HOPE, each extra memory level improves the score, and at three levels it nearly recovers the score from before the second language. More levels meant less forgetting. For question three, the authors write on page 40 that catastrophic forgetting is not solved in general, and describe nested learning as a roadmap rather than a destination.
Key takeaways
- HOPE is an architecture, not a training technique for existing models.
- Its claim is about the number of learning levels and their update speeds, not depth.
- Reported gains on perplexity and Lambada are real but narrow.
- Removing the continuum memory system costs less than a point of reasoning accuracy.
- On a 16,000-token retrieval test, HOPE scored 24.8 while a plain transformer scored 40.8.
- Adding memory levels reduced forgetting in the language experiment.
- The authors state that catastrophic forgetting is not solved in general.
Try it yourself
Next time a paper claims a model learns continuously, find the baseline row before you read the abstract. Ask which family of models it was compared against, and whether the strongest available model appears in the table at all.
Chapters
Full transcript(auto-generated, with timestamps)
Introduction: HOPE Architecture vs. Deep Learning Illusion
[0:00]I'm Udaye S. Ananwayne. Google published something called Hope, and most of the coverage calls it a fix for AI memory. The paper is more careful than that, and the careful version is the interesting one. First, what it is not. Hope is not a training technique you apply to a model you already have. It is an architecture, and the paper's actual claim is in its title, that deep learning, meaning stacking more layers, is an illusion hiding the thing that matters, not how deep the network is, how many levels of it are learning, and how fast each one updates. Three questions for any claim
Three Questions for Any Continuous Learning Claim
[0:30]That a model learns continuously. One, what updates and how often? Name the parts and their rates. Two, beaten against what? Because the comparison class is where most of these claims actually live. Three, did the authors say it is solved? The paper usually answers that one for you in a section the summary skip. Question one, a transformer has two memory speeds, the weights frozen after training and the attention window, which forgets everything the moment the context ends. Hope replaces that split with a spectrum, a continuum memory system, blocks that each update on their own clock. Fast ones take the immediate stuff, slow ones consolidate, and the update rule itself is learned. The model modifies its own memory, which is where the self-modifying label comes from. The numbers at 1.3 billion parameters trained on 100 billion tokens. Perplexity on WikiText 14.39 for Hope
How HOPE Works: Spectrum of Continuum Memory Systems - Benchmark Results: Perplexity & Ablation Accuracy Gains - The Retrieval Reality Check: HOPE vs. Standard Attention
[1:16]Against 15.60 for Titans, the architecture it is built on, and 17.92 for a tuned transformer. Lower is better. On Lambada accuracy 51% against 49.1 and 42.6. Real gains, narrow ones. A fair question is the memory spectrum doing the work, or is it decoration on a model that was already good? The paper answers it. Remove the continuum memory system and leave everything else. Average perplexity goes from 12.24 to 13.04, and reasoning accuracy drops from 58.1 to 57.3. So, it is load bearing by less than a point of accuracy. Question two, and this is where the headlines come apart. On the hardest retrieval test in the paper, find a random identifier buried in 16,000 tokens, Hope scores 24.8. It beats Titans at 21.2, and it
Buries the other recurrent models in single digits. A plain transformer scores 40.8. The paper's own sentence says it, "Comparing with other attention-free models, Hope achieves the best performance." Hope wins its class, attention still wins the task. So, what is it for? Here's the experiment that matters. Teach the model one new language in context, then a second one, then ask it to translate the first. In context learning collapses, the second language overwrites the first. That is forgetting in one trial. Then run Hope with one extra memory level two, three. The score climbs with every level added, and at three it nearly recovers what it scored when it had never seen a second language at all. More levels, less forgetting. That is the dose response
The whole paper is arguing for. Question three, page 40, in the authors' words, "Catastrophic forgetting is not solved in general." They call nested learning a roadmap rather than a destination, and they give the reason, "Forgetting is what finite capacity costs you." So, be excited that levels and update rates are design access anyone can now turn, and that the ablation says it works. Be skeptical of any sentence that says this fixes AI memory, because the people who built it wrote the opposite. Your turn. Next time a paper claims a model learns continuously, go find the baseline row before you read the abstract. Ask which family it was compared against, and whether the strongest available model was in the table at all. Levels, not layers.
More from Humanitarians AI Fellows
3:311 Model, 3 Benchmark Scores: Evaluating DeepSWE & Coding RL | Uday Sonawane
5:03Why a Coin Flip Beats This Learning Algorithm | Tanmay
6:19How Architecture Shapes Music: From Cathedrals to Streaming Intros | Tanmay Kulkarni
2:24Deepa Shenoy explains: Brutalist Video Submission & QC Process
2:01Why AI Models Can't Count Letters (And How Tokenization Explains It) | Sai Pranavi Jeedigunta
2:26