The Uncertain Eye: Teaching AI to Admit Doubt | The Uncertain Eye Ep. 4

Varun opens this episode of The Uncertain Eye with the idea that reorganized the whole project: what if the most valuable thing their glaucoma-detection AI produced wasn't its diagnosis, but its doubt? The episode explains what that means in practice and how it becomes a working triage system.

What a confidence score actually tells you

When the classifier reads an eye scan, it doesn't just output "glaucoma" or "healthy." It outputs a probability. Near the extremes, close to zero or close to one, the model is confident. Hovering near the middle, around a coin flip, it's genuinely unsure. Varun points out that this middle band isn't rare. It's exactly where a lot of scans land, including the early and ambiguous cases that matter most for screening.

Confidence tracks accuracy, sharply

The team measured how the model's accuracy tracked with its own confidence, and the pattern was sharp. When the model was confident, it was usually right. When it hovered in the uncertain middle, its accuracy fell off a cliff. That confidence number turned out to be information in its own right, a signal for when the model could be trusted and when it couldn't.

Calibration: making confidence mean something

Varun defines calibration directly: a well-calibrated model's confidence actually means something. If it says seven in ten, it should be right about seven times in ten. That property, not raw accuracy alone, is what the team built the rest of the system on.

Building triage around the confidence signal

With a calibrated confidence signal in hand, the system works in tiers. A fast first-tier classifier reads every scan and clears the ones it's confident about. The cases where it hesitates, the genuinely ambiguous eyes, get flagged and routed onward to a slower, deeper second opinion. Varun compares this directly to how a good clinic already operates: routine cases move fast, and the expensive resource, specialist attention, is spent only where it's actually needed.

The economics of escalation

The episode frames this as teaching the machine a kind of intellectual humility, knowing the edge of its own competence and raising a hand to say a case is beyond it. Varun is explicit that this has a cost: every flagged case is a specialist hour someone has to pay for, which is why the calibration has to be accurate rather than just cautious. An AI that knows what it doesn't know is framed as far safer than one that's confidently wrong, but only if the escalation rate stays economically workable.

Key takeaways

• A model's confidence score, not just its diagnosis, is treated as the most valuable signal it produces.
• Confidence and accuracy were shown to track together: the model was usually right when confident and much less reliable in the uncertain middle band.
• Calibration means a stated confidence level matches real-world accuracy at that level, for example being right seven times in ten when it says seven in ten.
• The resulting triage system uses a fast classifier to clear confident cases and routes only genuinely ambiguous ones to a deeper second opinion.
• Every escalated case carries a real cost in specialist time, which is part of why calibrated confidence matters more than raw accuracy.

Who this is for

Viewers following The Uncertain Eye series on AI-assisted glaucoma detection, and anyone interested in how confidence calibration turns a diagnostic model into a practical, cost-aware triage system rather than a single yes-or-no output.