Simbaremuteuro explains: Single-Retry Escalation and Cost-Ladder Dynamics

Introduction: The Single-Retry Gate Rule

[0:00]Hi, I am Simba, and this one's the fourth Mycroft Sprint, the one where the router finally gets to retry. One retry ever on the tier above, and the real question I built this Sprint to answer. Can a wrong answer in the right shape still slip through with nothing catching it and no retry helping? The rule, retry happens exactly once on the escalation tier triggered by a free check failing. There's no retry loop in

Free Validation Checks in `validators.py`

[0:26]The code, so a chain of retries isn't just discouraged, it's structurally impossible. validators.py holds five checks, one per task type. The answer is an allowed label, the JSON has its required keys, a contradiction verdict actually quotes the input, a summaries numbers all appear in the input, and a RAG answer sites a passage. Two more checks run on everything, no matter the

Non-Retryable Failures and Terminal Errors

[0:51]Task. The answer isn't empty, and it isn't cut off. gateway.py is the whole path. Route to a tier, call the model, run the check, and if it fails, retry exactly once on the tier above. That's the entire retry logic. No loop exists to walk back down or try a third time, and the adapters changed, too. Every failure now reports whether retrying

Sprint Results: 24 Fixtures and Zero Failures

[1:15]Could actually help. A bad API key, an unknown model, or a request that's simply too large are terminal. The gateway never retries those because no retry fixes them. bench/run.py runs every frozen fixture through the whole path and grades each answer against your key, writing both a log and a results file. Tests went from 96 to 146 over the

The Cost Ladder: Sticker Price vs. Real Token Spend

[1:41]Sprint. Final sweep, 24 fixtures. 24 requests became 26 attempts. Escalation happened on 8%, two fixtures. Failure, 0%. Total cost, $0.00250, about a hundredth of a cent per request. Latency, 393 milliseconds at the median, 734 at the 95th percentile. Of the 24, 16 were graded, 14 correct.

Your Turn: Auditing Your Cost Ladder Crossover

[2:07]This sprint was built around one worry, a model returning a wrong answer in the right shape, where no free check catches it and no retry helps. Across 24 fixtures, including every deliberate trap, that happened zero times. The two flagged fixtures turned out to be your answer key, not the model. Sent-001 and Sent-004 are keyed negative. Both models answered positive, and the models are right. The cheap 20B model even got Sent-004 correct. The exact headline where the negative words belong to a rival company, the same trap that once reached a finished brief. Two things broke early. My quote check was too strict. All three first sweep escalations were false alarms, because the models quoted both conflicting statements joined together. A reasonable answer the check rejected for wanting one contiguous span. Fixing the prompt dropped escalation from 11% to 8, and the strong tiers model just disappeared. QN/QN3.6-27B started returning 400 for a week after working fine. Grok's own

Deprecation page still recommends it. Replaced with QN3.8-27B. The replacement model was then refused on every call. It's 1,024 token budget exceeded the account's cap of 1,000 output tokens per minute. Lowered to 896, and it worked. And the summary itself was wrong once. A smoke run and the full sweep shared one dated log file, double counting three requests, 27 instead of 24. Log files are now stamped to the second. One more thing broke on purpose. A test that hard-coded 1,024 broke the moment the budget changed correctly. Two numbers are worth keeping. First, escalated requests cost 3.2 times a normal one, $0.000282 against $0.000088. Two of 24 requests took 23% of the spend. Retrying is cheap in absolute terms and expensive in relative terms, which is exactly why the trigger has to be right. Second, the cost ladder isn't monotonic. On the gate prompt, the strong tier cost 20% less than the cheap tier, despite a 13 times higher

Sticker price on output. Cheap spent 77 tokens thinking. Strong answered in two. The crossover sits around five output tokens. So, on real work, strong is far more expensive. What it proves is narrower and still useful. Sticker ratio and real ratio can point in opposite directions. So, one retry on the tier above and no loop for it to become a chain. Zero failures across 24 fixtures, including every deliberate trap. The two flags this sprint found were a wrong answer key, not a wrong model. And two numbers worth carrying forward. An escalation costs three times a normal request and the sticker price doesn't predict the real one. Both the retry trigger and the tier choice need real numbers, not list prices. Your turn. If you're pricing tiers by sticker rate alone, have you actually checked where your own cost ladder crosses over? It might not be where the price list says. One retry, never a chain. Simba for Humanitarian's AI.