Routing LLM Requests Without Rule Contradictions: The Mycroft Router

The risk of scattered routing rules across LLM cost tiers

[0:00]Hi, I am Simba, and this one's about the third Mycroft sprint, the router that decides for every request Mycroft gets, whether a cheap model can handle it or whether it needs something stronger. Six kinds of task, three cost tiers, and one rule I wanted to hold myself to. Nothing about how one task type is routed should be able to quietly

Locking task types and escalation thresholds in one policy file

[0:20]Contradict how another one is. The rule, task types and their routing rules live in exactly one file. Not scattered across the router, the prompts in someone's memory of what was decided last sprint. One file, so two rules can never quietly disagree. policy.json logs all six task types, sentiment, topic, structured extraction, contradiction detection, summarization, and a RAG

Building a pure router function without network calls or clock state

[0:47]Answer. For each one it records the answer format, the allowed labels, the check used to grade it, which tier it starts on, which tier it escalates to, and the input length that triggers that promotion. Each type points at a real file in the repo that proves Mycroft actually does that kind of work. And if that file ever disappears, a test catches it. The router itself is a pure function. No

Testing 24 synthetic fixtures: Human judgment vs. router predictions

[1:10]Network call, no clock, no model involved in deciding. It refuses anything pinned or unrecognized. Otherwise, every request starts on the tier its policy says and gets promoted one tier if the input is long, measured in characters, not tokens, because token counts depend on which model you'd even be asking. Every decision comes back with a one-sentence reason attached. One

The structural gap: Why format checks pass confidently wrong answers

[1:36]Thing is deliberately missing, a break-even rule that would route on cost versus accuracy trade-offs instead of just length. That's being left for later, on purpose. It's the clever version, and clever wasn't the goal this sprint. The bench folder holds three tools. fixtures.py checks the fixtures and freezes them. audit.py reports how much of the policy they actually cover. label.py is what I used to label them by hand. 24 fixtures in total, four per task type, two easy and two hard, all built around fictional companies. They got frozen on September 10th with a fingerprint of every file, so if anyone edits one later without meaning to, a test catches that too. Worth saying plainly, these fixtures are synthetic, not real. There's no real request data in this repo right now. 203 of 217 script samples are placeholders. The one data set that looks like sample data literally declares

Itself fake, and the Klarna mock data file is empty. So, these results can tell you something about how models handle these kinds of tasks. They can't yet tell you what Mycroft's real traffic actually looks like. The expected tier on each fixture is my own judgment. The cheapest tier I'd actually trust to get that answer right. I labeled every fixture before any model touched it, and the labeling tool never shows what the router would have picked. That ordering matters. It's the only way the comparison that's coming isn't just me agreeing with myself. Here's what it showed. My labels put six fixtures on cheap, 11 on mid, seven on strong. The router put eight on cheap, 16 on mid, and zero, not one, on strong. Of 24 fixtures, the router agreed with me on nine, and I want to be careful about that number. These are predictions being

Checked against a human's judgment. It isn't a measurement of real performance yet. That zero isn't a coincidence, it's structural. The simple router has exactly two ways to reach the strong tier, a long input or an escalation. There's no path from a short input straight to strong, and every single fixture I labeled as strong is a short input with a trap in it, a sentence that reads simply, but needs real reasoning to get right. As written, the router literally cannot agree with me on any of those seven. That's not a bug to patch quietly. It's the main gap this design has right now. Here's the risk underneath all of this. The free checks catch answers in the wrong format, not answers that are simply wrong. A model that completely misreads sarcasm can still return a label the policy allows. The check passes, and nothing escalates. The router has

No way to know the answer was bad if it was shaped correctly. That's the main design risk heading into the next two sprints. A couple of fixtures exist specifically to test the checks, not the models. In some dash-004, a genuinely correct unit conversion fails the check anyway. In extract-004, a flatly made-up answer passes it. Both are there on purpose. They're the honest edge cases the grading logic itself needs to survive. Two problems got fixed along the way. The labeling tool originally didn't show what each task was actually asking, so it now prints the task itself before every label. And I added a way to redo specific ones. And my very first labeling pass put everything on the cheap tier, so I relabeled those 10 using the same two questions every time. Is there a trap? And is there a reasoning step? Two more problems got found,

But not fixed here. Someone else's merge on August 30th deleted two entries from the run log, and the repo's conformance check can't find Python on Windows. Both are flagged as their own separate tasks. So, one file that can't disagree with itself, a router with no hidden state, and 24 fixtures frozen so nobody can quietly change the answer key. It agrees with human judgment on nine of 24, a starting point, not a benchmark. What's not yet done? Escalation only looks at length. The free checks only catch format, and every fixture behind these numbers is still synthetic, not real traffic. Your turn. If you're routing requests across cost tiers of your own, does your cheap check only catch the wrong shape of answer? Or would it actually catch a confidently wrong answer that happens to come back in the right shape? One file, no contradictions. Simba, for humanitarian's AI.