Mycroft Update: Running Two AI Agents Live

The silent failure of our first real multi-agent run

[0:00]Two weeks ago this project built something that runs two AI agents on the same company and flags when their numbers disagree. Last week both agents finally became real. This week it ran for the first time and the first thing it produced wasn't a disagreement. It was a number that came from nowhere. Fast version, cross-agent validation runs two independently reasoning agents on one company, pulls the numbers out of each conclusion and flags anything that doesn't match. Set arithmetic, no model, no judge. It started with one real agent and one fixture, a handwritten stand-in you can figure with whatever answer you want, used on purpose to prove

The detector works before pointing it at anything real. Last week the fixture was replaced with a second real grader reading the same SEC filing through a different lens, balance sheet versus earnings quality. 143 tests passed, zero lines changed in the comparator itself, but nobody had actually watched two real agents talk to each other yet. The second agent ran on a model set up locally. Once it ran, real filing data went in. A real independent model came back with a conclusion for each side. Producer A reading assets, revenue, and net income wrote calculated the debt-to-equity ratio as 0.34. Its input data contained no debt and no equity

Test 1: Claim verification and regex extraction blind spots

[1:10]Figures. Not close, not derived, the concept wasn't in the data at all. Producer B reading real earnings figures cited zero numbers. It described them consistent, significantly large, but never wrote one down. The system flagged a contradiction. It was right to, but not for the reason you'd hope. One surprising result could be noise. So five separate tests ran against it, not a quick pass or fail on each, but the full treatment. What the test is, why it was run, what a clean result should look like, exactly what was fed into it, what actually came back, and what that means. Here's all five. Test one, claim verification. Here's what it is. For every

Citation in a thought log that points at a real URL, this mechanism fetches that source and checks whether the numbers being claimed actually appear in it within 1% tolerance. Here's why it exists. This is the system's dedicated defense against exactly this failure, a citation that looks real but isn't backed by what it points to. Here's what a clean result looks like. A true number should confirm against the source. A fabricated number should come back checked and not found. Either outcome means the mechanism is doing its job. 100% pass rate isn't the goal. 100% honest verdict is. Here's what it was actually given. The citation under test was the debt-to-equity line pointing at

Test 2: Determinism and why fixed seed doesn't guarantee one answer

[2:21]The same SEC filing already fetched for producer A's real inputs. Here's what happened. Verification never got the chance to check that number at all. One layer upstream, claim extraction only recognizes figures shaped like a dollar amount, a percentage, a multiple, or basis points. A bare decimal ratio, 0.34, no symbol attached, matched none of those patterns. It was invisible before verification ever ran. Here's what it means. A verification layer is only as reliable as what gets handed to it. A gap in extraction can silently starve a working checker of the one claim it most needed to see. Test two determinism. What it is. Run the identical question through the identical model, same temperature, same fixed seed, five separate times, and compare all five answers. Why it matters. It separates two very different problems. A wrong answer that changes every time is noise, annoying but survivable. A wrong answer that repeats identically is a stable pattern in the model's reasoning,

Test 3: The consistency probe and catching the single-run outlier

[3:15]Worse but at least diagnosable. What a clean result looks like. Either all five converge on the same, hopefully correct, answer or if something's wrong, the same wrong answer keeps re- occurring because a repeatable failure is when you can actually target. What was actually given? The project's own local model adapter defaults to temperature zero and a fixed seed of 42. That part is confirmed straight from the adapter source code. Whether this specific run used those defaults unmodified isn't independently confirmed from this checkout, flagged plainly, not glossed over. What happened? Four of the five runs converged on the same invented pair of ratios, off by one digit in the last decimal place. The original debt-to-equity number, the one from the first run, never came back across the other four. What it means, fixing temperature and seed narrows the model's

Test 4 & 5: Format guardrails and testing breadth across 12 companies

[4:00]Behavior, but doesn't collapse it to one single deterministic answer. There's still an outlier, and the outlier was exactly the fabrication this video is about. Seen only once, it would have looked like the model's normal behavior instead of the one time it strayed furthest from it. Test three, the project's own consistency probe, a separate mechanism from determinism built for exactly this situation. What it is, run a second fully independent pass on the same input, then score how much the two conclusions actually overlap. weighted heavily at .6, because a real number is much harder to fabricate identically twice than a general phrase is. Why it exists, unlike verification, it needs no external source at all. Genuine reasoning from real evidence should converge across independent runs on its own. What a clean result looks like, high agreement, and this is the important part, a specific hard flag the moment any number shows up in one run and never in the other. No scoring required. What it was given, the same subject, the same agent identity, a fresh independent run. Weights of .4 on word overlap and .6 on number overlap. What happened, the debt-to-equity figure appeared exactly once across every run collected, never a second time. The hard divergence flag

The project redirect: Distinguishing disagreement from different topics

[5:10]Fires exactly the way it's designed to. What it means, of all five tests here, this is the one that worked precisely as intended with no workaround needed. It didn't need to know anything about the real world. It only needed the fabrication to fail at being consistent with itself, and it did. Test four, a guardrail stress test, and it's the odd one out. It isn't testing whether the agents are right. It's testing whether they can even speak the required format at all. Why that matters on its own, every other test here assumes the response actually parsed into a valid structure to begin with. If that layer breaks, nothing downstream even runs. What a clean result looks like, a first attempt parse success rate at or near 100% and zero forced halts. Proof the format enforcement layer holds up against real, not fixture generated model output. What it was given, 24 real calls across the live batch, the same directive version each time, first attempt only counted, a retry wouldn't

The honest verdict: Proof of plumbing, but not proof of judgment

[6:00]Count as a clean pass. What happened? 24 for 24, zero retries triggered, zero halts. What it means, whatever else is wrong this week, it isn't the format layer. That part held perfectly. At least at this sample size. The failure lives entirely in what the model reasoned, not in whether it could describe that reasoning in the shape the pipeline requires. Test five, breadth. What it is, run the same two producer comparison across 12 different real companies, not just the one from the first live run. Why? One company could be a fluke, an unusual filing, a missing field, anything. Testing across a spread of real companies is what tells you whether a pattern is general or one-off. What a clean result looks like, flags that correlate with actual factual contradictions. A low false positive rate across a genuinely diverse batch of

Real filings. What it was given, 12 real tickers, real SEC filings pulled for each, both producers run per company, the existing contradiction flag left completely unmodified. What happened? 11 of 12 got flagged. The one exception was the single company where neither agent cited a number at all, nothing to compare, so nothing to flag. Manually checking one of those 11 flagged cases showed both agents were completely correct. One reported assets and revenue, the other reported earnings per share. Different concepts, zero actual conflict, and the flag fired anyway. What it means, and this is the part that actually redirected the project this week. 11 out of 12 looked like an alarming contradiction rate at first glance. It was actually revealing that the flag can't yet tell the difference between two agents disagreeing and two agents simply talking about different things. Five

Tests, one upstream extraction gap, one informative outlier, one mechanism that worked exactly as designed, one clean structural pass, and one real project redirecting flaw in what the flag actually measures. The missing number problem turned out to be duplicated in three separate files, not one. The same rejects copied three times, all with the same blind spot. All three got wide in the same way, and an unfixed duplicate left behind would have been worse than not fixing it at all. The flag problem needed an actual decision, not a patch. The honest constraint, nothing in the system currently knows which real-world concept a number came from, only their number exists. So, a genuinely complete fix wasn't available without much bigger changes. The decision that shipped, a number cited by only one side, no longer counts as a contradiction by itself. That fixes the

Case where one agent simply wasn't asked about something. It does not fix two agents citing real, correct, unrelated numbers. That gap is named in the code, tested as a known limit, and left open on purpose, not quietly dropped. The fix wasn't just tested against hand-built examples. It was checked against the same 12 real companies, recalculated, not rerun. Under the old rule, 11 of 12 flagged. Under the new rule, seven of 12. Four companies where one agent simply hadn't quantified anything stopped being false alarms. The other seven where both agents cited real numbers about different things are still flagged. That's the honest, measured size of the fix, not everything. Something specific and countable. Two separate questions, two separate answers. Does the machinery work? Real filings, real independent reasoning, a full audit trail, a guardrail that held on every one of

24 real calls. Yes, provably, not just theoretically. Does the flag reliably mean these two agents disagree? Not yet. Most of what it flags right now is agents discussing different things, not agents contradicting each other. And the one real fabrication this system produced was caught by a person reading the output, not by the system itself. A system built specifically to distrust fluent, confident answers just produced one, and then measured exactly how wrong it was instead of hiding it. That's not a finished detector. It's a detector that's now told on itself twice in two days, once about a number, once about what its own flag actually measures. The smallest true claim, not the biggest. This proved the plumbing. It did not yet prove the judgment. The number that wasn't there was caught by a person, not the system. Signing off, Divij Paswha.