Mycroft: Cross-Agent Validation & The Number That Wasn't There

Running two independent AI agents on company filings

[0:00]Two weeks ago, this project built something that runs two AI agents on the same company and flags when their numbers disagree. Last week, both agents finally became real. This week, it ran for the first time, and the first thing it produced wasn't a disagreement. It was a number that came from nowhere.

The 0.34 debt-to-equity ratio fabricated from zero input data

[0:16]Fast version cross agent validation runs two independently reasoning agents on one company, pulls the numbers out of each conclusion, and flags anything that doesn't match. Set arithmetic, no model, no judge. It started with one real agent and one fixture. a handwritten standin

Producer A vs Producer B: Quantitative hallucinations vs qualitative summaries

[0:30]You can figure with whatever answer you want used on purpose to prove the detector works before pointing it at anything real. Last week the fixture was replaced with a second real grader reading the same SEC filing through a different lens balance sheet versus earnings quality. 143 tests passed. Zero lines changed in the comparator itself, but nobody had actually watched two real

Running five diagnostic tests to audit upstream extraction gaps

[0:52]Agents talk to each other yet. The second agent ran on a model set up locally. Once it ran, real filing data went in. A real independent model came back with a conclusion for each side. Producer A reading assets revenue and net income wrote calculated the debt to equity ratio as 0.34. Its input data contained no debt and no equity figures.

Updating flagging rules: Dropping false alarms from 11/12 to 7/12 companies

[1:10]Not close, not derived. The concept wasn't in the data at all. Producer B reading real earnings figures. Cited zero numbers. It described them consistent, significantly large, but never wrote one down. The system flagged a contradiction. It was right to, but not for the reason you'd hope. One surprising result could be noise. So, five separate tests ran against it. Not a quick pass or fail on each, but the full treatment. What the test is, why it was run, what a clean result should look like, exactly what was fed into it, what actually came back, and what that means. Here's all five. Test one, claim verification. Here's what it is. For every citation in a thought log that points at a real URL, this mechanism fetches that source and checks whether the numbers being claimed actually appear in it within 1% tolerance. Here's why it exists. This is the systems dedicated defense against exactly this failure. A citation that looks real but

Isn't backed by what it points to. Here's what a clean result looks like. A true number should confirm against the source. A fabricated number should come back checked and not found. Either outcome means the mechanism is doing its job. 100% pass rate isn't the goal. 100% honest verdict is. Here's what it was actually given. The citation under test was the debt to equity line pointing at the same SEC filing already fetched for producer A's real inputs. Here's what happened. Verification never got the chance to check that number at all. One layer upstream claim extraction only recognizes figures shaped like a dollar amount, a percentage, a multiple, or basis points. A bare decimal ratio 0.34. No symbol attached matched none of those patterns. It was invisible before verification ever ran. Here's what it means. A verification layer is only as reliable as what gets handed to it. A gap in extraction can silently starve a working checker of the one claim

It most needed to see. Test two determinism. What it is? Run the identical question through the identical model. Same temperature, same fixed seed five separate times and compare all five answers. Why it matters? It separates two very different problems. A wrong answer that changes every time is noise annoying but survivable. A wrong answer that repeats identically is a stable pattern in the model's reasoning. Worse, but at least diagnosable. What a clean result looks like. Either all five converge on the same hopefully correct answer or if something's wrong, the same wrong answer keeps recurring because a repeatable failure is when you can actually target what was actually given. The project's own local model adapter defaults to temperature zero and a fixed C of 42. That part is confirmed straight from the adapter source code. Whether this specific run used those defaults unmodified isn't independently confirmed from this checkout, flagged plainly, not glossed over. What happened? Four of the five runs converged

On the same invented pair of ratios off by one digit in the last decimal place. The original debt to equity number, the one from the first run, never came back across the other four. What it means fixing temperature and seed narrows the model's behavior, but doesn't collapse it to one single deterministic answer. There's still an outlier, and the outlier was exactly the fabrication this video is about. Seen only once, it would have looked like the model's normal behavior instead of the one time it strayed furthest from it. Test three, the project's own consistency probe, a separate mechanism from determinism built for exactly this situation. What it is, run a second fully independent pass on the same input. Then score how much the two conclusions actually overlap. Words weighted lightly at point4 specific numbers weighted heavily at 6. Because a real number is much harder to fabricate identically twice than a general phrase is why it exists. Unlike verification, it needs no

External source at all. genuine reasoning from real evidence should converge across independent runs on its own. What a clean result looks like. High agreement and this is the important part. The specific hard flag the moment any number shows up in one run and never in the other. No scoring required. What it was given? The same subject, the same agent identity. A fresh independent run. Weights of four on word overlap and 6 on number overlap. What happened? The debt to equity figure appeared exactly once across every run collected. Never a second time. The hard divergence flag fires exactly the way it's designed to. What it means of all five tests here, this is the one that worked precisely as intended with no workaround needed. It didn't need to know anything about the real world. It only needed the fabrication to fail at being consistent with itself, and it did. Test four, a guardrail stress test, and it's the odd one out.

It isn't testing whether the agents are right. It's testing whether they can even speak the required format at all. Why that matters on its own? Every other test here assumes the response actually parsed into a valid structure to begin with. If that layer breaks, nothing downstream even runs. What a clean result looks like. A first attempt parse success rate at or near 100% and zero forced halts. Proof the format enforcement layer holds up against real, not fixture generated model output. What it was given 24 real calls across the live batch. The same directive version each time. First attempt only counted. A retry wouldn't count as a clean pass. What happened? 24 for 24. Zero retries triggered. Zero halts. What it means, whatever else is wrong this week, it isn't the format layer. That part held perfectly. At least at this sample size, the failure lives entirely in what the model reasoned, not in whether it could describe that reasoning in

The shape the pipeline requires. Test five breadth what it is. Run the same two producer comparison across 12 different real companies, not just the one from the first live run. Why? One company could be a fluke, an unusual filing, a missing field, anything. Testing across a spread of real companies is what tells you whether a pattern is general or a one-off. What a clean result looks like. Flags that correlate with actual factual contradictions. A low false positive rate across a genuinely diverse batch of real filings. What it was given 12 real tickers. Real SEC filings pulled for each. Both producers run per company. The existing contradiction flag left completely unmodified. What happened? 11 of 12 got flagged. The one exception was the single company where neither agent cited a number at all. nothing to compare, so nothing to flag. And manually checking one of those 11 flag cases showed both agents were completely correct. One reported assets and revenue, the other

Reported earnings per share. Different concepts, zero actual conflict, and the flag fired anyway. What it means, and this is the part that actually redirected the project this week. 11 out of 12 looked like an alarming contradiction rate at first glance. It was actually revealing that the flag can't yet tell the difference between two agents disagreeing and two agents simply talking about different things. Five tests, one upstream extraction gap, one informative outlier, one mechanism that worked exactly as designed, one clean structural pass, and one real project redirecting flaw in what the flag actually measures. The missing number problem turned out to be duplicated in three separate files, not one. The same rejects copied three times, all with the same blind spot. All three got widened the same way. An unfixed duplicate left behind would have been worse than not fixing it at all. The flag problem needed an actual decision, not a patch. The honest constraint. Nothing in the system currently

Knows which real world concept a number came from. Only their number exists. So, a genuinely complete fix wasn't available without much bigger changes. The decision that shipped, a number cited by only one side, no longer counts as a contradiction by itself. that fixes the case where one agent simply wasn't asked about something. It does not fix two agents citing real, correct, unrelated numbers. That gap is named in the code, tested as a known limit, and left open on purpose, not quietly dropped. The fix wasn't just tested against handbuilt examples. It was checked against the same 12 real companies recalculated, not rerun. Under the old rule, 11 of 12 flagged under the new rule, seven of 12. Four companies where one agent simply hadn't quantified anything stopped being false alarms. The other seven where both agents cited real numbers about different things are still flagged. That's the honest measured size of the fix, not everything. Something specific and countable. Two separate

Questions, two separate answers. Does the machinery work? Real filings, real independent reasoning, a full audit trail, a guardrail that held on every one of 24 real calls. Yes, provably not, just theoretically. Does the flag reliably mean these two agents disagree? Not yet. Most of what it flags right now is agents discussing different things, not agents contradicting each other. And the one real fabrication this system produced was caught by a person reading the output, not by the system itself. A system built specifically to distrust fluent, confident answers just produced one and then measured exactly how wrong it was instead of hiding it. That's not a finished detector. It's a detector that's now told on itself twice in two days. Once about a number, once about what its own flag actually measures. The smallest true claim, not the biggest. This proved the plumbing. It did not yet prove the judgment. The number that wasn't there was caught by a person, not the system.