Project Mycroft: Building a Structural Enforcement Layer to Stop Silent AI Failures

The Unauditable Conclusion: Why a confident, fabricated AI explanation is a system failure

[0:00]Hi, I'm Diviage Power. Here's a problem I ran into. An AI system gave a conclusion with confidence and a good explanation. But what if that explanation was completely made up? What if the AI just hallucinated a reason that sounded right? An auditable conclusion is a system failure, no matter how accurate it actually is. I built a layer to stop that from happening silently. I built a layer that sits between any AI system and the person trusting its answer. It doesn't make the AI more accurate, that's a different problem. Instead, it makes it structurally impossible for a conclusion to sneak through without proof behind it. The layer enforces three things. Every conclusion must be sourced, the

Introducing Project Mycroft: Structural enforcement over accuracy

[0:35]Record must be permanent and unchangeable, and if anything breaks, the whole pipeline stops instead of guessing. Imagine you have four AI agents analyzing a company. One agent looks at finances, another checks patents, a third analyzes earnings, the fourth evaluates competition. They all send their findings to a final system that combines them into a single recommendation, buy, hold, or sell. Now

Mechanism 1: The Reasoning Object and Signed Source Line Affidavits

[0:56]Imagine the financial agent's analysis is based on a number it made up, a hallucinated value that never actually existed. Nothing stops it. That fake number flows right through the system and becomes part of an official investment decision. And here's the scary part. AI systems are really good at explaining wrong answers. Ask an AI why it made a mistake, and it will invent a plausible confident explanation for something that never actually

Mechanism 2: Append-Only Checkpointing to Prevent Database Tampering

[1:20]Happened inside the model. I looked at two other ways to solve this and rejected both. First, use another AI to check the first AI's reasoning. That doesn't work because the checking AI has the same blind spot as the one being checked. A perfectly crafted completely fake explanation can sound just as genuine to a checker as a real one. Second, mathematically trace through the AI's computation to verify the reasoning. Theoretically sound, but it forces you to abandon powerful AI models

Mechanism 3: Adversarial Arbitration and the "Affidavit of Disagreement"

[1:46]And use much weaker ones. Neither solution actually worked. So I built three different mechanisms. First, a reasoning object. Before any conclusion is allowed to propagate, before it reaches an investor or gets acted on, the AI must produce a signed affidavit. What did you conclude? How confident are you? What was the exact source line that led you there? What citations support this? If the AI can't point to the actual source that justified its conclusion, the state transition is

The XML Format Struggle: Why soft prompting failed and mechanical brackets saved the parser

[2:11]Rejected outright. Second, append-only checkpointing. Every record gets written to a database in a way that prevents updates or deletions. This is enforced by the database itself, not by application code, because code can have bugs or get bypassed. If a reviewer later flags a run as wrong, that flag goes into a separate table. The original record is never touched. You can always reconstruct exactly what the system saw and said at the time. Third, adversarial arbitration. When two agents genuinely disagree on something important, they debate once. If they still disagree,

ADR06 and the Honest Limit: Why structure does not guarantee truth

[2:42]That disagreement becomes a first-class output, an affidavit of disagreement, instead of being quietly averaged away into a meaningless middle ground. Here's where theory meets reality. I needed to force the AI to structure its output in a specific way. Two XML blocks, a thought log, and a conclusion, nothing else. Version one was a polite request, "Please format your output like this." I tested it against live Claude output. It failed. The model wrote a numbered preamble before the opening tag, and my parser couldn't find the blocks. So, the fix wasn't a softer

The Open Frontier: Proving if stated reasoning matches actual computational reasoning

[3:10]Prompt, it was a mechanical constraint. First character must be a bracket, no preamble allowed, period. Now, when parsing fails, it logs immediately as a parse failure with the broken response attached. One retry happens with an explicit corrective directive. If that fails, too, the entire pipeline halts, no grade is delivered. And here's the key, both the failure and the success

The Project Mycroft Challenge: Three tests to ensure your pipeline fails loudly

[3:30]Are part of history, not just the success. Now, here's the part I need to say clearly. Everything I just showed you enforces structure. It guarantees a thought log exists. It guarantees it's well-formed. It guarantees it's permanently recorded. But, it does not guarantee that the log reflects what actually happened inside the AI. A hallucinated conclusion can be accompanied by a perfectly coherent, perfectly formatted, completely fabricated thought log. That log will pass every structural check I've built. Every single one. I wrote this down explicitly. We call it ADR06 in our design log. The log is evidence of output, not evidence of process. This is not a bug I plan to fix later. It's a fundamental property of how large language models work. No logging layer by itself can solve this. I see

A lot of AI accountability framing that quietly slides past this distinction. It presents a structured trace as if structure equals truth. I'd rather say clearly what this system is and isn't. It's not a hallucination prevention system. It's not a fact verification system. It's a structural enforcement system with a known honest limit. So, where does that leave us? Structural enforcement is real and tested. The three mechanisms I built work. Ground truth checking is real where ground truth exists to check against like comparing what an AI claims about a financial filing to the actual filing from the SEC. Reproducibility checking is real. Run the same request twice and look for numbers that appear in only one of the two runs. And the hardest problem, proving that the stated reasoning is the actual

Reasoning, that one remains genuinely open and it might not be fully solvable with current AI models. But, here's what I can guarantee. When structural accountability fails, it fails loudly on the record instead of quietly becoming someone's investment decision. That's the difference this layer makes. Here's a challenge for you. Design a validation directive for your own AI pipeline that fails loudly instead of quietly accepting a bad answer. When you build it, check three things. First, does it log the failure immediately with the full broken response attached? Second, does the retry get its own separate record in the system? Third, does it halt the pipeline on a second failure instead of just guessing and moving on? If any of those three are missing, you found your accountability gap. Signing off, Div Vij Power.