Mycroft Finance Investigator: Reviewer-Guided Reinvestigation & Refusal Loops
Adwait Changan explains the reinvestigation loop in the Mycroft finance investigator, where each follow-up is bound by hash to an earlier run and replayed from scratch. Three requests produce three different outcomes, which is the point of an honest review loop.
This video looks at what happens after a reviewer asks a finance agent to go back and check its work. Adwait Changan describes the reinvestigation loop he built for the Mycroft finance investigator. Three follow-up requests went in and three different answers came back, and only one of them was yes. The video argues that this variety is the whole point of a review loop.
Binding follow-ups to a specific earlier run
A follow-up only means something if it points to a specific earlier run. Every reviewer request is therefore bound by hash to the exact investigation it questions. If the routing plan has changed, the run is rejected. If the underlying trend log has changed, it is rejected. A reviewer cannot quietly reopen a different investigation from the one they actually looked at.
The 18-line replay
The replay mechanism is 18 lines. It rehashes the plan, rehashes the trend log, then reruns the entire orchestration from scratch and compares the fingerprint of every specialist result against what was stored. Four separate conditions in the code end the run. The key design choice is that it never reads the old answer and agrees with it. It regenerates the answer and checks whether the new one still matches.
Three requests, three outcomes
The reviewer sends three different kinds of request: verify the evidence already held, test an explanation the reviewer believes is true, and go find evidence that does not exist yet. A system worth trusting has to answer these differently, and this one did:
- Verifying existing evidence came back verified, because the replay matched exactly.
- Testing a causal claim came back unsupported, because nothing in the evidence establishes the cause.
- Requesting additional evidence stayed open, because the operational records are simply not there.
One yes, one no, one not yet. The presenter says the middle row is the one he would check first if handed the system. A review loop that can only return verified is not reviewing anything, and he calls it a rubber stamp with extra steps.
Verified does not mean a business cause is proven
The sentence the presenter cares most about in the report is that verified means the evidence replayed exactly. It does not verify a business cause. Those are different claims, and the gap between them is where a finance tool either stays honest or quietly starts misleading people.
Fenced on five sides
The loop is guarded against five failures:
- A modified source file is caught.
- A missing task is caught.
- A duplicate request is refused.
- A citation pointing at evidence that does not exist is refused.
- A report overwritten after the fact is detected instead of absorbed.
Append-only closure packs
The output is an append-only closure pack that shows before and after side by side for every task. Nothing is edited in place, so a reviewer can see what changed and what did not. Closure status stays blocked pending human review, because the system finishes the checking and stops exactly where judgment begins. Final approval rests with a named human reviewer. The video notes that the easy version, returning verified on all three requests and closing the ticket, would have been faster to build and useless to a reviewer, since a result that is always yes carries no information.
Key takeaways
- Bind every reviewer follow-up by hash to the exact earlier run it questions, and reject it if the plan or log has changed.
- Regenerate results from scratch and compare fingerprints instead of trusting stored answers.
- A good review loop returns different outcomes: verified, unsupported and open.
- Verified means the evidence replayed, not that a business cause has been established.
- Guard against modified sources, missing tasks, duplicate requests, nonexistent citations and overwritten reports.
- Use append-only closure packs so before and after stay visible.
- Keep closure blocked until a named human reviewer decides.
Try it yourself
The video's exercise is to take a system you have built that rechecks its own work and list every outcome it is capable of returning. If the only outcome is success, write down what that costs a reviewer who relies on it, then find the smallest change that would let it return an honest no. If you cannot find a path that produces a refusal, the check is only a formality. Code and the full before-and-after report are in pull request 17 of the project repository.
Chapters
- 0:00Binding reviewer follow-ups by hash to exact earlier runs
- 0:15The 18-line replay mechanism: Regenerating and comparing fingerprints
- 0:30Three requests, three outcomes: Verified, rejected, and open
- 0:50Replay verification vs business cause: Fencing the system on five sides
- 1:10Append-only closure packs: Preserving before-and-after reports for human judgment
Full transcript(auto-generated, with timestamps)
Binding reviewer follow-ups by hash to exact earlier runs
[0:00]Hi, I am Adwait Changan. In this video is about what happens after a reviewer asks a finance agent to go back and check its work. This week I built the reinvestigation loop. Three follow-up requests went in. Three different answers came out and only one of them was yes. That is the whole point. A
The 18-line replay mechanism: Regenerating and comparing fingerprints
[0:15]Follow-up only means something if it points a specific earlier run. So, every request is bound by hash to the exact investigation it is questioning. If the routing plan changed, the run is rejected. If the underlying trend log changed, rejected. You cannot quietly
Three requests, three outcomes: Verified, rejected, and open
[0:30]Reopen a different investigation than the one the reviewer looked at. Here's that replay, 18 lines. Rehash the plan. Rehash the trend log. Then rerun the entire orchestration from scratch and compare the fingerprint of every specialist result against what was stored. Four separate conditions in there end the run. Notice it never reads the old answer and agrees with it. It regenerates the answer and checks
Replay verification vs business cause: Fencing the system on five sides
[0:51]Whether it still matches. Now, the reviewer sends three different kinds of request. Verify what you already have. Test this explanation I think is true. And go find evidence that does not exist yet. A system worth trusting has to answer those three differently. And it did. Verify existing evidence came back verified because the replay matched exactly. Test causal claim came
Append-only closure packs: Preserving before-and-after reports for human judgment
[1:12]Back unsupported because nothing in the evidence establishes the cause. Request additional evidence stayed open because the operational records simply are not there. One yes, one no, one not yet. That middle row is the one I would check first if someone handed me the system. A review loop that can only ever return verified is not reviewing anything. It is a rubber stamp with extra steps. This one rejected a causal claim that a reviewer proposed and it left a third request unfinished rather than closing it. And here is the sentence I care most about in the whole report. Verified means the evidence replayed exactly. It does not verify a business cause. Those are completely different claims and the gap between them is where a finance tool either stays honest or quietly starts lying to people. Which is why the loop is fenced on five sides, a modified source file is caught, A a missing task is caught, a duplicate request is refused,
A citation pointing at evidence that does not exist is refused, and a report that was overwritten after the fact is detected rather than absorbed. The output is an append-only closure pack, before and after, side by side for every task. Nothing is edited in place, so a reviewer can see what changed and what did not. And closure status stays blocked pending human review because the system finishes the checking and stops exactly where judgment begins. The easy version of this feature returns verified on all three and closes the ticket. That version would have been faster to build and completely useless to a reviewer because a result that is always yes carries no information at all. So, the verdict, a reviewer can now send work back and each follow-up is bound by hash to the exact investigation it questions. Every specialist result is regenerated from scratch and compared by fingerprint rather than trusted. Three requests produced three different answers. One verified by exact
Replay, one causal claim rejected as unsupported, and one left open because the operational evidence does not exist. The replay matched exactly and closure is still blocked because verified means the evidence replayed, not that anyone has established a cause. Final approval stays with a named human reviewer. Your turn, here is the prompt. Take a system I have built that rechecks its own work and tell me every outcome it is capable of returning. If the only outcome is success, explain what that costs a reviewer who is relying on it. Then show me the smallest change that would let it return an honest no. Run that on something you own. If you cannot find the path that produces a refusal, you have not built a check, you have built a formality. That was reviewer-guided reinvestigation and the closure pack for the Mycroft finance investigator. The code and the full before and after report are in pull request 17 at wait changan for humanitarian's AI.
More from Mycroft Financial AI
2:16The Real Cost of a Query | Aishwarya Patil
3:52Mycroft Finance Investigator: Evidence Gap Planner & Specialist Routing
2:37Mycroft: Reading the Claims
9:48Mycroft: Cross-Agent Validation & The Number That Wasn't There
1:52Mycroft Update: Broadening the Test
2:06