Mycroft: Testing Pipeline Shadows & The Stages That Stayed Dark

The happy path vs testing pipeline execution paths

[0:00]A passing test tells you what your system did. It tells you nothing about what it did on the way there. A travel insurance claim, three documents, a canceled flight, a hospital receipt in another language, a note asking for the cost of a missed tour. It goes into a pipeline. Intake,

Breaking a pipeline: Why output assertions pass on broken builds

[0:15]Extraction, coverage check, authorization, resolve. Five stages light up and the claim resolves. That is the whole happy path and it is the least interesting thing this pipeline does. Hi, this is Tanmay Kulkarni in for Humanitarian's AI. This is a reference implementation of a claims pipeline

Testing negative assertions: Verifying uncalled pipeline stage

[0:31]Built for case study on Zurich's Agentyc AI. 28 tests, all passing. I want to show you the eight assertions in it that don't check what it produced. Second claim, the hospital receipt comes back with an extraction confidence of 0.4 under the threshold. Intake passes, extraction halts. The claim escalates to a human with a named reason, low

The shrinking shadow: Matching assertions to early halts

[0:51]Extraction confidence, and three stages never run at all. Coverage check, authorization, resolve. They stay dark. Here is the question this whole video is about. How would you prove those three stages stayed dark? Not that the claim escalated, not that the reason was right, that the coverage check never ran. The obvious test checks the output. Run

Gate failures and identifying wasteful pipeline behavior

[1:10]The claim, assert the status is escalated, assert the reason is low extraction confidence. That test passes. It is a good test. Most test suites I have written are made almost entirely of tests like it and it cannot answer the question. So, let me break the pipeline and let me do it as gently as I can. One line moved up four lines. The coverage check now runs before the extraction halt is honored. Everything else is untouched. Run the same claim, same status, same reason, same detail string. The output is byte for byte identical and every output assertion still passes. Here is what changed. The broken pipeline fetched the customer's policy record for claim that should have stopped one stage earlier. For a claim nobody was ever going to decide because it was already on its way to a human. In a claim system, looking up a customer's policy is not free and it is not invisible. It is an event. One line catches it, not an assertion about

What came out, an assertion about what was never called. Patch the coverage check, run the claim, assert it was not called on the correct pipeline that passes. On the broken one, it fails, and it is the only test in the suite that does. 27 of the 28 still pass on a pipeline that is wrong. There are eight assertions like that one in the suite, and they are not eight arbitrary tests. Watch what they do. First claim, a document arrives without a language tag, so intake halts immediately. Three stages must stay dark, extraction, coverage, the gate. The test names all three. Second claim, this one gets further. Extraction runs, the confidence comes back low, it halts there. Now, only two stages must stay dark, coverage and the gate. Extraction is allowed to have run, it was supposed to run. Third claim, further still. Coverage runs, finds no matching policy, halts. One stage must stay dark, just the gate. Three, then two, then one. The further a claim travels,

The smaller the shadow behind it, and the assertion shrinks to match because the assertion is a description of the shadow. 3 + 2 + 1 is 6. There are eight. The last two belong to a fourth condition, and it does not fit this pattern at all. The fourth one is different, and it is the pair I would keep if I could keep only two. The authorization gate takes a decision function. If that function returns something malformed, not a decision, just a string that isn't one, the pipeline must reach neither outcome. Not resolve, not escalate. Escalating would look safe. A human gets the claim, everything continues, and a broken gate would be indistinguishable from a busy Tuesday. Nobody would ever find out. One more thing, and then I'll leave it alone. The one property Zurich confirms about this system is that it keeps an auditable trail of the reasoning behind its decisions. The broken pipeline's trail would be accurate. It really did fetch that record. A trail tells

You what happened. It cannot tell you that what happened shouldn't have. So, take your own pipeline. For every condition that stops it early, name the stage that must not run. Then, write the line that fails if it does. You pass if that line exists. You fail if your suite only checks outputs, cuz then you cannot tell a correct pipeline from a wasteful one that happens to end in the same place. The repository is in the description. 28 tests, standard library only, no Pytest needed. One boundary because it matters. On the happy path, the broken build and the correct build fetch the same record in the same order. No assertion in the suite can tell them apart. This technique only bites where a pipeline stops early. That is most of what a claims pipeline does, but it is not everything, and I would rather say so than let you think eight assertions cover the whole system. The stages that stayed dark, Tanmay Kolharkarney in for Humanitarian's AI, signing off.