Mycroft: Rebuilding Morgan Stanley's AI Tools (And Finding a Hidden Bug)

Rebuilding Morgan Stanley’s research searcher and meeting assistant

[0:00]One sentence from a Morgan Stanley press release. After the meeting, it summarizes key points, creates an email for an advisor to edit and send at their discretion, and saves a note into Salesforce. Three verbs in a row. One of them has 10 extra words attached. That's the whole difference, and I read straight past it. Hi, this is Tanmay Kulkarni, in for Humanitarian's AI. Morgan Stanley built two AI tools for its financial advisors. One searches the firm's research library. The other sits in on client meetings and writes them up. I built a working version of both using only what the firm has published. Along the way, I picked up one check

The three-question specification card to prevent design collapse

[0:40]That's worth more to me than anything else in that repository. I'm going to run it twice, once on Morgan Stanley's material, once on my own code. It finds something both times. Here's the check. Three questions, and you can run them on anything you've built from somebody else's spec. First, what does the source treat as two different things? Second, are they still two different things in the code? Third, is there a line that fails if someone makes them one? It catches one specific thing, two ideas in your source that quietly became one idea in your build. Keep the card in mind. It comes back. Pass one. Morgan Stanley is

Finding the difference: Autonomous Salesforce notes vs. human-edited emails

[1:16]Specific about results. 98% of advisor teams adopted the first tool. What they don't publish is how any of it works. Normal for a bank. It just means every line I wrote had to trace back to a published sentence. Question one. What does the source treat as two different things? Look at the verbs. It creates an email for an advisor to edit and send at their discretion. The advisor decides. Nothing happens without them. And it saves a note into Salesforce. No qualifier. Nobody to wait for. That's the only one of its outputs confirmed finished rather than waiting. The only time either tool writes into the firm's own records without an advisor deciding. And it's bounded. Internal filing. Nothing the client ever sees. And credit where it's due, it's right there in their own sentence in the open. Question two, were they still two different things in my code? Not in my first pass. My original design described both

The typo in the paraphrase: How "draft" hid a finished action

[2:10]Outputs as one thing, a draft waiting on the advisor's decision. Right about the email, wrong about the note. It took a finished action and filed it under waiting for a human. I caught it on review, checking the design back against the source and fixed it before any of it was final. That's the process working, but it's worth being precise about why it so nearly didn't. Morgan Stanley stated that distinction plainly. Three verbs in a row summarizes, creates, saves. Two of them just happen. The middle one carries 10 extra words. The verbs are parallel, so it reads as one list of things the tool does. The asymmetry isn't in the grammar at all. It's a qualifier hanging off the middle

Writing the "not equal" assertion test to lock in the difference

[2:51]Item. And here's the part that should worry you. The summary I first worked from said drafts. Morgan Stanley never says drafts, their word is creates. Draft sounds unfinished, creates does not. So the paraphrase was easier to read correctly than the original. Careful reading is good at finding what you skipped. It's much weaker on what you read and filed in the wrong place because rereading is the same instrument pointed at the same sentence by someone who's already decided what it says. Question three, and this is my favorite line in the repository. It says the email status is not equal to the sales first note status. That's the whole line. Look at what it doesn't do. It doesn't check the email is waiting or that the note is saved, other tests do that. It only asserts those two values can never become the same thing. So if somebody tidies this file next year and

Turning the check on my own repository: Finding the missing write-guard test

[3:40]Collapses them, the suite catches it, not a reviewer who might be tired at 4:00 in the afternoon. The suite, pass one done. Pass two, same three questions. New subject, my own repository. This is the part I didn't plan. Getting ready to make this video, I turned the card on my own code, expecting to walk you through a clean example. Both pipelines end the same way, and both carry a test that fails if a function with a dangerous name exists. Send, finalize, submit dispatch. The debrief side also checks for right. The assistant side doesn't. Same guard, two different word lists, and four of my own documents describe it as one. That's questions one and two answered in about a minute by a card I wrote for something else entirely. Now, let me be careful because this would be easy to oversell. I ran the full word list against all

Outro: Why an explicit test is better than relying on human memory

[4:30]Four modules. No matches, not one. The code is correct today. What's narrower is the coverage. Two of those modules are guarded and two happen to be fine. Add a right function to the debrief side, and a test fails immediately. Add the identical one to the assistant side, and everything passes. So, question three. Is there a line that fails if someone makes them one? On the assistant side, no. And the fix is one word. Add right to that list, and both halves are guarded the same way, which raises a fair question. And I'd rather answer it honestly than tidily. If nothing's broken, is this really a finding? By the card's own standard, yes. A difference held up by nothing but memory is worth a line of code before it costs you anything. But that's an argument about risk. I can't show you a failure because there isn't one. And the check has its own limit. It reads the module's own list of names, so it

Works partly because of how I happen to import things. Airtight is a stronger word than I've earned. So, your turn, same card, your name on it. Question one, what does the source treat as two different things? Open the spec, the docs, the ticket, and write the pairs down. Two states, two outputs, two permission levels. Question two, are they still two different things in the code? Question three, is there a line that fails if someone makes them one? If there isn't, that's the line to write today. Mine took 9 seconds. It's all on disk. Two pipelines, 12 modules, 29 tests. I ran them for this video. All 29 pass. It runs on made-up data. Nothing of Morgan Stanley's in it. And the write gap is still open as I record this on purpose. A check is more convincing when you can watch it catch something nobody tidied away first. This is Tanmay Kulkarni in for Humanitarian's AI. Go run the three questions on your own build.