Extract, Don't Decide: Using LLMs for Structured Signal Extraction | Mycroft

Introduction: Why Raw News Headlines Cannot Score Anything - The Boundary Line: Model Reads, It Never Decides

[0:00]Hello, this is Amuda. So far, my pipeline could gather news headlines about a company, but a raw headline is useless to a scoring system. Open AI safety employee resigns as just text. Is that a layoff and executive change? Nothing important. This week, I taught a language model to read those headlines and turn them into precise structured signals. But with one firm rule, the model reads, it never decides. Here's the gap. My agent collects headlines, but everyone comes in tagged the same generic way. News mentioned, that's not enough to score anything. Somebody has to read each headline and judge what kind of event is this, a funding round, a layoff, a security breach. Doing that by hand across hundreds of headlines doesn't scale. But handing the judgment entirely to an AI is exactly the thing this project refuses to do. So where's the line? So the ask drew the line carefully. Use a language model to extract structure, read the headline, decide what type of signal it is, pull out any amount or date, and rate its own

Enforcing Schema with LangChain: Typed Fields And Confidence Scores

[0:50]Confidence. But stop there. The model does not score risk. It does not decide the signal is true. It extracts and structures a human still validates before anything counts. I wired this with lang chain which forces the model to return clean typed fields instead of loose text. Here's the heart of it. The schema I forced the model to fill not a paragraph of text but exact typed fields. What signal type any dollar amount any date a confidence score from 0 to 100 and one line of reasoning and a

Testing Real Headlines: Executive Changes vs Plain News Mentions

[1:16]Hard instruction in the prompt. Never invent an amount or a date that isn't in the headline. If the model is unsure, it must say so with low confidence, not make something up. And here's the model actually reading two real headlines. First, open AI safety employee resigns. The model typed it as an executive change confidence 70 and explained why a resignation is a personnel change. Second, a vague headline about super intelligence. The model typed it as a plain news mention. Confidence 20 reasoning, no funding, no layoff, no

The Unvalidated Signal Rule: Human Validation Before Execution - Key Takeaway: Extract Structure, Don't Hand Over Decisions

[1:47]Real event. Look at that confidence gap. 70 versus 20. The model didn't just classify, it told me how much to trust it. And this is the line that keeps the whole thing honest. Both of those extracted signals, the confident one and the vague one, came out marked unvalidated. The model read them, typed them, scored its own confidence, and then stopped. It does not get to say they count. A human reviews the confidence, checks the reasoning, and decides the AI made the human faster, not absent. That's the entire difference between a tool you can trust and one you can't. So, what did week three teach? that the right way to use an LLM in a serious pipeline isn't to hand it the decision. It's to let it do the reading. Turn messy text into structured

Data, rate its own confidence, show its reasoning, and then hand all of that to a human who makes the call. The model did the tedious part at scale. The judgment stayed where it belongs. Extract. Don't decide. Your turn. If you're putting an LLM into a workflow, ask one question. Is it reading or is it deciding? Reading. Pulling structure out of messy text is where these models shine and where mistakes are cheap because a human checks them. Deciding, letting the model make the final call unchecked is where trust breaks. Use the LLM for the reading. Keep the deciding for a person. Teaching a model to read the news, extract structure, rate confidence, and leave the decision to a human. Next week, I run this on a model that lives entirely on my own machine. Thank you.