RAG Evaluation in Production: CI and CD Golden Set Gates and Live Monitoring

Introduction: Offline Evaluation vs Online Monitoring

[0:00]Namaste circumflex euros, this is Liam in for Amia. Build it with Claude then take it apart. You shipped the rag system and the numbers look good. Here's the question that actually matters. How do you know it still works tomorrow on traffic you've never seen? There are two kinds of evaluation and you need both. Offline and online evaluation are not the same job. Offline runs before you ship on a fixed set where you know the right answer, so you can compute correctness directly. Online runs in production on real questions with no labels where you cannot compute correctness at all. Same word, two completely different problems. Start with offline because it's the one you control. You build a golden set a circumflex euros, real questions with known good answers a circumflex euros, and you score your system against it. Retrieval quality with hit rate and MRR,

Offline Evaluation: Golden Sets, Hit Rate, MRR, and Faithfulness CI and CD Deployment Gates: Blocking Silent Quality Regressions Online Monitoring Without Labels: Watching Retrieval Score Distributions

[0:46]Generation quality with faithfulness. Because you know the truth, these numbers mean exactly what they say. Now make it a gate. Run that golden set on every deploy and compare to the baseline. If MRR or faithfulness drops, the deploy is blocked a circumflex euros, the same way a failing unit test stops a release. A silent quality regression after a prompt tweak is exactly the bug this catches before a single user sees it. Then you ship and the ground truth disappears. In production, a user asks a question you've never seen and nobody labels whether the answer was right. You can't compute correctness live. So online monitoring changes the question a circumflex euros from is this answer correct to is the system behaving like it did when it was healthy. Without labels, you watch proxies a circumflex euros, signals that move when something breaks. The retrieval score distribution, alert when the mean top one score starts declining because that means retrieval is

Drifting. Refusal rate, latency, and cost per request, and user feedback signals. None of these is correctness, but together they tell you when to go look. And the thing that makes both debuggable is the trace. For every request, a tool like LangSmith records the whole story. Which chunks were retrieved and at what scores, the exact prompt that was sent, what came back, and the latency of each stage a circumflex euros plus the tokens and cost. When a proxy alerts, the trace is where you find out why. In practice, the offline gate is small. Run the golden set, compute MRR, and fail the build if it slipped below the baseline. That's a handful of lines standing between you and a silent regression a circumflex euros, and it's the same eval harness the whole series has been building, now pointed at the deploy. So, the full picture is two loops. Before deploy, the offline golden set in CI gates the release. After

Deploy, online proxies watch for drift and traces explain it. And one rule ties them together, the same one from the whole series a circumflex euros debug retrieval first. When a number moves, look at the retrieved chunks before you touch the prompt. So, keep three things. Offline eval needs a golden set and runs before deploy where you can measure correctness. Online monitoring has no labels, so you watch proxies a circumflex euros score distributions, refusals, latency, cost, and a per request trace is what makes either one debuggable. Gate before, watch after. Your turn. Take this prompt to your own system. Build a 20-question golden set, wire it into CI so a drop blocks the deploy, then log the retrieval scores of live traffic, and watch the mean. The day that mean starts sliding, you'll know before your users do a circumflex euros, and the trace will tell you why. RAG a circumflex euros evaluation in production. Gate before you ship, watch

Production Proxy Signals: Refusal Rates, Latency, and Cost Tracking

[3:21]After. This is Liam in for Aamia.