Model Maintenance: Continuous Monitoring & Drift Detection
What happens to a model after it ships — drift detection, retraining triggers, and a CI/CD pipeline designed to catch a degrading model before real harm accumulates.
The Business Problem & What's at Stake
The Loan Approval model from Project 2 was trained, tested, and honestly evaluated — 82.9% test accuracy, a documented Go/No-Go decision, known limitations. But that whole project was a single snapshot in time. Nothing about it addresses what happens six months after launch, once applicant patterns shift, once new batches of data start arriving, once the model that was "good enough to reference" quietly stops being accurate. Most companies treat "we shipped the model" as the finish line. It isn't. This project is about the part that gets skipped: building a small, real system for noticing when a model starts to fail, and having a plan for what to do about it before real harm accumulates.
Undetected Drift
Model drift — when real-world accuracy degrades because incoming data no longer matches training data — is one of the most common, well-documented ways production ML systems fail. Not from a bad model at launch, but from a good model nobody kept watching.
Financial Cost
More bad loans approved as the model's known lean toward over-approval worsens — a direct, quantifiable financial loss, modeled later in this case study.
Fairness Risk
Any degradation could widen Project 2's already-unresolved gender accuracy gap — tying model health directly back to the unresolved compliance question from that project, not just financial cost.
Defining What 'Healthy' Means
Before building any monitoring script, "healthy" and "degraded" had to be defined in advance — a business judgment call, not a technical detail, since it's the rule the monitoring system would enforce automatically later.
Threshold
Accuracy within ~80–85% (near baseline)
Action
No action needed.
Threshold
Accuracy 75–80%, OR CM detects a statistically significant input shift
Action
Logged alert. Pause automatic retraining — investigate WHY the shift happened before retraining on it (see Architectural Discovery section).
Threshold
Accuracy below 75%, OR false positive rate spikes sharply, OR the subgroup fairness gap widens
Action
Stop using the model for new decisions. Escalate immediately — fairness-related Red triggers route to compliance, not just engineering.
These thresholds aren't arbitrary — they're anchored to this project's own validated baseline (82.9% accuracy, ~54% false positive rate from Project 2's real confusion matrix), following the Value Threshold / Cost Cap model described in The Profitable AI Advantage. Checked this reasoning against real industry practice afterward: it holds up as a legitimate simplified version of a real severity-tier system, though production systems typically name specific ownership per tier and use adaptive thresholds alongside static ones — both real gaps, addressed honestly in the Decision Framework section below.
The MLOps Pipeline
Four pieces, each doing one job: CI validates a model before it's trusted, CD promotes a validated model into production, CT retrains on new data, and CM watches for drift in the input data itself. Built and tested each one against real project data — including one real, working dependency: this project's validation gate is actively checked by the Loan Policy Assistant agent before it will load a model.
Real cross-project dependency: Loan Policy Assistant
The CD stage in this pipeline connects to the Loan Policy Assistant (Project 3) — a live dependency, not a hypothetical one. When a new model is promoted, the RAG assistant's prediction tool must be updated to use it. This diagram reflects that actual coupling.
Drift Simulation & Detection
Split the original 614 applicants into two batches to simulate a before/after: Batch 1 (first 400 rows, "day one") and Batch 2 (remaining 214 rows, "new applicants who arrived later").
Sanity Check — No Real Change
Ran a Kolmogorov-Smirnov test comparing ApplicantIncome between Batch 1 and the real, untouched Batch 2.
p = 0.681
Correctly reports no meaningful drift — both batches are genuinely similar, as expected.
Deliberate Shift — Real Drift
Artificially raised every ApplicantIncome value in Batch 2 by 40%, simulating a real-world shift like income inflation or a genuine change in applicant profile, then re-ran the same test.
p = 0.0000000011
An overwhelming, unambiguous "real drift detected" result.
The detector doesn't cry wolf on ordinary variation, and doesn't miss a real, meaningful shift when one exists — the two properties a monitoring system actually needs to be trustworthy.
A Real Architectural Gap: CT Doesn't Catch Its Own Drift
Notable Technical Finding
After building CI, CD, and CT, I tried something that seemed like an obvious next test: feed the artificially-shifted Batch 2 — the same +40% income data the drift detector correctly flagged above — into the CT retraining function, expecting the resulting model to fail CI or at least show a real difference. It didn't. The retrained model came back with identical numbers to the non-shifted version: 81.3% accuracy, 53.8% false positive rate, CI passed cleanly.
Why this happens, and it's not a bug: StandardScaler re-centers and re-scales whatever data it's given, every time it's fit. When CT retrains from scratch on the shifted data, the scaler simply adapts to the new income range rather than treating it as an anomaly — the model isn't "fooled" by drift, it just doesn't have a mechanism to notice the input distribution moved, because retraining re-normalizes around whatever it's handed.
The real finding this reveals: CI and CT, as built, only check whether a model performs acceptably on data drawn from its own — possibly already-shifted — training distribution. Neither one compares against the original baseline distribution. Only CM's drift test, which explicitly compares new data against the untouched Batch 1, actually catches a shift. This means the correct pipeline order isn't CI → CD → CT → CM as four independent, parallel checks; it should be CM running first, before CT retrains at all — if CM detects drift, that's a signal for a human to investigate the cause of the shift before blindly retraining on it, not a green light to retrain automatically.
Discovering that a system's pieces don't compose the way you initially assumed is a genuinely realistic, valuable outcome of building a monitoring pipeline for the first time — and it's a more honest, more interesting result than if CI had simply "caught" the drift the way I expected going in.
Business Impact — A Step-by-Step Calculation
My first pass at this section was a placeholder — a guessed percentage with a disclaimer attached, not a real calculation. Rebuilding it here with every assumption stated and every step shown, following the Value Threshold / Cost Cap structure from The Profitable AI Advantage.
Explicit Assumptions
500 loan applications/month (small lender) · £50,000 average loan size · 40% Loss Given Default
All explicitly assumed, not measured
From Project 2 — Real Numbers
68.3% baseline approval rate · ~54% false positive rate among should-be-denied cases · ~31.7% of applicants should genuinely be denied
From Project 2's actual model evaluation
| Step | Description | Result |
|---|---|---|
| 1 | Approvals per month | 500 × 68.3% ≈ 342 approved/month |
| 2 | Bad loans/month, healthy baseline | 158 should-be-denied × 54% ≈ 85 bad loans/month ≈ £1,700,000/month in expected losses (baseline reference cost, not the cost of drift) |
| 3 | Drift scenario: false positive rate worsens to 64% | 158 × 64% ≈ 101 bad loans/month (+16/month vs. baseline) |
| 4 | Cost of 6 months undetected | 16/month × 6 months × £20,000 ≈ £1,920,000 in additional losses |
| 5 | Savings from early detection (caught in ~1 week vs. 6 months) | ≈96% reduction in the damage window ≈ £1,843,000 saved |
Every number above traces back to either a real, measured project result or an explicitly labeled assumption — the value of this structure over a single guessed percentage is that every input can be challenged or refined with real figures the moment they exist.
Decision Framework: When to Retrain vs. Rollback
Every monitoring signal needs a predefined answer to "what happens next" — decided in advance, not improvised once a real alert fires. This builds directly on the Green/Yellow/Red tiers above, but adds a specific action for each — and critically, the correction from the Architectural Discovery finding about when retraining is even the right move.
Green
No action. The system keeps running as-is.
Action: Continue
Yellow
My original assumption was that a Yellow signal should trigger automatic retraining. The Architectural Discovery proved that's wrong: retraining on drifted data without understanding why just produces a model adapted to the drift, not one checked against it. Corrected action: CM's drift signal pauses automatic retraining and triggers human investigation into why the input distribution shifted — is it a temporary blip, a real population change, or an upstream data bug? Only once that's understood should CT run, and its output must still pass CI before promotion.
Action: Investigate First, Then Retrain
Red
Stop using the model for new decisions, escalate immediately — not a scheduled retrain. Given Project 2's unresolved gender fairness finding, a Red trigger on subgroup gap widening should route to whoever owns fairness/compliance review, not just engineering — this could be a legal exposure question, not only a technical one.
Action: Stop & Escalate
Rollback as a distinct fourth option, not just "retrain harder"
The deployment record built earlier makes rollback possible in principle — every promoted model has a timestamped record of its own validated performance, so a Red-tier failure in a newly-promoted model can be answered by reverting to the last known-good record and model file, rather than always reaching for a fresh retrain.
Retraining assumes new data will produce a better model; rollback assumes the previous model was fine and something about the promotion process failed. Conflating them — always retraining, never rolling back — is itself a design mistake worth naming explicitly.
What I'd Need from a Real Team & Reflection
What a Real Team Would Add
Model Owner
Accountable for the model's ongoing health — this framework's tiers currently route to 'a human' or 'compliance,' not a specific named role.
Data Engineer
To build the real, automated pipeline feeding new applicant data in, rather than the two hand-split CSV batches used to simulate it here.
Compliance / Legal Reviewer
Specifically for Red-tier fairness triggers, given Project 2's unresolved gender gap finding and its UK Equality Act framing.
MLOps / Infrastructure Engineer
To actually run this on a schedule, rather than the manually-triggered scripts built here — a deliberate scope decision made at the start of this project.
Reflection
The technical mechanics of this project — a checklist function, a promotion script, a retraining loop, a statistical test — were each individually simple. The genuinely hard part was discovering that four separate, working pieces don't automatically compose into a system that does what you assume it does just because each piece works correctly on its own.
The clearest example was the CT/drift finding above. I built CI, CD, and CT expecting the chain would eventually catch drifted data somewhere. It didn't — not because anything was broken, but because a real architectural property of how these pieces relate went untested until I tried the interaction directly. The CI threshold mistake was a smaller version of the same lesson: I set a false-positive-rate check at "under 30%" without first calculating what that number actually was on the model I'd already validated — it was 53.8%. The fix wasn't a better guess, it was anchoring to a real, measured baseline.
Checking this project against real, documented industry practice — rather than assuming a reasonable-sounding framework was sufficient on its own — surfaced genuine gaps every time I did it: the CI checklist against named escalation matrices used at real companies, the cost estimate against a real Value Threshold framework, the team structure against how prototype teams actually scale. That comparison habit is itself a useful discipline, independent of any single finding it produced.
The throughline connecting this project to the other three in this portfolio: the technical skill of building a model, an agent, or a monitoring pipeline is the easy part. The actual discipline is refusing to trust a number, a threshold, or a claim until it's checked against something real — a baseline, a documented case, or the plain arithmetic behind it — and being willing to say so plainly when an earlier version of the work didn't clear that bar.
Let's Connect
I am actively seeking Junior AI PM / Technical PM roles at companies building AI-powered products in media, events, e-commerce, or consumer applications. Let's connect if you're hiring or want to discuss AI product strategy.
Contact Info
© 2025 Ogbebor Osaheni. Built with Next.js, React, and Tailwind CSS.