Monitoring, MLOps & Security in Production
Keep an app healthy after launch: health checks, logs, metrics and alerts; what is special about ML in production (data drift, delayed labels, model registries, safe rollouts and retraining); and a security checklist.
Is the app healthy?
- Health checks: a cheap
/healthendpoint the platform calls every few seconds. - Uptime monitoring from outside, which emails or messages you when the site is down.
- Error tracking (for example Sentry) collects exceptions with their context, so you hear about bugs before users report them.
- Metrics: request rate, error rate (share of 5xx), and latency percentiles (p50, p95). Set alerts on the few that matter, such as "error rate above 2% for 5 minutes".
What is different about ML in production
A model can fail silently: the server returns 200 OK with confident predictions that have quietly become wrong, because the world changed.
- Data drift: the inputs change (new clinic, different patients, a changed form). Compare live feature distributions with the training data.
- Concept drift: the relationship changes (after SMS reminders were introduced, no-shows fell for everyone).
- Delayed labels: you learn whether a patient actually missed the appointment only days later. Join predictions with outcomes when they arrive and track real accuracy over time.
- Monitor the predictions too: a sudden change in the average predicted risk is an early warning, even before labels arrive.
Try it: spot data drift
The grey histogram is a feature (booking lead time) in the training data; the coloured one is this week's live traffic. Shift and stretch the live data and watch the Population Stability Index (PSI). A common rule of thumb: below 0.1 stable, 0.1 to 0.25 worth watching, above 0.25 a significant shift.
Interactive: this part runs in your browser.
Code: compute PSI (runs here)
PSI bins the training data, then compares the share of live data in each bin: the sum of (live − train) × ln(live ÷ train).
import numpy as np
def psi(train, live, bins=10):
edges = np.quantile(train, np.linspace(0, 1, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
t = np.histogram(train, edges)[0] / len(train)
l = np.histogram(live, edges)[0] / len(live)
t, l = np.clip(t, 1e-6, None), np.clip(l, 1e-6, None) # avoid log(0)
return float(np.sum((l - t) * np.log(l / t)))
rng = np.random.default_rng(0)
train = rng.gamma(2.0, 7.0, 5000) # booking lead time in days
same = rng.gamma(2.0, 7.0, 1000)
shifted = rng.gamma(2.0, 7.0, 1000) + 8 # a new policy: bookings made further ahead
print("same distribution :", round(psi(train, same), 3))
print("shifted by 8 days :", round(psi(train, shifted), 3))Releasing and retraining models safely
- Model registry: store every model with its version, training data, code version and evaluation results (MLflow is a popular open-source option). You must be able to answer "which model made this prediction, and how was it trained?"
- Shadow deployment: the new model receives real traffic and its predictions are logged, but users still get the old model's answers. Compare before switching.
- Canary release: send a small share of traffic (say 5%) to the new version, watch the metrics, then increase.
- Rollback: keep the previous version ready to switch back in one step.
- Retraining on a schedule or when drift is detected, always through the same evaluation gates as the first model, with a human signing off for high-stakes uses.
Security checklist before launch
- Debug mode off; secrets in environment variables; HTTPS only.
- All input validated; database access through the ORM or parameterised queries; templates escape output.
- Passwords hashed; access to every record checked against the current user; CSRF protection on forms.
- Rate limits on logins and expensive endpoints (especially model and LLM calls).
- Dependencies up to date (GitHub's Dependabot can open update pull requests for you); container runs as non-root.
- Backups of the database, and a tested way to restore them.
- Read the OWASP Top 10 ↗ for web apps, and, if you use LLMs, the OWASP Top 10 for LLM Applications.
Quiz
1. Your model API returns 200 OK for every request, but accuracy has fallen for weeks. Why did no alert fire?
2. In a shadow deployment, users get answers from…
3. PSI between training and live data for a key feature is 0.4. What should you do?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Made With ML (MLOps course) ↗ free course on testing, serving and monitoring ML systems
- MLflow documentation ↗ experiment tracking and a model registry
- OWASP Top 10 (web applications) ↗ the most critical web security risks