Serving a Machine-Learning Model
Save a trained scikit-learn pipeline, load it once when the app starts, build a /predict endpoint with input validation and a model version in every response, handle batches, measure latency, and serve LLM features safely.
From notebook to endpoint
- Train a full pipeline (preprocessing + model) so the server gets raw inputs, exactly as in training (see scikit-learn pipelines). Separate preprocessing code in the server is the classic source of training-serving skew.
- Save it with
joblib.dump, together with a version string and the list of expected features. Load files only from sources you trust: loading a pickle can run code. - Load once when the app starts, not on every request (loading can take seconds).
- Validate each request: right fields, types and sensible ranges.
- Predict and return JSON including the model version, so every prediction can be traced later.
- Log inputs, outputs and timing for monitoring (next part), without storing personal data you do not need.
Code: save the model (runs here)
Train a small no-show model on synthetic data and save it with its metadata. This part runs in the page.
import joblib, numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
rng = np.random.default_rng(0)
n = 2000
lead_days = rng.integers(0, 60, n); age = rng.integers(18, 90, n); sms = rng.integers(0, 2, n); prior = rng.poisson(0.5, n)
logit = -2 + 0.04 * lead_days - 0.01 * (age - 50) - 0.8 * sms + 0.7 * prior
y = rng.random(n) < 1 / (1 + np.exp(-logit))
X = np.column_stack([lead_days, age, sms, prior])
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X, y)
bundle = {"model": model, "version": "noshow-2026-11-16", "features": ["lead_days", "age", "sms_sent", "prior_noshows"]}
joblib.dump(bundle, "noshow.joblib")
loaded = joblib.load("noshow.joblib")
print(loaded["version"], loaded["features"])
print("P(no-show) for a 40-day lead, age 30, no SMS, 2 prior:", round(loaded["model"].predict_proba([[40, 30, 0, 2]])[0, 1], 3))Code: the /predict endpoint
Run on your computer next to noshow.joblib. Test it with: curl -X POST localhost:5000/predict -H "Content-Type: application/json" -d '{"lead_days": 40, "age": 30, "sms_sent": 0, "prior_noshows": 2}'
import time, logging, joblib
from flask import Flask, request
app = Flask(__name__)
logging.basicConfig(level=logging.INFO)
BUNDLE = joblib.load("noshow.joblib") # loaded ONCE, at start-up
MODEL, VERSION, FEATURES = BUNDLE["model"], BUNDLE["version"], BUNDLE["features"]
LIMITS = {"lead_days": (0, 365), "age": (0, 120), "sms_sent": (0, 1), "prior_noshows": (0, 50)}
def validate(row):
problems = []
for f in FEATURES:
v = row.get(f)
if not isinstance(v, (int, float)) or isinstance(v, bool):
problems.append(f"{f}: must be a number")
elif not LIMITS[f][0] <= v <= LIMITS[f][1]:
problems.append(f"{f}: must be between {LIMITS[f][0]} and {LIMITS[f][1]}")
return problems
@app.post("/predict")
def predict():
start = time.perf_counter()
body = request.get_json(silent=True)
rows = body if isinstance(body, list) else [body] # one row or a batch
if not rows or len(rows) > 500 or not all(isinstance(r, dict) for r in rows):
return {"error": "send a JSON object or a list of up to 500 objects"}, 400
problems = {i: p for i, r in enumerate(rows) if (p := validate(r))}
if problems:
return {"error": "invalid_input", "problems": problems}, 422
X = [[r[f] for f in FEATURES] for r in rows]
probs = MODEL.predict_proba(X)[:, 1].round(4).tolist()
ms = (time.perf_counter() - start) * 1000
app.logger.info("predict n=%d version=%s ms=%.1f", len(rows), VERSION, ms)
return {"model_version": VERSION, "predictions": probs}
@app.get("/health")
def health():
return {"status": "ok", "model_version": VERSION}Try it: call the model endpoint
A simulated version of the endpoint above. Change the inputs (including silly ones) and send the request.
Interactive: this part runs in your browser.
Speed, scale and LLM features
- Latency: measure it per request and watch the slowest 5% (p95), not the average.
- Batch requests when many predictions are needed at once; many small models answer thousands of rows in milliseconds.
- Heavy models (deep learning, large files) may need more memory, a GPU, a separate model server, or a managed endpoint from a cloud provider; quantised or smaller models help (efficient deployment).
- Background jobs: for slow work (retraining, big batch scoring), return immediately with a job id and process it in a task queue (such as Celery or RQ), instead of keeping the request waiting.
- LLM features: your Flask server calls the model provider with the API key kept on the server (never in the browser), streams the answer to the page, and adds the limits from the Generative AI safety topic: rate limits, token caps and escaped output.
Quiz
1. Why load the model at start-up instead of inside the /predict function?
2. Why include the model version in every response?
3. Training scaled the features, but the server feeds raw values to the model. This is…
4. Where should the API key for an LLM provider live in a web app?
📚 Further reading
Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.
- Made With ML (MLOps course) ↗ free course on testing, serving and monitoring ML systems
- FastAPI tutorial ↗ Official documentation