← Portfolio
GitHub ↗

Olist Delivery-Delay Prediction — MLOps Service

How a notebook model became a tested, containerized, monitored inference service. Every figure on this page comes from the project repository.

FastAPIMLflowDVCGreat ExpectationsDocker ComposeGitHub ActionsPrometheus · Grafana
Filter by stage:
Task sections
10 / 10
built and verified end to end
Automated tests
115
unit · data · model · monitoring · integration
Docker services
5
mlflow · register · api · prometheus · grafana
Model features
42
one fixed feature contract
Alert rules
3
errors · latency · prediction drift
Grafana panels
9
provisioned automatically

Pipeline architecture

1 · Training and versioning (offline)
Raw data9 Olist CSV files
DVC
Notebooks 01–06integration, labels, split, EDA, features, training
pandas · scikit-learn
Fitted artifactsmodel, imputer, encoder, scaler
DVC · git
Model registrymodel and metrics registered once
MLflow
2 · Serving a prediction (online)
Requestsingle order or batch
FastAPI · pydantic
Validationschema, ranges, nulls
Great Expectations
Featuresimpute → scale → encode, never re-fitted
src/features · src/data
Predictmodel loaded from the registry
RandomForest · MLflow
Logevery prediction stored
SQLite
3 · Delivery and monitoring
CIlint, format check, tests on every push
ruff · black · pytest
Build and publishimage pushed to the registry
GitHub Actions · GHCR
Metricsscraped every 10 seconds
Prometheus
Dashboards and alertslatency, errors, drift
Grafana

Model evaluation, read honestly

The headline accuracy of 0.93 is misleading. Only 6.6% of test orders are delayed, so a model can score high by predicting "on time" almost always. The number that matters is recall on delayed orders: 0.08, meaning the model catches 8% of real delays, with ROC AUC 0.61. This project demonstrates the engineering around a model; the model itself is a baseline to improve.

Held-out test set (14,472 orders)

13,515
On time: 13,515Delayed: 957 (6.6%)
ClassPrecisionRecallF1Orders
On time0.940.990.9613,515
Delayed0.310.080.13957
Macro average0.630.540.5514,472
Accuracy 0.93 · ROC AUC 0.61. Model: RandomForestClassifier, 100 trees, trained on 67,533 orders and validated on 14,471.

Model comparison on the validation set

Precision

Dummy
0.000
Log. Reg.
0.425
Rand. Forest
0.359

Recall

Dummy
0.000
Log. Reg.
0.162
Rand. Forest
0.163

F1

Dummy
0.000
Log. Reg.
0.234
Rand. Forest
0.224

ROC AUC

Dummy
–
Log. Reg.
0.765
Rand. Forest
0.737
Logistic Regression scored slightly higher than Random Forest on F1 and ROC AUC, yet Random Forest was the model saved in the notebook. The service serves what the notebook saved, and the repository states this openly.

What was built: the ten sections

01
Repository and configurationOne config file for every path, MLflow setting, host and port.
02
Notebooks to Python modulesPreprocessing and feature code packaged around the real 42-column contract.
03
Logging and error handlingStructured logs and controlled error responses.
04
Data versioning and validationRaw data and the model tracked with DVC; inputs checked with Great Expectations.
05
Experiment tracking and registryThe service loads its model and preprocessors from the MLflow registry.
06
Testing115 tests, including a leakage check and a notebook-versus-pipeline parity test.
07
APISingle and batch prediction endpoints with typed request and response schemas.
08
Docker and Docker ComposeFive services start together; batch predictions tested live through the containers.
09
CI/CDGitHub Actions: tests, then image build and push to GHCR.
10
MonitoringPrometheus and Grafana verified on a real Docker daemon with live predictions.

Monitoring and API

Alert rules

HighErrorRate
5xx error rate above 5% for 5 minutes.
HighLatency
p95 request latency above 500 ms for 5 minutes.
PredictionDriftDetected
Predicted-delay rate over the last 200 predictions outside the 1%–20% band (training baseline 6.6%) for 10 minutes.
A second drift signal, PSI on predicted probabilities, is exposed for diagnosis. Below 30 logged predictions the service reports "insufficient data" instead of a verdict.

Grafana dashboard panels

  • Predictions logged (total)
  • Current predicted-delay rate
  • Drift status
  • p95 latency (5 min)
  • Request rate by status
  • Request latency percentiles
  • Predictions by class
  • Model pipeline latency (p95)
  • Predicted-delay rate vs. baseline

API endpoints

POST /predictone order
POST /predict/batchmany orders in one call
GET /healthliveness check
GET /model-infowhich model version is serving
GET /monitoring/summarydrift report
GET /metricsPrometheus scrape target