MLOps CI/CD Pipeline: Versioning Code, Data and Models
Machine learning breaks every assumption your CI/CD pipeline was built on. Here's how to fix it.
Executive brief
- An MLOps CI/CD pipeline extends traditional CI/CD to version three artefacts instead of one: application code, training data and model weights, each with its own storage, lineage and promotion path.
- Standard CI/CD breaks for machine learning because a code change is no longer the only thing that can alter the output. A shift in training data, a hyperparameter tweak or a new feature column can change model behaviour without touching a single line of code.
- Data versioning tools like DVC track datasets alongside code in Git, giving every training run a reproducible snapshot of the exact data it used.
- Model registries (MLflow, for example) store trained model artefacts with metadata, metrics and lineage, so promotion from staging to production is an auditable, automated decision rather than someone copying a file.
- Shadow deployments route a copy of production traffic to a candidate model without serving its predictions to users, which lets you compare real-world performance before switching over.
- GPU cost governance requires treating compute as a first-class pipeline concern: tagging training jobs by team, setting per-project spend limits, and reclaiming idle capacity automatically.
Why CI/CD breaks for machine learning
An MLOps CI/CD pipeline is a continuous integration and delivery system designed to version, test and promote machine learning artefacts (code, training data and model weights) through automated stages, so that model changes reach production with the same reproducibility and governance as application code.
Traditional CI/CD assumes one thing changes at a time: code. A developer pushes a commit, tests run, and if they pass, the artefact is built and deployed. The entire system is deterministic. Same inputs, same outputs.
Machine learning breaks that assumption in at least three ways.
First, your output depends on data as much as code. If last week's training data included a seasonal spike and this week's doesn't, the model behaves differently even though nobody changed a line of code. Your pipeline needs to know which data produced which model, or you can't reproduce a result and you certainly can't debug a regression.
Second, training is non-deterministic in most frameworks. Random weight initialisation, GPU floating-point ordering, data shuffling: two runs from the same code and data can produce slightly different models. Your pipeline has to decide what "slightly different" means and whether that difference is acceptable.
Third, the artefact itself is opaque. A compiled binary either works or it doesn't. A model checkpoint is a blob of weights. You can't read it, you can't diff it meaningfully, and its quality is a statistical measurement, not a pass/fail assertion. Your promotion criteria have to be metric-based, with thresholds that someone has thought about and codified.
None of this means traditional CI/CD is wrong. It means it's incomplete. An MLOps CI/CD pipeline adds the layers that handle data lineage, experiment tracking, model registry and metric-based promotion on top of the source control, build and deploy pipeline you already have.
For a broader comparison of how MLOps differs from DevOps across the whole lifecycle, see MLOps vs DevOps: Key Differences and Best Practices.
Four questions every ML pipeline must answer
Stonetusker works with five architectural archetypes, and the compute-intensive non-deterministic archetype (which covers ML workloads) has to answer these four questions for every model change that reaches production:
What has to be true before this ships? (the gate)
For a model, the gate includes more than test coverage. Evaluation metrics (accuracy, precision, recall, or whichever metrics your business cares about) must meet or exceed the currently deployed model on a held-out test set. Data validation checks must confirm that the training data matches the expected schema, distribution and freshness. Bias and fairness checks must pass if your domain requires them.
How do we prove it was true six months later? (the evidence)
The experiment tracker logs every training run. The model registry records which run produced which artefact, with links to the dataset version and evaluation report. Six months later, when someone asks why this model was promoted, the answer is a URL to the registry entry, not a Confluence page written from memory.
What does undo cost? (the rollback)
With a model registry, rollback means promoting the previous model version. If you're serving with an inference server that supports model versioning, that switch is instant. Without a registry, rollback means finding the old weights on someone's laptop, hoping the serving code is compatible, and praying the feature pipeline hasn't drifted.
What does a test environment physically have to be? (the harness)
ML test harnesses need compute. You can't run a meaningful evaluation on a laptop if the model needs a GPU for inference. Shadow deployments and offline evaluation harnesses need production-representative data volumes. A staging environment that tests against a 1% sample will miss distribution-sensitive failures.
Reference architecture: code, data and model versioning
A complete MLOps CI/CD pipeline tracks three artefact types through three stages, and they don't always move in lockstep.
Stage 1: Source and data version control
Application code and ML pipeline code live in Git. Training data, which is too large for Git, lives in object storage (S3, GCS, Azure Blob) and is tracked by DVC (Data Version Control). DVC stores lightweight pointer files in Git alongside the code, and each pointer references a specific version of the data in the remote store.
This means a single Git commit captures the code, the DVC pointer to the exact dataset version, and the pipeline definition. Checkout that commit, run `dvc pull`, and you have everything you need to reproduce the training run.
Stage 2: Training and experiment tracking
When a training pipeline runs (triggered by a data change, a code change, or a scheduled retrain), it logs metrics, parameters and artefacts to an experiment tracker. MLflow is the most widely adopted open-source option. Each run records the Git commit hash, the DVC data version, all hyperparameters, evaluation metrics on each dataset split, and the path to the saved model artefact.
If a run's metrics pass the promotion gate (e.g. accuracy above threshold, no fairness regression), the pipeline registers the model in the MLflow Model Registry with a staging label. If the metrics fail, the run is logged but no promotion happens.
Stage 3: Model promotion and deployment
A model in the staging slot goes through a second round of validation: integration tests against the serving infrastructure, latency benchmarks, and optionally a shadow deployment against live traffic. If it passes, the pipeline promotes it to the production slot in the registry, and the serving infrastructure picks up the new version.
The key design choice here is that no human needs to copy a file, update a configuration or restart a service. The registry is the source of truth. The serving infrastructure polls the registry (or receives a webhook) and loads the new model version automatically.
Versioning: what to track and where
| Artefact | What changes | Where it's versioned | What triggers a pipeline run |
|---|---|---|---|
| Application code / pipeline code | Training logic, feature engineering, serving code, pipeline DAGs | Git | Push to main or merge request |
| Training data | New samples, corrected labels, schema changes, feature additions | Object storage, tracked by DVC pointer files in Git | DVC pointer change in Git, or scheduled data refresh |
| Model weights | Retrained checkpoint, fine-tuned weights, quantised variant | Model registry (MLflow) | Successful training run that passes the metric gate |
| Feature definitions | New features, changed transformations, deprecated columns | Feature store or Git (depending on architecture) | Feature schema change triggers retraining or compatibility check |
| Environment / dependencies | Python version, library versions, CUDA version, container image | Lockfile in Git (pip freeze, conda env export) or container registry | Dependency update triggers a rebuild and revalidation run |
The table looks orderly, but the hard part is the interactions. A new feature column changes the data schema, which means the training code needs to handle it, which means the serving code needs to expect it in the input. If any of those three is out of sync, the model either fails to train, fails to serve, or silently produces wrong predictions on the new feature's inputs.
Contract testing (covered in the progressive delivery article) applies here too. The training pipeline and the serving pipeline are effectively producer and consumer. If the training pipeline outputs a model that expects 47 input features and the serving pipeline sends 46, you want that caught in CI, not in production.
Model serving and shadow deployments
Getting a model into production isn't just about the file. You need an inference server that handles concurrent requests, manages model versions, and optionally runs pre/post-processing steps. Two open-source options dominate:
NVIDIA Triton Inference Server supports multiple frameworks (TensorFlow, PyTorch, ONNX, TensorRT), handles dynamic batching and model ensembles, and runs on both GPU and CPU. It's a good fit when you serve multiple model types behind one endpoint.
vLLM is purpose-built for large language model inference. It uses PagedAttention for efficient GPU memory management and supports continuous batching, which matters when you're serving an LLM with variable-length requests. If your workload is LLM inference specifically, vLLM is typically the faster path to production than a general-purpose server.
Shadow deployments for model validation
A shadow deployment sends a copy of production traffic to the candidate model without returning its predictions to users. Both the production model and the shadow model process the same requests. You compare their outputs offline.
This catches problems that offline evaluation misses. A model might score well on a held-out test set but perform poorly on the actual distribution of production requests, which shifts over time. Shadow deployments expose the candidate to real data patterns: request volumes, edge cases, seasonal shifts and adversarial inputs that no static test set captures.
The trade-off is cost. Running two models on every request doubles your inference compute for the duration of the shadow window. For teams serving on GPUs, that's not trivial, which connects directly to the next section.
GPU cost governance
ML pipelines consume expensive compute, and the bill scales with experimentation. More experiments, more GPU hours. More shadow deployments, more inference cost. A healthy MLOps practice ships more models faster, which means the cost goes up as the practice matures, not down.
That's fine, as long as it's intentional. The problem is unintentional cost: training jobs that run overnight because nobody set a timeout, abandoned experiments that still hold GPU reservations, shadow deployments that keep running after the evaluation window closes.
Three mechanisms keep GPU spend visible and bounded:
Tagging and attribution. Every training job, inference endpoint and shadow deployment carries a tag that maps it to a team, project and budget code. Without tags, your cloud bill is one number. With tags, you can see that the recommendations team spent 40% of the GPU budget on hyperparameter sweeps, and ask whether that was the best use of resources.
Per-project spend limits. Set a weekly or monthly GPU budget per team or project. When the budget is 80% consumed, alert the team lead. When it hits 100%, either pause new jobs or require manual approval. This doesn't prevent spending; it prevents surprise spending.
Automatic reclamation. Training jobs that haven't logged a metric update in 30 minutes are likely stuck or abandoned. Spot/preemptible instances that finish their work should release immediately, not idle until the reservation expires. Inference endpoints that serve zero requests for 24 hours should scale to zero. Each of these rules is simple to implement and hard to remember to implement without automation.
The LLM Hosting and Deployment Options article covers infrastructure choices for large model serving, including cost comparisons across cloud providers.
Regulated AI and the build-or-buy question
For teams building ML systems in regulated industries (healthcare, financial services, government), the MLOps CI/CD pipeline isn't just about speed. It's about evidence. Regulators want to know which data trained the model, whether the data was compliant with privacy rules, which humans approved the model's promotion, and whether there's an audit trail for every prediction the model made.
These requirements map to capabilities your pipeline either has or doesn't. Data lineage from DVC gives you the "which data" answer. The model registry gives you the "who approved and when" answer. Prediction logging in the serving layer gives you the audit trail.
What most teams underestimate is the policy-as-code layer. Automated gates that enforce data residency (training data doesn't leave a region), model approval workflows (a human must sign off before production promotion in certain risk tiers) and access controls (only named individuals can modify training pipelines for high-risk models) are the difference between "we have a process" and "we can prove we have a process."
For medical AI applications specifically, the regulatory compliance guide and DevSecOps for AI in healthcare cover the intersection of ML pipelines with FDA and MDR requirements.
Build or have it built
You can assemble an MLOps CI/CD pipeline from open-source components: Git, DVC, MLflow, Argo Workflows or Kubeflow for orchestration, and Triton or vLLM for serving. Every piece is mature and well-documented. The integration work is where teams stall.
Connecting DVC to your CI runner so that data version changes trigger retraining, wiring MLflow's model registry to your deployment automation so that promotion events trigger serving updates, and building the metric gates that sit between training and promotion: each of those integrations is a project, not a configuration change.
If your team has the platform engineering capacity and the ML infrastructure experience, build it. If your priority is getting models into production reliably and you'd rather spend your ML engineers' time on modelling rather than plumbing, bring in a team that's built this before.
Stonetusker's Platform Foundation practice covers the CI/CD, infrastructure-as-code and observability layers, while Secure Delivery handles the governance, compliance and audit trail requirements that regulated ML workloads demand. Take the readiness check to see where your current pipeline stands.
Frequently asked questions
What is the difference between MLOps and DevOps CI/CD?
DevOps CI/CD versions and deploys one artefact type: application code compiled into a binary or container image. MLOps CI/CD versions three artefact types: code, training data and model weights. It adds experiment tracking, data validation, metric-based promotion gates and model registry management on top of the standard CI/CD pipeline. The testing philosophy also changes: DevOps tests are deterministic (pass or fail), while ML model evaluation is statistical (accuracy above a threshold, no fairness regression, latency within budget).
Why do you need to version training data separately from code?
Training data is too large for Git and changes independently of code. A model's behaviour depends on the data it was trained on, so if you can't point to the exact dataset version that produced a given model, you can't reproduce the training run or diagnose a regression. Tools like DVC store lightweight pointer files in Git that reference the full dataset in object storage, so every Git commit captures both the code and the data version together.
What is a model registry and why does it matter?
A model registry is a versioned store for trained model artefacts, along with their metadata: which experiment produced them, what metrics they achieved, which dataset they were trained on, and their current lifecycle stage (staging, production, archived). It matters because without one, model promotion is a manual process of copying files and updating configurations, which is error-prone and leaves no audit trail. With a registry, promotion is an API call that changes a label, and every transition is logged.
What is a shadow deployment for ML models?
A shadow deployment routes a copy of live production traffic to a candidate model without serving its predictions to users. The candidate processes real requests in parallel with the production model, and you compare their outputs offline. This catches problems that offline test sets miss: distribution shifts, edge cases and performance characteristics that only appear under real traffic patterns. The downside is cost, because you're running two models on every request for the duration of the evaluation window.
How do you control GPU costs in an MLOps pipeline?
Three mechanisms: tagging every training job and inference endpoint by team and project so you can attribute costs; setting per-project GPU budgets with alerts at 80% and hard stops at 100%; and automatic reclamation of idle resources, such as terminating training jobs that haven't logged metrics in 30 minutes or scaling inference endpoints to zero when they receive no traffic. The goal isn't to minimise GPU spend overall but to make sure every GPU hour is intentional and attributable.
Building an ML pipeline that actually works in production is a different problem from building a model that works in a notebook. If your team is spending more time on plumbing than on modelling, it's worth a conversation about what a properly engineered pipeline looks like for your workload.
Talk to StonetuskerLast reviewed: October 2026. Tool versions checked on: October 2026 (DVC, MLflow, vLLM, NVIDIA Triton Inference Server).
30-minute call: Book with Subeesh
