AI/ML Infrastructure for Production: CI/CD Pipelines That Ship Models Safely

Your model worked in a notebook. Now it needs to work at 3 a.m. on a Saturday when nobody's watching. Here's what the pipeline between those two states actually looks like.

Why ML Infrastructure Is Not Just "DevOps With GPUs"

A software application has a deployment artifact: a container image, a binary, a bundle. You build it, test it, ship it. If the tests pass and the build succeeds, you have high confidence the code will behave the same way in production as it did in staging.

ML models don't work like that.

The "artifact" is a trained set of weights that depends on training data that may have shifted since you last looked at it, feature engineering code that transforms inputs in ways nobody documented, and hyperparameters that someone tuned in a Jupyter notebook three weeks ago. Deploying the model without validating all three is like deploying application code without running tests. Except the failure mode is worse: a broken application crashes. A broken model returns confident, wrong answers.

This is why standard CI/CD pipelines need fundamental modifications for ML workloads. The pipeline isn't just building and deploying code. It's orchestrating a system where data quality, model performance, infrastructure stability, and business metrics all intersect. Most teams discover this the hard way, about two months after shipping their first model to production, when a subtle distribution shift in input data causes the model to degrade and nobody notices for weeks.

The good news: production ML infrastructure is a solved problem, just not a simple one. The patterns exist, the tools are mature, and the failure modes are well-documented. What's missing for most organizations is the connective tissue between data science workflows and platform engineering reality.

The Anatomy of an ML CI/CD Pipeline

A production ML CI/CD pipeline is wider than an application pipeline. It touches more systems, triggers on different events, and has validation gates that application pipelines don't need.

Here's what actually goes into one:

Data Validation and Ingestion

Before any model touches new data, the pipeline validates it. Not just "is this CSV well-formed?" but statistical validation: has the distribution of key features changed beyond acceptable thresholds? Are there missing values in columns that were previously complete? Did a new data source introduce a category the model has never seen?

Tools like Great Expectations and TensorFlow Data Validation (TFDV) encode these checks as executable tests. They run before training starts, and if the data looks wrong, the pipeline stops rather than training a model on garbage.

Feature Store Synchronization

Feature stores solve the "training-serving skew" problem, which is one of the nastiest bugs in production ML. If the features used during training are computed differently from the features used during inference, the model performs well in evaluation and badly in production. A feature store like Feast or Tecton computes features once and serves them consistently to both training and inference.

The pipeline needs to sync with the feature store, making sure that any feature engineering changes are reflected in both the training and serving paths before a new model version ships.

Model Training and Retraining Triggers

Retraining isn't always on a schedule. Good pipelines trigger retraining on data drift (the input distribution has shifted), concept drift (the relationship between inputs and outputs has changed), or performance degradation (the model's accuracy on live traffic has dropped below a threshold).

Schedule-based retraining is a starting point, not a destination. A recommendation engine at an e-commerce company might need daily retraining. A fraud detection model might need retraining only when transaction patterns shift, which could be weekly or could be after a specific event.

Model Registry

Every trained model gets registered with its metadata: training data version, hyperparameters, evaluation metrics, the commit hash of the training code, and who triggered the training run. MLflow Model Registry and Weights & Biases are the two most common choices here.

The registry is the model's birth certificate. Without it, when something goes wrong in production, you can't trace back to figure out what data the model saw, what code produced it, or what metrics it achieved before deployment.

A/B Testing and Canary Deployments

Models are deployed the same way you'd deploy a risky application change: gradually. A canary deployment routes 5% of traffic to the new model while the old one handles 95%. If the new model's metrics (latency, error rate, business metrics like click-through or conversion) hold up, traffic shifts gradually.

A/B testing serves a different purpose. It tells you whether the new model is actually better by the metric the business cares about, which is often different from the metric the model was optimized for. A model trained on accuracy might be more accurate but slower, and the latency hit costs more in user engagement than the accuracy gain provides.

Model Monitoring: Drift Detection and Performance Tracking

After deployment, the pipeline's job isn't done. It needs to watch:

Monitoring tools like Evidently AI, Arize, and WhyLabs specialize in this. Infrastructure monitoring (Prometheus, Grafana) handles the systems layer. The two need to talk to each other because a latency spike in the serving infrastructure and a drift alert in the model layer might have the same root cause.

Triple Versioning: Code, Data, and Models

In application development, Git handles versioning. Commit the code, tag a release, done. ML requires versioning three things simultaneously, and they have different characteristics.

Code versioning is the easiest part. Git works. The training scripts, feature engineering code, inference server configuration, and pipeline definitions all live in a repo and follow standard branching strategies.

Data versioning is where it gets hard. Training datasets can be gigabytes or terabytes. You can't put them in Git. DVC (Data Version Control) is the most widely adopted answer: it stores metadata and pointers in Git while keeping the data itself in S3, GCS, or Azure Blob. When you check out a specific commit, DVC pulls the corresponding data version. LakeFS takes a different approach, building Git-like branching and merging directly into object storage.

Model versioning tracks the trained artifact (weights, config, tokenizers) along with the provenance chain: which code version + which data version + which hyperparameters = this model. The model registry handles this, and the critical thing is that it can reconstruct the exact conditions that produced any model version currently in production or in the last N deployments.

The three versioning systems need to be linked. A model artifact in the registry should point to the Git commit of the training code and the DVC hash (or LakeFS commit) of the training data. Without this chain, reproducibility is impossible. For a deeper look at how to structure this, our guide on MLOps CI/CD pipeline versioning for code, data, and models covers the patterns in detail.

Infrastructure Patterns and Tooling

The ML infrastructure space has settled into recognizable layers, each with a few mature tools. Here's what the stack looks like across those layers.

Model Serving

The serving layer handles inference requests: model in, prediction out. The right choice depends on the model type, latency requirements, and scale.

For most Kubernetes-based teams, KServe is the pragmatic default. It provides serverless autoscaling (including scale-to-zero for cost savings), supports multiple frameworks, and integrates with Istio for traffic splitting during canary deployments.

Feature Stores

Feast is open-source and composable: you plug in your own offline store (BigQuery, Redshift, Snowflake) and online store (Redis, DynamoDB). Tecton is managed and opinionated, with stronger streaming feature support. For teams just starting, Feast is fine. For teams processing real-time feature streams at high volume, Tecton pays for itself in engineering time saved.

Experiment Tracking

MLflow is the open-source standard. Self-hosted, it tracks experiments, logs metrics, and manages model versions. Weights & Biases (W&B) is the SaaS alternative with stronger visualization, collaboration features, and an artifact management layer that's hard to replicate with MLflow alone. Most teams start with MLflow and migrate to W&B when collaboration across a large ML team becomes the bottleneck.

Pipeline Orchestration

The orchestrator triggers, sequences, and monitors the steps in the ML pipeline. Three contenders:

Kubeflow Pipelines is purpose-built for ML on Kubernetes. It handles distributed training, hyperparameter tuning (via Katib), and pipeline DAGs. It's complex to set up and maintain.

Argo Workflows is a general-purpose workflow engine on Kubernetes that ML teams adopt because it's lighter than Kubeflow and the platform team is often already running it. Less ML-specific tooling, more flexibility.

Apache Airflow is the most common choice when the data engineering team already uses it for ETL. Adding ML pipeline DAGs to an existing Airflow installation is path of least resistance, though Airflow's scheduler wasn't designed for the GPU-intensive, long-running jobs that training involves.

The pattern that keeps emerging in healthy organizations is: the platform engineering team provides the orchestration layer and the compute primitives (Kubernetes clusters, GPU node pools, storage), while the ML team defines the pipeline DAGs and model logic. For more on how this division of responsibility works, MLOps vs. DevOps: Operational Differences walks through the boundaries.

Bridging the MLOps Gap Between Platform and ML Teams

The most common failure pattern in production ML isn't technical. It's organizational.

ML teams build models in notebooks, validate them with offline metrics, and hand them to a platform or DevOps team to deploy. The platform team has no context on what the model does, what its failure modes look like, or what monitoring matters. The ML team has no context on how the deployment environment works, what SLOs the service needs to meet, or how the infrastructure handles load.

Both teams are competent. The problem is the handoff.

Organizations that get this right share ownership of the deployment path. Here's what that looks like concretely:

Platform teams own: the Kubernetes clusters, GPU scheduling, serving infrastructure, CI/CD pipeline infrastructure, secrets management, networking, and the model serving framework (KServe, Triton). They provide golden paths for model deployment: a standardized interface where the ML team provides a model artifact and a serving config, and the platform handles the rest.

ML teams own: model code, training pipelines, evaluation criteria, monitoring thresholds for model-specific metrics (drift, performance), and the decision about when a model is ready for production.

Both teams own together: the model deployment config (resource requests, autoscaling thresholds, health check endpoints), SLOs for inference latency and availability, and the incident response runbook for model failures.

An internal developer platform (IDP) that includes ML deployment as a first-class workflow makes this joint ownership practical. The ML team shouldn't need to write Kubernetes manifests. The platform team shouldn't need to understand model evaluation. The IDP is the contract between them. MLOps in Practice covers how to structure these workflows with the right tooling.

Governance and Compliance for Production Models

Regulated industries have been grappling with model governance for years. Now the rest of the industry is catching up, driven by the EU AI Act, expanding FDA guidance on Software as a Medical Device (SaMD), and increasing pressure from enterprise customers who want to understand how models that affect their operations make decisions.

Model Cards

A model card documents what a model does, what data it was trained on, its intended use, its limitations, and its performance across different population subgroups. Google introduced the format in 2018. Now it's becoming a practical requirement, not just an academic best practice. The EU AI Act requires comparable documentation for high-risk AI systems.

Model cards should be generated as part of the CI/CD pipeline. When a new model version is trained, the pipeline should auto-generate a model card from the training metadata (data version, metrics, architecture), with human-written sections for intended use and limitations pulled from a template in the model's repo.

Decision Audit Trails

For models making decisions about people (credit scoring, hiring, insurance pricing, medical diagnosis), regulators want to know why a specific decision was made. This means logging prediction inputs and outputs in a queryable store, with the model version that produced each prediction recorded alongside it.

The technical implementation is straightforward: log to a structured data store (BigQuery, Snowflake, a time-series database) at inference time, with a schema that includes the request ID, timestamp, model version, input features (or a hash for privacy), prediction, and confidence score. The hard part is retention policy and access control, which are compliance concerns, not infrastructure concerns.

Explainability

SHAP values, LIME, and attention visualization are tools the ML team already knows. The infrastructure concern is making explanations available at serving time without destroying latency. Pre-computed explanations for common inputs, cached SHAP values, and lightweight surrogate models are patterns that work. Computing SHAP values on every inference request does not.

For healthcare and financial services organizations, our guide to DevSecOps for AI model deployment in healthcare covers the specific regulatory requirements and how to meet them in a CI/CD context.

Cost Management: GPUs Don't Have to Bankrupt You

GPU compute is expensive. An A100 instance on AWS (p4d.24xlarge) costs roughly $32.77 per hour on-demand. A training run that takes 8 hours costs $262. Run that daily and you're spending $95,000 a year on a single experiment cadence.

Three strategies cut this without cutting capability:

Spot Instance Training

Training jobs are batch workloads. They don't need guaranteed uptime. AWS Spot instances, GCP Preemptible VMs, and Azure Spot VMs offer the same GPU hardware at 60 to 90% discounts. The tradeoff: your instance can be reclaimed with 2 minutes' notice.

The mitigation is checkpointing. Save model state to persistent storage (S3, GCS) every N steps. If the instance gets reclaimed, the orchestrator spins up a new spot instance and resumes from the last checkpoint. Kubeflow and Argo both support this pattern natively. A well-checkpointed training job running on spot instances costs a fraction of on-demand, with wall-clock time increasing only 10 to 20% from occasional restarts.

Model Compression for Inference

The model you trained doesn't have to be the model you serve. Quantization (reducing weight precision from FP32 to INT8 or FP16) typically reduces model size by 2x to 4x with less than 1% accuracy loss. Knowledge distillation trains a smaller "student" model to mimic the larger "teacher" model. Pruning removes weights that contribute little to the output.

Serving a quantized model on a T4 GPU instead of an A100 can cut inference costs by 80% or more. The pipeline should include a compression step between training and deployment, with validation that the compressed model still meets the performance threshold on the evaluation dataset.

Right-Sizing Inference Infrastructure

Not every model needs a GPU for inference. Decision trees, linear models, and small neural networks run fine on CPUs. Even models that benefit from GPU acceleration often don't need the latest hardware. A T4 GPU ($0.526/hour on AWS) handles most inference workloads that an A100 ($4.10/hour) would be overkill for.

KServe's autoscaling, especially scale-to-zero for models with intermittent traffic, avoids paying for idle GPU time. A model that receives 100 requests per hour doesn't need a GPU running 24/7.

Building Your ML Pipeline in 5 Stages

If you don't have a production ML pipeline today, don't try to build everything at once. Here's a staged approach that gets you to production safely.

1. Stage 1: Version Everything (Week 1 to 2) Set up Git for code, DVC for data, and MLflow for experiment tracking and model registry. Every training run gets logged. Every dataset gets a version. This is the foundation everything else builds on. At the end of this stage, you should be able to answer: "which data and code produced model X?"
2. Stage 2: Automate Training (Week 3 to 4) Move training from notebooks to reproducible scripts triggered by your orchestrator (Argo Workflows if your platform team already runs it, Kubeflow if you're building an ML-specific stack). Add data validation as the first step. At the end of this stage, a single command should produce a trained, validated, registered model.
3. Stage 3: Automated Deployment with Canary (Week 5 to 7) Deploy the model through KServe or Seldon Core with canary routing. Start at 5% traffic to the new model, with automated rollback if latency or error rate exceeds thresholds. Add the model serving endpoint to your existing infrastructure monitoring (Prometheus and Grafana). This is when you want platform and ML teams to co-own the deployment config.
4. Stage 4: Monitoring and Drift Detection (Week 8 to 10) Add model-specific monitoring: data drift detection, prediction confidence tracking, and business metric correlation. Connect drift alerts to retraining triggers. At the end of this stage, the pipeline should be able to detect when the model is degrading and kick off a retraining run without human intervention.
5. Stage 5: Governance and Optimization (Week 11 to 14) Add model card generation, decision logging for audit trails, and cost optimization (spot training, model compression). This is also when you integrate with a feature store if your inference pipeline has training-serving skew risk. At the end of this stage, you have a production ML pipeline that's versioned, automated, monitored, governed, and cost-efficient.

For a broader look at how CI/CD pipeline architectures work across different deployment patterns, including ML-specific patterns, see our guide to CI/CD pipeline design patterns.